Migrating rsyslog to Splunk

I recently decided to migrate my server cluster's logging mechanism from rsyslog to Splunk. My previous setup was to use rsyslog to centralise all logging onto /var/remotelog/ on a central server. I then configured the Splunk indexer to index both /var/log/ as well as /var/remotelog/, this allowed me to monitor all security events from a single Splunk dashboard. However, there were a few minor issues with such a setup. Firstly, some services such as Apache do not natively support syslog, hence a workaround is to use /usr/bin/logger to perform the remote logging. This results in a log file with an additional hostname field that does not play nice with Splunk's pretrained source logs. Secondly, Splunk's log forwarding is supposedly more robust. The forwarder can buffer logs when the indexer is offline and is able to better buffer writes, thus reducing disk usage.

With that, I started my migration. It was a relatively simple process. Firstly, I modified syslog configuration to log locally to /var/log/, installed splunk forwarder, and configured it to listen for changes to /var/log/ , forwarding entries to the central server. Over on the central server (x.x.x.x), I set it up to listen on port yyyy.

1
2
3
4

  dpkg -i /tmp/splunkforwarder-0.1.2.3-linux-0.1-amd64.deb
  /opt/splunkforwarder/bin/splunk add forward-server x.x.x.x:yyyy
  /opt/splunkforwarder/bin/splunk add monitor /var/log
  /opt/splunkforwarder/bin/splunk start

And the logs started coming in... However, the sourcetype was still wrong. I had too_small appended behind each of the sourcetypes. After some googling, I realised that because I had just switched to local logging, my log files were really small and Splunk was not able to analyse it and assign it one of the pretrained sourcetypes. At this point, there are 2 options. The first is to wait for the logs to naturally populate and setup Splunk another day. The alternative is to pad the files with appropriate content. If auth.log only had 200 lines, you could repeatedly copy those 200 lines until the file is approximately 100KB. Splunk will then be able to correctly detect the sourcetype. The annoying part about this is that Splunk would not recategorise the sourcetype automatically after the file is of a decent size, hence what I had to do was to uninstall the Splunk forwarder, reinstall and set it up once again.

Happy logging!