Kyh's blog

Kyh's random blog of stuff

Setting Up Splunk With Remote Forwarders

Splunk is a fancy log monitoring application. It lets you pull in all kinds of data from logs and various other inputs, then provide you with a fancy web GUI to generate graphs, reports and alerts.

The free license allows you to process 500MB of data a day, which is fine for a few machines.

Splunk supply linux tarballs/deb/rpm packages, so just go ahead and grab the latest version from the splunk download page and install using your package manager. By default splunk listens on port 8000 on http, which means we’re passing our login in the clear, this is bad, so lets go ahead and enable SSL before we login to it.

You need to start splunk so that it generates a bunch of configs/keys automagically, then edit /opt/splunk/etc/system/default/web.conf and change enableSplunkWebSSL to “true”

You can either go ahead and replace the key/cert paths to your own SSL certs or use the newly generated ones. In this case, if we want to use the one that splunk generated for us, we can check the fingerprint by running openssl x509 -fingerprint -in /opt/splunk/etc/auth/splunkweb/cert.pem and verifying the self-signed cert fingerprint matches.

Login with the default login and change your password to something secure.

Splunk provide a headless version of splunk called splunkforwarder that will ship off logs to your main splunk instance (called receivers), so after installing that on our “client” box, we need to tell it where to ship said logs off to as well as enabled “receiving” on our receiver.

In the webgui, you can do this under Manger->Forwarding and Receiving->Configure receiving-> Add new. Splunk can either listen on a single port for all forwarders or you can setup a port per forwarder. I’d recommend firewalling the receiving ports to only allow access from the forwarder hosts and/or use something like openvpn, since splunk listens on 0.0.0.0 by default.

Since we don’t want our logs going over the internets as plaintext we need to change the input type. Edit splunk/etc/apps/search/local/inputs.conf and change [splunktcp://3333] with [splunktcp-ssl:3333] and then add

1
2
3
[SSL]
serverCert=$SPLUNK_HOME/etc/auth/server.pem
rootCA=$SPLUNK_HOME/etc/auth/cacert.pem

If you have reverse DNS setup for the forwarding host, you can also change connection_host to “dns” so it’ll use that instead of the IP address.

At this point we need to configure our forwarder to use a data input, in this case i’m just going to use all of /var/log/.

Edit