Put the Splunk Universal Forwarder on the Raspberry Pi

Splunk released the ARMv6 version of the Universal Forwarder and now you can install it on the Raspberry Pi as a result. To put the Splunk Universal Forwarder on the Raspberry Pi there are some delicate details that I cover in this post. You are better off giving it a read before attempting to do it on your on!

Author’s note: Splunk does not need a sudo user to run but I find it is more convenient during setup at least to just use sudo. You can make a “splunk” user later to actually run Splunk.

Here’s What You Need

• Splunk Enterprise Free ( A running Splunk server serving up the Splunk Web application )

Don’t have it? Get it by following this easy guide.

• Raspberry Pi 3 B+ 
• Splunk Blog Tutorial

Step 1: Get the Universal Forwarder

Go to splunk.com. Login with your account. Click on Free Splunk in the right corner of the site. (You won’t miss it) The link Splunk suggests you use to get the ARMv6 Universal Forwarder from Splunk base actually doesn’t work, go figure. ( You will see this isn’t the first time that Splunk’s tutorial will be incorrect or missing details you need. That’s why I point these parts out and provide the solutions 🙂 ).

# In Downloads 
# This will unpack to a folder /opt/splunkforwarder
sudo tar xvzf splunkforwarder.....64.tgz /opt

Start the forwarder. But First set SPLUNK_HOME in your PATH.

# Set your environment variable for convenience
sudo nano /etc/profile 
export SPLUNK_HOME=/opt/splunkforwarder
source /etc/profile
echo $SPLUNK_HOME

Now start the forwarder.

# running for first time requires account setup
sudo $SPLUNK_HOME/bin splunk start --accept-license

If setup is successful you will see configurations flying around and a cheeky line from Splunk like “winning the war on error”.

Step 2: Setup Receiving in Splunk Enterprise Free

In the GUI of your Splunk Enterprise Free(Splunk Web) go to Data then Forwarding and Receiving then Receive data and finally Add new. 

Step 3: Configure the Universal Forwarder to Send to Splunk Enterprise Instance

To do this step you will be editing a configuration file (.conf) in the local dir of the /etc/sytem path.

cd $SPLUNK_HOME/etc/system/local
sudo nano inputs.conf 
# by default the file looks like this
# a [text] is a stanza you will hear this a lot
[default]
host = rapsberrypi
# add these lines
[script:///opt/Splunkforwarder/bin/scripts/getcpudata.sh]
index = main
interval = 60
source = cpu_details
sourcetype = pi_system_scripts

Now that we have defined a script we need to make one.

cd $SPLUNK_HOME/bin/scripts
sudo touch getcpudata.sh

Add these lines to our script file, from the Splunk Blog post.

sudo nano getcpudata.sh

# add these lines
#! /bin/sh
for c in $(seq 1 60)
do
TIME=$(date +"%D %T.%3N %Z")
TEMP=$(vcgencmd measure_temp)
VOLTS=$(vcgencmd measure_volts)
CPU_TEMP=`echo $TEMP | cut -d \' -f 1`
VOLTS=`echo $VOLTS | cut -d V -f 1`
echo "time=$TIME|$VOLTS|$CPU_TEMP"
sleep 1
done

The Splunk blog article suggests going to $SPLUNK_HOME/etc/apps/Splunkuniversalforwarder and adding these lines to outputs.conf, but I disagree. You don’t want to edit default configuration files if you don’t have to, because it’s not good practice. One update cloud blow away your configurations if they are in a default directory. Go to $SPLUNK_HOME/etc/system/local and create an outputs.conf file that you can now add these lines to it instead. 

defaultGroup=my_indexers

# new stanza 
[tcpout:my_indexers]
# this is the ip of your Splunk Enterprise 
server=10.X.X.XX:9997

Go ahead and ping the IP address of your Splunk Enterprise to be sure you can reach it. 

Restart splunk UF

sudo $SPLUNK_HOME/bin splunk restart

Do a search for last 15 minutes and try looking for getcpudata.sh or index=main and locate the new logs coming in.

HOWEVER

Following the steps in this guide did not leave me with a working setup. I wasn’t surprised really…expecting a script in that folder to just run like that.

Here is how you fix it:

# If the output of this command is empty you are missing a step!
sudo ./splunk list forward-servers
# this is to a data input via the monitor command 
sudo ./splunk add monitor /var/log/messages

And here is a working setup finally after Splunk’s blog tutorial lead us in circles.

And the full view of the results shown below.

Conclusion: So we did change the chartered course a bit by not using the suggested bash script but there is plenty to do with the data we receive from /var/log/messages.

Extra Step

Turn on Forwarder Monitoring

Go to Settings > Monitoring Console > Forwarders > Setup

Enable “Forwarder Monitoring Setup”

Splunk will auto-detect the live connection between the Universal Forwarder on the Raspberry Pi and the Splunk server.