Thursday, February 11, 2016

Adding DD-WRT Sourcetype

Sample Data

Here I'm going to show you how to add  a new source type into the Home Monitor App using dd-wrt as an example.  I'm planning on building a Splunk Technology Add-on (TA) for the Home Monitor app so that these extractions can be used by other apps or in a distributed environment.

First, let's take a look at the data.  We can see here that the data is already in Name = Value pairs with some extra fields that we can extract using the interactive Splunk Extraction tool.
2016-01-10 14:59:57 Kernel.Warning 192.168.28.1 Jan 10 06:59:57 kernel: ACCEPT IN=vlan2 OUT=br0 MAC=78:54:2e:4e:13:c9:00:17:10:85:ab:92:08:00:45:00:00:8f SRC=218.15.145.194 DST=192.168.28.57 LEN=143 TOS=0x00 PREC=0x00 TTL=43 ID=4934 PROTO=UDP SPT=14392 DPT=19598 LEN=123 MARK=0xa000
2016-01-10 14:59:57 Kernel.Warning 192.168.28.1 Jan 10 06:59:57 kernel: ACCEPT IN=br0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:1d:ba:67:d7:f2:08:00 SRC=192.168.28.11 DST=192.168.28.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=23255 PROTO=UDP SPT=137 DPT=137 LEN=58 MARK=0x35400
2016-01-10 14:59:57 Kernel.Warning 192.168.28.1 Jan 10 06:59:57 kernel: ACCEPT IN=vlan2 OUT=br0 MAC=78:54:2e:4e:13:c9:00:17:10:85:ab:92:08:00:45:00:00:84 SRC=123.26.105.194 DST=192.168.28.57 LEN=132 TOS=0x00 PREC=0x00 TTL=113 ID=15843 PROTO=UDP SPT=10538 DPT=19598 LEN=112 MARK=0xa000
2016-01-10 14:59:57 Kernel.Warning 192.168.28.1 Jan 10 06:59:58 kernel: ACCEPT IN=br0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:1d:ba:67:d7:f2:08:00 SRC=192.168.28.11 DST=192.168.28.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=23351 PROTO=UDP SPT=137 DPT=137 LEN=58 MARK=0x35400
I'm going to start by making this CIM compliant, so that means that this source will be able to populate the Home Monitor Dashboards without having to re-write the searches.

*** WARNING *** 
If you are going to build your own sourcetypes, please make all your changes in the $SPLUNK_HOME/etc/apps/homemonitor/local directory.  Making changes to the default directory will cause unnecessary grief since any updates I push will overwrite all your hard work.

Let's start with the Interactive Field Extractor.

First, let's add the sample data into Splunk.

Upload the sample file


Click on Save As 


See that the source type is now dd-wrt 


For testing purposes, create a test index and send this sample data into it


Let's Extract Fields


Notice that we are extracting fields for the source type dd-wrt.
Click on one of the events.


Select Regular Expression

Highlight 'ACCEPT' and name it action

Now you'll see the newly extracted field

Make sure to select permission for App

Now run a search and you'll see the newly created field.



Next step, let's add some logic to the props.conf : Go ahead and open your $SPLUNK_HOME/etc/apps/homemonitor/local/props.conf

You should see this entry [dd-wrt] :

[dd-wrt]
pulldown_type = 1
EXTRACT-action = (?i) .*?: (?P<action>\w+)(?= )

Now, let's 'normalize' the fields for Source IP, Source Port, Destination IP and Destination Port.

FIELDALIAS-dst = DST as dest_ip
FIELDALIAS-dpt = DPT as dest_port
FIELDALIAS-proto = PROTO as protocol
FIELDALIAS-SPT = SPT as src_port
FIELDALIAS-SRC = SRC as src_ip
EVAL-direction = if(match(OUT,"eth*"), "out", "in")



This stanza will create a new field called direction and base the direction on what interface is being used by the field 'OUT.' If the OUT field has eth0, then it will determine the direction as 'out,' otherwise it will set the direction to 'in.' 



Finally, we can add some lookups to enrich the data. The first lookup normalizes the action to either ACCEPT or BLOCK. The second lookup actually does reverse lookup for IP addresses. 

LOOKUP-action_lookup = action_lookup action OUTPUT action2
LOOKUP-rdns = dnsLookup ip AS dest_ip OUTPUTNEW host as rdns_host

The final props.conf entry should look like this :

[dd-wrt]
pulldown_type = 1
EXTRACT-action = (?i) .*?: (?P<action>\w+)(?= )
FIELDALIAS-dst = DST as dest_ip
FIELDALIAS-dpt = DPT as dest_port
FIELDALIAS-proto = PROTO as protocol
FIELDALIAS-SPT = SPT as src_port
FIELDALIAS-SRC = SRC as src_ip
EVAL-direction = if(match(OUT,"eth*"), "out", "in")
LOOKUP-action_lookup = action_lookup action OUTPUT action2
LOOKUP-rdns = dnsLookup ip AS dest_ip OUTPUTNEW host as rdns_host



I will be adding this to the default props.conf for the next release of Home Monitor 4.4.3. 

Sunday, February 7, 2016

Getting Bandwidth Data into Home Monitor App version 4.4.2

Super Bowl 50 is in the books and so I decided to update the Home Monitor app to include average bandwidth for your home network.  I got the idea after reading the post about the Comcast customer that was tweeting  their complaints each time their bandwidth fell below certain rate.  I decided to add this feature to the app to help show users their average Download, Upload and Ping rates.

First, I downloaded the scripts necessary to get the speedtest data, found here : https://github.com/sivel/speedtest-cli

Next, I tweaked the python script so that it displayed everything as a Name = Value pair, making it easy to Splunk :

Download=85 Mbps
Upload=60 Mbps
Ping=8ms

speedtest_cli.py

Changed Download: to Download=, Upload: to Upload= and Ping: to Ping=

Once that was done, I created the inputs for both Windows and Linux :

# Bandwidth Input for Linux Machines
[script://$SPLUNK_HOME/etc/apps/homemonitor/bin/speedtest.sh]
disabled = true
interval = 1800.0
sourcetype = bandwidth_test

# Bandwidth Input for Windows Machines
[script://$SPLUNK_HOME\etc\apps\homemonitor\bin\speedtest_cli.py --simple]
disabled = false
interval = 1800.00
source = bandwidth_test
sourcetype = bandwidth_test

Now, all you have to do is go into Settings -> Data Inputs -> Scripts and enable the proper script for your OS.  I would keep it to 15 minutes (1,800 seconds) since you don't want to be running a speedtest on a high frequency which could degrade your networks performance.

Thanks and enjoy,
Kam


Saturday, December 26, 2015

Adding Quantam Sourcetype

props.conf
[syslog]
TRANSFORMS-changesourcetype = asus, fios, link sys, mikro, netgear, openwrt, pfsense, quantum, sophos, skyhub, tomato
[quantum]
FIELDALIAS-dst = DST as dest_ip
FIELDALIAS-dpt = DPT as dest_port
FIELDALIAS-proto = PROTO as protocol
FIELDALIAS-SPT = SPT as src_port
FIELDALIAS-SRC = SRC as src_ip
EXTRACT-action = ^[^\]\n]*\]:\s+(?P<action>\w+)
EVAL-direction = if(match(OUT,"eth*"), "out", "in")
pulldown_type = 1
LOOKUP-action_lookup = action_lookup action OUTPUT action2
LOOKUP-rdns = dnsLookup ip AS dest_ip OUTPUTNEW host as rdns_host

transforms.conf

[quantum]
# Make sure that this matches the hostname of your router, quantum is just an example.
REGEX = quantum
SOURCE_KEY = MetaData:Host
FORMAT = sourcetype::quantum
DEST_KEY = MetaData:Sourcetype

For whatever reason, when I paste this into the comments bar it does not properly show the '<action>' extraction.

Monday, October 26, 2015

Adding Splunk Stream to your home network

One Level Deeper with Splunk Stream

The data that you get from your firewall or home network device only tells you half the story and even that story is kind of boring.  For the real details of your home network, you'll have to start digging into your network a little more.  You can start by adding a managed switch to your environment which will allow you to span your network and collect some interesting data points.

Setting up Port Mirroring

"Port Mirroring is a method used to monitor your network traffic." Basically, your managed switch will replicate the data going through your network onto one port. You will 'tap' this port and listen to all the packets using Splunk Stream.  Since most home networks are not terribly large, you can leverage a computer with not a great deal of horse power.  I'm using the same hardware I used to build my pfsense firewall to build this Stream forwarder / proxy server. 

I'm going to walk you through setting up the Cisco Switch along with the Stream Forwarder to capture this wire data.  Let's start by logging into the Switch and setting up Port Mirroring:


First, add the ports you want to mirror.


Step 1 - Enable the Ports


Step 2 - Enable the mirrored ports and the Admin Port.  In this example, I've mirrored ports G1-G7 onto port G8.  I will plug my Splunk Stream Forwarder to this port (G8).


Setup Splunk Stream Forwarder

Step 1 - Download Splunk Stream

Step 2 - Setup Spunk Stream

I will not go deeply into this setup since it is well documented on the Splunk Stream Docs page. (http://docs.splunk.com/Documentation/StreamApp/latest/DeployStreamApp/InstallSplunkAppforStream) 

Once you've setup your Splunk Stream Server, you should go to your machine's Stream page. For example, if your stream server's hostname is "stream," simply go to https://stream:8889 and you should see this page:


This will show you that you are collecting packets from your 'SPAN port' and that you should have data in your Splunk indexer.  Run this simple search command to see if you are collecting data :

index=main sourcetype=stream* | stats count by sourcetype

This will show you if you are receiving data from your Splunk Stream Forwarder.  

Hope that helps and as always happy Splunking! 

Suggested Parts List

Here's what I bought to make all of this possible :


If you don't have a dual NIC server, then I would suggest you build one like this which I used to build my pfsense firewall. 



Sunday, September 20, 2015

Troubleshooting home | monitor > app

Setting Up Splunk

Linux / Mac OSX Users

Let's start by setting up your Splunk instance on a *NIX system.  First, install the Splunk binaries (.dmg, .rpm, .deb, or .tgz) and start your Splunk instance.  If you want Splunk to start on reboots, just run the boot-start -user splunk as root for your instance:

my-host$ sudo opt/splunk/bin/splunk start enable boot-start -user splunk

Firewalls

If you have iptables enabled, make sure to open the ports for syslog in-bound (UDP 514) and out-bound TCP 8000, 8089.  If you are going to use Splunk Stream enable TCP port 8889.  Lastly, if you're going to enable forwarding, you should open TCP port 9997. 

Windows Users

Create a firewall rule and include the following ports:

In-bound
UDP port 514

Out-bound
TCP port 8089, 8000, 8889, 9997

Getting Data In

Syslog

First, let's make sure your router / device can send data in via syslog.  Most devices send data over UDP port 514, which is the default syslog. On your device, you will to set the IP Address of your Splunk server as the recipient of the syslog data. 

On your Splunk server, you will see that the UDP port 514 has already been enabled but you might have trouble collecting the data. On Linux, you have to be running as root in order to listen to port 514.  If you do not want to run as root, you will need to modify the port on which syslog is sending data. 

(http://answers.splunk.com/answers/310324/home-monitor-how-to-configure-a-netcomm-modem-rout.html#answer-310832)

If you're modem allows you to change the port from 514 to 1514 (UDP), then all you have to do is modify the input in Splunk to reflect the new UDP port.

You can also just move the inputs.conf file from the $SPLUNK_HOME/etc/apps/homemonitor/default/inputs.conf and put it into $SPLUNK_HOME/etc/apps/homemonitor/local and just make the following changes:

 [udp://514]
 connection_host = dns
 sourcetype=syslog
 index = homemonitor
 disabled = 0
to

 [udp://1514]
 connection_host = dns
 sourcetype=syslog
 index = homemonitor
 disabled = 0

To do this from the Web GUI, go to Settings -> Data Inputs -> UDP and click on Add. For the port put in 1514 and then click Next.

In the next screen, under Index, make sure you select the dropdown and select Homemonitor. Click Review and then click Submit.

That should get you to start collecting data into your home monitor app. To check, go to the Search in the homeomitor app and type in index=homemonitor and hit enter. You should start to see data streaming into Splunk.

Data is in, but no dashboards are populating?!

First step in troubleshooting is to make sure that the data is flowing into the app.  Let's start by running a simple search. Open the home | monitor > app and click on Search.  Now run the following search :

index=homemonitor | stats count by sourcetype

If you do not see any events, then the data is not coming in and you should double check 1) the Splunk IP on your device 2) Firewall rules and that UDP 514 is allowed

OK, I got events, but they are all just coming in as syslog! 

What is happening is that the data is coming in as syslog but is not being transformed into the source type of your device. You have some options to fix this.

Option 1: Hard Code the source type on the input:

Via the Web UI click on Settings -> Data Inputs -> UDP -> 514 . From here, it will open that port and you can change the Manual setting of syslog to "From list" and then select your device type (e.g. asus, linksys, fios, pfsense, etc.).  Click Save. 

From the command line (CLI) edit the $SPLUNK_HOME/etc/apps/homemonitor/local/inputs.conf 
 [udp://514]
 connection_host = dns
 sourcetype=syslog
 index = homemonitor
 disabled = 0

Change "syslog" to your corresponding device type.

Once it's been saved then you will start receiving the data as the source type of your device.

Option 2: Change the hostname to match the modem type:

Go into your device and change the hostname from the default to just the vendor name of your modem.  For example, if you have an Asus router, just make the hostname "asus".  This will allow Splunk to automatically set the source type to asus via the transforms.conf file.  

Hopefully this should cover most of the problems that you'll encounter setting up the Splunk app for home | monitor >.  I've been meaning to setup this page for some time now.  I will be posting a new video shortly going over setting up the new 4.x app with Splunk 6.x.

Adding OpenWRT sourcetype

I've made some modifications to the props.conf and added the new source type:

[syslog]
TRANSFORMS-changesourcetype = fios, pfsense, asus, netgear, skyhub, linksys, mikro, openwrt
[openwrt]
# Based on Asus RT-N66U router syslog output.
FIELDALIAS-dst = DST as dest_ip
FIELDALIAS-dpt = DPT as dest_port
FIELDALIAS-proto = PROTO as protocol
FIELDALIAS-SPT = SPT as src_port
FIELDALIAS-SRC = SRC as src_ip
EXTRACT-action = ^[^\]\n]*\]\s+(?P<action>\w+)
pulldown_type = 1
LOOKUP-action_lookup = action_lookup action OUTPUT action2

And made a quick change to the transforms.conf to include openwrt :

[openwrt]
# Make sure that this matches the hostname of your router, openwrt is just an example.
REGEX = openwrt
SOURCE_KEY = MetaData:Host
FORMAT = sourcetype::openwrt
DEST_KEY = MetaData:Sourcetype


Special thanks to @LodiHensen [twitter] for helping test out this source type on OpenWRT.

I will add these updates to the next release of the home | monitor > app, but for now you can copy these entires for your props.conf and transforms.conf files.

Tuesday, June 16, 2015

Sophos Sourcetype Added

Here are the configuration changes you'll need to make to add Sophos firewalls to the home | monitor > 4.0 .  Please note that the direction field does not exist, so some of the pfsense dashboards will not fully populate.

transforms.conf 

[sophos]
REGEX = sophos
SOURCE_KEY = MetaData:Host
FORMAT = sourcetype::sophos
DEST_KEY = MetaData:Sourcetype


props.conf

[sophos]
FIELDALIAS-srcip = srcip as src_ip
FIELDALIAS-srcport = srcport as src_port
FIELDALIAS-dstip = dstip as dest_ip
FIELDALIAS-dstport = dstport as dest_port
FIELDALIAS-dstmac = dstmac as dest_mac
FIELDALIAS-proto = proto as protocol
FIELDALIAS-fwrule = fwrule as firewall_rule

action_lookups.csv

drop, BLOCK
accept,ACCEPT

After you add these entries, make sure to restart your Splunk instance.  I'll update the default conf and csv files for a later release (4.0.2).