Sunday, February 8, 2015

Home Monitor V.3.1.0 for Splunk Released

I've finished the latest version of the Home Monitor App 3.1.0 (https://apps.splunk.com/app/1214/) and it now supports the Asus router as well as pfSense firewalls.  I have not tested the functionality with Netgear or Skyhub routers, please let me know how they work and if they need any modifications to the app. I have not tested OpenWRT either, but please feel free to let me know if any changes need to be made.

I've also updated my GitHub Repo : https://github.com/amiracle/homemonitor.git where you can feel free to look at the components and make any changes to the app.

So, what's new?

Source type by hostname or manual

This version of the app can either rely on your router's hostname to configure the sourcetype, or you can select it manually on the Data Inputs page. For example, if you have a fios router, and the hostname is fios, then the props.conf and transforms.conf will work together to change the sourcetype to fios.  (The reason I did this was that it helepd during my testing having Splunk automatically pickup and change the sourcetype on the fly for me.)

Once the data input is in (more on that below), you will be able to see all of the dashboards populate with your data.  I even normalized the fields and the output of some of the fields using a lookup. This allows my Asus router and my pfSense firewall to have the same output as my FiOS router. You'll see that there are two fields, 'action' and 'action2' in the interesting fields.  The lookup, named action_lookup.csv, will convert the action to a normalized BLOCK or ACCEPT instead of DROP or pass. This allows all the dashboards to populate regardless of your router. There are some dashboards that WILL NOT populate since they have FiOS specific fields in the search.

How it works:

The props.conf has the following entry:

[syslog]
TRANSFORMS-changesourcetype=fios, pfsense, asus, netgear, skyhub

Then the transforms.conf file takes the source type and changes it depending on the hostname of your router:

[fios]
# Make sure that this matches the hostname of your router, fios is just an example.
REGEX = fiosSOURCE_KEY = MetaData:Host
FORMAT = sourcetype::fios
DEST_KEY = MetaData:Sourcetype

The key here is that your router's hostname either starts with or contains 'fios' in order for the change to occur automatically.  Otherwise, you'll see your data come in as syslog and it will NOT have any of the proper field extractions.

You can also manually change the sourcetype to fios, asus, pfsense, netgear or skyhub.

Normalized Data : 

In order to get the dashboards to populate regardless of router, I had to normalize the fields from all the routers.  Specifically, the 'action' field which told me if the traffic was 'BLOCKED' or 'ACCEPTED.'  Since the Asus router used 'DROP' instead of BLOCKED and the pfSense firewall used 'block' instead of BLOCKED', I had to use a lookup and create a new field, 'action2.'

With this in place, I was able to populate all of my dashboards with blocks and accepts regardless of the router I used.

Updated Dashboards : 
New updated dashboards - I've gone through and vetted all the dashboards to make sure they make some logical sense.  I stopped using the 'process' field since it did not exist in all the routers syslog data.  Instead, I determined that outbound connections were iniated by src_ip = 192.168.* and inbound connections were iniated by NOT src_ip=192.168.* .

Future Updates: 

I was thinking about creating a setup page, so more advanced users can configure the app to suite their customized networks.  When I get more motivation, I'll work on setting up tags for local networks and a setup page that allows you to change some of the inputs or specify your local network IP address space.

8 comments:

  1. - I've finished the latest version of the Home Monitor App 3.1.0 and it now supports the Asus router as well as pfSense firewalls. I have not tested ... monitorsspeakers.blogspot.com

    ReplyDelete
  2. I've finished the latest version of the Home Monitor App 3.1.0 (https://apps.splunk.com/app/1214/) and it now supports the Asus router as well as ... asusrouter.blogspot.de

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. Based on the above transforms.conf, all syslog messages coming from my pfsense instance are being assigned the sourcetype 'pfsense'. I have a need for the snort logs to be assigned different sourcetype. Admittedly, I'm new to Splunk, but how can I avoid the snort alert logs from getting assigned this sourcetype since I have need to assign them a different one?

    Thank you.! Great app!

    ReplyDelete
  5. You can create an eventtype called snort-alerts. This way, you can still get the data in as sourcetype=pfsense, but the application can be snort. What are you trying to accomplish with your snort alerts? Again, I can help you out if you let me know more about your use cases.

    ReplyDelete
    Replies
    1. For now, frequency of alerts, source ips, types of alerts, etc. Once I can separate the data I'll figure out what to do with it. I guess where im stuck is how do I differentiate all the syslog data coming from the pfsense box on 514/tcp from the snort-specific stuff also coming from it on 514/tcp. Being relatively new to both, I'm kind of stuggling.

      Delete
  6. First of all, great work!
    I start some days ago to test with my RT-n56U and it looks good, but I noticed the several events are saved as one record (multiple lines) and then several of them are not reported.
    Is there some setting I need to change?

    ReplyDelete
    Replies
    1. You might want to check and make a change in your props.conf. Go to $SPLUNK_HOME/etc/apps/homemonitor/local/props.conf and add this stanza:

      [asus]
      SHOULD_LINEMERGE=false

      If there is no props.conf in that local folder just create it and add that line from above. If the props file exists, then just add the SHOULD_LINEMERGE =false to the Asus stanza.

      Restart Splunk and let me know if that fixes the issue.

      Delete