Saturday, December 26, 2015

Adding Quantam Sourcetype

props.conf
[syslog]
TRANSFORMS-changesourcetype = asus, fios, link sys, mikro, netgear, openwrt, pfsense, quantum, sophos, skyhub, tomato
[quantum]
FIELDALIAS-dst = DST as dest_ip
FIELDALIAS-dpt = DPT as dest_port
FIELDALIAS-proto = PROTO as protocol
FIELDALIAS-SPT = SPT as src_port
FIELDALIAS-SRC = SRC as src_ip
EXTRACT-action = ^[^\]\n]*\]:\s+(?P<action>\w+)
EVAL-direction = if(match(OUT,"eth*"), "out", "in")
pulldown_type = 1
LOOKUP-action_lookup = action_lookup action OUTPUT action2
LOOKUP-rdns = dnsLookup ip AS dest_ip OUTPUTNEW host as rdns_host

transforms.conf

[quantum]
# Make sure that this matches the hostname of your router, quantum is just an example.
REGEX = quantum
SOURCE_KEY = MetaData:Host
FORMAT = sourcetype::quantum
DEST_KEY = MetaData:Sourcetype

For whatever reason, when I paste this into the comments bar it does not properly show the '<action>' extraction.

4 comments:

  1. Hey,

    So the errors stopped! That definitely did the trick for splunk complaining on startup. However, all graphs, panels, etc. are all still "No Results Found", "0", or "Waiting for Input" except for the Total Number of Events from the Home Network Overview (which does change based on the time query). I have validated that the router is sending the data to syslog, and that data is getting into splunk (index=homemonitor). Below are props.conf and transforms.conf:

    /opt/splunk/etc/apps/homemonitor/local:cat props.conf
    [syslog]
    TRANSFORMS-changesourcetype = quantum

    [quantum]
    FIELDALIAS-dst = DST as dest_ip
    FIELDALIAS-dpt = DPT as dest_port
    FIELDALIAS-proto = PROTO as protocol
    FIELDALIAS-SPT = SPT as src_port
    FIELDALIAS-SRC = SRC as src_ip
    EXTRACT-action = ^[^\]\n]*\]:\s+(?P\w+)
    EVAL-direction = if(match(OUT,"eth*"), "out", "in")
    pulldown_type = 1
    LOOKUP-action_lookup = action_lookup action OUTPUT action2
    LOOKUP-rdns = dnsLookup ip AS dest_ip OUTPUTNEW host as rdns_host

    /opt/splunk/etc/apps/homemonitor/local:cat transforms.conf
    [portlookuptable]
    filename = application_protocals.csv

    [quantum]
    # Make sure that this matches the hostname of your router, fios is just an example.
    REGEX = 192\.168\.136\.1
    #For the hostname I am just using the IP as that is what shows up in Splunk - there is no DNS entry for this IP. Since it is a Regex field, I am using the "\" in front of "." although I have also tried it without the "\" with the same results.
    SOURCE_KEY = MetaData:Host
    FORMAT = sourcetype::quantum
    DEST_KEY = MetaData:Sourcetype
    FORMAT = homemonitor
    DEST_KEY = _MetaData:Index

    #Lookup normalizing Action
    [action_lookup]
    filename = action_lookup.csv

    [dnsLookup]
    external_cmd = external_lookup.py host ip
    fields_list = host, ip

    And, since this file was also changed, here is the action_lookup.csv:
    cat ../lookups/action_lookup.csv
    action,action2
    BLOCK,BLOCK
    ACCEPT,ACCEPT
    pass,ACCEPT
    block,BLOCK
    DROP,BLOCK
    Accepted,ACCEPT
    Blocked,BLOCK

    I am probably missing something so stupid ... but for the life of me I cannot see what it is. Any ideas? Thanks again!

    ReplyDelete
    Replies
    1. Let's do some simple searches and make sure the data is properly coming into the app. First, run this search :

      index=homemonitor sourcetype=quantum | stats count by action2
      (This will give you the total number of accepts and blocks that your router has logged.)

      If this search does not populate, then we have to make sure that the REGEX is working properly to extract the action field.

      Next, let's run this search :

      index=homemonitor sourcetype=quantum | stats count by direction

      If that comes up with no results, then run this search instead

      index=homemonitor sourcetype=quantum |eval direction=if(match(match(OUT,"eth*"), "out", "in") | stats count by direction

      This should give you a breakdown of in vs. out bound traffic.

      The only issue you're probably running into now is the field extractions, these searches will just help narrow down where the problem might be and which fields to extract.

      Thanks,
      Kam

      Delete
    2. Kam, Strange -- even though we are telling it to use a sourcetype of quantum, they are showing up with a sourcetype of syslog. Whatever, just odd.

      Narrowed all queries to a 30 minute window.

      index=homemonitor sourcetype=syslog | stats count by action2
      440 of 440 events matched, but no results in current time range.

      index=homemonitor sourcetype=syslog | stats count by direction
      440 of 440 events matched, but no results in current time range.

      index=homemonitor sourcetype=syslog |eval direction=if(match(OUT,"eth*"), "out", "in") | stats count by direction
      in: 120
      out: 300

      Thanks!

      Delete
    3. The reason it's not picking up the source type is that it's trying to match the hostname and you might have set it to DNS instead of IP. Regardless, you can either manually override the input or just change the REGEX stanza to be the hostname of your router in the transforms.conf.

      Once you get this to work, then the extractions will happen and the dashboards should populate. Check out the wiki (https://github.com/amiracle/homemonitor/wiki) and see the section called Configuring the App.

      Delete