Sunday, September 20, 2015

Troubleshooting home | monitor > app

Setting Up Splunk

Linux / Mac OSX Users

Let's start by setting up your Splunk instance on a *NIX system.  First, install the Splunk binaries (.dmg, .rpm, .deb, or .tgz) and start your Splunk instance.  If you want Splunk to start on reboots, just run the boot-start -user splunk as root for your instance:

my-host$ sudo opt/splunk/bin/splunk start enable boot-start -user splunk

Firewalls

If you have iptables enabled, make sure to open the ports for syslog in-bound (UDP 514) and out-bound TCP 8000, 8089.  If you are going to use Splunk Stream enable TCP port 8889.  Lastly, if you're going to enable forwarding, you should open TCP port 9997. 

Windows Users

Create a firewall rule and include the following ports:

In-bound
UDP port 514

Out-bound
TCP port 8089, 8000, 8889, 9997

Getting Data In

Syslog

First, let's make sure your router / device can send data in via syslog.  Most devices send data over UDP port 514, which is the default syslog. On your device, you will to set the IP Address of your Splunk server as the recipient of the syslog data. 

On your Splunk server, you will see that the UDP port 514 has already been enabled but you might have trouble collecting the data. On Linux, you have to be running as root in order to listen to port 514.  If you do not want to run as root, you will need to modify the port on which syslog is sending data. 

(http://answers.splunk.com/answers/310324/home-monitor-how-to-configure-a-netcomm-modem-rout.html#answer-310832)

If you're modem allows you to change the port from 514 to 1514 (UDP), then all you have to do is modify the input in Splunk to reflect the new UDP port.

You can also just move the inputs.conf file from the $SPLUNK_HOME/etc/apps/homemonitor/default/inputs.conf and put it into $SPLUNK_HOME/etc/apps/homemonitor/local and just make the following changes:

 [udp://514]
 connection_host = dns
 sourcetype=syslog
 index = homemonitor
 disabled = 0
to

 [udp://1514]
 connection_host = dns
 sourcetype=syslog
 index = homemonitor
 disabled = 0

To do this from the Web GUI, go to Settings -> Data Inputs -> UDP and click on Add. For the port put in 1514 and then click Next.

In the next screen, under Index, make sure you select the dropdown and select Homemonitor. Click Review and then click Submit.

That should get you to start collecting data into your home monitor app. To check, go to the Search in the homeomitor app and type in index=homemonitor and hit enter. You should start to see data streaming into Splunk.

Data is in, but no dashboards are populating?!

First step in troubleshooting is to make sure that the data is flowing into the app.  Let's start by running a simple search. Open the home | monitor > app and click on Search.  Now run the following search :

index=homemonitor | stats count by sourcetype

If you do not see any events, then the data is not coming in and you should double check 1) the Splunk IP on your device 2) Firewall rules and that UDP 514 is allowed

OK, I got events, but they are all just coming in as syslog! 

What is happening is that the data is coming in as syslog but is not being transformed into the source type of your device. You have some options to fix this.

Option 1: Hard Code the source type on the input:

Via the Web UI click on Settings -> Data Inputs -> UDP -> 514 . From here, it will open that port and you can change the Manual setting of syslog to "From list" and then select your device type (e.g. asus, linksys, fios, pfsense, etc.).  Click Save. 

From the command line (CLI) edit the $SPLUNK_HOME/etc/apps/homemonitor/local/inputs.conf 
 [udp://514]
 connection_host = dns
 sourcetype=syslog
 index = homemonitor
 disabled = 0

Change "syslog" to your corresponding device type.

Once it's been saved then you will start receiving the data as the source type of your device.

Option 2: Change the hostname to match the modem type:

Go into your device and change the hostname from the default to just the vendor name of your modem.  For example, if you have an Asus router, just make the hostname "asus".  This will allow Splunk to automatically set the source type to asus via the transforms.conf file.  

Hopefully this should cover most of the problems that you'll encounter setting up the Splunk app for home | monitor >.  I've been meaning to setup this page for some time now.  I will be posting a new video shortly going over setting up the new 4.x app with Splunk 6.x.

29 comments:

  1. What your doing with this app is great! I am trying to get this working with my Tomato router. I am getting my data into splunk just fine but am not able to get the dashboards working in the app. I am assuming something is wrong with the field aliases but i am not quite sure what. I can send you a sample log file from my router if that would help you out or if you have any ideas that would be great, i would love to have this setup at home.

    ReplyDelete
  2. Sure, please post some samples of the data and we can work it out. What kind of router are you using?

    ReplyDelete
  3. The router is a ASUS RT-N66U running the tomato firmware.

    ReplyDelete
    Replies
    1. Sample logs:

      <12>Oct 1 07:48:39 kernel: DROP IN=ppp0 OUT= MAC= SRC=196.212.151.226 DST=174.25.218.236 LEN=78 TOS=0x00 PREC=0x00 TTL=114 ID=27577 PROTO=UDP SPT=49164 DPT=137 LEN=58
      <12>Oct 1 07:48:49 kernel: ACCEPT IN=br0 OUT=ppp0 SRC=192.168.1.53 DST=176.32.99.246 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15905 DF PROTO=TCP SPT=46225 DPT=443 SEQ=795615877 ACK=0 WINDOW=14600 RES=0x00 SYN URGP=0 OPT (020405AC0402080A0AB71CA40000000001030306)
      <12>Oct 1 07:49:32 kernel: ACCEPT IN=ppp0 OUT=br0 SRC=129.176.197.27 DST=192.168.1.9 LEN=52 TOS=0x00 PREC=0x00 TTL=118 ID=19646 DF PROTO=TCP SPT=26634 DPT=443 SEQ=3415705785 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405AC0103030801010402)
      <12>Oct 1 07:49:32 kernel: ACCEPT IN=ppp0 OUT=br0 SRC=129.176.197.27 DST=192.168.1.9 LEN=52 TOS=0x00 PREC=0x00 TTL=118 ID=19647 DF PROTO=TCP SPT=26635 DPT=443 SEQ=3102524028 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405AC0103030801010402)

      Delete
  4. That looks like the ASUS or OpenWRT source type. The easiest way to get this to work is to follow the steps I posted for this sourcetype in this blog post: http://amiracle19.blogspot.com/2015/09/adding-openwrt-source-type-ive-made.html

    Let me know how that works for you. I will create an update that includes the OpenWRT sourcetype natively for home | monitor > .

    ReplyDelete
    Replies
    1. I'm also trying to get my tomato router (Netgear R7000) to properly populate inside of the home monitor app. I followed the suggestion of setting the datatype to openwrt and I'm getting nothing but the "Events" counter being populated. It doesn't seem to key off of inbound/outbound events or accepts/drops. Here's a couple of sample logs:

      Dec 18 07:45:13 kernel: ACCEPT IN=br0 OUT=vlan2 SRC=172.16.1.251 DST=17.254.0.59 LEN=80 TOS=0x00 PREC=0x00 TTL=126 ID=9761 PROTO=UDP SPT=64893 DPT=53 LEN=60
      Dec 18 07:49:53 kernel: DROP IN=vlan2 OUT= MACSRC=80:e0:1d:02:50:22 MACDST=e4:f4:c6:0d:33:ca MACPROTO=0800 SRC=74.125.226.68 DST=76.118.200.184 LEN=121 TOS=0x00 PREC=0x20 TTL=54 ID=60621 PROTO=TCP SPT=443 DPT=42549 SEQ=581254237 ACK=2025359033 WINDOW=414 RES=0x00 ACK PSH URGP=0 OPT (0101080A2977922C2F8D8202)

      Seems like the key difference between this log and the previous sample is the IN/OUT interfaces on my router are br0 and vlan2 vs. eth0 and ppp0

      Any suggestions on how to tweak the props.conf? splunk is seeing it as the datatype openwrt when I search the database.

      Delete
    2. I did a bit more tinkering around and the tomato router output can be parsed using the asus type as opposed to the openwrt type. When doing this it starts picking up on the ACCEPT and DROP values.

      However, I still can't seem to get the direction field to evaluate properly. The props.conf has an EVAL-direction line in it for the asus type that uses the match property based on "OUT" having "eth*" as a value. I modified this based on the interface names on my router and tried doing a search on events: index=hometype | stats count by direction This yields no results.

      If I put the following into the search: index=homemonitor sourcetype=asus | eval direction=if(match(IN,"vlan2"),"in","out") | stats count by direction
      It will yield the "In" and "out" values for direction in this case and the counts are appropriate in terms of "inbound" vs. "outbound" traffic.

      How can I get the props.conf EVAL-direction to behave correctly. Fairly new to plunk so I'm experimenting :)

      Delete
    3. I see that you figured out how to fix your issue in the next post. Good job and let me know if there is anything else that you need help with.

      Delete
  5. using wireshark I can see log data coming in from my fios router which includes source and destination IPs, but in the home monitor dashboard the only item with any data is "average events." Everything else in every display says either "search is waiting for input" or "no results found." How can I troubleshoot this?

    ReplyDelete
    Replies
    1. First thing you should try and do is run this search :

      index=homemonitor sourcetype=fios | stats count by src_ip

      See what kind of results you get. If you don't get any results (No Results Found), then you might want to run :

      index=homemonitor | stats count by sourcetype

      If you don't see fios as a sourcetype, then we will need to fix your input and set the sourcetype in the UDP/514 to fios.

      Start with that, and then we can continue to troubleshoot to get you going.

      -Kam

      Delete
    2. I did some testing on the FiOS router and found that these two fields have changed (action and direction). Here is how you can fix this :

      Open the props.conf in the default directory ($SPLUNK_HOME/etc/apps/homemonitor/default/props.conf) and edit the [fios] stanza:

      [fios]
      EXTRACT-direction = ^(?:[^ \n]* ){11}(?P\w+)
      EXTRACT-action = ^(?:[^ \n]* ){12}(?P\w+)

      You can double-check and see that these fields using the Interactive Field Extractor. (Click on Settings -> Fields -> Field Extractions -> Open Field Extractor.) From there, type in "fios" as the sourcetype and you can see how the fields are lined up.

      I will be fixing that for a later release of the home | monitor > app.

      Thanks,
      Kam

      Delete
    3. This comment has been removed by the author.

      Delete
    4. It's waiting for you to set the sourcetype. I did that so you can pick the source type if you have multiple sources.

      Delete
  6. For those interested, I did some tinkering and have a tomato configuration based on the asus configuration. Kam - if you want to add this to the next version feel free!!

    In props.conf, add the following:
    [tomato]
    #Based on the ASUS model provided in the default props.conf
    FIELDALIAS-dst = DST as dest_ip
    FIELDALIAS-dpt = DPT as dest_port
    FIELDALIAS-proto = PROTO as protocol
    FIELDALIAS-SPT = SPT as src_port
    FIELDALIAS-SRC = SRC as src_ip
    EXTRACT-action = (?i) .*?: (?P\w+)(?= )
    EVAL-direction = if(match(IN,"vlan2"), "in", "out")
    pulldown_type = 1
    LOOKUP-action_lookup = action_lookup action OUTPUT action2
    LOOKUP-rdns = dnsLookup ip AS dest_ip OUTPUTNEW host as rdns_host

    NOTE: on the EVAL-direction, you may have to change the evaluation parameter on the IN from vlan2 to whatever your "external" interface is named. That way it will properly evaluate the data to determine traffic coming inbound.

    In transforms.conf:
    [tomato]
    REGEX = tomato
    SOURCE_KEY = MetaData:Host
    FORMAT = sourcetype::tomato
    DEST_KEY = MetaData:Sourcetype

    Then set your source type to be tomato and good to go!

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. Thanks Reuben! Now that I have some time to devote back to this project, I will gladly add this to the next release of the app. I will also post some videos on how to setup and troubleshoot some common issues.

    ReplyDelete
  9. Hello,

    I am working on setting up a FiOS Quantum G1100 Router and here is where I am at:

    My transforms.conf file has been changed to:
    [fios]
    # Make sure that this matches the hostname of your router, fios is just an example.
    REGEX = 192.168.136.1
    SOURCE_KEY = MetaData:Host
    FORMAT = sourcetype::fios
    DEST_KEY = MetaData:Sourcetype
    FORMAT = homemonitor
    DEST_KEY = _MetaData:Index

    The idiots that built the logging function could not keep with the industry standard of src_ip, dst_ip, etc. They had to go and change things, and much of what they changed looks like it resembles the skyhhub router. So, I changed the fios section of the props.conf file to the following:
    [fios]
    FIELDALIAS-dst = DST as dest_ip
    FIELDALIAS-dpt = DPT as dest_port
    FIELDALIAS-proto = PROTO as protocol
    FIELDALIAS-SPT = SPT as src_port
    FIELDALIAS-SRC = SRC as src_ip
    EXTRACT-action = (?i) kernel: (?P[^\-]+)
    pulldown_type = 1
    LOOKUP-action_lookup = action_lookup action OUTPUT action2
    LOOKUP-rdns = dnsLookup ip AS dest_ip OUTPUTNEW host as rdns_host
    #EXTRACT-direction = ^(?:[^ \n]* ){9}(?P\w+)
    #EXTRACT-action = ^(?:[^ \n]* ){10}(?P\w+)
    #EXTRACT-state,protocol,src_ip,src_port,dest_ip,dest_port = ^[^\]\n]*\]\s+(?P\w+\s+\w+)[^:\n]*:\s+(?P\w+)\s+(?P[^:]+):(?P[^ ]+)[^\[\n]*\[(?P[^:]+):(?P\d+)
    #EXTRACT-src_ip,src_port,dest_ip,dest_port = ^[^\(\n]*\(\w+\s+(?P[^:]+):(?P\d+)\->(?P[^:]+):(?P\d+)
    #LOOKUP-fios = action_lookup action OUTPUTNEW action2
    #LOOKUP-rdns = dnsLookup ip AS dest_ip OUTPUTNEW host as rdns_host

    This gets data into Splunk under the homemonitor index. It also populates the "Total Number of Events" section on the Home Network Overview Dashboard, but everything else is either 0 or "No Results Found". Here are some lines of the logs from syslog:

    Dec 25 20:35:36 192.168.136.1 ulogd[570]: Accepted IN=br-lan OUT=eth0 MAC=48:5d:36:24:c6:ba:00:17:f2:06:8c:06:08:00 SRC=192.168.136.236 DST=109.201.154.156 LEN=1521 TOS=00 PREC=0x00 TTL=63 ID=13349 PROTO=UDP SPT=60721 DPT=8080 LEN=1501 MARK=2
    Dec 25 20:35:36 192.168.136.1 ulogd[570]: Blocked IN=eth0 OUT= MAC=48:5d:36:24:c6:ba:cc:e1:7f:90:41:ca:08:00 SRC=71.252.0.12 DST=108.28.179.247 LEN=125 TOS=00 PREC=0x00 TTL=248 ID=60337 DF PROTO=UDP SPT=53 DPT=33456 LEN=105 MARK=0
    Dec 25 20:35:42 192.168.136.1 ulogd[570]: Accepted IN=br-lan OUT=eth0 MAC=48:5d:36:24:c6:ba:00:17:f2:06:8c:06:08:00 SRC=192.168.136.236 DST=109.201.154.156 LEN=1537 TOS=00 PREC=0x00 TTL=63 ID=60854 PROTO=UDP SPT=60721 DPT=8080 LEN=1517 MARK=2
    Dec 25 20:36:16 192.168.136.1 dhcpd: DHCPACK on 192.168.136.95 to 19:3c:dc:7e:72:59 via br-lan
    Dec 25 20:37:30 192.168.136.1 IGMP: leave group 224.0.1.60 on if 2
    Dec 25 20:37:30 192.168.136.1 IGMP: join failure group 224.0.1.60
    Dec 25 20:40:07 192.168.136.1 ulogd[570]: Blocked IN=eth0 OUT= MAC=48:5d:36:24:c6:ba:cc:e1:7f:90:41:ca:08:00 SRC=14.158.112.131 DST=108.28.179.247 LEN=60 TOS=00 PREC=0x00 TTL=53 ID=10995 DF PROTO=TCP SPT=29835 DPT=23 SEQ=450789324 ACK=0 WINDOW=5808 SYN URGP=0 MARK=0
    Dec 25 20:40:10 192.168.136.1 ulogd[570]: Blocked IN=eth0 OUT= MAC=48:5d:36:24:c6:ba:cc:e1:7f:90:41:ca:08:00 SRC=14.158.112.131 DST=108.28.179.247 LEN=60 TOS=00 PREC=0x00 TTL=53 ID=10996 DF PROTO=TCP SPT=29835 DPT=23 SEQ=450789324 ACK=0 WINDOW=5808 SYN URGP=0 MARK=0

    I am guessing more changes need to do be made and appreciate any assistance you can provide. I can provide additional log files if this would help.

    Thanks in advance!

    ReplyDelete
    Replies
    1. This was a fun one to fix. First, let's start with the props.conf file. Here is what I created for your Quantum modem :

      [quantum]
      FIELDALIAS-dst = DST as dest_ip
      FIELDALIAS-dpt = DPT as dest_port
      FIELDALIAS-proto = PROTO as protocol
      FIELDALIAS-SPT = SPT as src_port
      FIELDALIAS-SRC = SRC as src_ip
      EXTRACT-action = ^[^\]\n]*\]:\s+(?P\w+)
      EVAL-direction = if(match(OUT,"eth*"), "out", "in")
      pulldown_type = 1
      LOOKUP-action_lookup = action_lookup action OUTPUT action2
      LOOKUP-rdns = dnsLookup ip AS dest_ip OUTPUTNEW host as rdns_host

      Your transforms.conf file is fine, but just keep in mind that you don't need to do too much here, it's only meant to convert syslog to your modem based on the hostname.

      Lastly, make a change to your lookup file, found in the $SPLUNK_HOME/etc/apps/homemonitor/lookups/action_lookup.csv , just add the following lines to it :

      Accepted,ACCEPT
      Blocked, BLOCK

      I would recommend that you make the changes to your props.conf in your local directory ($SPLUNK_HOME/etc/apps/homemonitor/local/props.conf) since any changes to your default directory will be overwritten when I come out with a new version of the app.

      I hope that helps out, I was able to test it with the sample data you gave and it seemed to have worked. Let me know if you run into any issues.

      Thanks,
      Kam

      Delete
    2. Thanks for your quick reply to this. I knew this was not going to be fun or easy, so I figured I would get the assistance of the person who knows it best, thank you.

      I am still running into one problem when splunk restarts, I get the following on conf file check:

      Checking conf files for problems...

      Bad regex value: '^[^\]\n]*\]:\s+(?P\w+)', of param: props.conf / [quantum] / EXTRACT-action; why: unrecognized character after (?P

      One or more regexes in your configuration are not valid. For details, please see btool.log or directly above.

      I am looking at REGEX pages, but thought you would know immediately what the problem may be.

      Thanks again for your assistance!

      Delete
    3. Correct, the line should be :
      EXTRACT-action = ^[^\]\n]*\]:\s+(?P\w+)

      Thanks,
      Kam

      Delete
    4. OK, that's weird each time I paste it in this window, it removes the '' from the line. Let me try and post it again.

      Delete
    5. http://amiracle19.blogspot.com/2015/12/adding-quantam-sourcetype.html

      Delete
  10. This comment has been removed by the author.

    ReplyDelete
  11. For OSX, it is critical that splunk be launched in sudo mode to access UDP 514

    $sudo /.../bin/splunk start

    This took lots of time and searching on the internet to discover an oblique reference to this fact when I could not get homemonitor to work. I recommend adding some emphasis on this critical element in the setup reference instructions. Thanks for putting this App together. I am enjoying using it now.

    ReplyDelete
  12. I really appreciate the information shared above. It’s of great help. If someone wants to learn Online (Virtual) instructor lead live training in Splunk TECHNOLOGY, kindly contact us http://www.maxmunus.com/contact
    MaxMunus Offer World Class Virtual Instructor-led training on TECHNOLOGY. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ pieces of training in India, USA, UK, Australia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Pratik Shekhar
    MaxMunus
    E-mail: pratik@maxmunus.com
    Ph:(0) +91 9066268701
    http://www.maxmunus.com/

    ReplyDelete
  13. After months of avoiding splunk updates and the home monitor update. This is the error after upgrading to Splunk Enterprise 6 Splunk Version 6.6.1 Splunk Build aeae3fe0c5af and home monitor4.5.1 at initial configuration.
    i cant get past the set up page in home monitor app.....which is step one.......i've googled the hell out of this and have come up empty:
    i click save. and this happens at the home monitor inital config page:

    Encountered the following error while trying to update: Cannot find item for POST arg_name="/data/inputs/script/%24SPLUNK_HOME%252Fetc%252Fapps%252Fhomemonitor%252Fbin%252Fspeedtest.sh/disabled"

    splunk server is running on a mac mini osx 10.12.5 sierra.

    any ideas on how to proceed past this??

    Thanks in advance!

    ReplyDelete
    Replies
    1. Let me know if this helps you out: https://github.com/amiracle/homemonitor/wiki/Issues-with-Setup-Page-%28404-Error%29-Fix---work-around

      Thanks,
      Kam

      Delete
    2. This comment has been removed by the author.

      Delete
    3. thanks man, that seems to have helped.....i'm reviewing the dashboards now. thanks!

      Delete