Setting up Verizon FiOS Router for Home Monitor **UDPATED**
Log into your router and select Advanced, click Yes to proceed. |
Select "System Settings" |
Scroll down and enable System Logging and Security Logging. Next, Enter the IP Address of your Splunk Server. Now select the Firewall Settings |
Click on the Security Log |
Put the check box on all the items you want logged in Splunk, then click Apply. |
Now log into your Splunk instance and go to the Manager. |
In Splunk, select Add Data and then select Data Inputs. |
Click on UDP |
Follow each step and MAKE SURE to click on the check box for More settings. When complete, click Save. |
When done, your Data inputs page should look like this. |
The homemonitor index was not automatically created when installing the app. I had to manually create it.
ReplyDeleteJust fixed it in the latest version 1.2. Thanks and good catch!
DeleteThis comment has been removed by the author.
ReplyDeleteI'm getting no results found..
ReplyDeleteOn your Splunk server, can you do a tcpdump and look for traffic on udp port 514?
ReplyDeleteNo I'm on Windows 8. I went in and allowed port 514 UDP through the firewall and still no dice.
ReplyDeleteI would run wireshark on your Splunk instance and validate that you are seeing traffic from your FiOS router. You can also validate if Splunk is receiving data by taking a look at the homemonitor index. Search index=homemonitor * All Time.
ReplyDeleteI can see traffic... homemonitor index event count shows only 15 from yesterday all with the following:
ReplyDeleteJan 15 22:58:21 192.168.1.1 Jan 15 23:58:20 2013 Wireless_Broadband_Router Unknown PTR name format
host=192.168.1.1 Options| sourcetype=syslog Options| source=syslog Options
This could mean a couple of things. First, what version router do you have? (Model number and firmware version). Next, what options did you select on your routers configuration page, Information, Warn, or Error?
ReplyDeleteRouter Info:
ReplyDeleteModel Name: MI424WR-GEN3I
Firmware Version: 40.19.36
Hardware Version: I
Both System and Security logging are set to information
In your Splunk instance, go to Manager->Data Inputs->UDP Click on 514 and when it opens, make sure that "Set Sourcetype" is either set to "From a List" and syslog is selected, or you can use "Manual" and type in syslog. Let me know if that works.
ReplyDeleteThe first thing I would do is just do a search in Splunk like this: index=homemonitor and go ahead and do it for All Time. If you see results, then the data is properly flowing into the index. If not, we need to revisit your setup. Are you running your Splunk server on Windows or Linux (Mac OS X)?
ReplyDeleteDid you try running tcpdump on your Splunk box to validate that you're receiving traffic from the FiOS router? (tcpdump -i eth0 UDP) Also, is there a reason you're not running Splunk as root on your Ubuntu box?
ReplyDeleteWhat does the output look like when you run index=homemonitor in Splunk?
ReplyDeleteI figured out what the problem was with the app. It's a simple fix, you just have to go into your Data Inputs (Settings -> Data Inputs -> UDP -> click on the 514). Change the sourcetype from syslog to fios. That will make all your dashboards populate. I've been working on something that will fix that in a later release. Good catch!
ReplyDeleteI just updated the app to version 3.0.4 that has all the fixes listed above. Please feel free to download it and test it out with your system. Thanks for pointing out the issues that were in the app.
ReplyDeleteLet's try and edit the panels and see if we can get them to populate with your data. First, let's make sure you're getting the data with the correct sourcetype. Go into search and type: index=homemonitor | stats count by sourcetype
ReplyDeleteYou can do it for the last 15 minutes.
You should see the sourcetype fios and no syslog.
Send me an email, kamilo@gmail.com and we can do a quick webex or Google Hangout.
Home monitor and pfsense 2.2.1. The search arguments in home monitor are not compatible with the input from pfsense. I have lots of input from pfsense but the surch arguments in the panels. Give zero hits.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteI have a vanilla install of Splunk on a Macbook Pro. I have only installed Stream and Home Monitor 4.0.1. I have reviewed the installation video on youtube for version 2.x and read the blogs. I followed the installation instructions provided. The router was configured to forward syslog to my laptop and I can verified 514 traffic is being received with tcpdump. I created the UDP Data Input, but didn't see FIOS as a source type from the pick list. It's set for syslog. I'm not sure what I'm missing
ReplyDeleteThis comment has been removed by the author.
DeleteThis comment has been removed by the author.
DeleteI have a vanilla install of Splunk inside a CentOS7-vm on my Macbook Pro, I have fusion8pro installed there where the VM runs. I have followed the directions above to the T and I am not seeing anything in the dash board. I have configured the inputs to account for up\514 and I did select the index. One nuance I did notice was that a UDP connection for syslog was in data inputs. So I had to modify that one for the advanced settings. no clue how to fix, would love to see the data......
ReplyDeleteI'm working on a video that will walk you through the install of Splunk, the home | monitor > app and get the data into Splunk from your router / modem.
ReplyDeleteThe easiest way to fix the input is to go into the $SPLUNK_HOME/etc/apps/homemonitor/default and copy the inputs.conf file to the $SPLUNK_HOME/etc/apps/homemonitor/local directory and make the modification there. Once the changes have been made (change the port or whatever change you want to make) then it should start receiving the data.
Make sure that you've either disabled iptables (sudo service iptables stop) on your Linux server OR just add an entry for UDP 514 in-bound (modify /etc/sysconfig/iptables/). Let me know if that works and if you don't see any data.
Can you add support for D-Link router, mine is DIR-655. thanks
ReplyDeleteI am familier with Splunk but new to Home Monitor App. First of all your app is amazing, thank you.
ReplyDeleteSo I got my dd-wrt log to populate and its working great. Then I moved to fios router. I noticed there is no sourcetype where it was mentioned in earlier comments. Do I need to create that sourcetype and if so is the parsing parameters documented somewhere, I couldn't find it?
Currently using Splunk 6.3.3 and Home Monitor 4.4.2
Thank you in advance!
Yebro,
DeleteThanks for downloading and using the app. If you want to change back your sourcetype, you can either do this via the Web GUI or modifying a configuration file using the CLI. The WebUI option is good if you're going to stick with your FIOS sourcetype; however if you’re going to be going back and forth between different routers, it might make more sense to make the change to the configuration files.
The WebUI option is simple, just go into Settings -> Data Inputs -> UDP 514 and then over ride the source type as fios.
The CLI option is a more scalable solution if you’re going to be jumping around from different sources. First, get the hostname for your router. For this example, I’m going to use “home.fios” as the hostname. To make the change, log into your Splunk server and go to $SPLUNK_HOME/etc/apps/homemonitor/local and edit the transforms.conf file. The transforms.conf file typically does not exist in this directory, just simply create it and add the following entry. Under the REGEX = field, just make sure that you set the entry to your hostname.
[fios]
# Make sure that this matches the hostname of your router, fios is just an example.
REGEX = home.fios
SOURCE_KEY = MetaData:Host
FORMAT = sourcetype::fios
DEST_KEY = MetaData:Sourcetype
Once you’re done making the changes, go ahead and restart your Splunk instance. Now, let’s say you want to add a dd-wrt router (hostname = ddwrt.home), you can add this line to the same transforms.conf file :
[dd-wrt]
# Make sure that this matches the hostname of your router, fios is just an example.
REGEX = ddwrt.home
SOURCE_KEY = MetaData:Host
FORMAT = sourcetype::dd-wrt
DEST_KEY = MetaData:Sourcetype
Now, anytime a machine sends data in as syslog then Splunk will look at the hostname. If it matches ddwrt.home, it will set the source type to dd-wrt; or if it’s home.fios then it will set the source type to fios all automatically.
I know it’s a long winded answer, but that should cover you. Let me know if you have any additional questions.
Thanks,
Kam
Thank you Kam,
DeleteAs for props.conf file, how would the fios parsing parameters look like. I wasn't able to see "fios" as a sourcetype however when I tried creating it, it said I was already created.
Thank you again
If you want to see how the fields are created for the fios sourcetype, go to $SPLUNK_HOME/etc/apps/homemonitor/default/props.conf . Don't modify the file there, any changes will be overwritten when I make any updates.
DeleteThank you Kam,
DeleteSo I tried using those fields for my fios box but did not match. Here are few examples from my fios box:
Feb 17 00:22:29 192.168.1.1 Feb 17 00:22:28 2016 fios IN: ACCEPT [57] Connection closed ( : UDP 8.8.8.8:53 <-->8.8.8.8:53 [192.168.1.2:59059] br0 Route Incoming UNSECURED )
Feb 17 00:22:12 192.168.1.1 Feb 17 00:22:11 2016 fios OUT: ACCEPT LAN-OUTBOUND [38] Wireless Broadband Router initiated traffic (UDP 100.29.130.190:1024->68.254.0.13:53 on eth1)
Feb 17 00:22:12 192.168.1.1 Feb 17 00:22:11 2016 fios Empty name
Got it to work. I know not to tweak this file /opt/splunk/etc/apps/homemonitor/default/props.conf but I noticed no way to get it working with current config. so here are my changes/correction.
Delete[fios]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
category = Custom
disabled = false
pulldown_type = true
EXTRACT-direction = ^(?:[^ \n]* ){9}(?P[^:]+)
EXTRACT-protocol = ^[^\(\n]*\(\s+:\s+(?P\w+)
EXTRACT-action = ^(?:[^ \n]* ){10}(?P\w+)
EXTRACT-state = ^[^\]\n]*\]\s+(?P\w+\s+\w+)
EXTRACT-src_ip = ^[^\(\n]*\(\s+:\s+\w+\s+(?P\d+\.\d+\.\d+\.\d+)
EXTRACT-src_port = ^(?:[^:\n]*:){7}(?P\d+)
EXTRACT-nat_ip = ^[^>\n]*>(?P\d+\.\d+\.\d+\.\d+)
EXTRACT-dest_ip = ^(?:[^\[\n]*\[){2}(?P\d+\.\d+\.\d+\.\d+)
EXTRACT-dest_port = ^(?:[^:\n]*:){9}(?P\d+)
EXTRACT-interface = ^(?:[^\]\n]*\]){2}\s+(?P[^ ]+)
EXTRACT-reason = ^[^\[\n]*\[(?P\d+)
EXTRACT-config_change_user = ^[^\(\n]*\(\w+\s+\w+\s+(?P\w+)
LOOKUP-fios = action_lookup action OUTPUTNEW action2
LOOKUP-rdns = dnsLookup ip AS dest_ip OUTPUTNEW host as rdns_host
This comment has been removed by the author.
ReplyDeleteThat's great, if you want just copy that stanza to your $SPLUNK_HOME/etc/apps/homemonitor/local/props.conf file. This way when I make an update it will not overwrite your changes.
ReplyDeleteHello Kamilo,
ReplyDeleteIs google maps no longer available via Splunk Apps or add ons? I'm not longer able to find it through splunk enterprise, the latest version 6.3.3. I was trying to install it as instructed by the video. Any insight?
thanks in advance!
I think I just answered my own question, It's not available in Splunk Enterprise 6.x - so that makes me wonder about home monitor and the latest 6.x enterprise version and mapping....just wondering if you're now using built in splunk maps or some other method? keep up the good work!
DeleteYes I started using the defaul maps that come with Splunk enterprise.
DeleteSo when you say default maps that come with Splunk Ent, that means that i do NOT have to download other apps to install correct? I cannot get any of the maps to work unfortunately. My input is fios, when i click on Map of Connections.....just sitting stale on Waiting for input.... is there any trick to getting these maps to populate on that dashboard? any files to manipulate? thanks in advance.
DeleteKamilo,
ReplyDeleteappologies, reposting as its own post now.
just wondering if there is any update to Craigs issue, I didn't see a remedy/response in this thread. I've been poking around many related threads and I'm still having issues.
I have a similar problem, I'm using the latest version of Splunk Enterprise (6.3.3) on a mac mini, El Capitan, with the latest version of Home Monitor (4.4.2).
I've been reading several blogs and searched this one and the videos. That said, I think I have it correctly configured, please let me know if i'm way off; In Data Input - I've configured the UDP 514 as instructed, but with the update of using 'fios' in the place of 'Source name override' , set source type manual, picked from the list 'syslog', then I checked the more settings, and for host, selected the button for custom. I've tried whatever popped up as default here, left it blank, and since one of your posts says use the router hostname (i take it that that means the fios host name, your router hostname, found in Fios system settings as 'Wireless System Hostname' in which i put as 'fios')
note from https://4dd0p3r470r.wordpress.com/2015/01/31/how-to-splunk-home-monitor-for-fios-routers/
I followed this as well as this was a little vague in repeated postings: IMPORTANT UPDATE(!): in the field “Source name override” put “fios” instead of the “syslog” - i did this - let me know if thats wrong too.
which leaves the final thing in the Data set , i chose homemonitor as the index, and hit save....
unfortunately, nothing is populating in my homemonitor. Not sure what I'm doing wrong here.
I also saw this thread; https://answers.splunk.com/answers/310324/home-monitor-how-to-configure-a-netcomm-modem-rout.html#answer-310832
in which you answer about Mac osx , 'If you're running Splunk on Linux (or Mac OS X), then you'll either have to run Spunk as root or change the port that your modem is sending syslog to 1514 UDP.'
-I assume I am running as root, im the Admin on the machine, i even enabled the root user, but not sure if thats an issue with simply executing the Splunk icon when i launch it. so I think it would run as me being the admin which should be root - let me know if i'm mistaken.
'change the port that your modem is sending syslog to 1514 UDP' - I assume that you are saying change your Fios router to send the syslog via port 1514? I cannot figure out how to change my Fios router to change that......(i also tried changing the Data Set back and forth to 514 and 1514 via the web UI and tried hard coding into the .conf files) niether of which made a difference.....still no traffic coming to the splunk server from Fios router.
I also used wireshark to check for anything UDP coming from my Fios router, and i havent seen a single packet for hours using 514 and 1514, i even cleared my logs and started them over to generate current traffic. My NTP is synched too on fios and splunk.
I guess that might explain why i cant get home monitor to show anything as well on Data Summary and trying in the search index=homemonitor sourcetype=fios, and index=homemonitor | stats count by sourcetype
I've burned 8 hours on this today - really not sure what else to try....any ideas? by the way, great work! I really love what it can do and cant wait to get it working! thanks in advance.
First, when you run Splunk you are probably not running it as root. In order to run splunk as root you will have to go into your terminal and type sudo "$SPLUNK_HOME/bin/splunk start" . That will start splunk as root on your machine. If you want to have it start automatically as root, you will need to run "sudo $SPLUNK_HOME/bin/splunk enable boot-start -user root"
DeleteI've used the environment variable $SPLUNK_HOME, if it's in your profile, then you should be fine to run the commands as listed above, otherwise go to your splunk home directory (/Applications/Splunk I assume) and run sudo bin/splunk start . It will prompt you for your admin password and then run the command.
Once you've finished that, then you should start seeing data come into your Mac on port UDP 514.
(http://docs.splunk.com/Documentation/Splunk/latest/Installation/InstallonMacOS)
Let me know if you have any additional questions.
This comment has been removed by the author.
DeleteThanks Kamilo! I was able to get data after running it as root on the mac mini. I thought I was the whole time, but I was mistaken. Thanks for the step by step. I still dont see any 514 or 1514 traffic in wireshark, but I guess thats another story.
ReplyDeleteI am getting data from 'fios' and most of my dashboards are populating.
Under Overview Dashboards >Network Event Overview is the only one that is NOT populating, in that category of dashboards.
The others not populating are the Map of Connections , and the Bandwidth Overview is not populating (followed your instructions to enable the data input on this blog)
I also wanted to note that i see nothing in Search>Data summary - i dont think thats correct.
thanks in advance! love what i see so far!
This comment has been removed by the author.
ReplyDeleteHey Kam, I really love your app and after reading some posts I have got Most of it working except for the fios router config dashboard, not sure what I am doing wrong, pretty much everything else is working
ReplyDeleteDepending on what Verizon did they might have made some changes to the way the app can extract the fields necessary for the dashboards.
ReplyDeleteI'd recommend looking at the extractions in the props.conf and seeing which ones work and fix the ones that don't. If you look on the wiki for my github, it shows how to troubleshoot adding extractions.
Let me know if that helps.
Kam
Well, I have done alot, ill keep looking for something that relates to the state. Another issue I am having is on the home network overview and bandwidth overview I cant get the bandwidth panels with the avg upload or download or ping to work. I changed the python script to what you have on github but still nadda.
ReplyDeleteurls = [
ReplyDelete'https://www.speedtest.net/speedtest-servers.php',
'://www.speedtest.net/speedtest-servers-static.php',
'://www.speedtest.net/speedtest-servers.php',
'://c.speedtest.net/speedtest-servers-static.php',
'://c.speedtest.net/speedtest-servers.php',
]
Any ideas on the issue with the bandwidth overview yet?
ReplyDeleteKam, I'm trying to get the app to work for the first time. I believe I have the router and application configured correctly but I do not see any data. I'm using Windows10 and have configured my router to log to the local machine running Splunk. I've set UDP source name to fios and alternatively to syslog. I've reviewed the blog and youtube video but still can't get it to work. Could it be an issue with a windows10 setting?
ReplyDeleteThanks, Gilbert
Hello Kamilo,
ReplyDeleteLooks like I have a different Fios log format, as such the dashboards are not populating. This seems to be caused by the "Destination" field is not being properly extracted. My 2 log formats ("Accepted" and "Blocked"), captured in SPLUNK are as follows:
Accepted:
12/29/16
10:11:40.000 AM
Dec 29 10:11:40 192.168.0.1 ulogd[582]: Accepted IN=br-lan OUT=eth0 MAC=11:1d:11:1f:11:11:11:f1:1f:ce:11:a1:11:11 SRC=192.168.0.152 DST=172.111.1.11 LEN=40 TOS=00 PREC=0x00 TTL=127 ID=21567 DF PROTO=TCP SPT=43573 DPT=443 SEQ=3940399687 ACK=2770573242 WINDOW=255 ACK URGP=0 MARK=2
Blocked:
12/29/16
10:10:39.000 AM
Dec 29 10:10:39 192.168.0.1 ulogd[582]: Blocked IN=eth0 OUT= MAC=11:1d:11:1f:11:11:11:f1:1f:ce:11:a1:11:11 SRC=175.117.3.42 DST=192.168.0.51 LEN=40 TOS=00 PREC=0x00 TTL=58 ID=354 PROTO=TCP SPT=443 DPT=45784 SEQ=2546652287 ACK=0 WINDOW=0 RST URGP=0 MARK=0
From what I see is that "Accepted IN" is outbound traffic on br-lan. And "Blocked IN" is inbound traffic on eth0. Any suggestions on updating the Destination REGEX?
Thanks
I'm getting bandwidth populated but that's it; running on mac. one thing to note is when running the root command in terminal i'm getting, "sudo: /bin/splunk: command not found" could that be the problem?
ReplyDeleteTo validate that the script works, go to the splunk home directory (where you installed Splunk) and then navigate to the etc/apps/homemonitor/bin directory. Once there, you should be able to run the speedtest.sh command and see some results:
Deletemac-mbp:bin$ sh speedtest.sh
2017-01-19 17:11:51
Ping=27.083 ms
Download=100.19 Mbit/s
Upload=47.13 Mbit/s
This also assumes that the SPLUNK_HOME environment variable has been defined, which is the case if you installed splunk. Just run 'env' command to see the variables.
Great post! I got it up and running and with Splunk not as root (which is recommended by Splunk). One just needs to set up a rule to forward 514 to say 5514. So, for Centos7x64 this would be:
ReplyDeletefirewall-cmd --zone="public" --add-forward-port=port=514:proto=udp:toport=5514
Now going through all the other Help stuff to get the other widgets working.
Just installed splunk 6.6.2 on my Mac and installed home monitor to look at FiOS router. I tried to follow your setup above, but I'm not getting any data into splunk. Note with 6.6.2 when I configure the UDP setting and select more settings the input to set address of the router is not available unless unless I select custom. Tried that, but I'm still no seeing data.
ReplyDeleteReading about I saw you suggested executing the speedtest as a test. The SPLUNK_HOME ENV wasn't set although I did install splunk. I set SPLUNK_HOME and ran the speedtest. Received below. Suggestions?
2017-07-19 09:12:23
Ping=9.973 ms
Download=57.12 Mbit/s
Upload=37.00 Mbit/s
I'm new to Splunk and figured Home Monitor would be a good starting point... I have a HomeLab running on Windows2016 Standard for the VMworkstation, Splunk and a Bastion Host. A pfSense v2.3.4 as a VM, I have a Splunk Ent v7.1.1 running in a VM, I installed Home Monitor 4.5.1 and configured it per the video and Blog... but no Data in the Index="homemonitor". I set up Kiwi syslog server on a third system and had pfSense configured to send syslogs to both the Splunk Ent and to the Kiwi system. So I'm getting logs to the Kiwi server and to confirmed that syslogs are getting to the Splunk system via Wireshark. Is it phSense vs FIOS or is there something else you could suggest?
ReplyDeleteSo after going back to Ent v6.2.3 I was able to get data into the index=homemonitor. :-) But all the fields in "Home Network Overview" show "Waiting For input or data", Any thought on how I should proceed?
DeleteFirst question is did you setup Splunk to listen on UDP 514? By default Splunk is listening to that port once you enable it via the UI (Settings -> Data Inputs ->UDP->514 enabled). If this is a Linux based machine then you’ll need to be root to enable that port. Another option is to setup a Splunk forwarder to send data from your syslog server into Splunk.
ReplyDeleteI actually have both pfsense and FiOS quantum data flowing into Splunk via a syslog (rsyslog) server and a Splink forwarder.
Let me know if that helps, and we can work on it from there.
So after going back to Ent v6.2.3 I was able to get data into the index=homemonitor. :-) But all the fields in "Home Network Overview" show "Waiting For input or data", Any thought on how I should proceed?
DeleteSo the pfSense FW has "Remote Syslog Contents" has Everything checked,
DeleteAll you need to do now is pick a sourcetype (pfsense, quantum etc.) and the dashboards should populate assuming there is data there.
ReplyDelete