Splunking Power Mains

What happens when you take Splunk, a wireless smart meter and an api interface? Splunking your entire home’s power main, without touching a single power wire! This allows a whole house view of all the power being drawn in real time. The only requirements are raspberry pi on a network, a splunk instance somewhere and a usb adapter plugged into the PI. Hold on, hold on, whats a smart meter? Recently the major power companies have been going around and replacing the old power meters on everybody’s house with a new fancy smart meter. This is essentially just a zigbee enabled power meter than allows power vans to roll down the street and collect all the readings quickly, without having a meter reader go house to house on foot. This page or the page for the a similar provider can be useful when identifying what kind of meter is installed. Then identify the usb dongle that is compatible to the smart meter, most sites have a list of those devices. Then order it up and get it registered via power company. A simple process however it requires paperwork and some lead time. The device that will be used here is the Rainforest EMU-2 Energy Monitoring Unit. / https://www.comed.com/SmartEnergy/SmartMeterSmartGrid/Pages/ForYourHome.aspx https://www.amazon.com/gp/product/B00BGDPRAI Enough of the prework time to dive in. The bulk of the work will be done with a python script that reads from the usb adapter, which is recognized as a serial port on the pi. Then we will install the python script as a service on the pi, which will allow it to run on boot and be easily restarted, and monitored. The python script: There are two requirements here pyRaven and Splunk_hec_handler, those can be found here: The splunk hec handler can also be found in pypi, however at the time of writing the version on pypi was broken and would not pip install, go after the source instead, then do the https://github.com/vavarachen/splunk_hec_handler https://github.com/nonspecialist/pyraven python setup.py install Great we have the requirements and the script. The script is pretty straight forward it connects to the serial port, it will listen to the serial firehose and parse when the data flows across. This is where the usb device is just terrible it has options to query the api but then there is a 4 second lag, during which time the same data will spit out automatically and then again from the query. It is really easier to sit there and listen to the firehose. Regardless the pyraven library does all the work and we are plugging in to it to grab the json object and pass it to a splunkhec function. The splunkhec function receives the data and appends the needed wrapper and posts the event. The posting of the event is then handled by the splunkhec library because why reinvent the wheel and deal with rest posts when we can use a library to do it. What this gives us is a nice json object being loaded into splunk. After we examine the data we can see that a simple splunk search, will give us the graph below. host=gate | timechart avg(event.raw_demand) as Load span=1m The service: In order for the pi to run this script automagically a service template is used and then installed. The service is assuming the the emu.py from above is going to be store in folder. /home/pi/ Alright the next step is to make sure the python script is executable and then install the service template and get it running. That can all be done with the snippet below. Thats all there is to it, the service is running the python script and outputting a whole bunch of nice data to play with. Once the data is acquired then the possibilities are endless for monitoring and reporting out of splunk for high usage or anomalous usage. ***BONUS: (JQ is required for parsing) In order to get the data in to splunk a HTTP collector token is needed. Use curl to generate a token: LINKS: _Splunk indexes and makes searchable data from any app, server or network device in real time including logs, config…_dev.splunk.com Use the CLI to administer HTTP Event Collector | Splunk _This repository allows you to access your Rainforest Automation Emu2 through it's serial interface …_github.com rainforestautomation/Emu-Serial-API _A Python Logging Handler for Splunk HTTP Event Collector (HEC). - vavarachen/splunk_hec_handler_github.com vavarachen/splunk_hec_handler _Interface with a Rainforest Automation RAVEn USB device - nonspecialist/pyraven_github.com nonspecialist/pyraven