Christian Fetzer

Sun and Moon Datasource Plugin for Grafana

2016-07-03T00:00:00+00:00

While working on atMETEO, I found it useful to bring the measured temperature and atmospheric pressure values into relation with the position of the sun or the current moon phase and see how they correlate. Back then when Grafana only supported the Graphite datasource this was done with a simple script that calculated those values every minute and stored them in the Graphite database along with the other measurements.

Now that Grafana supports a sophisticated plugin framework (starting with v3.0), I have created the Sun and Moon Datasource Plugin for Grafana which uses SunCalc to calculate the position of sun and moon on demand when rendering on the frontend. Additional features are the calculation of the moon illumination and annotations for various events such as sunrise or sunset.

Like all Grafana plugins the Sun and Moon Datasource Plugin is now hosted on Grafana.net and the source code is available in a GitHub repository. An example dashboard can be found as well.

Technical aspects

The main advantage of doing the calculation in a datasource on the client side is that the values don’t have to be pregenerated which is not optimal with Graphite because it does not allow to submit future values and therefore the calculation has to be done just in time.

From a technical point of view the plugin is interesting because as of today it is the only Grafana plugin that doesn’t rely on a server providing the data. Therefore it might be a useful example for new plugin authors.

Feedback

When submitting the plugin to Grafana.net I received very positive feedback from raintank, the company behind Grafana and Torkel Ödegaard even demoed the datasource in his talk at this year’s Monitorama which happened to take place two days after I created the Pull Request on GitHub. Thanks for this! A recording is available on YouTube.

Outlook and further work

The current version of the plugin implements all features that SunCalc currently offers. Nevertheless there are already a few ideas for future improvements.

I could imagine that the plugin might be useful when monitoring photovoltaic systems (maybe even solar parks). For this the calculation of the clear sky radiation could be added. Additionally the annotations could be extended to be able to show additional astronomic events such as full and new moon or solstices/equinoxes and perihelion/aphelion.

If you have additional ideas or want to help out for example with the calculation of the mentioned values, please use the issue tracker on GitHub.

Monitoring sensor values of an HP ProLiant MicroServer G7 N54L

2015-11-15T00:00:00+00:00

I’m running a small Linux server (HP ProLiant MicroServer G7 N54L) in my home network as backup drive for desktops and notebooks. The health of the server is critical and therefore I wanted to keep track of some system metrics such as temperatures, voltages and system fan speed in the hope that they can detect hardware problems in advance.

These metrics are typically offered by hardware monitor chips and can be accessed in Linux with the lm-sensors utilities (sensors-detect and sensors). Unfortunately sensors-detect was not able to detect the respective chip (Nuvotem/Winbond W83795ADG). After a bit of research, it turned out that the SMBus/I2C driver for the chipset failed to detect the sensor. Luckily there was already a [email protected]/msg06757.html">patch available that fixed the driver but supported only an older kernel version. The patch was never merged into the mainline Linux kernel and meanwhile a few things have changed in that module. Since I think it’s a valuable change, I ported it to the current kernel version, cleaned it up and re-sent it to the mailing list where it’s currently being reviewed. (UPDATE: The patch has been merged and is included in Linux 4.5.)

For the time being, the patched driver can be found in this GitHub repository including a more detailed description and installation instructions. This should make it simple for anyone interested to test it and use it on their systems.

After installation of the updated driver and loading the modules, sensors is able to show the measurements:

dg-i2c-1-2f
Adapter: SMBus PIIX4 adapter SDA2 at 0b00
Vcore:        +0.88 V  (min =  +0.50 V, max =  +1.40 V)
Vdimm:        +1.51 V  (min =  +1.42 V, max =  +1.57 V)
+3.3V:        +3.30 V  (min =  +2.96 V, max =  +3.63 V)
3VSB:         +3.26 V  (min =  +2.96 V, max =  +3.63 V)
System Fan:   679 RPM  (min =  329 RPM)
CPU Temp:     +34.8°C  (high = +109.0°C, hyst = +109.0°C)
                       (crit = +109.0°C, hyst = +109.0°C)  sensor = thermal diode
NB Temp:      +43.2°C  (high = +105.0°C, hyst = +105.0°C)
                       (crit = +105.0°C, hyst = +105.0°C)  sensor = thermal diode
MB Temp:      +20.8°C  (high = +39.0°C, hyst = +39.0°C)
                       (crit = +44.0°C, hyst = +44.0°C)  sensor = thermistor

The sensor readings can be easily aggregated with collectd’s sensor plugin. Unfortunately the current version of the plugin has one minor limitation and tracks the sensor readings only by metric names (e.g “temp1”) but not by the more descriptive labels (e.g. “CPU Temp”). To address this shortcoming I’ve created a pull request on GitHub. (UPDATE: The pull request has been merged and the feature is included in collectd 5.6.0.)

The following grafana screenshot shows the result:

The drive temperatures were collected using the HDDTemp plugin.

atMETEO - An ATmega based weather station

2015-05-20T00:00:00+00:00

Besides logging and visualizing internet connection statistics I wanted to track also local weather conditions and later eventually integrate the data into a home automation system. While today there are proprietary and free projects available that facilitate this, the topic seemed to be a perfect candidate for a small spare time open source project, which I called atMETEO.

Project goals and introduction

Programming an ATmega based weather station not only allowed to deepen my knowledge in electronics and microcontrollers, at the same time the project served as practical accompaniment while reading Modern C++ Design and C++ Templates and enabled me to explore the advantages and limits of modern C++ on 8 bit hardware.

The essential concept of atMETEO is to read and interpret data from sensors connected to the microcontroller and to transfer this information to a connected PC for further processing, storing and visualization.

Ideally atMETEO should hereby be able to access my already existing Hideki TS53 RF sensors. Therefore a preliminary step was to reverse engineer their data format.

Conducting test automation is essential in order to develop a stable product which can be left running unattended. For this purpose a Jenkins server has been set up that runs Clang Static Analyzer, builds all commits and executes unit tests (including code coverage generation). In addition it flashes new software periodically and maintains statistics on successful and failed attempts to read sensor data for a given time range to identify possible race conditions.

Technical aspects

Even though microcontroller projects are usually hardware centric, the main emphasis has been placed on the software part. My current setup utilizes an Arduino Uno, but atMETEO is prepared to be built for different ATmega boards with only minor adaption. Currently there are 5 sensor types supported: Hideki TS53 Thermo/Hygrometer, DHT22 / AM2302 temperature and humidity module, Bosch BMP180 Digital pressure sensor, Melexis MLX90614 Infrared thermometer and Figaro TGS 2600 air contaminant sensor. The RF receiver as well as the other sensors are connected to the microcontroller as shown in the following breadboard circuit (created with Fritzing).

On the software side the project is divided into two main parts. A libsensors library contains the target / platform independent functionality and algorithms and ships with unit tests that can be executed on the host. All utilities for accessing ATmega hardware features (such as pins, timers, UART, I2C (TWI), SPI) as well as an Ethernet driver (WIZnet W5100) are part of libtarget. The main application makes then use of both libraries in order to send the measured sensor data in JSON format over UART or Ethernet (UDP) to the host.

atMETEO uses the CMake build system which controls cross compilation (including flashing) as well as unit tests (including code coverage generation) and Doxygen documentation.

Detailed information can be found in the project’s readme and the documentation.

Graphical user interface

Graphite and grafana are two excellent tools for logging and graphing time series data. With the command line client atMETEO integrates nicely into this setup as it can be configured to transfer measurements to graphite’s carbon daemon.

The following screenshot shows the atMETEO dashboard I am using to access the measurements from PC, tablet or phone.

Outlook and further work

atMETEO provides weather data now since more than 10 months, even though hardware and circuit are still just built on a breadboard. One of the very next steps therefore is to solder the sensors on a circuit board and fit everything into a small enclosure.

The measurements are transferred to the host in JSON format. While this is relatively easy to generate on the microcontroller, it requires to run an atMETEO client application on the host to interpret the data and process it. Therefore it would be beneficial to switch to a standard format such as MQTT.

Reverse engineering 433 MHz sensors

2014-11-15T00:00:00+00:00

This article focuses on how to decode data sent by proprietary RF 433 MHz sensors using the example of a wireless thermo/hygrometer. Understanding how the sensor works is a first step towards logging and analyzing the data on a computer.

Note that there are already implementations for many popular sensors available online so that there’s a good chance that you will not have to reverse engineer or implement anything on your own.

Visualizing the data

At the very beginning it is essential to understand how the basic RF signal being transmitted looks like. This can be accomplished either with an oscilloscope/logic analyzer or simply with a sound card, Audacity and a voltage divider circuit limiting the 5V from the RF receiver to <1V.

When the experimental setup is functional the first step is to collect a set of meaningful samples for the subsequent analysis. An essential part thereby is that the original receiver is available so that the samples can be annotated with the reference values from the receiver’s display. It proved to be useful to record samples that differentiate only in exactly one value. This technique allows to easily isolate length and position of that metric in the raw signal. Assuming that the length of a metric remains constant for the whole message it can be used to split the message into blocks.

For the example sensor the first analysis reveals the position of temperature and humidity as well as a block size which is illustrated by reference lines in the figure below. It is also noticeable that the two last blocks change as soon as temperature or humidity changes. This indicates that these blocks contain CRC values.

The recordings should also give a good overview on how often the sender transmits and in which time periods. The exemplary thermo/hygrometer sensor sends three times in a row every 43 seconds.

Modulation, bit decoding and interpreting the payload

The next step in the analysis is to determine which modulation is used. Typically RF 433 MHz sensors use line codes such as Manchester, Differencial Manchester or Biphase Mark coding. Other sensors use On-off keying.

Decoding the particular sections from the example above using Biphase Mark coding, where a long pulse represents a binary 1 and a short pulse a binary 0, yields the following results.

111110010 000001001 011100111 011110011 001010101 010000111 101001001 110111111 000111001 011111100
111110010 000001001 011100111 011110011 001010101 010000111 011001001 110111111 110111001 101111010
111110010 000001001 011100111 011110011 011010100 010000111 011001001 110111111 100111000 101100100

The binary representation consists of ten blocks each with nine bits, which indicates that there are eight data bits followed by one parity bit. A cross check reveals that even parity is used.

Subsequently the binary representation has to be transformed into data bytes by applying the correct bit numbering such as MSB or LSB. Electronic systems commonly use Binary-coded decimal encoding a decimal digit in four bits (nibble). Therefore bit numbering variants with reversed nibbles are further viable options.

#1	Display	MSB 0
1	25.4° - 25%	`F9 04 73 79 2A 43 A4 DF 1C 7E`
2	25.4° - 26%	`F9 04 73 79 2A 43 64 DF DC BD`
3	25.6° - 26%	`F9 04 73 79 6A 43 64 DF 9C B2`

#2	Display	MSB 0 (reversed nibbles)
1	25.4° - 25%	`9F 40 37 97 A2 34 4A FD C1 E7`
2	25.4° - 26%	`9F 40 37 97 A2 34 46 FD CD DB`
3	25.6° - 26%	`9F 40 37 97 A6 34 46 FD C9 2B`

#3	Display	LSB 0
1	25.4° - 25%	`F9 02 EC E9 45 2C 52 BF 83 E7`
2	25.4° - 26%	`F9 02 EC E9 45 2C 62 BF B3 DB`
3	25.6° - 26%	`F9 02 EC E9 65 2C 62 BF 93 D4`

#4	Display	LSB 0 (reversed nibbles)
1	25.4° - 25%	`9F 20 CE 9E 54 C2 25 FB 38 7E`
2	25.4° - 26%	`9F 20 CE 9E 54 C2 26 FB 3B BD`
3	25.6° - 26%	`9F 20 CE 9E 56 C2 26 FB 39 4D`

For the LSB bit numbering a closer look on the values in hex shows already the expected decimal values. The humidity can be found in byte seven (0x25 for 25%) and the temperature is located in bytes five and six. Therefore LSB 0 with reversed nibbles seems to be the appropriate candidate for all further proceedings.

The step of working out the correct modulation and bit numbering can be very time consuming because the process only succeeds when at the end a correlation between the byte value and the reference data can be found. This can make several iterations with different parameter combinations necessary.

At the end of this process the complete user payload can be decoded which lays the foundation for starting a basic implementation. The transmitted data typically contains more information which can make the implementation simpler or more robust. Examples of this are described in the next section.

Additional information encoded in the payload

RF 433 MHz receivers usually perform automatic gain control to adjust the reception level to a suitable value. While this is needed to receive data over longer distances and to support weaker signals, it also increases the noise level and complicates detecting the beginning of a message. Especially because messages are only transmitted rarely in order to save energy. To compensate this effect the messages are usually prefixed by a static pattern. For the exemplary sensor every message starts with 0x9F.

Distinguishing sensors

If the original proprietary receiver supports multiple senders at the same time, the protocol needs to be capable of distinguishing sensors using an ID. The thermo/hygrometer sensor sends its ID in the second byte. The ID changes when the battery is removed or when pushing the sensor’s reset button.

Checksums (CRCs)

CRCs are used to ensure that the data has been received correctly. Depending on the used algorithms, they not only allow to recognize transmission errors but also in which area of the payload it appeared and in some cases CRCs even allow to recalculate the correct bit value. These criteria makes it interesting trying to reverse engineer the CRCs as well.

An indication for a CRC value is a byte that changes as soon as any other bit in the payload changes. For the exemplary sensor, this is true for the last two bytes.

The CRC mechanisms being used can be reverse engineered using CRC RevEng by just feeding the tool with some recorded samples.

Using the example data from the thermo/hygro sensor to decipher first CRC1 in byte nine and then CRC2 in byte ten outputs the following CRC algorithms that can later be implemented as explained in this article.

$ reveng -w8 -s 9F20CE9E54C225FB38 9F20CE9E54C226FB3B 9F20CE9E56C226FB39
width=8  poly=0x01  init=0x9f  refin=false  refout=false  xorout=0x00  check=0xae  name=(none)
width=8  poly=0x01  init=0xf9  refin=true  refout=true  xorout=0x00  check=0xae  name=(none)

$ reveng -w8 -s 9F20CE9E54C225FB387E 9F20CE9E54C226FB3BBD 9F20CE9E56C226FB394D
width=8  poly=0x07  init=0xf9  refin=true  refout=true  xorout=0x00  check=0x58  name=(none)

Sensor specific data

In addition to the fields mentioned above proprietary RF 433 MHz protocols eventually contain more data. These are highly sensor specific so that there is no general approach for reverse engineering those. The data includes:

Sensor type (for generic protocols supporting different sensor types)
Payload length (if sensor sends different message types)
Sensor status
Battery information
Message id (current number of message for repeated transmission)

Conclusion

Reverse engineering proprietary RF 433 MHz sensors is possible even with basic knowledge in electronics if the illustrated aspects are taken into account. Especially for a first preliminary implementation not all the protocol details are required and it is often enough to start with only the reference data decoded. More advanced specifics like CRCs can be introduced in a later step or left out completely if the implementation matches the quality criteria.

fritzcollectd - A collectd plugin to monitor FRITZ!Box routers

2014-08-23T00:00:00+00:00

Recently I set up collectd, graphite and grafana to gather and visualize statistics for the home network.

At this, I was particularly interested in monitoring my AVM FRITZ!Box router, especially because I experience some stability problems lately.

The router exposes its status information via UPnP and fortunately there was already a Python package available which allows to access the data from Python scripts: fritzconnection. Hence I decided to implement a module that can feed the data into collectd for further processing: fritzcollectd. (see the GitHub page for installation instructions)

Since a picture is worth a thousand words, this is how it looks:

This dashboard is accessible on Grafana.net.

Alternatives

At the time of writing, there were mainly two alternative approaches documented that are worth mentioning.

At first, there was only a snippet available which is based on a Perl script. Besides the statistics available over UPnP the example also collects additional data scraped from the router’s web interface. Unfortunately even the simple version wasn’t fast enough in my environment to retrieve the data reliably using a 10 second interval.

Secondly, if you cannot use Python or Perl plugins it is also possible to use collectd’s cURL-XML plugin to call the respective SOAP actions directly and parse the results with XPath.

Christian Fetzer

Sun and Moon Datasource Plugin for Grafana

Technical aspects

Feedback

Outlook and further work

Monitoring sensor values of an HP ProLiant MicroServer G7 N54L

atMETEO - An ATmega based weather station

Project goals and introduction

Technical aspects

Graphical user interface

Outlook and further work

Links

Reverse engineering 433 MHz sensors

Visualizing the data

Modulation, bit decoding and interpreting the payload

Additional information encoded in the payload

Header

Distinguishing sensors

Checksums (CRCs)

Sensor specific data

Conclusion

fritzcollectd - A collectd plugin to monitor FRITZ!Box routers

Alternatives