Hive Data Gathering

New house, new opportunities

Blog posts and website have been on hold for a while due to the stresses and general time consumption of moving house. I am surprised at just how much time gets consumed with the whole packing and unpacking process. Next time (not any time soon I hope) we'll hire people to just do it all for us!

One of the features of our new house is that the heating and hot water are heated from a lovely Rayburn Nouvelle aga. Luckily it is a gas powered model, so I don't need to worry about feeding it solid fuel or getting an oil tank in the garden topped up. While it certainly looks great in the traditional style kitchen, it irks me somewhat that it isn't a very efficient. It isn't even a condenser boiler, so a considerable amount of energy is wasted as hot, moist air.

Having just moved house, we're not relishing the idea of fitting a modern boiler, as well as working out where it could reasonably be placed.

In our previous house we had a Nest thermostat. It looked great on the wall and worked flawlessly for all the years we had the device. I had written a little Flask app to allow guests to control the heating through a web interface when they were connected to the house WiFi without needing to be given details to my nest account, or download an app to their phones. Unfortunately, Nest didn't support the control of hot water boosting through this API, so I was not able to include the "boost hot water" functionality to my app. Not without reverse engineering the API that their web app used anyway. I also captured and logged the temperature and humidity information that I could get out through the API. While the data was interesting, it only gave me the data about a single room in the house.

So when it came to controlling the heating in our new house, I went with the Hive Home system instead. This system comes with far more options than the Nest family of products. I ordered the thermostat, hub and a set of smart thermostatic radiator valves (TRVs), enough for each radiator already fitted with a thermostatic valve.

The way this is intended to work is that each radiator valve can call for a heating boost when it is below its desired temperature. As the valves in sufficiently warm areas will close, hot water from the heating system will bypass their radiator and only the cold areas will receive the heat.

Thermostat

Like the Nest thermostat, it is split into two parts. On the left is the thermostat, it's the bit you put on the wall and interact with. It is rather underwhelming compared to the design and feel of the Nest, but I expect it works just fine. I find it odd that nearly all of the official photos of the thermostat make it look like it has a dark finish, when in reality it has a mirrored finish, so is harder to read than it would be if it was dark.

On the right is the receiver, which is the part that connects to the heating system and calls for heating or hot water.

Both have buttons to call for a heating or hot water boost.

Hub

The hub is an additional device used to connect Hive devices together, and to allow the devices to be controlled from the internet. I believe that it used to have a direct API when accessed from the internal network, but it has since been revoked. Here is hoping that Hive don't go out of business, or decide to end of life the devices I currently have. I am hoping that as Hive is a British Gas venture, they won't just disappear.

Perhaps a future project may be to see if it can be accessed by an internal UART interface or something, but I hope I don't have to worry about that for a long time.

Smart thermostatic radiator valve

The thermostatic radiator valves (TRVs) are where I'm hoping efficiencies can be made - I'll be able to only heat downstairs rooms during the day, and I don't need to heat downstairs at night.

It will also allow me to only heat the lounge/office when I'm at home on my own working during the day, and not the other downstairs rooms that I'll not use.

The TRVs are powered by 2x AA batteries, and on the whole seem to have lasted for a few months on their original batteries, though some have consumed their battery charge faster than others. They seem to drain their batteries fastest when they are unable to connect to the hub.

The TRVs apparently contain multiple thermometers to attempt to determine the temperature of the room despite their rather non-ideal positioning - near the floor, next to a radiator. From the values I've seen from my set of TRVs, they seem to report a few degrees low as I can't quite believe that I'd be comfortable being sat in a 15ºC room. One day I'll take a better thermometer round to get an idea of the actual room temperatures.

A slight annoyance in respect to controlling room temperatures is that while Hive allows grouping of light bulbs, they don't allow TRVs to be grouped. Our living room has three radiators in it, and manually adjusting the temperature of the room requires altering three separate TRV entities through the app.

Signal booster

I found that I could cover all the Hive devices with the hub alone, if the hub was placed in the center of a window sill in the playroom. Had I opted for the more expensive wifi-enabled hub 360, then this may have been a worthy option, though rather at risk of being pulled off the window sill by a 2 year old. As I had bought the standard hub, it only had ethernet connectivity, so this meant that I also needed a wifi device on the window sill. This worked in the short term as the extra wifi device was needed repeat the wifi signal to the other side of the house.

Since I have solved the wifi coverage issue with a couple of Unifi access points, I no longer needed the extra wifi device on the window sill, so the hub would not be able to stay there either. So, the easiest solution was to buy a signal booster and put the Hive hub under the stairs with the server and other equipment then plug the signal booster in to a socket near the edge of the range of the hub to extend it to the far side of the house. I also hope that using the signal booster will mean that the battery-powered TRVs won't need to shout so loud to communicate with the hub, and therefore extend the lifetime of their batteries.

Data gathering

I wanted to get an idea of how temperatures changed in the house as, and how much of the time the boiler was active. There is a chance that this new setup could cause the boiler to be on more of the time than just using a timer.

There doesn't seem to be an official API to access the data, but the app and web interface both use an API. There are a number of projects that make use of this API. I decided to use pyhiveapi as it looked like I could get something up and running pretty quickly.

I effectively wrote a simple wrapper around the pyhiveapi library and handled persisting authentication tokens between executions, and formatted the data returned for whatever would consume the data. My hivereader project can be found on github.

This hivereader script is distributed by Ansible plays, and the metrics gathering and battery check running is scheduled by Sensu go. Sensu then processes the metric responses and ships the data off to Graphite Go.

The script allows for login, metrics, and battery commands.

Login

Before you can start using the API, you will need to authenticate. This is done with the login command:

./hivereader.py login --username USERNAME --password PASSWORD

When using the login command, the username and password options are required. This command handles the two factor authentication via SMS feature, so you'll be asked for the code sent to you by Hive.

Metrics

The metrics output is the Graphite format, only because that's the time series database I currently use. It ought to be straight forward to support other formats if needed in the future.

The script gathers these metrics where available:

• temperature: the current temperature as recorded by the device
• target: the current target temperature that the device is set to
• boost: if the device is currently boosted
• battery: if the device has a battery, the percentage charge is returned

./hivereader.py metrics
hive.aga.temperature 17.7 1615845710
hive.aga.target 7 1615845710
hive.aga_heating.boost 0 1615845710
hive.d_hall.temperature 15 1615845710
hive.d_hall.target 7 1615845710
hive.d_livingroom_dining.temperature 15.4 1615845710
hive.d_livingroom_dining.target 15 1615845710
hive.d_livingroom_garden.temperature 15 1615845710
hive.d_livingroom_garden.target 7 1615845710
hive.d_livingroom_side.temperature 21 1615845710
hive.d_livingroom_side.target 7 1615845710
hive.d_lounge.temperature 14.1 1615845710
hive.d_lounge.target 7 1615845710
hive.d_toilet.temperature 11.7 1615845710
hive.d_toilet.target 11 1615845710
hive.u_benjamin_bedroom.temperature 15.3 1615845710
hive.u_benjamin_bedroom.target 14 1615845710
hive.u_master_bedroom.temperature 14.8 1615845710
hive.u_master_bedroom.target 14 1615845710
hive.u_samuel_bedroom.temperature 15.4 1615845710
hive.u_samuel_bedroom.target 14 1615845710
hive.u_spare_bedroom.temperature 15 1615845710
hive.u_spare_bedroom.target 7 1615845710
hive.aga_hotwater.boost 0 1615845710
hive.d_lounge.battery 40 1615845710
hive.d_hall.battery 80 1615845710
hive.d_livingroom_dining.battery 80 1615845710
hive.d_toilet.battery 80 1615845710
hive.d_livingroom_garden.battery 80 1615845710
hive.d_livingroom_side.battery 0 1615845710
hive.u_master_bedroom.battery 100 1615845710
hive.u_samuel_bedroom.battery 100 1615845710
hive.u_spare_bedroom.battery 80 1615845710
hive.u_benjamin_bedroom.battery 100 1615845710
hive.aga.battery 80 1615845710

The d_ and u_ prefixes are merely to group upstairs and downstairs together in the app to make it easier to manage.

Battery

The battery command is a check of battery charge. The warning and critical threshold can be configured with --warning WARNING_THRESHOLD and --critical CRITICAL_THRESHOLD.

The default warning threshold is 20, and the default critical threshold is 5. I've not looked to see what percentage triggers a device to show the low battery icon.

If a single battery charge percentage is lower than the critical threshold, the check will result in a critical status. If a single battery's charge is lower than the warning threshold, the check will result in a warning status. Otherwise an OK status is returned.

In all cases, the check output lists battery levels for batteries at critical or warning values.

./hivereader.py battery
Critically low batteries:
- D Livingroom side: 0%

Yeah, I really need to replace that battery!

Data visualisations

Once the data has made it to Graphite Go, I can draw nice graphs using Grafana.

Some things to note from these graphs are that when the battery goes flat, the TRVs seem to read a consistent 21ºC, which effectively switched them off.

I also have a graph of battery levels, but it is the most uninteresting graph I have. It is just a series of horizontal lines over the three days or so I have been collecting the data for.

Other thoughts and plans

I have realised that the hot water boost metric does not work yet.

I should probably find a way to record the outside temperature to go with this data, as I'm sure it has quite an impact on how much of the time the heating needs to be active for.

Yes, the downstairs toilet is pretty cold. It has a poorly insulated door to the garage, so doesn't retain heat well in the winter. It would be a cold room with a normally heated house, and to avoid the heating being on all the time to keep the toilet at a reasonable temperature, I decided to allow the room to be cold until we get round to replacing the door to the garage.

I feel I should update the login command to not require the credentials to be included in the command, especially as it is already an interactive command to deal with the SMS 2FA.

I think I'm going to add a low temperature check to ensure something hasn't gone wrong and a room temperature has dropped too low.