HiveMQ Cloud for Home and Community IoT Projects

The rise of smart homes and IoT technology has made it easier than ever to monitor and optimize various aspects of our daily lives. With over 30 billion IoT devices projected to be in use by 2025, the need for efficient data transmission protocols like MQTT is more critical than ever. MQTT plays a crucial role in IoT home projects by providing a lightweight and efficient protocol for data transmission, ensuring seamless communication between various devices. This makes it ideal for managing and optimizing energy usage and other critical metrics in smart homes.

In this insightful interview, Gerhard Müller, Senior Lead Software Engineer at Software AG, shares his personal journey into the world of IoT with Kudzai Manditereza, Senior Industrial Solutions Advocate at HiveMQ. Despite his professional focus on business process management and dashboarding, Gerhard embarked on a passion project to monitor and optimize energy consumption in his home and local churches using HiveMQ Cloud Serverless. His story is a testament to the power of MQTT and cloud technologies in creating impactful, real-world applications.

Below, we explore Gerhard's experiences and his insights on using HiveMQ Cloud Serverless.

Transcript of the Interview

Kudzai Manditereza: Gerhard, thank you for taking the time to join me this afternoon to talk about your project on how we are using our HiveMQ Cloud Serverless option for your project. So maybe before we begin, can you tell us a bit about yourself? What do you do and things like that? 

Gerhard Müller: Yeah, sure. So my main job is not related to MQTT at all. It's more a business software. I work for Software AG, which is a well-known company in Germany with a long tradition. It is more than 50 years old.

My work is more related to business process management. Particularly, I work for dashboarding, which is mainly for business KPIs. But this dashboarding, for example, could also be used to display line charts from my sensor data and things like that.

Regarding MQTT, that is more of a private project. I do it for my own house, monitoring my heating system, the solar thermal system, a lot of sensors, even air quality carbon dioxide in the living room and things like that. I also do similar monitoring for two churches in our region – in my hometown and another in the neighboring town, which is mainly for saving energy costs where those heating systems are very old and we try to find out where to save energy and money. For that purpose, I'm using the HiveMQ Cloud.

Kudzai Manditereza: That's good to know. And it's super interesting that you are what you call like an unrelated field where you're like working with dashboarding and like building all these reporting tools, which I found very interesting because a big part of IoT involves that aspect of having to visualize. You can wire up all your sensors, you can wire up all your systems, even like in industrial applications, at least I've come across situations where now you need to like present that information in an intelligible way. Whoever is consuming that information, I would imagine this kind. And, you have been bringing your expertise from the dashboarding world to the IoT space to provide visualization for your IOT project. 

Do you have visualizations that you could share or is it is it a web-based platform where you're able to visualize all of this stuff? 

Gerhard Müller: Yeah, I have actually several options for doing that because I'm using this field of programming also for learning new stuff, for example, just recently, I wanted to learn a bit about REACTJS to UA framework.

And yeah, I thought what could I do for a first experiment? And I found that it might be interesting to also visualize my sensor data. And that was mainly because I'm also a bit familiar with Android’s programming and Android apps, but most of the people in our church community are using Apple iPhones.

And, so I decided to do it web-based and that was the thing I was doing. Besides that, I'm also using more sophisticated Grafana dashboards, which I directly connect to my Postgres database, which is historian in my architecture, and besides that, the first thing I implemented for reporting was just sending out emails on a daily basis.

And here I included an HTML report with vector graphics and also just commerce CSV files, which I can then import into Excel or whatever. 

Kudzai Manditereza: I see. So maybe let's take a step back a bit and start from so let's start from your home project.

And, So your idea here was to say, you want to look, get some visibility into energy usage, and identify areas where you could serve some energy. What was the first step? Was it identifying the sensors to say, okay, how could I get data out of this system?

Can you talk to us about what systems energy systems are you harvesting data from? And how did you choose the sensors to get that data out before you even thought about putting it into MQTT? Can you talk to us about that first part? 

Gerhard Müller: Yes, sure. Actually, I started with my heating system. So I did that very early, to be honest.

So my house was built in 2004 or we moved to the new flat or the new house in 2005. And, yeah I saw that my heating system was all the time starting up, shutting down, starting up, shutting down. And so that happened in the starter of the burner in the heating system.

It was about 20 times per hour. And that is a lot. And so I found out that something must be wrong with the parameters in the heating system. And I started to install some sensors just to the flow and return of my heating system. And yeah, I found out that just reducing heating reduces too much energy for the house.

It does not need so much power. By reducing the power of the heating system I found a way to slow it down. And now I have maybe one or two Burner starts per hour, which is a lot better. The heating is now running in a manner that really makes sense. And I can see that the temperature of the flow also reacts to the outside temperature and things like that.

And yeah, then I started to do the same measurements for my solar thermal system. And, from the beginning, I did that with the D-S-A-T-P 20 sensors from tele semiconductor, but I think in the meantime the company changed the names or don't know.

Yeah. First I did that with a USB adapter, but now I'm doing the same with Raspberry Pi and I'm now using TinkerForge hardware to connect the one way up us. And that is really very comfortable and reliant works out pretty good.

Kudzai Manditereza: Okay. So are you reading that OneWire from, let's say from your solar system, what sort of interface are you reading analog signals directly? Or are you reading via a serial? What sort of interface are you using in Raspberry Pi to get data from your solar system?

Gerhard Müller: Yeah, as I said, I'm using I think porch hardware, which does the communication with the bus, actually, so I don't have to care for that. And TinkerForge is very strong in providing comfortable APIs for developers. It's very easy to connect to the TinkerForge hardware with Java or Python or whatever language you want.

And I think it even supports MQTT directly on their boards, but I'm not sure. So I'm currently using Python to read the data. 

Kudzai Manditereza: Oh, okay. So basically TinkerForge gives you those device libraries for all the different sensors that you can then use within your Python environment to just read the data, connect to those things without really even worrying about the underlying protocol, how all of that information has been, which really I think it's a good plug and play system to work with. So now, and then you're moving from your data from your actual physical hardware using your TinkerForge sensors into your Raspberry Pi into your Python environment.

Now we need to push this data out into HiveMQ cloud serverless. First of all, can you speak to us about the MQTT library that you're using for that upstream connection? 

Gerhard Müller: Yeah, I think in Python, but yeah, I'm not a hundred percent sure to be honest. Yeah, but I think it is.

Kudzai Manditereza: Yeah. That's the most common one, at least that I've seen a lot of people use the library on Python. So are you collecting data? How are you deciding to call it to collect it? So are you just pushing all of this because I know you mentioned that you've got a Postgres database, which is like testing as a historian.

So are you first aggregating the data on the Raspberry Pi? Are you performing some calculations, or are you just moving all of that data straight into the cloud? 

Gerhard Müller: Yeah so what I'm doing on the Raspberry Pi is I read the data sensor by sensor every minute or some every two minutes and then I'm attaching a timestamp to each value. I know that I have two timestamps. One is the timestamp of the read out of the sensor. And I have a timestamp when it's received on my Linux machine and the Internet so that I can check what delays are involved. On the Raspberry Pi, I then create a data structure in JSON, which contains some meta data about the sensor, for example, the unit of value and timestamp. And then I compress this JSON bundle of all the sensors and the payload in MQTT, and that is again encoded in base 64.

Kudzai Manditereza: Okay, I see. Can you talk to us about what was compelling about MQTT? What made you say, okay, I want to use MQTT for this data transportation. Why not HTTP? Or can you talk to us about your decision there? 

Gerhard Müller: Yeah. On one hand MQTT is leightweight. So, I also have some really small devices, for example, ESP8266, which I use for measuring the gas consumption and also the electrical power consumption.

And implementing on these devices, you have to program in C. I know that it's not fun to program HTTP clients and things like that in C. On the other hand, it is really easy to use MQTT library on those tiny devices. That, I can directly connect to my cloud broker and almost the same as applies also to the Raspberry PI.

It's all kinds of small devices and just using this MQTT library and pushing everything to the broker. The natural way for me in the meantime is I would not even think about a different way to do that. 

Kudzai Manditereza: Yeah, absolutely. That's super interesting. I want us to talk also about how we're going to move from the internet side of things to the cloud side of things.

But before we do that can you just provide context on what exactly for your house, what sensor readings are you specifically collecting? Is it about air quality? You can just give us a list of the sensor readings that you're collecting.

Gerhard Müller: Yeah. It would also be possible to show you one of those dashboards if that's interesting for you. Yes. 

Kudzai Manditereza: Yes. I can see your web browser. 

Gerhard Müller: Yeah to sum it up, this is now the total energy consumption of the whole house. Every piece of hardware, which is connected to to my house is summed up in this chart, you may see here, we have some higher peaks like here. Most probably that's the dishwasher or the laundry. I don't know what exactly it is, but it's those high peaks that are making the devices in our house really consume a lot of energy and as you can see here, it's just the base consumption and by reducing.

So when I started to do this kind of measurement I found out that at no point in time my house was not consuming less than 100 Watts. And so I started to investigate a bit and in the meantime I have reduced that. Now, I have 60 or 70 watts. So that means this base consumption I reduced it to about 30 Watts every day.

And, if you calculate that over the whole year, that's a lot. I was surprised. And that shows more or less the way how I approached to things. I was surprised how much energy really the house consumes even when nothing's ON. I think it's the internet router and the satellite receiver and things like that that are consuming energy 24 seven the whole year.

That's one of the things I measure and maybe a bit more complicated dashboard is this one. So that shows all the data that I was talking about. For example, here you can see the solar thermal system at the end. Here you can see that currently my solar thermal system is really a harvest energy because temperatures are rising and this is the heating.

And here you can see that I had some starts of the burner this morning because it was below 14, 15 degrees, I think. And here we had the heating do some charging of the hot water tank. And yeah, that was mainly because you can see that the temperature in my tank fell down a lot and most probably someone had a shower and that's why the temperature of the warm water was too cold so that it was heated up here by the heating.

And round the clock, the solar thermal system is at work. [00:17:00] Yeah. And these are the things I'm measuring and also some indoor temperature and even the water temperature of my aquarium, the fish tank. I'm checking here from time to time. 

Kudzai Manditereza: Yeah. That's some really impressive graphics. So you seem to have really put some thoughts into this.

And I imagine this is something that you have like building over time because already you've got all the data coming in. Now you're able to then think about, okay, I just want to see data from the fish tank right now. I just want to see data from this and that. So I want this information.

I would imagine you're sending it as different. So you mentioned that you're packaging all this into a JSON payload. So these are all different JSON payloads that you send at different intervals. Or is it coming in one JSON payload information about everything every minute, like you mentioned? 

Gerhard Müller: No, there are different ones.

For example, what I showed here – this one sent the sensor data directly from an ESP 8266 to the club broker so that is just one Jason structure containing one single value or there are two honestly, because power and energy. But that's one device which directly publishes to the broker.

And here we also have different devices. For example, these are temperatures that are measured with these DS80B20 sensors, but these are wireless sensors in different rooms. For example this – heating and solar terminal, that's all packed into one JSON structure.

And I'm sending that I think every two minutes to the cloud broker as a bundle. 

Kudzai Manditereza: Okay. I see. All right, so you're sending all these data points to the MQTT broker. And then what we're looking at right now is the data that is coming out of the Postgres database that you're pulling out into Grafana.

So what I would like to find out is. How are you getting your MQTT data from the broker into Postgres SQL? 

Gerhard Müller: Yeah, I can show you. I have a kind of drawing that shows everything that we were talking about. That is one of the devices that are collecting data, this is the one that collects the ambient light, carbon dioxide and air quality, and also the Outdoor and indoor weather data from these wireless sensors, and this is the device that we talked about first, which collects the temperatures from the heating and the solar terminal.

On the wire bus, there are two devices missing – one which collects the data from the gas meter and one which collects the data from the power meter. And those are the ESP devices and all are sending MQTT to this to my cloud broker. And here I have a virtual server running in the from a host. It's just a a small machine, I think for six or seven Euro per month. On that machine, I'm running a Docker. And on the Docker, I have for each project such a kind of data sink, which is just a small Java program that encapsulates the MQTT client, and this one subscribes to the broker and puts the data into the Postgres database. And from there I read all my HTML and CSV reports. And the Grafana dashboard is also connected to this Postgres database. Besides that's I have a Java application running on a Tomcat server which is an in memory buffer that contains the data of the last 24 hours. And from here, on the one hand, this in-memory buffer is able to check for certain alerts when some temperatures or whatever measure is reaching a limit. Okay. Then I can send alarms currently that is just via email, but I'm planning to do that also via SMS.

And on the other hand, I have a react JS app that I have also mentioned which connects to the in-memory buffer with a REST API. And with that, I can also show the data on mobile phones. I think I also have the output of this app. As you can see here, it contains all the different groups of sensors.

For example, here's the heating. Now let's have a look at the solar thermal system. You can also see that starting at around 10 o'clock, the system is heating up the hot water tank. That's basically it. So that's these are all the components involved.

Kudzai Manditereza: Oh, there's some really impressive work, Gerhard. I might say this is really an interesting architecture. And I see you could really use MQTT, like the full power of MQTT to really build your systems around MQTT. So maybe let's talk about HiveMQ Cloud in particular. So you mentioned, obviously, MQTT for you was a no brainer, right?

Gerhard, how else you'd be doing it. What specific feature about HiveMQ Cloud Serverless do you find so compelling and what made you choose HiveMQ Cloud Serverless? 

Gerhard Müller: Yeah, so first of all, I was looking for an MQTT broker and I know HiveMQ for a long time because I also did some innovation projects in my main company where I used HiveMQ broker, but at that point in time, it was just a locally installed instance. So I was just checking the website and somehow I ended up on HiveMQ Cloud. I started to have look and I don't remember exactly, but it was also a no-brainer.

I found it on the website and created an account. And started playing around with it and yeah, I found out that it's really easy to connect. So I think that the first thing I did was a small example or test project on Java. And, later on, I also tried with Python. All the things came together easily on this cloud instance because it's quite handy. I don't have to care. It's just working. It's reliable. It's fast. And that's just the perfect choice. 

Kudzai Manditereza: Oh, yes, absolutely. It's good to know. As far as the dealing with MQTT or maybe HiveMQ cloud service, what were the challenges that you maybe come across that maybe would have made your life better if you had discovered those earlier, or maybe is it around the education of MQTT?

Is it around the installation of the libraries? Were there any stumbling blocks along the way into MQTT? 

Gerhard Müller: So I have some MQTT experience already for a longer time. And. So my first experiments with Java and Python just worked. So, I did not face any problems, but then when it came to installing client on the ESP32 and tiny devices, that was challenging.

What I particularly can mention is that in the Arduino environment. It was not so easy to connect the MQTT client to the cloud broker because it requires encryption, TLS, and to get the certificates on this tiny machine that was the first challenge that I faced with MQTT.

Later on, I also did some experiments with in Raspberry Pi Pico. And here I used CircuitPython from Adafruit. Unfortunately, I started with CircuitPython version eight. With this version, there is a known bug which caused my programs not work as expected. I had a lot of errors when I was running this loop. And, one of the developers of the Python of the MQTT library in CircuitPython gave me that hint that the issue might be gone with CircuitPython 9 and yeah that was the case indeed.

After installing the Python nine, everything worked just fine. And that was much easier to connect compared to the ESP devices. Currently the hardware of my choice is a Raspberry Pi Pico with CircuitPython, which is very easy to connect with.

Kudzai Manditereza: Awesome stuff. Awesome. So I know you mentioned that you had already been working with MQTT for a while before you started on this project. By any chance, are there any new things that you learned from this new project about MQTT that maybe somehow you never knew about before the project? Anything you learned? 

Gerhard Müller: I have to be honest, not really in my project, but what helped me a lot I think in couple of months – I started to work on certificates that are provided by the HiveMQ University. Here, actually, I learned a lot. And there are certain things I want to implement also in my project. For example, I learned about a testament and last will and things like that. So that's something I'm currently not using. But I definitely have plans to implement that as well because it really helps, for example, when one of the devices breaks down or connection fails or something like that. I would be informed then and currently I'm not. I'm just informed when I check my dashboards and something is not working or showing up as expected, but I think that's something I definitely want to change so that I have a kind of device management or the device monitoring that shows me at one glance that a device that is not working as expected. 

Kudzai Manditereza: Ah, nice. That's good to know. It's good to know that you find some of the material on HiveMQ University useful for that. 

Gerhard Müller: Yeah, definitely. That helped a lot. Yeah. 

Kudzai Manditereza: Cool stuff. Yeah. Is there anything else that you want to say? Maybe as a closing thoughts, anything else that you want to say?

Gerhard Müller: No, not really. I think I have shown you everything I did. 

Kudzai Manditereza: Perfect. Which is very impressive, I must say. Really some well documented and well presented on the Grafana dashboard. So this is really some commendable work. And I'm sure a lot of our community members will find all of this information useful. By any chance, do you have this on a GitHub page or is it still really within your environment? Do you have a GitHub repo? 

Gerhard Müller: Yes. All is stored in repos on GitHub. But most of the repos are private, not accessible to the public. That's mainly because not everything is as it should be from documentation point of view and things like that.

Yeah. But as soon as I have something really to show for example, currently I'm working on another device, which is based on the Raspberry Pi Pico. I want to exchange the Raspberry at least for the OneWire with a Pico. And yeah. I'm thinking about making it public because that's quite a clean project, which I can show. The code, for example, for these ESP devices is really ugly and dirty. So that's something I would not show publicly, but, 

Kudzai Manditereza: Perfect. But as it stands right now, I think you've really shared so much valuable information with the community. So I really appreciate that. Thank you so much. 

Gerhard Müller: Yeah. Thank you to you too. 

Kudzai Manditereza: Thank you so much for taking the time to join me on this call today. 

Gerhard Müller: Yeah. Thanks for having me.

Next Steps

Gerhard’s project showcases the practical applications of MQTT and cloud technologies in optimizing energy usage and monitoring environmental conditions. His use of HiveMQ Cloud Serverless highlights the ease and reliability of this platform for IoT enthusiasts and professionals alike. Whether you are looking to monitor your home systems or community projects, Gerhard’s insights provide a valuable roadmap.

Are you inspired by Gerhard's IoT journey? Start your own project today with HiveMQ Cloud Serverless or HiveMQ Cloud Starter. Sign up for a free account and explore the possibilities of MQTT in your home or community projects.