Integrating ESP32 with LoRaWAN and HiveMQ MQTT Broker for Advanced IoT
In the dynamic sphere of the Internet of Things (IoT), the integration of various technologies is key to unlocking new and advanced capabilities. At the heart of such integration is ESP32, a powerful microcontroller renowned for its versatility in IoT projects. In this comprehensive guide, we venture into developing an ESP32 program that not only communicates via LoRaWAN – a protocol celebrated for its long-range and low-power benefits – but also interfaces seamlessly with HiveMQ MQTT Broker, an enterprise-grade MQTT platform designed to facilitate efficient and reliable message exchange.
This tutorial serves as a detailed roadmap for developers looking to explore the synergy between ESP32, LoRaWAN, and MQTT technologies. We will guide you through each step of the development process — from the initial setup and configuration of your ESP32 with LoRaWAN to the intricacies of connecting, exchanging, and normalizing data with HiveMQ Platform. You will gain a deep understanding of how to harness these technologies together, creating IoT solutions that are not only innovative but also robust and scalable. So, gear up with your ESP32 module, and let's dive into this exciting world of interconnected IoT technologies!
To follow this hands-on tutorial, we assume that you have already read our previous article on the integration of ChripStack LoRaWAN Network Server and HiveMQ Platform and that you have your environment up and running.
Connecting ESP32 to LoRaWAN and MQTT
In this hands-on lab, we walk you through the below steps:
Create the Device profile and declare a new Device in the Application “Environment sensors” already created during the first hands-on guide.
Create a simple firmware for our ESP32 chips to read the temperature and humidity value from the DHT11 sensor and send the values via LoRaWAN to HiveMQ Platform.
Create a HiveMQ Data Hub transformation script on HiveMQ Platform to simplify and normalize the data received in the LoRaWAN Application.
Use the HiveMQ Enterprise Extension for PostgreSQL to record sensor data.
Prerequisites
To create the firmware for our ESP32 chip, we use the Arduino IDE. If you don’t have it already installed, follow the instructions for your operating system:
We will need some extra configuration on the Arduino IDE to be able to work with the ESP32 board and the sensors.
To add the ESP32 boards support on the IDE, we navigate to Arduino IDE > Preferences (Windows File > Preferences), and fill in "Additional Boards Manager URLs" with this URL as shown below:
https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json
Then, navigate to Tools > Board > Boards Manager, search for esp32, select the latest version of esp32, and install it.
We will also need the DHT11 library that works with our sensor. You can import the SeeedStudio DHT11 library by adding this package in Arduino IDE.
To import the custom library, go to
Sketch > Include Library > Add .ZIP Library...
You will be prompted to select the library you want to add. Navigate to the .zip file’s location and open it.
Hardware
There are many manufacturers supplying boards with ESP32 chips. Here we're going to take a look at a solution provided by Seeed Studio, which produces boards with Grove connectors, making the development of embedded solutions much easier.
We will use the following components:
Grove-Wio-E5 Wireless Module (868 Mhz for EU)
For reference, the shield with XIAO ESP32 offers the below connectivity.
Connect Hardware
We need to wire the components to the shield as follows:
XIAO ESP32 on the ESP32 position on the shield
Wio-E5 on UART connector (A7, D7)
DHT11 on A0 connector
Once wired, connect the XIAO ESP32 to your computer via USB and configure it in the Arduino IDE. On the Arduino IDE, you can select the port directly. Search for XIAO_ESP32S3 in the development board on the left. Select XIAO_ESP32S3.
Note: If you are not sure that the connection is working properly, you can test it by deploying the blinking LED test code from the Arduino samples.
ChirpStack Configuration
Before diving into the code, we will configure our ChirpStack Network Server to allow our chip to send data by:
Creating a new device profile
Adding our device to the existing application for environmental sensors
Create a Device Profile
To create a device profile, go to Tenant > Device profiles.
Create a new profile by entering the name and the region configuration.
On the Join tab, check that “Device supports OTAA” is activated.
Then click “Submit” to create the device profile.
Declare a New Device
In the ChirpStack Network Server Web UI, go to the Application previously created (in the first hands-on post) and add a new Device based on the device profile we’ve just created.
Note: To find the Device EUI of your LoRaWAN chip, look at the sticker under the Wio-E5.
We submit the form to create the device. Then, we need to generate an OTAA key to finalize the configuration. To do so, go to the device property page and the tab OTAA keys. Use the round arrow to generate an MSB Application Key and copy it for later. Click on “Submit” to save it.
Save the key for later.
Arduino Development
ESP32 chips support the execution of several types of code and firmware. These include Arduino code execution. It has the advantage of being easy to use and has a very large community. We will use it to create our firmware.
Open the Arduino IDE and create a new sketch to add to the following code blocks.
Import the required libraries:
# include <Arduino.h>
# include "DHT.h"Define DHT PIN and TYPE and initialize the instance:
# define DHTPIN A0
# define DHTTYPE DHT11
DHT dht(DHTPIN, DHTTYPE);We will then add a code block that will handle some variables and the serial communication via UART to our Wio-E5 module to enable LoRaWAN communications:
static char recv_buf[512];
static bool is_exist = false;
static bool is_join = false;
static int led = 0;
static int at_send_check_response(char *p_ack, int timeout_ms, char*p_cmd, ...)
{
int ch;
int num = 0;
int index = 0;
int startMillis = 0;
va_list args;
memset(recv_buf, 0, sizeof(recv_buf));
va_start(args, p_cmd);
Serial1.printf(p_cmd, args);
Serial.printf(p_cmd, args);
va_end(args);
delay(200);
startMillis = millis();
if (p_ack == NULL)
{
return 0;
}
do
{
while (Serial1.available() > 0)
{
ch = Serial1.read();
recv_buf[index++] = ch;
Serial.print((char)ch);
delay(2);
}
if (strstr(recv_buf, p_ack) != NULL)
{
return 1;
}
} while (millis() - startMillis < timeout_ms);
return 0;
}
static void recv_prase(char *p_msg)
{
if (p_msg == NULL)
{
return;
}
char*p_start = NULL;
int data = 0;
int rssi = 0;
int snr = 0;
p_start = strstr(p_msg, "RX");
if (p_start && (1 == sscanf(p_start, "RX: \\"%d\\"\\r\\n", &data)))
{
Serial.println(data);
led = !!data;
if (led)
{
digitalWrite(LED_BUILTIN, LOW);
}
else
{
digitalWrite(LED_BUILTIN, HIGH);
}
}
p_start = strstr(p_msg, "RSSI");
if (p_start && (1 == sscanf(p_start, "RSSI %d,", &rssi)))
{
Serial.println("rssi:");
Serial.println(rssi);
}
p_start = strstr(p_msg, "SNR");
if (p_start && (1 == sscanf(p_start, "SNR %d", &snr)))
{
Serial.println("snr :");
Serial.println(snr);
}
}We now add the Setup block to initialize the necessary components. In this block, you will need to replace <DEVICE EUI> and <ADD THE OTAA KEY> by your values:
void setup(void)
{
Serial.begin(115200);
Serial.setDebugOutput(true);
Serial.print("HIVEMQ ESP32 LORAWAN LAB\\r\\n");
pinMode(LED_BUILTIN, OUTPUT);
digitalWrite(LED_BUILTIN, HIGH);
Serial1.begin(9600,SERIAL_8N1,44,43);
// Init Wio-E5
if (at_send_check_response("+AT: OK",100, "AT\\r\\n"))
{
is_exist = true;
at_send_check_response("+ID: AppEui", 1000, "AT+ID\\r\\n");
at_send_check_response("+MODE: LWOTAA", 1000, "AT+MODE=LWOTAA\\r\\n");//select mode
at_send_check_response("+DR: EU868", 1000, "AT+DR=EU868\\r\\n");//select frequency band
at_send_check_response("+CH: NUM", 1000, "AT+CH=NUM,0-2\\r\\n");//select sub band
at_send_check_response("+ID: DevEui", 1000, "AT+ID=DevEui,\\"<DEVICE EUI>\\"\\r\\n");//write deveui
at_send_check_response("+ID: AppEui", 1000, "AT+ID=AppEui,\\"0128944F83E7AF55\\"\\r\\n");//write AppEui
at_send_check_response("+KEY: APPKEY", 1000, "AT+KEY=APPKEY,\\"<ADD THE OTAA KEY>\\"\\r\\n");//write appkey
at_send_check_response("+CLASS: C", 1000, "AT+CLASS=A\\r\\n");//write device class
at_send_check_response("+PORT: 8", 1000, "AT+PORT=8\\r\\n");
delay(200);
is_join = true;
}
else
{
is_exist = false;
Serial.print("No Wio-E5 module found.\\r\\n");
}
dht.begin();
}We finally add the Loop block:
void loop(void)
{
float temp = 0;
float humi = 0;
temp = dht.readTemperature();
humi = dht.readHumidity();
Serial.print("Humidity: ");
Serial.print(humi);
Serial.print(" %\\t");
Serial.print("Temperature: ");
Serial.print(temp);
Serial.println(" *C");
if (is_exist)
{
int ret = 0;
if (is_join)
{
ret = at_send_check_response("+JOIN: Network joined", 12000, "AT+JOIN\\r\\n");
if (ret)
{
is_join = false;
}
else
{
at_send_check_response("+ID: AppEui", 1000, "AT+ID\\r\\n");
Serial.print("JOIN failed!\\r\\n\\r\\n");
delay(5000);
}
}
else
{
char cmd[128];
sprintf(cmd, "AT+CMSG=\\"T:%d-H:%d\\"\\r\\n", (int)temp, (int)humi);
ret = at_send_check_response("Done", 5000, cmd);
if (ret)
{
recv_prase(recv_buf);
}
else
{
Serial.print("Send failed!\\r\\n\\r\\n");
}
delay(5000);
}
}
else
{
delay(1000);
}
}Compile the firmware and deploy it on your ESP32.
Once executed, the serial monitor should return the following:
HIVEMQ ESP32 LORAWAN LAB
AT
+AT: OK
AT+ID
+ID: DevAddr, 50:20:BC:4B
+ID: DevEui, 2C:F7:F1:C0:42:90:02:11
+ID: AppEui, 01:28:94:4F:83:E7:AF:55
AT+MODE=LWOTAA
+MODE: LWOTAA
AT+DR=EU868
+DR: EU868
AT+CH=NUM,0-2
+CH: NUM, 0-2
AT+ID=DevEui,"2CF7F1C042900211"
+ID: DevEui, 2C:F7:F1:C0:42:90:02:11
AT+ID=AppEui,"0128944F83E7AF55"
+ID: AppEui, 01:28:94:4F:83:E7:AF:55
AT+KEY=APPKEY,"35d313e21071d5f41f651da500712677"
+KEY: APPKEY 35D313E21071D5F41F651DA500712677
AT+CLASS=A
+CLASS: A
AT+PORT=8
+PORT: 8
Humidity: 49.00 % Temperature: 17.60 *C
AT+JOIN
+JOIN: Start
+JOIN: NORMAL
+JOIN: Network joined
+JOIN: NetID 000000 DevAddr 00:CD:31:69
+JOIN: Done
Humidity: 49.00 % Temperature: 17.60 *C
AT+CMSG="T:17-H:49"
+CMSG: Start
+CMSG: Wait ACK
+CMSG: FPENDING
+CMSG: ACK Received
+CMSG: RXWIN1, RSSI -94, SNR 6.0
+CMSG: DoneYou should be able to see the frames in the ChirpStack.
You should also be able to see the payload of the application by connecting to HiveMQ MQTT Broker.
Data Normalization (Data Transformation)
Now that our ESP32 device is sending the temperature and humidity regularly, we need to use HiveMQ Data Hub transformation to extract the values from our Application payload as shown below:
{
"deduplicationId":"65b0841c-ec95-4f74-9caf-7a027b5dc8e2",
"time":"2024-01-15T15:19:48.917438315+00:00",
"deviceInfo":
{
"tenantId":"52f14cd4-c6f1-4fbd-8f87-4025e1d49242",
"tenantName":"ChirpStack Lab",
"applicationId":"f12b0a65-9637-44ad-a138-2106fc650910",
"applicationName":"Environment sensors",
"deviceProfileId":"4ff8106e-5d00-4258-bc84-51625e32b57c",
"deviceProfileName":"XIAO ESP32S3",
"deviceName":"XIAO ESP32S3",
"devEui":"2cf7f1c042900211",
"deviceClassEnabled":"CLASS_A",
"tags":{}
},
"devAddr":"0101848e",
"adr":true,
"dr":5,
"fCnt":33,
"fPort":8,
"confirmed":true,
"data":"VDoxOS1IOjQ2",
"rxInfo":
[{
"gatewayId":"2cf7f11053100024",
"uplinkId":50862,
"nsTime":"2024-01-15T15:19:48.709423081+00:00",
"rssi":-103,
"snr":2.5,
"rfChain":1,
"location":{},
"context":"O89wuA==",
"metadata":{"region_config_id":"eu868",
"region_common_name":"EU868"},
"crcStatus":"CRC_OK"
}],
"txInfo":{"frequency":868100000,
"modulation":
{
"lora":
{
"bandwidth":125000,
"spreadingFactor":7,
"codeRate":"CR_4_5"
}
}
}
}What we need to extract is the field "data":"VDoxOS1IOjQ2" that is sent as base64 string. What we want to get after transformation is a payload as shown below:
{
temperature: <value>,
humidity: <value>,
deviceName: <value>,
time: <value>
}We will capture all messages on the following topic:
application/f12b0a65-9637-44ad-a138-2106fc650910/device/+/event/upHiveMQ Data Hub Prerequisites
To configure the HiveMQ Data Hub policy, you need:
The HiveMQ API enabled on the broker
The HiveMQ MQTT CLI (latest version)
If you followed the first article to set up your broker, you will need to update the config file for the broker to reflect the following sample:
<?xml version="1.0" encoding="UTF-8" ?>
<hivemq xmlns:xsi="<http://www.w3.org/2001/XMLSchema-instance>"
xsi:noNamespaceSchemaLocation="config.xsd">
<listeners>
<tcp-listener>
<port>1883</port>
<bind-address>0.0.0.0</bind-address>
</tcp-listener>
</listeners>
<control-center>
<enabled>true</enabled>
<listeners>
<http>
<port>8080</port>
<bind-address>0.0.0.0</bind-address>
</http>
</listeners>
</control-center>
<rest-api>
<!-- Enables or disables the HiveMQ REST API-->
<enabled>true</enabled>
<!-- Enables or disables authentication and authorization for the HiveMQ REST API-->
<auth>
<enabled>false</enabled>
</auth>
<listeners>
<http>
<port>8888</port>
<bind-address>0.0.0.0</bind-address>
<!-- Defines listener name to help distinguish between multiple listeners -->
<name>api-http-listener</name>
</http>
</listeners>
</rest-api>
<anonymous-usage-statistics>
<enabled>false</enabled>
</anonymous-usage-statistics>
</hivemq>This will allow you to interact with the API (http://broker-ip:8888) without authentication and see the Policies in the Control Panel (http://broker-ip:8080) with the default credentials (Username: admin and password: hivemq).
Remember that this setup is ok for a lab but not for production, so we encourage customers to use our Enterprise Security Extension to protect all services of HiveMQ Broker including HiveMQ Control Center and API.
After updating the configuration, restart the broker to take it into account.
Validation Schema
To be sure that we are receiving a JSON object, we will create a very simple policy that will just check that the payload is actually a JSON object. Create a file called “pre-schema.json” and add the following content:
{
"description": "Generic JSON schema, it requires just a JSON, nothing further specified",
"type": "object"
}We also need a schema to validate the object that will be generated after our transformation. Create a new file called “post-schema.json” and add the following content:
{
"type": "object",
"properties": {
"temperature": {
"type": "string"
},
"humidity": {
"type": "string"
},
"deviceName": {
"type": "string"
},
"time": {
"type": "string"
}
},
"required": [
"temperature",
"humidity",
"time"
]
}Then using HiveMQ MQTT CLI, upload the schemas to the broker:
mqtt hivemq schema create --id pre-lorawan-application --type json --file pre-schema.json
mqtt hivemq schema create --id post-lorawan-application --type json --file post-schema.jsonNote: If you are executing the command remotely, don’t forget to specify ‘—url https://your-broker-address:8888’
Transformation Script
The next step after being sure that the payload will be in JSON format is to proceed with the transformation, which will consist in extracting the base64 data coming from the sensor and converting it to a simple JSON payload. Create a file called “script.js” and add the following content:
function decodeBase64(base64String) {
const base64Chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
let base64 = base64String.replace(/=+$/, '');
let binaryString = '';
for (let i = 0; i < base64.length; i++) {
let binaryChar = base64Chars.indexOf(base64[i]).toString(2);
binaryChar = binaryChar.padStart(6, '0');
binaryString += binaryChar;
}
let decodedString = '';
for (let i = 0; i < binaryString.length; i += 8) {
let byte = binaryString.substring(i, i + 8);
decodedString += String.fromCharCode(parseInt(byte, 2));
}
return decodedString;
}
function transform(publish, context) {
console.log(publish.payload.data)
let values = decodeBase64(publish.payload.data).split('-')
let temperature = values[0].split(":")[1]
let humidity = values[1].split(":")[1]
const newPayload = {
topic: publish.topic,
payload: {
"temperature": temperature,
"humidity": humidity,
"time": publish.payload.time
},
userProperties: [{ name: "transformed", value: "true" }]
}
return newPayload;
}In a nutshell, the transform function is called by the Data Hub transformation to manipulate the payload, and the decodeBase64 function is called from the transform function to decode the data in the original payload. We then build the new payload respecting the Publish Object format.
Upload the script to the broker using HiveMQ MQTT CLI:
mqtt hivemq script create --id lorawan-transformation --type transformation --file script.jsNote: If you are executing the command remotely, don’t forget to specify ‘—url https://your-broker-address:8888’
Policy
Now that we have both the validation schema and the transformation script, we can create the policy that will handle the messages. Create a file called “policy.json” and add the following content:
{
"id": "lorawan-application-simplifier",
"matching": {
"topicFilter": "application/f12b0a65-9637-44ad-a138-2106fc650910/device/+/event/up"
},
"validation": {
"validators": [
{
"type": "schema",
"arguments": {
"strategy": "ALL_OF",
"schemas": [
{
"schemaId": "pre-lorawan-application",
"version": "latest"
}
]
}
}
]
},
"onSuccess": {
"pipeline": [
{
"id": "deserialize",
"functionId": "Serdes.deserialize",
"arguments": {
"schemaVersion": "latest",
"schemaId": "pre-lorawan-application"
}
},
{
"id": "lorawan-transformation",
"functionId": "fn:lorawan-transformation:latest",
"arguments": {}
},
{
"id": "serialize",
"functionId": "Serdes.serialize",
"arguments": {
"schemaVersion": "latest",
"schemaId": "post-lorawan-application"
}
}
]
},
"onFailure": {
"pipeline": [
{
"id": "drop-invalid-message",
"functionId": "Mqtt.drop",
"arguments": {
"reasonString": "Your client ${clientId} sent invalid data according to the schema: ${validationResult}."
}
}
]
}
}Upload the policy to the broker using HiveMQ MQTT CLI:
mqtt hivemq data-policy create --file policy.jsonNote: If you are executing the command remotely, don’t forget to specify ‘—url https://your-broker-address:8888’
If your device is already connected and sending data to the broker, you should instantly see the new payloads as shown below:
The new structure is now:
{
"temperature":"18",
"humidity":"37",
"time":"2024-01-16T14:14:31.292536457+00:00",
"deviceName":"XIAO ESP32S3"
}Database Injection
An additional step that you can do if you have a database available (Postgres in our case, but you can do the same with MySQL) is to save the history of the values in a database. In our lab, we will create a database and its table with the following script:
create database sensorshistory with encoding 'UTF8';
\c sensorshistory;
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
SET default_tablespace = '';
SET default_table_access_method = heap;
CREATE TABLE public.sensors_history (
topic text,
payload text
);Configure the HiveMQ Enterprise Extension for PostgreSQL
On the broker, we need to update the configuration file for HiveMQ Enterprise Extension for PostgreSQL in order to collect our MQTT messages and save them into the database. Stop the broker and create the config.xml file with the below content in the extension directory (/opt/hivemq/extensions/hivemq-postgresql-extension/conf):
<?xml version="1.0" encoding="UTF-8" ?>
<hivemq-postgresql-extension xmlns:xsi="<http://www.w3.org/2001/XMLSchema-instance>"
xsi:noNamespaceSchemaLocation="postgresql-extension.xsd">
<postgresqls>
<postgresql>
<id>postgresql-backend</id>
<host>**YOURHOST**</host>
<port>5432</port>
<database>sensorshistory</database>
<username>**YOURUSERNAME**</username>
<password>**YOURPASSWORD**</password>
</postgresql>
</postgresqls>
<mqtt-to-postgresql-routes>
<mqtt-to-postgresql-route>
<id>sensors-to-postgresql-route-insert</id>
<postgresql-id>postgresql-backend</postgresql-id>
<mqtt-topic-filters>
<mqtt-topic-filter>application/f12b0a65-9637-44ad-a138-2106fc650910/device/+/event/up</mqtt-topic-filter>
</mqtt-topic-filters>
<processor>
<insert-statement>
<table>sensors_history</table>
<columns>
<column>
<name>topic</name>
<value>mqtt-topic</value>
</column>
<column>
<name>payload</name>
<value>mqtt-payload-utf8</value>
</column>
</columns>
</insert-statement>
</processor>
</mqtt-to-postgresql-route>
</mqtt-to-postgresql-routes>
</hivemq-postgresql-extension>Note: Don’t forget to remove the DISABLE (/opt/hivemq/extensions/hivemq-postgresql-extension/DISABLE) file in the extension folder.
Start the broker and monitor the logs. If you don’t see any error on startup regarding the PostgreSQL extension (such as a connection issue to the PostgreSQL instance), you should shortly see data from your ESP32 device being inserted in the database.
Wrap Up
In this guide, we’ve not only successfully created a simple firmware based on Arduino for our ESP32 board to send sensor values via our LoRaWAN private network, but we’ve also used the powerful data transformation feature (that comes with Data Hub on HiveMQ Platform) and the Enterprise Extension for PostgreSQL database to historicize our sensor data. This allows scenarios where the data is prepared before being sent to backend services and/or saved into databases. If you need more examples of concrete implementations of Data Hub transformations, see the GitHub repository shared by our engineering team with some samples.
As we conclude, remember that the world of IoT is ever-evolving. Stay curious, keep experimenting, and don't hesitate to push the boundaries of what your IoT network can achieve. ESP32 and STM32 chips can be used in all kinds of scenarios, from environmental monitoring to end-user products.
Resources
Code and files from this article are available on my GitHub repository: ESP32-DATAHUB.
If you want to learn more about SeeedStudio XIAO products, here’s their official wiki.
Anthony Olazabal
Anthony is part of the Solutions Engineering team at HiveMQ. He is a technology enthusiast with many years of experience working in infrastructures and development around Azure cloud architectures. His expertise extends to development, cloud technologies, and a keen interest in IaaS, PaaS, and SaaS services with a keen interest in writing about MQTT and IoT.
