Hands-on Guide to Using MQTT and Eclipse Ditto for Digital Twins
In the rapidly evolving world of Internet of Things, the quest for efficient, scalable, and robust communication solutions is unending. Enter the dynamic duo of Eclipse Ditto and HiveMQ broker – a combination that's redefining the boundaries of IoT connectivity and data management. In this tutorial, we delve into how Eclipse Ditto, an open-source framework for creating and managing digital twins, seamlessly integrates with HiveMQ Broker, a state-of-the-art MQTT broker, to revolutionize IoT applications. If you want to know more about the benefits of having them working together, read the article HiveMQ and Eclipse Ditto: Friends or Foes?.
What is Our Lab Use Case?
Let's have a quick look at the overall picture to understand what we will set up in our lab. We will deploy HiveMQ Broker and Ditto on a Docker host to simplify all the installation process. We will simulate an ESP32 device with sensors (temperature, gaz and a led) that communicate with a light payload to optimize the resources and data transfer. Our goal is to be able to have our Ditto Thing updated when there are sensor updates on our ESP32 device.
Key Terminologies Around Eclipse Ditto
Before going into the deployment and the configuration, we need to understand some concepts of Ditto to understand what we will configure.
Digital Twin: A digital twin is a virtual representation that serves as the real-time digital counterpart of a physical object or process.
Ditto Policy: A Policy enables developers to configure fine-grained access control for Things and other entities easily. A specific policy provides someone (called subject) permission to read and/or write a given resource.
Ditto Thing: Things are very generic entities and are mostly used as a “handle” for multiple features belonging to this Thing. For example,
A physical device, such as a lawn mower, a sensor, a vehicle, a lamp, etc.
A virtual device, such as a room in a house, a virtual power plant spanning multiple power plants, the weather information for a specific location collected by various sensors, etc.
A transactional entity, such as a tour of a vehicle (from start until stop), a series of measurements of a machine, etc.
A master data entity, such as a supplier of devices or a service provider for devices, an entity representing a city/region, etc.
Anything else – if it can be modeled and managed appropriately by the supported concepts/capabilities.
Ditto Connections: A connection represents a communication channel for the exchange of messages between any service and Ditto. It requires a transport protocol, which is used to transmit Ditto Protocol messages. Ditto supports one-way and two-way communication over connections. This enables consumer/producer scenarios as well as fully-fledged command and response use cases.
Payload Mapping: Ditto comes with a default message format which is called Ditto Protocol JSON. By default any connection (communication channel) is using the default Ditto mapper which expects the Ditto message format to be able to update the Thing. If you are in a situation where you already have defined message payloads, you can use the JavaScript mapper to convert the incoming payload to a Ditto formatted payload.
Prepare the Lab for HiveMQ MQTT Broker and Eclipse Ditto
To run our lab, we prepare a fresh Ubuntu Desktop 22.04 LTS with the following specifications:
4 CPU
8 Gb RAM
100 GB Disk
Note: You can run it in a virtual machine, or you can also use Windows or MacOS with Docker Desktop to run HiveMQ and Eclipse Ditto.
To run all the containers, we will need:
Docker desktop, you can find the installation instructions on the official documentation.
Docker Compose, you can also find the installation instructions on the official documentation.
Deployment of Eclipse Ditto
For Ditto’s deployment, we assume that you are using Docker with Docker Compose. Note that Ditto can be deployed on a variety of target systems. Have a look at the GitHub repository for templates. We will use the docker compose template to deploy all containers needed to run Ditto.
Download the latest docker-compose.yml from Ditto’s GitHub repository.
mkdir -p ditto/docker
wget https://raw.githubusercontent.com/eclipse-ditto/ditto/master/deployment/docker/docker-compose.yml -o ditto/docker/docker-compose.ymlTo facilitate the communication between HiveMQ and Ditto, we will add the following block in the docker-compose.yml file under the service section to start the HiveMQ Broker with Ditto:
hivemq:
image: hivemq/hivemq4
mem_limit: 1024m
restart: always
networks:
default:
aliases:
- hivemq
ports:
- 8181:8080
- 1883:1883Note: We change the default Control Center port to 8181 to avoid conflict with Eclipse Ditto UI.
In order to start Ditto with docker-compose, you need to have the docker “compose” plugin or “docker-compose” standalone installed.
cd /deployment/docker/
sudo docker-compose up -dAfter waiting for all tasks to complete, you can access the web interface via http://localhost:8080/ui/
You can refer to the official Eclipse Ditto documentation for more detailed information.
To check that HiveMQ is running, navigate with your browser to http://localhost:8181 and open the HiveMQ Control Center. If the HiveMQ Control Center is available, HiveMQ is up and running. You can login with the default credentials:
User: admin
Password: hivemq
Any MQTT client can now be connected to HiveMQ on port 1883.
Remember to refer to the official HiveMQ documentation for detailed configuration options and other advanced settings.
Configuration of Eclipse Ditto
Now that all the needed services are running, we need to configure the following:
A Ditto Policy to allow the connected identity to access Ditto’s entities
A Ditto Thing that represents the state of our ESP32 Device
A Ditto Connection to have a communication channel via MQTT to our HiveMQ Broker and with our device.
End-to-End Communication Flow
Before going into the configuration, let’s have a look at the following schema to illustrate the end-to-end flow.
Our ESP32 device is connected to our HiveMQ Broker via MQTT. It publishes the sensor metrics into the topic “devices/<deviceDittoId>” (We keep it simple for the lab). Our Eclipse Ditto is subscribing to “devices/#” to be able to update all the Things that could be connected via MQTT. Since our ESP32 device is working with non-Ditto-compliant messages, we need to put in place a payload mapper in JavaScript on the connection created on Ditto. Then our backend application can consume information on Ditto via the Ditto Rest API.
Authentication on Eclipse Ditto
Ditto works with authentication and authorization. By default, the users are pre-authenticated using the reverse proxy in front of Ditto. In our lab, we rely on Nginx to provide HTTP Basic Authentication and set the “x-ditto-pre-authenticated” header to nginx:<username> to forward the information to Ditto.
For more details on the way Ditto works, dive into the official Ditto documentation.
1. Create a Ditto Policy
By default, Ditto allows anyone to create a new entity (policy or thing) in any namespace. However, this behavior can be customized, and the ability to create new entities can be restricted. The official documentation provides additional information on how to restrict it. If you start on an empty platform you first need to create a policy to allow interactions with things, messages and policies. In our lab, we create a policy to Read and Write information on Things, Messages and Policies. The policy will be called “default.ditto:policy”.
Here is the curl command to create the policy using an API call:
curl -X PUT 'http://localhost:8080/api/2/policies/ditto.default:policy' -u 'ditto:ditto' -H 'Content-Type: application/json' -d '{
"entries": {
"owner": {
"subjects": {
"nginx:ditto": {
"type": "nginx basic auth user"
}
},
"resources": {
"thing:/": {
"grant": [
"READ","WRITE"
],
"revoke": []
},
"policy:/": {
"grant": [
"READ","WRITE"
],
"revoke": []
},
"message:/": {
"grant": [
"READ","WRITE"
],
"revoke": []
}
}
},
"connection": {
"subjects": {
"connection:hivemq-mqtt": {
"type": "Connection to HiveMQ MQTT broker"
}
},
"resources": {
"thing:/": {
"grant": [
"READ","WRITE"
],
"revoke": []
},
"message:/": {
"grant": [
"READ","WRITE"
],
"revoke": []
}
}
}
}
}'Note: As you might have noticed, I’m executing the curl locally with the default user (ditto). If you need to configure it remotely, you will need to change the address.
We can check the creation in the Ditto Explorer:
Note: You can also update the policy via Ditto Explorer once it has been created. If you do it remotely, you will need to take the case of the CORS configuration in NGINX.
2. Create Our First Thing
Once we have our Policy, we can create our first Thing and assign the previously created policy. Our Thing will have features declared to map properties of the device to Ditto twins:
Temperature Sensor: temp_sensor
Gaz Sensor: gaz_sensor
Alert Led: alert_led
We create our Thing called “sensor.test:hivelab” via an API call with curl command:
curl -X PUT 'http://localhost:8080/api/2/things/sensor.test:hivelab' -u 'ditto:ditto' -H 'Content-Type: application/json' -d '{
"policyId": "ditto.default:policy",
"attributes": {
"name": "HiveMQ Sensor Lab",
"type": "ESP32 Sensor board"
},
"features": {
"temp_sensor": {
"properties": {
"value": 0
}
},
"gaz_sensor": {
"properties": {
"value": 0
}
},
"alert_led": {
"properties": {
"value": 0
}
}
}
}After creating the first Thing, we can have a look at Ditto Explorer to see the details and interact with it.
3. Create the MQTT Connection
Now that we have the minimum requirements (Policy and a Thing), we can create our first MQTT connection between our HiveMQ Broker and Ditto. This will open a communication channel that will allow us to get the information sent by our ESP32 board.
We create our connection called “sensor.test:hivelab” via an API call with curl command:
curl -X POST ‘http://localhost:8080/api/2/connections' -u 'devops:foobar' -H 'Content-Type: application/json' -d '{
"name": "HiveMQ",
"connectionType": “mqtt",
"connectionStatus": "open",
"failoverEnabled": true,
"uri": "tcp://hivemq:1883",
"sources": [{
"addresses": ["devices/#"],
"authorizationContext": ["connection:hivemq-mqtt"],
"qos": 1,
"enforcement": {
"input": "{{ source:address }}",
"filters": [
"devices/{{ thing:id }}"
]
}
}],
"targets": [{
"address": "devices/{{ thing:id }}/downlink",
"topics": [
"_/_/things/twin/events",
"_/_/things/live/messages"
],
"authorizationContext": ["connection:hivemq-mqtt"],
"qos": 1
}]
}'At this point, you should have an active connection open from Ditto to HiveMQ Broker. You can check it in Ditto Explorer connections view.
If it’s not the case, you can have a look at the “Connection Status, Metrics and Logs section” to see errors, and if needed, to activate logs.
4. Payload Mapping
Depending on the type of device that you are using, you might need to optimize the payload to limit compute and data transfer on the embedded platform (ESP32 or STM32 for example). In that case, you will need to create and assign a mapping function to the connection in order to convert the payload received to a payload with Ditto-supported format.
As an example, the Ditto-formatted payload for our device looks like the following:
{
"thingId": "sensor.test:hivelab",
"policyId": "ditto.default:policy",
"features": {
"temp_sensor": {
"properties": {
"value": 21.12
}
},
"gaz_sensor": {
"properties": {
"value": 40.97
}
},
"alert_led": {
"properties": {
"value": 0
}
},
}
}In our case, our ESP32 device sends a compressed payload like the following:
{
"temp": 30.67,
"gaz": 60.341,
"alert": 0,
"thingId": "sensor.test:hivelab"
}As you might have understood, we will need to create a kind of middleware to convert our source payload to the target payload for Ditto. This is easily made in our Ditto connection with mapping functions. To simplify our demonstration, we've already prepared the mapping function.
function mapToDittoProtocolMsg(headers, textPayload, bytePayload, contentType) {
const jsonString = String.fromCharCode.apply(null, new Uint8Array(bytePayload));
const jsonData = JSON.parse(jsonString);
const thingId = jsonData.thingId.split(':');
const value = {
temp_sensor: {
properties: {
value: jsonData.temp
}
},
gaz_sensor: {
properties: {
value: jsonData.gaz
}
},
alert_led: {
properties: {
value: jsonData.alert
}
},
};
return Ditto.buildDittoProtocolMsg(
thingId[0], // your namespace
thingId[1],
'things', // we deal with a thing
'twin', // we want to update the twin
'commands', // create a command to update the twin
'modify', // modify the twin
'/features', // modify all features at once
headers,
value
);
}In a nutshell, we are extracting the values from the original payload by parsing it and rebuilding a new payload with the Ditto structure sensor_name > properties > value.
To use the mapping function in our connection, we need to update it in order to use the mapping engine. We will add the following section:
"javascript": {
"mappingEngine": "JavaScript",
"options": {
"incomingScript": "function mapToDittoProtocolMsg(headers, textPayload, bytePayload, contentType) {const jsonString = String.fromCharCode.apply(null, new Uint8Array(bytePayload));const jsonData = JSON.parse(jsonString); const thingId = jsonData.thingId.split(':'); const value = {temp_sensor: {properties: {value: jsonData.temp}},gaz_sensor: {properties: {value: jsonData.gaz}},alert_led: {properties: {value: jsonData.alert}},};return Ditto.buildDittoProtocolMsg(thingId[0],thingId[1],'things','twin','commands','modify','/features',headers,value);}"
}
}The final connection string will look as follows:
{
"id": "29888d30-6b3d-40a3-8891-c18c60a3b00f",
"name": "HiveMQ",
"connectionType": "mqtt-5",
"connectionStatus": "open",
"uri": "tcp://hivemq:1883",
"sources": [
{
"addresses": [
"devices/+"
],
"consumerCount": 1,
"qos": 1,
"authorizationContext": [
"connection:hivemq-mqtt"
],
"enforcement": {
"input": "{{ source:address }}",
"filters": [
"devices/{{ thing:id }}"
]
},
"headerMapping": {
"content-type": "{{header:content-type}}",
"reply-to": "{{header:reply-to}}",
"correlation-id": "{{header:correlation-id}}"
},
"payloadMapping": [
"javascript"
],
"replyTarget": {
"address": "{{header:reply-to}}",
"headerMapping": {
"content-type": "{{header:content-type}}",
"correlation-id": "{{header:correlation-id}}"
},
"expectedResponseTypes": [
"response",
"error"
],
"enabled": true
}
}
],
"targets": [
{
"address": "devices/{{ thing:id }}/downlink",
"topics": [
"_/_/things/twin/events"
],
"qos": 1,
"authorizationContext": [
"connection:hivemq-mqtt"
],
"headerMapping": {
"reply-to": "devices/{{ thing:name }}/downlink",
"correlation-id": "{{ header:correlation-id }}",
"Content-Type": "application/vnd.eclipse.ditto+json"
},
"payloadMapping": [
"Ditto"
]
}
],
"clientCount": 1,
"failoverEnabled": true,
"validateCertificates": true,
"processorPoolSize": 1,
"mappingDefinitions": {
"javascript": {
"mappingEngine": "JavaScript",
"options": {
"incomingScript": "function mapToDittoProtocolMsg(headers, textPayload, bytePayload, contentType) {\r\n const jsonString = String.fromCharCode.apply(null, new Uint8Array(bytePayload));\r\n const jsonData = JSON.parse(jsonString); \r\n const thingId = jsonData.thingId.split(':'); \r\n const value = { \r\n temp_sensor: { \r\n properties: { \r\n value: jsonData.temp \r\n } \r\n },\r\n gaz_sensor: { \r\n properties: { \r\n value: jsonData.gaz \r\n } \r\n }, \r\n\t\t\t\talert_led: { \r\n properties: { \r\n value: jsonData.alert \r\n } \r\n }, \r\n }; \r\n return Ditto.buildDittoProtocolMsg(\r\n thingId[0], // your namespace \r\n thingId[1], \r\n 'things', // we deal with a thing\r\n 'twin', // we want to update the twin\r\n 'commands', // create a command to update the twin\r\n 'modify', // modify the twin\r\n '/features', // modify all features at once\r\n headers, \r\n value\r\n );\r\n}\r\n",
"outgoingScript": ""
}
}
},
"tags": []
}Note: Be careful if you copy and paste the string into Ditto; there are extra characters that you need to remove (like: \r\n)
To update the existing connection string, you can use Ditto Explorer:
After updating the connection, we are ready to test the update of the Thing by sending a message on the HiveMQ Broker with the following command (using HiveMQ CLI):
#> mqtt pub -t devices/sensor.test:hivelab -m '{"temp": 23, "gaz": 58, "alert": 0, "thingId": "sensor.test:hivelab"}' -q 1 -h localhost -p 1883The previous command establishes a connection to our lab HiveMQ Broker and sends a message on the topic that Ditto is expecting to receive messages to update our Thing “sensor.test:hivelab”.
You can then come back into Ditto Explorer to validate that the value has been updated on our Thing by Ditto:
The connection logs also show that the message was processed properly:
Secure Integration Between HiveMQ and Eclipse Ditto
As you noticed, we don’t apply any security in our lab on our HiveMQ Broker. Integrating Eclipse Ditto with HiveMQ Broker in production for IoT applications demands a strong focus on security. Given the sensitive nature of IoT data and the potential risks associated with its transmission and management, it's crucial to address several key security aspects when configuring this integration.
1. Authentication and Authorization:
HiveMQ Broker Security: Use the HiveMQ Enterprise Security Extension to ensure that devices and clients communicating via HiveMQ are authenticated. This can be achieved through various mechanisms like username/password, client certificates, or integrating with external authentication services.
Eclipse Ditto Authentication: Eclipse Ditto requires authentication for clients accessing its API. This can be done using basic authentication (username and password) or through tokens.
Authorization: Both platforms should implement fine-grained authorization to control access to resources. This means setting up permissions for what each authenticated client is allowed to do, such as which topics they can publish or subscribe to in HiveMQ, and what operations they can perform on digital twins in Ditto.
2. Secure Communication:
TLS/SSL Encryption: Encrypting data in transit is critical. Use TLS (Transport Layer Security) for both HiveMQ and Eclipse Ditto to secure the communication channels. This prevents interception and tampering with the data as it moves between devices, HiveMQ, and Ditto.
Firewalls and Network Segmentation: Protect your network with firewalls and segment the network to isolate the IoT environment from other parts of your network. This reduces the risk of lateral movement in case of a security breach.
3. Regular Updates and Patch Management:
Both HiveMQ and Eclipse Ditto, like any software, need regular updates to patch security vulnerabilities. Establish a process for regularly updating and patching these systems.
4. Monitoring and Auditing:
Implement logging and monitoring for both Eclipse Ditto and HiveMQ. This helps in detecting and responding to potential security incidents promptly.
Regular audits can help identify security gaps and ensure compliance with security policies and standards.
In a nutshell, securing the integration between HiveMQ Broker and Eclipse Ditto involves a comprehensive approach covering authentication, encryption, network security, and regular security monitoring and updates.
Wrap Up
In conclusion, the digital twin model of Eclipse Ditto allows for a detailed and real-time representation of physical devices in the digital domain, facilitating improved monitoring, analysis, and interaction. Meanwhile, HiveMQ's efficient handling of MQTT protocol ensures reliable and secure data transmission, even in environments with thousands or millions of devices.
This integration not only enhances the efficiency and scalability of IoT systems but also opens up new possibilities for innovation in various sectors, from smart homes and industrial automation to healthcare and smart cities. As the IoT landscape continues to evolve, the synergy between Eclipse Ditto and HiveMQ is poised to play a crucial role in shaping the future of interconnected, intelligent devices.
I'd also like to thank Thomas Jäckle for his advice on integrating HiveMQ and Ditto. If you'd like to find out more about Ditto's capabilities, feel free to browse the documentation and talk to the community via the forum or the GitHub discussions.
Anthony Olazabal
Anthony is part of the Solutions Engineering team at HiveMQ. He is a technology enthusiast with many years of experience working in infrastructures and development around Azure cloud architectures. His expertise extends to development, cloud technologies, and a keen interest in IaaS, PaaS, and SaaS services with a keen interest in writing about MQTT and IoT.
