MQTT Data Visualization with Grafana
Operational data from IoT devices, industrial systems, and real-time applications is the new oil as it provides critical insights into system performance and business metrics. But that data is only valuable if it’s easily accessible and interpretable. MQTT offers an efficient way to collect and transport this telemetry data at scale. By pairing MQTT with Grafana's visualization capabilities, organizations can transform raw operational data streams into actionable dashboards. This tutorial walks through building a complete MQTT-to-Grafana pipeline for real-time analytics. We show how easy this pipeline can be set up and put to use.
Why Integrate MQTT and Grafana
MQTT data visualization is essential for transforming raw MQTT message streams into meaningful insights, enabling users to monitor, analyze, and act on real-time data effectively.
The combination of MQTT (Message Queuing Telemetry Transport) with HiveMQ brokers, Postgres databases, and Grafana offers a robust, efficient, and flexible solution for gathering, visualizing and analyzing data.
MQTT, a lightweight and efficient messaging protocol, is specifically designed for resource-constrained networks and devices, making it ideal for IoT applications.
Grafana is a powerful visualization and analytics platform that excels at turning raw data into meaningful insights through customizable and interactive dashboards.
To do long-term storage and offer direct integration with visualization systems such as Grafana, an intermediate PostgreSQL database is used. Both HiveMQ brokers and Grafana integrate very well with this open source platform.
By integrating MQTT and Grafana, you can create a seamless pipeline for collecting real-time data from sensors, devices, or applications, and then visualizing it in an intuitive and user-friendly interface. This combination is particularly effective in scenarios such as smart home systems, industrial automation, environmental monitoring, and any application where timely and actionable insights are critical.
This synergy offers several benefits:
Real-time Data Streaming: MQTT ensures fast and reliable data delivery, while Grafana enables real-time visualization, helping users monitor systems as they operate.
Scalability: Both technologies handle scaling efficiently, supporting small setups to enterprise-grade deployments.
Flexibility: With MQTT’s topic-based architecture and Grafana’s wide range of data source integrations, you can tailor the solution to fit your specific needs.
Ease of Integration: HiveMQ brokers can directly submit data to (time-series) databases that work well with Grafana, simplifying the implementation process.
Cost Efficiency: Both MQTT and Grafana are open-source and widely supported, providing a low-cost, high-value solution.
How MQTT and Grafana Work Together
Now let’s explore how MQTT and Grafana work together to empower data-driven decisions and create a transparent, scalable, and insightful monitoring ecosystem.
Technology Highlights Used in the Implementation
1. HiveMQ Data Hub
(provide robust data validation and content additions)
HiveMQ Data Hub as a schema validator ensures the integrity, consistency, and reliability of data in MQTT ecosystems. By validating messages against predefined schemas, it prevents malformed or inconsistent data from propagating through the system, reducing errors and ensuring compatibility with downstream applications like databases, analytics platforms, and visualization tools.
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"properties": {
"SensorID": {
"type": "string"
},
"temperature": {
"type": "number"
} },
"required": [
"SensorID",
"temperature" ]
}A JSON Schema definition.
HiveMQ Data Hub can also add data JSON fields into incoming messages providing, in this case, Data Hub inserts adequate timestamping in the sensor-generated, MQTT JSON-encoded messages by executing a JavaScript function.
function transform(publish, context) {
const unixTime = Math.floor(Date.now() / 1000);
publish.payload.unixtime = unixTime.toString(10);
const utcTime = new Date().toISOString();
publish.payload.isotime = utcTime; //"2011-12-19T15:28:46.493Z"
return publish;
}A JS function to add timestamps to a MQTT payload.
2. HiveMQ Enterprise Extension for PostgreSQL
(directly move data from JSON/MQTT into database tables)
The HiveMQ Enterprise Extension for PostgreSQL enables seamless integration of HiveMQ with PostgreSQL, allowing MQTT (JSON serialized) messages to be stored directly into a PostgreSQL database. This extension provides a powerful way to persist MQTT data for long-term storage, analysis, and direct integration with other (visualization) systems such as Grafana.
3. PostgreSQL’s ability to extract and insert JSON data objects
(deserialise JSON and insert)
Using PostgreSQL to do direct JSON insertion allows us to store semi-structured or unstructured data directly in a relational database. And since the HiveMQ postgres extension can be configured to use a custom defined SQL insertion script deserializing and table insertion can be done quickly and efficiently.
INSERT INTO tempdata (sensorid,isotime, unixtime, temperature)
SELECT
json_data->>'SensorID' AS sensorid,
(json_data->>'isotime'):: timestamp AS isotime,
(json_data->>'unixtime')::numeric AS isotime,
(json_data->>'temperature')::numeric AS temperature /* casting from text to numic value ! */
FROM (
VALUES
( ${mqtt-payload-utf8}::jsonb)
) AS input(json_data);A custom insert SQL script executed by the HiveMQ database extension that decodes the JSON-formatted MQTT payload and inserts the JSON payload data.
The Setup
To showcase a data pipeline, starting from a MQTT-data-generating device all the way to visualization in Grafana, we set up a number of Docker containers that intercommunicate.
The first container will run the HiveMQ MQTT Broker that will ingest the sensor data as MQTT payload. This payload is formatted by the device as a simple 2-line JSON object and is being published to topic temp/test
{ "SensorID": "CMD-Line",
"temperature": 22.54 }Since we deal with a very simplistic sensor device, the timestamp is missing in the JSON data. Once the broker determines the sensor data conforms to the input data JSON schema, it will add two timestamps to the JSON payload and re-publish the message on a new topic "Temp-TS/temp/test"
{ "unixtime": "1736445953",
"SensorID": "CMD-Line",
"temperature": 22.54,
"isotime": "2025-01-09T18:05:53.505Z" }The new message will be picked up by the HiveMQ PostgreSQL database extension and propagated to a database running in a second Docker container. Since we’re able to write our own SQL-insert scripts, we can make use of the sophisticated JSON decoding methods provided by PostgreSQL, thus inserting individual datafield into their respective database columns.
The last step is having Grafana, running in a third container, to query the database and displaying the data in a time-series graph. For that, we use a simple query as shown below.
All described steps along with the container structure are depicted in the following graph:
Running Your Own Setup in Docker Desktop
So you can implement this setup yourself, a Github repo is available here.
Just clone this and make sure your local docker desktop backend is up-and-running. Now run the ‘docker-compose’ command on your commandline.
git clone https://github.com/hivemq/Blog-Hive-PostG-Grafana
cd Blog-Hive-PostG-Grafana
docker-compose up --build -d Completion of this command will result in three docker containers inter-communicating over SQL and MQTT, as described above.
HiveMQ broker container:
You can access the HiveMQ CommandCenter GUI locally at http://127.0.0.1:8080. We need its Command Center GUI (or the also enabled API) to upload the appropriate JSON Schemas, the JavaScript, and the data-policy. The default username / password is admin / hivemq.
Enable Data Hub and upload validation schemas, scripts, and policies.
As stated you can do this in the GUI interactively or use the HiveMQ CLI tool.
CLI: First enable the Datahub trail modus:
curl -X POST localhost:8888/api/v1/data-hub/management/start-trial
CLI: And then upload all the datahub assets with the CLI tool:
mqtt hivemq schema create --id=mytemp-in-schema --file=mytemp-in-schema.json --type=json
mqtt hivemq schema create --id=mytemp-out-schema --file=mytemp-out-schema.json --type=json
mqtt hivemq script create --id=add_timestamp --file=add_timestamp.js --type=transformation
mqtt hivemq data-policy create --file=add_ts_policy.json
GUI: If you decide to use CommandCenter (http://127.0.0.1:8080) for uploading the above the default username / password is admin / hivemq. Please click the enable Data Hub trial mode on the Data Hub top menu before proceeding.
Enabling the HiveMQ PostgreSQL Extension along with creation of the ‘insertion’ SQL SQL file is already done by the docker-compose.
Send Data
So, in order to generate data you can send JSON data manually to the broker by using the HiveMQ CLI tool. The data-policy expects data to be published on “temp/#”
mqtt pub -t temp/test --message-file=mytemp.json
You can also make use of the data-generating MQTT python script sim-temp.py in the repo or even build your own hardware sensor.
(If you're interested in ESP or Pico based hardware options for this goal, just drop me an email)
Once the data policy processes the incoming message on temp/# it will validate it, and if ok, it will add two timestamp fields to it.
A PUB-ed message on ‘temp/sensor’ containing (as in file mytemp.json) this:
{ "SensorID": "CMD-Line",
"temperature": 22.54 }Will be transformed by the broker Datahub script into...
{ "unixtime": "1736445953",
"SensorID": "CMD-Line",
"temperature": 22.54,
"isotime": "2025-01-09T18:05:53.505Z" }… and will be re-PUB-ed on the topic “Temp-TS/sensor”. If you want to keep an eye on that process, use the following (non-returning, so use a separate CLI session) command:
mqtt sub -t ‘#’ -J
The Database
The HiveMQ database extension will filter on topic ‘Temp-TS/#’ and will call a SQL insertion script (statement.sql) as shown below in the DB extension config file.
<!-- insert every message using the STATEMENT TEMPLATE processor -->
<mqtt-to-postgresql-route>
<id>my-mqtt-to-postgresql-route-template</id>
<postgresql-id>my-postgresql-id</postgresql-id>
<mqtt-topic-filters>
<mqtt-topic-filter>Temp-TS/#</mqtt-topic-filter>
</mqtt-topic-filters>
<processor>
<statement-template>conf/statement.sql</statement-template>
</processor>
</mqtt-to-postgresql-route>PostgreSQL DB extension config file
The custom insert SQL script executed by the HiveMQ database extension that decodes (the JSON-formatted MQTT payload) and inserts the JSON payload data into table ‘tempdata’.
INSERT INTO tempdata (sensorid,isotime, unixtime, temperature)
SELECT
json_data->>'SensorID' AS sensorid,
(json_data->>'isotime'):: timestamp AS isotime,
(json_data->>'unixtime')::numeric AS isotime,
(json_data->>'temperature')::numeric AS temperature /* casting from text to numic value ! */
FROM (
VALUES
( ${mqtt-payload-utf8}::jsonb)
) AS input(json_data);The SQL insert script ‘statement.sql’
You can use appropriate tooling (e.g. PGadmin4) to peek into this table and make sure everything works as described.
The Grafana Container
Once you log in Grafana (http://localhost:3000), select the dashboard option in the left main menu. You will find it populated with a ‘Temp Dashboard’ visualizing the ingested temp data along the current timescale X-axis.
Conclusion
MQTT data visualization is crucial for converting raw MQTT message streams into actionable insights, empowering users to monitor, analyze, and respond to real-time data with precision.
By combining MQTT (Message Queuing Telemetry Transport), the reliability of HiveMQ brokers, PostgreSQL databases, and Grafana, users can create powerful, efficient, and flexible pipelines for data collection, visualization, and analysis.
HiveMQ’s Data Hub enhances data quality by validating and enriching data, while HiveMQ Broker’s seamless integration with upstream databases and visualization platforms ensures unmatched simplicity and effectiveness.
Kamiel Straatman
Kamiel is part of Solutions Engineering team at HiveMQ. Based out of the Netherlands, Kamiel is an IoT enthusiast with many years of experience working in the data center domain and Azure & AWS cloud architectures. He specializes in ICT infrastructure consulting with a strong focus on cloud, cloud automation, datacenter transformation, security, storage and networking.
