Bidirectional IoT Messaging between MQTT and Apache Kafka

Many companies use Apache Kafka as their central messaging and real-time streaming platform. A pivotal question for many of these companies is how to efficiently process the data they collect from IoT devices. Using MQTT to send IoT data via Apache Kafka for further processing in a system of your choice is certainly a very attractive answer.

So what is the best way to connect an MQTT broker with a Kafka cluster? There are several options, but the fastest and most reliable solution is achieved when you have full control over both the MQTT broker and the Kafka cluster.

The HiveMQ Enterprise Extension for Kafka gives you exactly this kind of control over your systems. As a tailor-made enterprise extension for the HiveMQ MQTT broker, the extension hooks directly into the MQTT implementation and can speak fluent MQTT. On the other side, it implements the Apache Kafka Protocol and serves as a full Kafka client. This versatile implementation can process hundreds of thousands of IoT messages between HiveMQ and Apache Kafka.

The initial release of our HiveMQ Enterprise Extension for Kafka provided a fast and reliable way to ingest MQTT data into Kafka. However, sending data from HiveMQ to Kafka isn’t enough for many of our customers’ use cases. Usually, our customers want to read data from Kafka topics and send that information back to IoT devices as MQTT messages. Today, we are thrilled to announce that the new version of the HiveMQ Enterprise Extension for Kafka now supports bidirectional message sending!

What are the main new features?

The theme of this release is clear: Bidirectionality. In addition to the existing message ingestion of MQTT messages into Kafka, the new version adds support for sending messages from Kafka to IoT devices. Let’s dive deeper into the new possibilities:

Publish messages from Kafka to IoT devices

The use cases of many customers require forwarding Kafka records to IoT devices. This ability makes it possible to take full advantage of both systems. Backend systems can respond to messages from IoT devices, process the information, and generate new data. In return, IoT devices can respond in real-time to the newly generated data that Kafka provides.

To offer this functionality the HiveMQ extension reads information from specific Kafka topics and transforms the data into MQTT messages. The HiveMQ MQTT broker processes these MQTT messages and publishes them to the appropriate MQTT topics and the subscribing IoT devices. Because the extension is deeply integrated into HiveMQ, the sending of these converted messages is fast and reliable. Our customers benefit from native implementations of both the MQTT and the Kafka protocols. The HiveMQ extension honors all message-delivery guarantees of both protocols in both directions.

Dynamic MQTT message generation

In the most basic use case, the Kafka record is transposed to an MQTT message using the Kafka topic and Kafka value. In other use cases, it can be desirable to use some of the metadata from the MQTT publish, such as the Quality of Service, retained flag, or user properties. The HiveMQ extension makes it possible to statically configure these settings or dynamically generate MQTT messages based on the information extracted from the Kafka record.

In this example, the Kafka key is used as part of the MQTT topic and is replaced at runtime with the key of the corresponding Kafka record. The Kafka value contains two fields, a message and a QoS. These two fields can be used as placeholders in the MQTT PUBLISH packet and replaced with the real value at runtime.

The dynamic message creation requires the Kakfa messages to follow a specific schema. The current expectation is that the Kakfa messages are in AVRO format.

Message Multicasting

Publishing Kafka records from one or more Kafka topics to a defined number of MQTT topics is useful. However, in some cases it is more useful to be able to generate the MQTT topics dynamically depending on the Kafka value in the Kafka record.

Reusing the concept of placeholders in the MQTT topic section of the configuration makes it possible to multicast one Kafka record to multiple IoT devices. The topics are generated at runtime with values extracted from the Kafka record. This method allows you to send one Kafka record to a large number of IoT devices thus saving storage, computing power and bandwidth in your Kafka cluster.

With HiveMQ 4.1 or greater and the HiveMQ Enterprise Extension for Kafka you can now use the HiveMQ broker to bring data from IoT devices into Kafka and also retrieve information from Kafka topics and distribute them to specific MQTT clients. If you want to try it out (free trial available), download HiveMQ and the HiveMQ Enterprise Extension for Kafka from the HiveMQ Marketplace. To see more details about the new features and to get deeper insights into how to use them, visit our documentation. As always, if you have questions, don’t hesitate to contact us. We are happy to help.