Stream IoT Data Between MQTT and Kafka with HiveMQ Cloud

MQTT and Apache Kafka are often used together to enhance the functionality of IoT and Machine-to-Machine communications. You commonly see them paired in the following use cases:

• Data collection: MQTT is used to collect data from IoT devices and publish it to a Kafka broker, where it is processed, analyzed, and stored for future use.

• Real-time processing: Using MQTT and Kafka, organizations build real-time data processing pipelines that handle large amounts of incoming data from IoT devices.

The HiveMQ Cloud Kafka Integration

There are five (5) easy steps to ingest data from your IoT devices with the Apache Kafka service of your choice. These can be broadly divided into:

• Connection configuration parameters

• Topic mapping parameters 

The connection configuration parameters help establish a secure connection between HiveMQ Cloud and your Apache Kafka cluster. Topic mappings let you set up the bidirectional data flow between your MQTT cluster and Apache Kafka.

But first, you must find the Kafka extension in the “Integrations” tab inside your HiveMQ Cloud cluster. This integration is available with HiveMQ Cloud Starter. 

Integrations tab in HiveMQ Cloud

Configuring Your Kafka Integration

Now you are ready to dive into the five steps:

1. Connect HiveMQ Cloud with the Kafka service of your choice: To connect, you need a list of bootstrap servers for your Kafka cluster so the integration can fetch the initial metadata about your Kafka cluster.

2. Secure the connection: Now you need to add your Kafka credentials. This helps ensure there is a secure connection between HiveMQ Cloud and Kafka.

3. Send data from HiveMQ to Kafka: Once you set up and secure the connection, you can choose what data to forward from your IoT devices. This requires mapping topics from HiveMQ Cloud to your Kafka cluster.

The source topic is the MQTT topic you want to send from your HiveMQ cluster. The destination topics are the Kafka topic receiving the messages that your HiveMQ cluster sent. 

4. Establish bidirectional communication: For bidirectional communication between Kafka and HiveMQ, you can configure the Kafka cluster to HiveMQ Cloud similarly as you define the topic mapping from HiveMQ Cloud to your Kafka cluster. You can also use ${KAFKA_TOPIC}, ${KAFKA_KEY}, or ${KAFKA_VALUE} as variable part in the MQTT topic. This helps you to efficiently map one Kafka topic to one or more MQTT topics.

In this case, the source topic represents the Kafka topic from which the integration should read messages. These messages are then published with the defined destination topic on your HiveMQ Cloud MQTT broker cluster.

5. Enable the configuration: You can start the data flow between the HiveMQ Cloud cluster and your Kafka cluster by selecting the “enable” button. Once active, you see a green banner at the top, indicating that your integration is active.

If you've followed these five steps, you should now be able to employ Apache Kafka with HiveMQ Cloud to use data from your IoT devices for bidirectional communication. 

Validating the Data Flow With the Web Client

Now that you have enabled your integration, you can use the inbuilt web client to quickly validate the data flow between HiveMQ Cloud and Kafka server. 

For that, you need to enter your credentials and add the same topic name under “Publish message” section as the one entered in the configuration. Then, you can add test message payload and click the Publish button as shown in the image below. 

Once you see the message being published, you can go to your Kafka server and validate that the data has been flown into the mapped Kafka Topic. For this blog post, we used Confluent cloud but you can use any Kafka server of your choice.  

We’ve captured this information for you in a 2 minute video as well.

Get Started 

To access this Kafka-HiveMQ Cloud functionality, all you need to do is sign up for a 15-day free trial with HiveMQ Cloud Starter.