Real-time Data Processing with Amazon Kinesis Integration on HiveMQ Cloud

We are pleased to announce the addition of Amazon Kinesis integration to HiveMQ Cloud Starter. Amazon Kinesis integration enhances our MQTT platform by providing IoT data integration and real-time processing capabilities to HiveMQ Cloud users. This is valuable for individuals and organizations focusing on IoT and data-driven decision-making because it enables users to effortlessly collect, process, and analyze data from many IoT devices in real-time, leading to faster, more informed decision-making and a boost in operational efficiency for AWS-based processes. 

Amazon Kinesis Integration  

What is Amazon Kinesis? 

Amazon Kinesis is a comprehensive suite of tools on AWS to facilitate real-time data streaming and analytics. Amazon Kinesis Data Streams enables the continuous capture and storage of data streams such as IoT device data, supporting ingestion and enabling real-time analytics on this streamed data. As such, there is often a need for data transport from MQTT broker to AWS data pipeline using Kinesis Data Streams. 

What is Included in the Integration? 

This integration supports communication between IoT devices to Amazon Kinesis Data Streams. Once the data is integrated, it can be processed, stored, analyzed, or visualized for AI/ML projects — all within AWS.

The Amazon Kinesis integration is available under the Integrations tab in your Starter Cluster. 

What are the Benefits of Using Amazon Kinesis Integration with HiveMQ Cloud?

The Amazon Kinesis integration with HiveMQ Cloud is particularly valuable for individuals and organizations focused on IoT and data-driven decision-making. As a user, it enables you to effortlessly transport data from your cloud broker instance to your AWS services. This allows you to process and analyze data in near real-time, leading to faster, more informed decision-making and a boost in operational efficiency for AWS-based processes. 

Configuring Amazon Kinesis Integration 

Configuring Amazon Kinesis integration consists of three easy steps: 

1. AWS Authentication 

You need an AWS credentials file. Your file should contain aws_access_key_id and aws_secret_access_key 

2. MQTT to Topic Filter 

The next step is to create a mapping from MQTT to AWS Kinesis. Currently, HiveMQ Cloud Starter supports unidirectional mapping from MQTT to the Kinesis stream. You can select the MQTT topics you want to stream data to Amazon Kinesis by defining an MQTT topic filter. You can utilize MQTT wildcards here. 

3. Amazon Kinesis Stream and Region Configuration 

Finally, add details such as your Amazon Kinesis data stream name and your Kinesis region to complete your configuration. This helps you to connect your broker to the right Kinesis data stream.

Here is a video that walks you through the integration: 

Summary 

The Amazon Kinesis integration provides a simple and easy way to connect your HiveMQ Cloud broker with AWS workflows, leading to faster, more informed decision-making. This integration is a simplified version of the Enterprise Kinesis extension which is available with Cloud Enterprise Self-hosted plans.