Underutilized MQTT 5 Features that Enhance Modern IoT Data Flows
MQTT 5 introduces a range of features designed to improve the flexibility, efficiency, and functionality of the MQTT protocol, especially for IoT (Internet of Things) and high-frequency API applications. Key enhancements include the Topic Alias and the Request-Response pattern, both of which aim to optimize messaging and enhance secure communication.
Together, the Topic Alias and the Request-Response pattern make MQTT 5 a lean, more robust, and versatile protocol, empowering developers to build applications that are both resource-efficient and responsive to real-time needs.
While these MQTT 5 features have been around for a while, they are still not widely familiar to everyone. This blog aims to clarify its features, offering explanations and sample Python code that demonstrates MQTT 5 capabilities using the latest Paho Python libraries, ready to run out of the box. This blog also serves as a practical example of implementing some of the features discussed in the blog, Optimizing Data Cost Efficiency in MQTT-Based IoT and Connected Systems, by Francisco Menéres.
MQTT Request-Response Pattern – “Lets shake hands”
The MQTT Request-Response Pattern enables a structured method for bidirectional communication, resembling a traditional client-server model within the MQTT framework. Unlike the standard MQTT publish/subscribe approach, the Request-Response pattern allows clients to send targeted requests and receive specific responses on a preset response topic, enabling more interactive and synchronous operations. This is ideal for applications where direct, immediate feedback or command execution is essential, such as remote device management, monitoring, and troubleshooting.
Topic Alias – “You don't want your topic to contain more data than your payload.”
The Topic Alias feature greatly minimizes the size of packets transmitted over the network by enabling clients to substitute lengthy topic names with shorter, reusable aliases. This reduction in data payload is particularly beneficial in environments with limited bandwidth, such as many IoT scenarios. Additionally, the widespread use of Unified Namespace(UNS), which often involves complex and lengthy topic structures, can see significant advantages from this feature.
By optimizing data transmission, Topic Alias helps MQTT clients operate more efficiently, improving both network performance and reducing latency in communication.
MQTT Request-Response Pattern
Ensuring Message Delivery is NOT Ensuring Message Processing
MQTT has three Quality of Service (QoS) levels that help ensure messages are delivered at the desired level of reliability. In a Request-Response setup, QoS can help guarantee that requests and responses are reliably delivered, which is critical in environments with intermittent connectivity, such as IoT.
Why QoS 2 is Reliable
The four-step handshake process in QoS 2 ensures that, even if a network issue or a disconnection occurs during the message transfer, the client or broker will know which step was last completed. If a step fails, the protocol can resend the last unconfirmed packet without reprocessing or duplicating the message.
But QoS 2 is not enough sometimes.
The publish/subscribe model of MQTT fully separates the sender and receiver of messages, enabling asynchronous message delivery. However, this approach has a drawback: even with QoS levels 1 and 2, the publisher can only confirm that the message reached the server without any assurance that the subscriber actually received it. In cases where requests or commands are issued, the publisher may need feedback on the outcome at the receiving end.
In summary, QoS level 2 in MQTT guarantees that a message is successfully delivered to the broker once, but it doesn’t ensure that it reaches a specific subscriber or is processed correctly on the receiving end, such as completing a database commit. To confirm this end-to-end processing, the receiving subscriber needs to send an MQTT acknowledgment message back to the original sender. This is where the MQTT Request-Response pattern is valuable, enabling reliable acknowledgment from the subscriber to the sender. Let me walk you through how this works.
Components of MQTT Request-Response
To implement request-response in MQTT, on the client or publisher side, we use the following entities that are part of the MQTT 5 standard, which are available in the latest Paho libraries:
Request Topic: The client making the request publishes a message to a "request topic." This is the topic to which the server backend is subscribed.
Response Topic & Correlation ID: The client making the request defines and sets a unique response topic where it expects to receive a response. It also sets a unique Correlation ID so it can chain the reply to the request that was sent earlier.
To ensure the expected response, the client should subscribe to its specified response topic. By matching received correlation IDs on this topic, a complete SEND-ACK/NACK handshake can be achieved at the application level. This approach removes the need for QoS 2 on the MQTT level, allowing for the faster QoS 0 instead. As long as a valid ACK, via MQTT request/response mechanism is not received, no assumption can be made that the data has been sent or processed.
On the backend or subscriber side, we use the following entity:
Request Topic: This is the topic the server backend is subscribed to. Messages received here will be decoded and processed. If an acknowledgment is appropriate to report successful processing, an appropriate MQTT message is published to the Response Topic as set in the original message along with the original Correlation ID that was in the original request, ensuring the sender can receive and link the response to the original request.
Example Workflow of Request Sequence
Let's say a Sender wants the temperature data from the Backend.
Sender publishes a message to
temperature/requestwith:Payload:
{ "request": "current_temperature", "response_topic": "sender/response", "correlation_id": "12345" }
Backend (subscribed to
temperature/request) receives this request, reads theresponse_topicasclientA/response, andcorrelation_idas12345.Backend processes the request and publishes a response to
sender/responsewith:Payload:
{ "correlation_id": "12345", "temperature": "22°C"}
Sender receives the response on
sender/response, checks thecorrelation_id, and processes the temperature data.
Topic Alias
In MQTT, a Topic Alias is a feature that allows clients to use shorter identifiers (aliases) for topics in message exchanges. This can help reduce the size of the messages being sent, which is especially useful in environments with bandwidth limitations or when dealing with numerous topics, which is becoming more standard in UNS situations.
Mapping is actually very easy.
When a client subscribes to or publishes a message to a topic, it can use a numeric alias instead of the full topic name once the topic is associated with the alias. The topic alias must be between 1 and 65535; a value of '0' is not allowed.
The broker and client maintain a mapping between the alias and the actual topic name, which helps reduce the number of bytes transmitted with each message.
To establish this mapping, when the client first publishes a message, it sends both the topic name and the Topic Alias. After this initial transmission, the topic name is no longer needed in subsequent messages, as shown in the example below.
Topic = 'sender-topic/cmd/ridiculously_long_topic_name/sensor'
publish_properties.TopicAlias = 3 # arbitrary number
# now topic alias for Topic is set to 3 at the brokerside
client.publish(Topic, "initial payload", 1, properties=publish_properties)
# now topic alias for Topic is set to 3 and no topic is explicitly set
client.publish('' , "subsequent Payload on alias 3", 1, properties=publish_properties)Important Notes
Persistence: The Topic Alias is usually valid only within a session. Once the connection is closed, the alias-to-topic mapping is typically lost. A non-clean, reconnect session will retain the aliases.
Topic Alias Limits: The HiveMQ broker can impose limits on the number of topic aliases a client session can handle at a time, as it requires maintaining a mapping. The number of topic aliases available per client is set to 5 by default, but can be changed as seen below in the configuration snippet:
<mqtt>
<topic-alias>
<enabled>true</enabled>
<max-per-client>5</max-per-client>
</topic-alias>
</mqtt>Code Example for Implementing Topic Alias and Request-Response Pattern
The accompaniment sample code is available on GitHub, and demonstrates the two MQTT features we discussed in this blog. Please update the endpoint in the first few lines of each code snippet so this points to your MQTT broker.
Topic Alias Sample Code
As shown in the example below, we set the TopicAlias, and from that moment on, there is no need to set the topic as long as the alias is set.
# Publish
Topic3 = 'sender-topic/cmd/ridiculously_long_topic_name/3'
Topic7 = 'sender-topic/cmd/another_long_topic_name/7'
publish_properties = Properties(PacketTypes.PUBLISH)
publish_properties.TopicAlias = 3
# now topic alias for Topic3 is set to 3 at the brokerside
client.publish(Topic3, 'Initial payload with alias 3', 1, properties=publish_properties)
publish_properties.TopicAlias = 7
# now topic alias for Topic is set to 7 at the brokerside
client.publish(Topic7, "Initial payload with alias 7", 1, properties=publish_properties)
publish_properties.TopicAlias = 3
# now topic alias for Topic is set to 3 and no topic is explicitly set
client.publish('', "subsequent Payload on alias 3", 1, properties=publish_properties)
publish_properties.TopicAlias = 7
# now topic alias for Topic is set to 7 and no topic is explicitly set
client.publish('', "subsequent Payload on alias 7", 1, properties=publish_properties)This will result in the publishing of the following payloads:
MQTT Request-Response Sample Code
The demo demonstrating the MQTT Request-Response sequence involves two main components. First, there’s the backend, which listens to the topic sender-topic/cmd/#. When a message arrives on this topic, a command_dispatcher is triggered to decode and process the message. Based on the outcome of this processing, an ACK or NACK message is sent to the ResponseTopic along with the associated CorrelationData.
To send instructions to the awaiting backend, the Sender.py code transmits a JSON-formatted command to it. This MQTT message includes a response topic specific to the client (response/ + client_id) as well as CorrelationData. Upon receiving an ACK or NACK, the client then waits for a response from the backend.
Conclusion
In conclusion, implementing an MQTT Request-Response pattern and Topic Aliases can greatly enhance communication reliability and efficiency in IoT and messaging systems. By setting up well-defined topics, using CorrelationData, and handling ACK/NACK responses, you can ensure that messages are processed accurately and that clients stay informed on the status of their requests. This approach helps build robust, responsive applications, especially in environments where devices and servers need to communicate seamlessly.
If you're looking to improve message handling and control in your systems, experimenting with MQTT's Request-Response pattern is a powerful step forward. With proper implementation, you can streamline data exchange, minimize errors, and create a responsive, dependable user experience. Contact us if you need help with your MQTT deployments.
Kamiel Straatman
Kamiel is part of Solutions Engineering team at HiveMQ. Based out of the Netherlands, Kamiel is an IoT enthusiast with many years of experience working in the data center domain and Azure & AWS cloud architectures. He specializes in ICT infrastructure consulting with a strong focus on cloud, cloud automation, datacenter transformation, security, storage and networking.
