MQTT Session Expiry and Message Expiry Intervals – MQTT 5 Essentials Part 4

Welcome to Part 4 of our MQTT 5 Essentials series. In Part 3 of this series, MQTT 5: Seven Reasons to Upgrade to it from MQTT 3.1.1, we illuminated the primary motivations for MQTT users upgrading to version 5, underscoring the enhancements that make MQTT 5 an attractive upgrade. In this article, we focus on two closely-related features: the Session Expiry Interval and the Message Expiry Interval. These functionalities represent key improvements in MQTT 5, and a thorough understanding of them is crucial for optimal protocol utilization.

How do Session Expiry Interval and Message Expiry Interval Work in MQTT 5?

Let’s break down and analyze these two integral expiry interval features individually.

Session Expiry Interval in MQTT 5

The Session Expiry Interval is a parameter a client sets during the CONNECT packet stage, specified in seconds. This parameter indicates the duration the broker retains the client’s session information. If this interval is set to zero, or if the CONNECT packet does not specify an expiry value, the session data is promptly deleted from the broker as soon as the client’s network connection terminates. Notably, the maximum session expiry interval is UINT_MAX (4,294,967,295), enabling an offline session to persist for an extended duration of just over 136 years following client disconnection.

The Session Expiry Interval can be defined in the connect packet

Message Expiry Interval in MQTT 5

Within MQTT 5, clients can set a unique Message Expiry Interval in seconds for each PUBLISH message. This interval establishes the duration the broker preserves the PUBLISH message for subscribers that match the topic but are currently offline. If the interval isn’t defined, the broker must indefinitely hold the message for matching subscribers, yet disconnected subscribers. Furthermore, if the ‘retained=true’ option is selected during the PUBLISH message, the interval also dictates the length of time a message is retained on a particular topic.

Publish Packet with Message Expiry Interval set to 120 seconds. The retainFlag is set to "true", so the message will also be retained for 120 seconds.

Why Were the Expiry Intervals Introduced in MQTT 5?

Understanding the functionality of these expiry intervals merely scratches the surface. It’s also essential to explore the reasons that prompted incorporating these features into the MQTT 5 specification, along with their practical applications.

In our article Introduction to MQTT 5 Protocol, we captured the spirit of the OASIS committee’s deliberation process, which culminated in the establishment of MQTT 5 as a standard in March 2019. HiveMQ, a proud member of this committee, actively gathered user feedback, comprehended the needs of long-standing users of the MQTT 3.1 and 3.1.1 versions, and determined how best to evolve the protocol. The main driving force was to introduce features that expanded the realm of possibilities for users and enhanced the simplicity of their interactions with the protocol.

MQTT 5’s Session Expiry Interval

The Session Expiry Interval in MQTT 5 masterfully fulfills two critical needs simultaneously. Earlier versions of MQTT offered a single avenue for removing persistent sessions as defined by the specification: a client bearing the same client ID as the session to be discarded would have to connect with the cleanSession=true flag set.

Consider scenarios where some IoT devices never reconnect due to decommissioning, destruction, or leftover sessions from inadequate cleanups post-load-testing. These residual sessions could impose an unnecessary burden on a broker’s persistence. Enterprise-grade MQTT brokers like HiveMQ come equipped with sophisticated administrative tools such as the HiveMQ Control Center to facilitate the management of idle sessions. Nonetheless, it is still crucial to discern which sessions can be effectively removed.

Beyond this automated cleanup capability, the session expiry interval has significantly simplified session state management. Ian Craggs graciously provided two diagrams illustrating the reduction in complexity in state transitions. This visual representation underscores how this new feature aids in streamlining state transitions and bolstering user efficiency.

De-cluttering of state transition in MQTT 3.1.1 and MQTT 5 (courtesy of Ian Craggs)

MQTT 5’s Message Expiry Interval

Much like its counterpart, the session expiry’s interval inception was fueled by a need for an automated maintenance mechanism. Consider the myriad of IoT devices, such as connected cars designed to weather extended periods of internet disconnection. MQTT offers a lifeline for these scenarios with persistent sessions and message queueing.

Messages crafted for offline devices are stored on the broker, waiting for the device to regain connectivity for delivery. In grand-scale deployments, where the count of connected devices escalates into thousands or even millions, it’s imperative to constrain the offline message queue for each client individually.

The longevity of message relevance to IoT devices can exhibit significant fluctuations. Take the example of the connected car: traffic updates maintain their relevance only briefly. Yet, when we consider over-the-air firmware upgrades, these need to be executed even if the car remains offline for an extended period, stretching into several weeks.

Example: traffic jam alerts usually become obsolete after 1-2 hours, but a firmware update should be available for many weeks. Message Expiry Interval is the perfect feature to define these different periods of time.

By assigning an optimal message expiry interval to messages with a limited relevance window and leaving perpetual relevance messages without an expiry, we ensure effective utilization of broker resources for clients that could be offline for a considerable duration. This strategy also spares clients from being inundated with irrelevant messages upon reconnection.

One crucial aspect to note, however, is that when a client’s session expires, all queued messages for that client also expire in sync with the session, irrespective of their individual message expiry status. This “GOTCHA” is an important reminder of the interconnectedness of sessions and their queued messages in the MQTT protocol.

Important Tips While Using MQTT 5 Session Expiry Interval and the Message Expiry Interval

Here are some important information:

• Both the Session Expiry Interval and the Message Expiry Interval are instrumental in refining resource management on the MQTT broker.

• Reflecting on the past, many MQTT 3 users expressed a need for expiry features. HiveMQ responded to this demand by introducing session and message expiry as supplementary features in our MQTT platform, predating their standardization in MQTT 5.

• The equivalent for cleanSession=true in the CONNECT packet of MQTT 3 is sessionExpiry=0 (or absent) and cleanStart=true in MQTT 5.

• Likewise, cleanSession=false from the MQTT 3 CONNECT packet finds its counterpart in MQTT 5 as a sessionExpire value exceeding zero, coupled with cleanStart=false.

• MQTT brokers, such as HiveMQ, offer the capability to configure a maximum value for these expiry intervals on the server side. This feature is handy in multi-vendor projects, particularly when the broker operator may lack authority over the MQTT client settings.

The advent of MQTT 5 has ushered in a host of new features designed to enhance the protocol’s usability, flexibility, and efficiency. The Session Expiry Interval and the Message Expiry Interval are prime examples of this, acting as invaluable tools in resource management and ensuring the smooth functioning of your MQTT broker. Both these features truly embody the user-centric evolution of the MQTT standard, demonstrating its responsiveness to the demands and challenges of its users.

Learn More About MQTT 5

As we continue to delve into the myriad features of MQTT 5, our journey will take us even deeper into the protocol’s mechanisms. In Part 5, we will explore Improved client feedback & negative ACKs (NACKs) — an important innovation designed to increase reliability and error handling within the MQTT protocol. We will explain how they play a crucial role in the future of MQTT and IoT, offering intriguing possibilities and further broadening MQTT’s appeal in the IoT world.

Sign up for our newsletter to get regular updates. Subscribe to our RSS feed here to stay updated. We encourage you to visit our MQTT Glossary for an in-depth understanding of the essential MQTT terminologies. It will equip you with the necessary vocabulary to grasp the complexities of MQTT and its various versions. Watch the video below that complements the concepts discussed in this article.