MQTT 5.0: The Next Generation of MQTT

In the previous blog post, we covered standardization of MQTT and the ambitious timetable set by the committee to make it happen. Now we’ll share the story of the improvements and enhancements to the protocol that shape MQTT as it’s known today.

During the MQTT 3.1.1 standardization, thought was given to future versions. There were tens of suggested improvements or new MQTT features that did not fit into the TC’s charter, and were recorded as “futures” issues. Throughout 2015 these issues were discussed and recorded. They were split and reorganized so as to provide a coherent set of improvements which could be prioritized and grouped.

In July 2015, several additional people from Microsoft joined the TC, with the goal of influencing a new version of MQTT. A year before, Clemens Vasters, a Microsoft software architect, wrote a blog post excoriating MQTT 3.1.1. Clemens was writing from the viewpoint of an experienced AMQP standardizer and practitioner, and MQTT looked, to him, horribly limited and deficient. In that post he surmised that MQTT was intentionally being kept limited. In a sense that was true, but not for the reasons Clemens supposed. The business case for MQTT is to support the lightest weight devices possible, in development as well as use. Its limitations are therefore a key selling point, supported by the MQTT community as a whole.

Issues

Between 15th October and 4th November 2015, a TC ballot was held to decide which issues should be prioritized. Each TC member was asked to choose up to 10 issues. The results of the ballot were:

There were 11 further issues that received no votes.

To define the approach to the next version of MQTT, two 3 day face to face workshops were scheduled. The first at IBM Hursley Park from 27th to 29th January and the second at Microsoft Bellevue from 11th to 13th April 2016.

For some of the original participants, their goal had already been achieved. At the TC meeting of February 25th 2016, Geoff Brown and the other M2MI participants resigned from the secretary role and left the TC. Geoff Brown sent this message:

“M2Mi feels that it is healthy to allow others to participate as Secretary rather than M2Mi continuously serving in the role as default... It's been fun, however I consider the spec where it needs to be and thus our input and support is no longer needed.”

New Charter

At the TC convener meeting of June 16th 2016, the first gathering under the new charter was held. New chairs, secretary and editors were elected, these being:

• Richard Coppen and Brian Raymor

• Ian Craggs

• Ken Borgendale, Andrew Banks and Rahul Gupta

respectively. Brian joined Richard as co-chair, replacing Raph Cohn. Andrew and Rahul, who had done sterling work as editors on MQTT 3.1.1 were joined by Ken Borgendale. Ian Craggs volunteered to be secretary, and was unopposed, not surprisingly, as it is not a sought after role. 

Raph Cohn was thanked for his efforts and commitment as co-chair during the 3.1.1 standardization period. 

The new charter allowed all sorts of changes which were previously outlawed for 3.1.1. Only these items were declared as out of scope:

• Mappings of MQTT to any programming language or particular messaging middleware.

• Reference implementations.

• Prescribing the payload format of messages.

• MQTT topics or topic space definition and classification.

• Additional mandatory security features - transport level (TCP/IP) security is assumed.

The enhancement suggestions that had been worked on and documented in JIRA issues over the previous year were collected together and sifted into several themes:

• Enhancements for Scalability and Large Scale Systems

• Improved Error Reporting

• Formalize commonly observed patterns - for example request response

• Extensibility Mechanisms - for example message formats

• Performance Improvements - for example, constrained devices

Version Number

MQTT has only one byte for the version number. MQTT 3.1.1 used a value of 4 in the connect packet to distinguish it from version 3. That lead to confusion when talking about implementations, so the decision was taken to align the standard version number with the connect packet for all future versions. As 4 had already been used, 5 was the next in line. If there is ever a follow on version, it will be 6.

At the end of June, Richard put together an example MQTT 5 timeline which showed a completion date in April 2017. That aggressive timetable was a noble aim, but given all the extra possible changes over 3.1.1, was optimistic.

There was a further three day face-to-face workshop from 26th to 28th September 2016 at IBM Hursley, to make further quick progress on issues and the specification working draft. Between the end of this workshop and the middle of 2017, the remaining issues were resolved and incorporated, and detailed text of the specification was revised and corrected. Anyone who has been involved in drafting documents which must be clear and accurate will know how much effort the last stages can be.

On 9th August 2017, the first public review of the new MQTT version 5.0 was announced, ending on 8th September. Given the number of changes that went into this update, that was a successful outcome.

There was a good amount of feedback as a result of this review. The most significant issue was the omission of properties on the will message, which was something that had just not been thought about. The rest were inconsistencies, omissions and minor typographical corrections. This freely given feedback is invaluable to improving the standards, and thanks goes to all those who make the effort.

A committee note on the handling of disallowed unicode code points was published on April 19th 2018. This affected both versions of MQTT and had been observed in the 3.1.1 timeframe.

By July 2018, a new draft of MQTT, responding to all the previous concerns, was ready. The TC was waiting for statements of use - a necessary part of the OASIS standardization process. The TC had already received them from HiveMQ and VerneMQ. IBM was preparing its own. Ian Craggs on behalf of the Eclipse Paho project was waiting for approval from the Eclipse Foundation.

A public review of the new OASIS candidate standard was announced on 15th November 2018, ending on 14th January 2019. There were some minor queries and comments but none that were deemed significant enough to delay publication.

The MQTT V5.0 standard was published on 21st March 2019, It is important to note that MQTT 3.1.1 was not deprecated as a result, both versions coexist and are expected to be deployed in future solutions.