How MQTT 3.1.1 Was Standardized

In our last article, we talked about the pre-standardization era of MQTT. Now we’ll cover the pivotal years and monumental effort that led to the protocol’s standardization.

Calls for the standardization of MQTT were coming from numerous sources by this time, including IBM clients, industry providers and users. Its adoption was hindered by its IBM ownership. IBM was unlikely to change MQTT in an abrupt or unexpected way, but the possibility existed.

On 26th March 2013, the kick off meeting of the OASIS MQTT Technical Committee (TC) took place at EclipseCon in Boston, along with an online meeting for those who couldn’t attend in person. Scott De Deugd of IBM gave an introduction, and Chet Ensign welcomed everyone on behalf of OASIS. Richard Coppen of IBM and Raphael Cohn were elected as co-chairs. Geoff Brown, the founder of M2MI, was elected as secretary. A number of M2MI representatives sat on the TC, including Julien Niset and Louis-Philippe Lamoureux who between them documented the minutes during the MQTT 3.1.1 standardization period. Minute-taking is a necessary but time-consuming and unglamorous task, so their efforts were much appreciated.

There were representatives from IBM, M2MI Corp, dc-square (later HiveMQ), Cisco Systems, WSO2, Eurotech, Software AG, Open Geospatial Consortium, Solace Systems, Landis+Gyr, Red Hat, INETCO and Cimetrics present at that first meeting. Richard proposed meetings every other week starting on Thursday 11th April 2013 - and the convention of holding TC meetings on Thursdays was born.

At this point, MQTT was an abbreviation for “Message Queuing Telemetry Transport” which was incorporated in the name of the OASIS TC. During standardization it was agreed by the TC that from then on MQTT would not be an abbreviation but the protocol name in its own right. 

A very aggressive timeline was set with the goal of producing a standardized MQTT in less than a year. To this end, only minor changes to the input MQTT 3.1 specification were considered, as outlined in the committee’s charter, which was principally written by Peter Niblett of IBM:

• backward compatibility with MQTT 3.1 was crucial

• minimize disruption to existing implementations

• any requirements or recommendations which would break backward compatibility to be documented for a future major revision

• other than clarifications and editorial changes, updates were allowed only to the Connect command in a backward compatible way

To ensure only the core features were considered and avoid potentially lengthy discussions which could derail the timeline, some topics were explicitly defined as out-of-scope:

• Mappings of MQTT functions to any programming language or particular messaging middleware.

• Reference implementations of the protocol

• Payload format of messages published according to the specifications (except for the values and fields directly related to the MQTT protocol)

• Standardized MQTT topic names

• Any MQTT-specific mechanism or convention to classify topics or topic spaces.

• No security features will be added over and above the input specification.

In an interview with Richard Coppen, co-chair, he said there were a lot of processes to learn. Both Richard and Raphael Cohn, the other co-chair, were OASIS novices. Peter Niblett, who had extensive previous OASIS experience, and the OASIS staff, were very helpful in supporting the chairs in navigating the standardization journey. Over the next year, they would hold over 50 TC meetings, mostly by teleconference, as the various team members were located around the world. 

Experienced OASIS staff, such as Chet Ensign, thought that completing the first standardization in one year would be impossible - they had never seen it done before. The fact that it was largely achieved (reviews and error corrections took it to 18 months) is a testament to the motivation and commitment of all involved, especially the co-chairs for creating the momentum needed, and the editors Andrew Banks and Rahul Gupta of IBM for the consistent effort they made. 

There were consistent and major contributions from many members of the TC. Of those not already mentioned in this context, I’d like to highlight some more of the people involved in identifying and resolving issues during this period: Sarah Cooper of M2MI, Nick O’Leary and Peter Niblett of IBM.

Security Sub-Committee

Geoff Brown and M2MI were very focussed on the security needs and implications of MQTT and were key to the establishment of a Security Sub-Committee (SC). Because MQTT overlays TCP/IP and TLS much of the attention is given to protecting the communications at that level. 

The work of the SC resulted in the Security chapter and the Committee Note titled “MQTT and the NIST Cybersecurity Framework Version 1.0” also referenced in the specification. The key members were, from M2MI: Geoff Brown, Sarah Cooper, Julien Niset and Louis-Philippe Lamoureux, and Allan Stockdill-Mander of IBM. 

The specification chapter outlines the risks that MQTT implementations will face, along with suggestions of how to reduce them with reference to other security and cryptography standards. Even though this chapter lists the types of attacks that should be considered when creating an MQTT solution, a number of CVEs were raised after publication of the standard on exactly those topics.

The Committee Note introduces “implementors and senior executives to the NIST Framework for Improving Critical Infrastructure Cybersecurity and its relationship with the MQTT security recommendations”. Louis-Philippe Lamoureux and Allan Stockdill-Mander were thanked explicitly for the “great work they put in on the draft”.

Technical Advisory Board Review

Between February 2nd and 6th 2014, Patrick Durusau and Jacques Durand of the OASIS Technical Advisory Board (TAB) reported over 57 issues as a result of their review of the first MQTT Committee Specification Draft (CSD). The TC did not expect such detailed feedback, assuming that the MQTT draft was almost ready to go. The TC then scheduled more frequent weekly meetings for the next two months to get through the newly created issue backlog. The specification was improved significantly as a result of the TAB’s diligence, which was greatly appreciated.

Final Steps and Reception

After the TAB review had completed, and the issues it raised resolved, the second public review draft was completed in April 2014. At the TC meeting of May 29th 2014, statements of use were received from:

• Software AG

• IBM

• HiveMQ

• 2lemetry

and a candidate standard was balloted on and approved. At least three statements of use from different parties are required by OASIS to show that the candidate standard is implementable and broadly achieves the aims it sets out, which includes interoperation with other implementations. The first candidate OASIS standard was produced in June 2014. A 60 day Public Review, another OASIS requirement, ran from 7th July 2014 until 4th September. A number of small changes were suggested as a result, but none significant enough to warrant another public review period. 

ISO Submission

On April 7th 2015 a TC ballot concluded unanimously in favor of submitting the MQTT 3.1.1 specification to the ISO/IEC JTC1 committee. There followed a 30-day ISO membership review concluding on 30th May - no comments were received. After due process the ISO/IEC standard was published in June 2016. This was a shortened time frame and simplified process for new technical standards - it would not be available for subsequent MQTT versions.

MQTT 3.1.1 standardization was no small feat but a pivotal milestone in the history of the protocol. We’ve gone from 1999 to 2014 so far in our series and up next: MQTT 5.0 (2015-2018).