Building a Specification on top of MQTT to Meet Your Industry Requirements

Introduction

HiveMQ has extensive expertise with MQTT. We are dedicated to advancing the protocol, hence our long-standing involvement with associated steering committees and working groups. Over more than a decade working in this space, we’ve seen many organizations in various industries - from oil and gas to manufacturing to smart cities - looking for guidance on implementing MQTT to meet their specific industry needs and requirements.

MQTT is the de-facto standard for IoT data movement due to its ability to connect millions of devices securely and reliably. However, during the implementation phase, companies may struggle to define the MQTT specification without a roadmap. Some specifications for MQTT-based communication already exist, such as MQTT Sparkplug, which is now successfully speeding adoption in the Industrial IoT space. However, other industries and use cases could use a similar boost that Sparkplug gave. We’re here to help.

This guide aims to help IT Architects, IoT developers, solutions architects, and engineers take the right steps to deploy MQTT for IoT data movement and create the standard, or specification, for their unique implementation. Once you put the right foundation in place and define a uniform approach to communication via MQTT, all data publishers and subscribers can easily communicate with each other regarding their common use case.

The following steps will help you define the MQTT specification for your needs to build value upon a robust and flexible foundation for IoT messaging.

Key Steps to Building an MQTT Specification

An MQTT specification should map all facets of machine-to-machine (M2M) communication as completely as possible, only leaving out those areas for which no uniform approach can be defined. One should also consider future growth or potential implementation or use case changes.

Root the initial definition of an MQTT specification on the primary use case. For example, suppose the primary application is a smart building or a communication to a fleet of ships. In that case the specification will develop differently than if it was a connected car use case. Each industry vertical has unique requirements.

To build an MQTT specification, start by defining the following requirements for your IoT project.

• Interoperability requirements

• Evolution requirements

• Infrastructure and deployment requirements

• Communications requirements

• Data integrity requirements

• Functional data requirements

• Maintainability requirements

Next, take the following key steps:

Step 1 - Define a Topic Namespace

The free and dynamic use of topic names is a major benefit of MQTT, but the protocol doesn’t dictate the message topic namespace or data representation. In production environments, it is necessary to define a topic namespace, so all devices and applications involved know, understand, and use the same terms and relationships in that space.

Step 2 - Specify the Payload Data Structure

Similar to the definition of the topic namespace, a schema for the contents of the messages, or the payload, must be defined to use MQTT in production environments. This can be implemented elegantly with the native concepts of MQTT 5, described in detail in the next section.

Step 3 - Define the State and Data Flow Management

Knowing the state of any device is relevant for most use cases. MQTT has useful features for this purpose, which will be discussed in detail later in this paper. Since MQTT was originally developed to monitor real-time systems, it inherently handles network failures, low bandwidth, and high latency well for data flow management.

Step 4 - Define the Security Requirements

IoT messaging communication must guarantee that each device is trusted and authorized to exchange information. A secure access validation process should be part of the specification.

Step 5 - Set Boundaries

Finally, the specification should also state the scope of its validity. These provisions may specify the audience, define the industry or use case, and outline the limitations of its use.

Building the Specification

Once you have defined the requirements and understand the key steps that will be required to successfully build and implement the MQTT specification, we recommend you work through the following detailed considerations.

Enable Versioning

A simple but essential specification detail is versioning, or allowing the specification to evolve with controlled change. Without versioning you cannot change the specification in the future if you have compatibility issues. Therefore, every specification should also use the version in its topic structure and in its payload schema, as is usually done for Rest APIs, for example.

Connectivity and Secure Communications

The details of the connection setup is an important topic. Here are a few questions to consider regarding connectivity: Do the participants work in stable or often unreliable network environments? What type of connection can be used by all parts? The environment is important to understand for security decisions. For example, can all participants use certificates and communicate with TLS, or can the rule be defined to communicate via secure websockets? Is the whole setup in a private VPN, in which case, certificates might not be necessary, or is the use case in a public space where security is extremely important?

One result of the above validation could be:

• All participants use Secure Websockets with JWT tokens managed by an OAUTH service.

• All participants running in a private VPN use simple TLS with a ClientID for authentication that corresponds to a specific address range.

If it is impossible to define a uniform method for establishing the connection, define variants per subscriber group.

For example - mobile clients use TLS with JWT, and applications that operate in the internal VPN use TLS with Active Directory. If possible, a specific (default) port should also be defined here for each variant, or group of clients.

The design specifications for authentication/authorization should describe the required data and its use. Data transferred upon successful authentication can be used for authorization. For example, for authentication with JSON web tokens, use role mapping claims.

Client Identifier

The structure of the client identifier (ClientID) is critical for many decisions. The ClientID can have an important role in defining rights via device-based access, where the ClientID is dynamically integrated into the topical structure. Thus, a rule could be defined where the ClientID is always specified as a particular part of the topics architecture for which the client has subscribe or publish rights.

First, it must be verified whether ClientIDs of all participants can be identified and validated or whether they are random numbers from which no meta information can be taken.

If the ClientID is always the same when the client logs in again, consider the use of persistent sessions, which are especially helpful for unstable network conditions. With persistent sessions, client-specific information can be stored at the broker. This information includes subscriptions or accruing QoS 1 or 2 messages that the client missed offline or has not been fully acknowledged. This minimizes possible message loss and saves bandwidth.

Suppose a ClientID can be validated with respect to the membership of a group of participants. In that case, this meta-information can also be further used, for example, as authorization. In addition, validating a ClientID for specific patterns can help increase system security. Therefore, specify common patterns as thoroughly as possible.

Example

Clients are only allowed to subscribe to their own topics and realized if the topic structure contains their own ClientID.

Possible Permission Definition

Permission applied:

Another benefit of using the ClientID is the possibility of providing the status of a device for all participants in the system. This only makes sense if the ClientID is reusable.

Suppose status information is required but cannot be implemented with the ClientID. In that case, it may be possible to implement this with a username or other information from the CONNECT package for identification.

Another benefit of using the ClientID is the possibility of providing the status of a device for all participants in the system. This only makes sense if the ClientID is reusable.

Suppose status information is required but cannot be implemented with the ClientID. In that case, it may be possible to implement this with a username or other information from the CONNECT package for identification.

MQTT Version - Interoperability, Functional Requirements

For a specification based on MQTT, choosing which MQTT version to implement is a key step. The requirements of the implementation from both an MQTT broker and client standpoint should dictate which version is chosen so once you understand what is needed for the use case then you can decide.

With MQTT 5, for example, certain process paths can be solved more efficiently using the core features like user properties or shared subscriptions.

Simultaneously, MQTT 5 implementations have several optional features that weakened it compared to MQTT 3. For example, the feature, persisting retained messages at the broker, is optional in MQTT 5. This means a (broker/client) implementation can label itself MQTT 5 compatible without implementing all optional features.

For this reason, the specification should specifically state which features are expected from the various participants including which features the broker must support. To determine the necessary feature list, reference the processes for the individual use case.

Some Typical Examples

• If a client status mechanism is required, using Last Will & Testament principle (and Retain Flag) is necessary. (MQTT3 Standard, MQTT5 Optional)

• To ensure fail-safety on the client side, use shared subscriptions for backend-applications to implement client-based load balancing to minimize possible downtime (for example, during upgrades).

• If different payload formats are to be used and validated, the MQTT 5 features such as User Properties and Payload Format Indicator are beneficial.

• If end-to-end delivery guarantees are needed, the request/response pattern from MQTT5 is suitable.

State Management

As already described, if a client status mechanism is necessary, this can be fully implemented with LWT. If the status message is stored as a retained message, it is also available for newly added MQTT clients.

Life Cycle & Clients State

Payload Format and Related Opportunities

First, determine whether there is one payload or different payload formats in use. Ideally, the format must be specified. Whether binary, compressed, or human-readable formats are used depends on the specific application.

Binary formats like Protobuf or Avro are structured and compressed, saving bandwidth transmission. When direct readability is required, XML, JSON, or YAML structures are suitable.

Schema should be present to check and extract parts of the payload.

The current schema could be stored as a retained message in an initialization process at a topic defined for this purpose. This approach makes it easy to manage different schemas for groups of clients with the same properties.

Meta Data Usage

When using MQTT5, employ the Content-Type property to identify the payload type without reading the binary stored payload first.

Transfer additional metadata for the payload via the user properties.

Example Publish Message

Key Features:

• Human readable

• Schema validation possible

• Parsable

• Parsing necessary to get value

Example Publish Message

Key Features:

• Binary Payload

• Direct Read possible

• Metadata for interpretation outside Payload available.

Message characteristics - Delivery Guarantees

An important point is the definition of the delivery guarantees, Quality of Services for the PUBLISH (and Subscribe) messages.

When selecting QoS, it is crucial to remember that the delivery guarantees in an event-based system with the broker as the central element always refer to only part of the communication between different clients.

An end-to-end delivery guarantee from a client to an app cannot be mapped with QoS alone.

An excellent way to select the right delivery guarantee for a specific instance is to focus on the following typical cases:

• High-frequency data

  • Data that is only relevant for a short time and is overwritten at high-frequency should be sent with QoS 0. This generally includes telemetry data.

• Status, Business data

  • Status information or all data essential to the functionality of the use case, where handling duplicates is not critical, so most use cases require QoS 1.

• Business-critical data

  • The difference between QoS 1 and 2 is mainly the avoidance of duplicates. However, this also requires a higher communication overhead, consisting of 4 steps instead of 2.

• Door Opening UC

  • An end-to-end delivery guarantee can best be implemented using the request-response pattern and the correlation ID (MQTT 5). QoS 1 can also be used for this.

Payload and Report by Exception

Publish messages are usually responsible for the majority of data volume. Therefore, it makes sense to take a special look here at how to minimize data redundancy. An effective method to implement is, Report By Exception, which consists of the following steps.

• Establish a client connection

• With the first message, define all values this client can send to a specific topic as a template. (Initial, Retain)

• After that, only messages with values that change are sent.

• Clients subscribed to this information thus get a statement of all possible values (template) plus changes.

Topic Design - Ontology

Possibly the toughest stage is the definition of the topic structure. Because its nature depends mainly on the use case, it is difficult to provide concrete tips.

Define (in advance) the categories and the relationships between them, regardless of which structured information representation format is chosen. No structured format of data representation escapes this upfront investment. The basic idea here is that semantics is already contained in the topic, so the information about the communication structures is available outside the payload. This is also necessary to enable filtering.

Various Variants

Of course, there is always the option to use existing specifications guides already in the industry, like ISA95.

The topic structure can be primarily oriented to the existing communication hierarchy of the participants in the use case. This is true in constant environments such as factory floors, production lines, and similar conditions. Here the Client Id is used in the Topic.

For static participants (IIoT), including the physical location(region, place, area…) is helpful if communication beyond the location is planned.

In the case of mobile subscribers (autonomous transport), in some instances, it is beneficial to map the position within a particular area using the topic structure. The granularity of the possible values (division into tiles) is decisive for the number of dynamically generated topics.

If possible, the grouping into types of clients should always be included, as this simplifies rights and role management. If several use cases are planned for the future, a topic level can also be available for this.

Another helpful feature is the introduction of message types at a given topic level. Sparkplug uses this by defining the different types of messages, such as commands to clients or values from clients.

Combining different variants is the most sensible and solid way to implement the optimal topic design. The application of rights management to the topic structure is always important.

Topic structures can only ever capture semantic properties from a particular point of view. Define segments to unify the different perspectives of all participants via the domain model in a meaningful and future-proof way.

Only the combination of Topics and Topic Filters enables resource-efficient data routing at the application level between publishers and consumers. Topic filters can be matched exactly to the concrete topic or access news subtrees using wildcards (+, #).

In addition to the above examples, there are some general aspects of topic design to consider. A good overview is given in the article: MQTT Topics, Wildcards, & Best Practices.

Summary

Even though each specification intends to address a particular industry with a particular set of use cases, some recurring aspects are worth considering as outlined in this paper. The definition of the topic space, the payload structure, the state management, security, and a delimitation of what the specification can deliver are essential. Mapping functionalities are easier to facilitate with MQTT 5 than with MQTT 3. The topic space’s definition is closely linked to the respective use case.

Important considerations for the topic structure include:

• Extensibility for future use cases

• A simple relationship between the access rights of the devices and the topic structure. General patterns can also be taken into account here, such as using the ClientID as a sub level in a topic to which the device must subscribe.

We have developed this decision-making helper that lists all MQTT features and how they connect to aid the implementation and definition process.