MQTT Packets: A Comprehensive Guide

In MQTT, control packets represent the basic unit of data transfer. These packets are exchanged between servers and MQTT clients to perform various functions, including subscribing to topics and publishing messages. Whether you are an IoT solution architect, an IoT developer, or anyone trying to understand the MQTT protocol in depth, comprehending MQTT control packets is crucial for effectively working with IoT and MQTT-based systems. By grasping the structure and purpose of each packet type, you can better design, implement, and troubleshoot MQTT-based applications. 

In this blog post, we will explore the MQTT packet structure and its various control packets to help you gain a thorough understanding of this essential IoT communication protocol.

MQTT Packet Structure and Format Explained with Examples

Imagine MQTT packets as a futuristic postal service using smart envelopes. These envelopes are clever and adapt to the type of message they're delivering. Each smart envelope in this system is made up of three possible parts:

• Fixed Header (Imagine a smart envelope): This part is always there, just like the physical envelope in a postal system. It includes crucial information for every delivery, such as the message type and size.

• Variable Header (Imagine a dynamic shipping label): This smart label is unique because it can appear or disappear based on the message type. For some straightforward deliveries, it may not be needed. When it's present, it provides extra routing and handling details.

• Payload (Imagine the contents of the envelope): This is the actual message or data being sent, similar to a letter or package in a traditional postal service.

Here's how an MQTT packet looks like. 

MQTT Packet Fixed Header

The Fixed Header is like the smart envelope — always present and essential for delivery. MQTT has different types of messages or control packets, like CONNECT, DISCONNECT, PUBLISH, SUBSCRIBE, etc. The Header needs to represent all these different types of messages. This is what the Fixed Header is responsible for. 

The Fixed Header contains two crucial pieces of information:

1. Control Field: This is 1 Byte long and the first Byte. It is like the type of mail service (e.g., First Class, Priority, Express). The first 4 bits of the first byte indicate the message type. Because a 4-bit binary number produces 16 unique decimal numbers, it means that MQTT has a way of representing 16 different message types.

2. Remaining Length: This is the second byte of the Fixed Header and indicates the length of the remaining bytes within the current packet. Essentially, it indicates the size of the Variable Header and Payload combined. 

MQTT Packet Variable Header

The Variable Header is like the dynamic shipping label on your envelope. While it isn't always required for simple tasks, such as internal memos or direct handovers, it becomes essential for more complex deliveries. It contains packet-specific information.

For example: 

• For a CONNECT MQTT Control packet: It's like including your return address and any special handling instructions.

• For a PUBLISH MQTT Control packet: It's akin to writing the recipient's address and any tracking numbers.

MQTT Packet Payload: 

The Payload is the actual content of your message — what's inside the smart envelope. Note that not all MQTT packets need a payload.

For example: 

• A PUBLISH MQTT Control packet's payload is like the letter or package contents.

• A SUBSCRIBE MQTT Control packet's payload contains the topics to subscribe to, similar to filling out a form to sign up for different magazine subscriptions.

Summary of MQTT Packet Structure

While MQTT packet format refers to the structure of packets at the byte level, the MQTT control packets are specific packet types that are used for different operations in the MQTT protocol. Each type of control packet has a specific purpose and follows a specific structure.

Here's how different control packets utilize these components:

1. CONNECT: Uses all three components

• Fixed Header: Identifies it as a CONNECT packet

• Variable Header: Contains protocol name, version, and connection flags

• Payload: Includes client ID, will topic, username, password

2. PUBLISH: Uses all three components

• Fixed Header: Identifies it as a PUBLISH packet

• Variable Header: Contains topic name and packet identifier

• Payload: The actual message being published

For example, a packet publishing a temperature reading of 25.5 to the topic "a/b" with a Packet Identifier of 1 is as shown below: 

Under Fixed Header, 0011 represents the PUBLISH message type, 0000 are the flags, and 00001010 (10 in decimal) is the Remaining Length. 

3. SUBSCRIBE: Uses all three components

• Fixed Header: Identifies it as a SUBSCRIBE packet

• Variable Header: Contains packet identifier

• Payload: List of topics to subscribe to and their QoS

4. PINGREQ: Uses only the Fixed Header

• Like sending an empty envelope just to check if the postal service is working

5. DISCONNECT: Uses only the Fixed Header

•  Similar to sending a pre-stamped postcard to inform the post office you're moving

By structuring packets this way, MQTT ensures efficient communication, only including necessary information for each type of message, much like how different types of mail (letters, packages, postcards) are handled differently by the postal service. Now, let’s dive deep into each MQTT control packet.

MQTT Control Packets Explained With Examples

MQTT defines 15 types of control packets, which are exchanged between MQTT clients and brokers for various operations such as subscribing to topics and publishing messages. Here's a list of all MQTT control packets:

MQTT CONNECT and MQTT CONNACK Control Packet 

The CONNECT packet is sent by an MQTT client, such as a sensor, a Raspberry Pi board, ESP8266, etc. to initiate a connection with the MQTT broker. It includes information such as the client ID, username, password, and various connection flags. For instance, a sensor device (client) sends a CONNECT packet to start communication with the IoT platform (server). Then, the server responds with a CONNACK packet to indicate the result of this connection attempt. Here’s a structured visualization of CONNECT control packet in tabular format:

Here's a simplified example of a CONNECT packet:

Here is a breakdown of each part of the above packet:

Fixed Header: 10 1F

10: Type and flags for CONNECT

1F: Remaining length (31 bytes)

Variable Header: 00 04 4D 51 54 54 05 C2 00 3C

00 04: Protocol Name Length (4 bytes)

4D 51 54 54: Protocol Name ("MQTT")

05: Protocol Level (5)

C2: Connect Flags (Clean Session, Username, Password)

00 3C: Keep Alive (60 seconds)

Payload:

Client ID: 00 0B 63 6C 69 65 6E 74 2D 30 30 31 ("client-001")

Username: 00 08 75 73 65 72 6E 61 6D 65 ("username")

Password: 00 08 70 61 73 73 77 6F 72 64 ("password")

Here’s a structured visualization of CONNACK packet in tabular format:

MQTT PUBLISH Control Packet

The PUBLISH control packet is used to send messages from the client to the server. For instance, a fitness tracker (client) publishes the user's heart rate data to the health monitoring server. Additionally, four other packets are used to acknowledge QoS 1 and 2 messages, ensuring reliable delivery. Here’s a structured visualization of PUBLISH packet in tabular format: 

MQTT Control Flags in PUBLISH and CONNECT Packets

Based on the MQTT specification, there are several important control flags that you can use under control field of the Fixed Header, as shown below:

1. DUP (Duplicate) flag: It is used in the PUBLISH packet. It indicates if the message is a duplicate or being republished. It’s set to 1 when the message is being re-delivered due to lack of acknowledgment.

2. QoS flag: It is used in the PUBLISH packet. It uses 2 bits and indicates the QoS level of the message. It can be 0 (At most once), 1 (At least once), or 2 (Exactly once).

3. RETAIN flag: It is used in PUBLISH packet. It indicates if the message should be retained by the broker. When set to 1, the broker will store the message and send it to future subscribers.

4. Clean Session flag: It is used in the CONNECT packet. It Indicates whether the client wants to establish a persistent session or not.

5. Will flag: It is used in the CONNECT packet. It indicates if a Will message should be stored on the server.

6. Username flag: It is used in the CONNECT packet and indicates if a username is present in the payload.

7. Password flag: It is used in the CONNECT packet and indicates if a password is present in the payload. 

It's important to note that while the PUBLISH packet makes the most use of flags in its control field, other packet types like CONNECT also use flags in their Variable Headers to specify various connection options. The correct use and interpretation of these flags are crucial for proper MQTT communication between clients and brokers.

MQTT SUBSCRIBE, MQTT UNSUBSCRIBE,  MQTT SUBACK, and MQTT UNSUBACK Control Packets

The SUBSCRIBE packet lets the client subscribe to specific topics. For example, a home automation controller (client) subscribes to the 'temperature/updates' topic. Conversely, the UNSUBSCRIBE packet is used to cancel these subscriptions. The server sends SUBACK and UNSUBACK packets to confirm the results of these subscription and unsubscription requests, respectively. 

• SUBACK is sent by the broker in response to a SUBSCRIBE packet, confirming the subscription and specifying the granted QoS levels for each topic.

• UNSUBACK is sent by the broker in response to an UNSUBSCRIBE packet, confirming the unsubscription.

For example, after subscribing to temperature updates, the home automation controller receives a SUBACK packet to confirm the subscription. Later, if it unsubscribes, it receives an UNSUBACK packet.

The Fixed Header of a SUBSCRIBE packet is always 2 bytes. The Variable Header for a SUBSCRIBE packet consists of 3-4 bytes. The payload contains the list of topics and QoS levels to which the client wants to subscribe. Each topic filter is a UTF-8 encoded string, preceded by its length, followed by the QoS level. Here’s an example of how a SUBSCRIBE packet looks like at the byte level:

Here’s a structured visualization of SUBACK packet in tabular format:

Here’s a structured visualization of UNSUBACK packet in tabular format: 

Here’s a structured visualization of UNSUBSCRIBE MQTT packet in tabular format with an example where Topic 1 is sensor/temperature and Topic 2 is sensor/humidity.  

MQTT DISCONNECT Packets Explained

If communication needs to be terminated, either due to a routine shutdown or an error, the client and server can send a DISCONNECT packet to close the network connection. For example, a smart thermostat might send a DISCONNECT packet before shutting down for maintenance. The DISCONNECT packet is sent by either the client or the broker (in MQTT 5.0) to indicate that they are closing the connection. Here’s a structured visualization of DISCONNECT packet in tabular format:

MQTT PUBACK, MQTT PUBREC, MQTT PUBREL, and MQTT PUBCOMP Control Packets 

MQTT PUBACK, PUBREC, PUBREL, and PUBCOMP control packets are used for implementing Quality of Service (QoS) levels 1 and 2. Here’s a structured visualization of each packet in a tabular format:

The purpose of PUBACK is to acknowledge the receipt of a PUBLISH packet (QoS 1).

• Sent By: The subscriber (or the broker, depending on the implementation).

• When: After receiving a PUBLISH packet with QoS 1.

• Packet Structure:

  • Fixed Header

  • Variable Header (contains the Packet Identifier)

The purpose of PUBREC is to acknowledge the receipt of a PUBLISH packet and inform the sender that the message has been received but not yet completely processed (QoS 2).

• Sent By: The subscriber (or the broker, depending on the implementation).

• When: After receiving a PUBLISH packet with QoS 2.

• Packet Structure:

  • Fixed Header

  • Variable Header (contains the Packet Identifier)

The purpose of PUBREL is to acknowledge the receipt of a PUBREC packet and to release the message to the subscriber (QoS 2).

• Sent By: The publisher (or the broker, depending on the implementation).

• When: After receiving a PUBREC packet in response to a PUBLISH packet with QoS 2.

• Packet Structure:

  • Fixed Header

  • Variable Header (contains the Packet Identifier)

The purpose of PUBCOMP is to acknowledge the receipt of a PUBREL packet, indicating that the QoS 2 message flow is complete.

• Sent By: The subscriber (or the broker, depending on the implementation).

• When: After receiving a PUBREL packet.

• Packet Structure:

  • Fixed Header

  • Variable Header (contains the Packet Identifier)

QoS 2 Message Flow Using PUBLISH, PUBREC, PUBREL, and PUBCOMP   

The QoS 2 message flow, which guarantees exactly-once delivery, follows this sequence:

1. PUBLISH (QoS 2): The publisher sends a PUBLISH packet with QoS 2.

2. PUBREC: The subscriber (or broker) acknowledges receipt of the PUBLISH packet with a PUBREC packet.

3. PUBREL: The publisher sends a PUBREL packet to release the message for processing by the subscriber.

4. PUBCOMP: The subscriber (or broker) acknowledges receipt of the PUBREL packet with a PUBCOMP packet, completing the flow.

MQTT PINGREQ and PINGRESP Packets 

PINGREQ is sent by the client to check if the connection is still alive. PINGRESP is the broker's response, confirming that the connection is active. In summary, to maintain the connection, the PINGREQ and PINGRESP packets are used for Keep Alive purposes. The client periodically sends a PINGREQ packet to the server to show it is still active. The server replies with a PINGRESP packet to confirm its status. For example, a remote weather station (client) sends a PINGREQ to the monitoring system (server) and awaits a PINGRESP in return. 

Here’s visualization of PINGREQ packet in tabular format: 

Here’s visualization of PINGRESP MQTT packet in tabular format:

MQTT AUTH Packets Explained

In MQTT 5.0, the AUTH packet was introduced to enhance authentication. This packet ensures more secure verification between clients and servers. For instance, an industrial gateway might use an AUTH packet to authenticate itself securely to the central server. Here’s visualization of AUTH packets in tabular format:

Data Flow in MQTT Via Different Packets

Here's a quick overview of how information flow is transmitted and managed in the MQTT protocol using various types of packets:

1. Sending (Publishing): Here client publishes a message to a topic on the broker. Similar to how a sender drops a letter in a mailbox.

2. Processing (Broker): Here the broker receives the message and filters based on topic. Similar to how a post office sorts mail based on addresses.

3. Receiving (Subscribing): Here the subscribed clients receive messages from the broker. Similar to how recipients collect mail from their mailboxes.

4. Acknowledgment: Here depending on QoS, acknowledgments may be sent. Similar to how you get delivery confirmation for registered mail.

5. Security: This includes encryption and authentication mechanisms. Similar to how envelopes are secured with sealed envelopes, registered mail, signatures, etc.

Difference Between MQTT 3 and MQTT 5 Packets

MQTT 5.0 introduced several new features and improvements over MQTT 3.1.1:

1. Enhanced error reporting

2. Shared subscriptions

3. Message expiry

4. Topic Alias

5. User properties

6. Subscription identifiers

7. Request/response pattern support

8. Server-side disconnect

9. Authentication exchange

These enhancements make MQTT 5.0 more flexible and powerful for IoT applications. 

Here’s an overview:

What’s the Maximum Size of an MQTT Packet?

The maximum size of an MQTT packet is 256 MB (268,435,455 bytes). This limit is due to the 4-byte length field in the packet header, which can represent a maximum value of 2^28 - 1.

What’s the Maximum Length of MQTT Topic Name and Message ID?

The maximum length of an MQTT topic name is 65,535 bytes. This limit is because the topic name length is represented by a 2-byte field in the SUBSCRIBE and PUBLISH packets.

Common Errors Encountered in MQTT Message Formats

1. Incorrect packet type or flags

2. Invalid remaining length encoding

3. Malformed Variable Header or payload

4. Exceeding maximum packet size

5. Using reserved packet types or flags

Conclusion

The beauty of MQTT's packet structure lies in its flexibility and efficiency. By only including what's necessary for each packet type, MQTT keeps its messages lean and fast, making it ideal for IoT and other low-bandwidth scenarios. As the IoT landscape continues to evolve, MQTT remains a key protocol, and mastering its intricacies will undoubtedly be a valuable skill in your IoT toolkit.