MQTT Packets: A Comprehensive Guide
In MQTT, control packets represent the basic unit of data transfer. These packets are exchanged between servers and MQTT clients to perform various functions, including subscribing to topics and publishing messages. Whether you are an IoT solution architect, an IoT developer, or anyone trying to understand the MQTT protocol in depth, comprehending MQTT control packets is crucial for effectively working with IoT and MQTT-based systems. By grasping the structure and purpose of each packet type, you can better design, implement, and troubleshoot MQTT-based applications.
In this blog post, we will explore the MQTT packet structure and its various control packets to help you gain a thorough understanding of this essential IoT communication protocol.
MQTT Packet Structure and Format Explained with Examples
Imagine MQTT packets as a futuristic postal service using smart envelopes. These envelopes are clever and adapt to the type of message they're delivering. Each smart envelope in this system is made up of three possible parts:
Fixed Header (Imagine a smart envelope): This part is always there, just like the physical envelope in a postal system. It includes crucial information for every delivery, such as the message type and size.
Variable Header (Imagine a dynamic shipping label): This smart label is unique because it can appear or disappear based on the message type. For some straightforward deliveries, it may not be needed. When it's present, it provides extra routing and handling details.
Payload (Imagine the contents of the envelope): This is the actual message or data being sent, similar to a letter or package in a traditional postal service.
MQTT Packet | Analogy Example of a Futuristic Postal Service Using a Smart Envelope |
---|---|
Fixed Header | A smart envelope, which is always present. Contains essential information for delivery (for example, message type, size). |
Variable Header | Dynamic shipping label, which is optional. Appears or disappears based on the message type. Provides additional routing and handling instructions when needed. |
Payload | Envelope contents, which is the actual message or data being transmitted. |
Here's how an MQTT packet looks like.
MQTT Packet Fixed Header
The Fixed Header is like the smart envelope — always present and essential for delivery. MQTT has different types of messages or control packets, like CONNECT, DISCONNECT, PUBLISH, SUBSCRIBE, etc. The Header needs to represent all these different types of messages. This is what the Fixed Header is responsible for.
The Fixed Header contains two crucial pieces of information:
1. Control Field: This is 1 Byte long and the first Byte. It is like the type of mail service (e.g., First Class, Priority, Express). The first 4 bits of the first byte indicate the message type. Because a 4-bit binary number produces 16 unique decimal numbers, it means that MQTT has a way of representing 16 different message types.
2. Remaining Length: This is the second byte of the Fixed Header and indicates the length of the remaining bytes within the current packet. Essentially, it indicates the size of the Variable Header and Payload combined.
MQTT Packet Variable Header
The Variable Header is like the dynamic shipping label on your envelope. While it isn't always required for simple tasks, such as internal memos or direct handovers, it becomes essential for more complex deliveries. It contains packet-specific information.
For example:
For a CONNECT MQTT Control packet: It's like including your return address and any special handling instructions.
For a PUBLISH MQTT Control packet: It's akin to writing the recipient's address and any tracking numbers.
MQTT Packet Payload:
The Payload is the actual content of your message — what's inside the smart envelope. Note that not all MQTT packets need a payload.
For example:
A PUBLISH MQTT Control packet's payload is like the letter or package contents.
A SUBSCRIBE MQTT Control packet's payload contains the topics to subscribe to, similar to filling out a form to sign up for different magazine subscriptions.
Summary of MQTT Packet Structure
While MQTT packet format refers to the structure of packets at the byte level, the MQTT control packets are specific packet types that are used for different operations in the MQTT protocol. Each type of control packet has a specific purpose and follows a specific structure.
Here's how different control packets utilize these components:
1. CONNECT: Uses all three components
Fixed Header: Identifies it as a CONNECT packet
Variable Header: Contains protocol name, version, and connection flags
Payload: Includes client ID, will topic, username, password
2. PUBLISH: Uses all three components
Fixed Header: Identifies it as a PUBLISH packet
Variable Header: Contains topic name and packet identifier
Payload: The actual message being published
For example, a packet publishing a temperature reading of 25.5 to the topic "a/b" with a Packet Identifier of 1 is as shown below:
Fixed Header: 0011 0000 | 00010100
Variable Header: 00000011 | 'a/b' | 00000001
Payload: "temperature:25.5"
Under Fixed Header, 0011
represents the PUBLISH message type, 0000
are the flags, and 00001010
(10 in decimal) is the Remaining Length.
3. SUBSCRIBE: Uses all three components
Fixed Header: Identifies it as a SUBSCRIBE packet
Variable Header: Contains packet identifier
Payload: List of topics to subscribe to and their QoS
4. PINGREQ: Uses only the Fixed Header
Like sending an empty envelope just to check if the postal service is working
5. DISCONNECT: Uses only the Fixed Header
Similar to sending a pre-stamped postcard to inform the post office you're moving
By structuring packets this way, MQTT ensures efficient communication, only including necessary information for each type of message, much like how different types of mail (letters, packages, postcards) are handled differently by the postal service. Now, let’s dive deep into each MQTT control packet.
MQTT Control Packets Explained With Examples
MQTT defines 15 types of control packets, which are exchanged between MQTT clients and brokers for various operations such as subscribing to topics and publishing messages. Here's a list of all MQTT control packets:
For Connecting/Disconnecting | For Publishing | For Subscribing/Unsubscribing |
---|---|---|
CONNECT | PUBLISH | SUBSCRIBE |
CONNACK | PUBACK | SUBACK |
DISCONNECT | PUBREC | UNSUBSCRIBE |
AUTH (MQTT 5.0 only) | PUBREL | UNSUBACK |
PINGREQ | PUBCOMP | |
PINGRESP |
In layman's terms, here’s what control packets generally behave like:
CONNECT: The MQTT client says "Hello, I want to connect!"
CONNACK: MQTT Broker replies, "Hello, you are connected!"
PUBLISH: The MQTT client or Broker sends a message.
PUBACK: Receiver says, "Got your message!"
PUBREC: Receiver says, "Received your message, will process."
PUBREL: Sender says, "Please release the message."
PUBCOMP: Receiver says, "Message processing complete."
SUBSCRIBE: The MQTT client says, "I want to receive updates about this topic."
SUBACK: The MQTT Broker replies, "You are subscribed."
UNSUBSCRIBE: The MQTT client says, "I don't want updates about this topic anymore."
UNSUBACK: The MQTT Broker replies, "You are unsubscribed."
PINGREQ: The MQTT client says, "Are you there?"
PINGRESP: The MQTT Broker replies, "Yes, I'm here."
DISCONNECT: The MQTT client says, "Goodbye, I'm disconnecting."
MQTT CONNECT and MQTT CONNACK Control Packet
The CONNECT packet is sent by an MQTT client, such as a sensor, a Raspberry Pi board, ESP8266, etc. to initiate a connection with the MQTT broker. It includes information such as the client ID, username, password, and various connection flags. For instance, a sensor device (client) sends a CONNECT packet to start communication with the IoT platform (server). Then, the server responds with a CONNACK packet to indicate the result of this connection attempt. Here’s a structured visualization of CONNECT control packet in tabular format:
Here's a simplified example of a CONNECT packet:
10 1F
00 04 4D 51 54 54 05 C2 00 3C
00 0B 63 6C 69 65 6E 74 2D 30 30 31
00 08 75 73 65 72 6E 61 6D 65
00 08 70 61 73 73 77 6F 72 64
Here is a breakdown of each part of the above packet:
Fixed Header: 10 1F
10: Type and flags for CONNECT
1F: Remaining length (31 bytes)
Variable Header: 00 04 4D 51 54 54 05 C2 00 3C
00 04: Protocol Name Length (4 bytes)
4D 51 54 54: Protocol Name ("MQTT")
05: Protocol Level (5)
C2: Connect Flags (Clean Session, Username, Password)
00 3C: Keep Alive (60 seconds)
Payload:
Client ID: 00 0B 63 6C 69 65 6E 74 2D 30 30 31
("client-001")
Username: 00 08 75 73 65 72 6E 61 6D 65
("username")
Password: 00 08 70 61 73 73 77 6F 72 64
("password")
Here’s a structured visualization of CONNACK packet in tabular format:
Fixed Header (2 Bytes) | Variable Header (2 Bytes) | Payload (0 Bytes) |
---|---|---|
Control Packet Type (2) | Connect Acknowledge Flags | NA |
Remaining Length (2) | Reason Code | NA |
MQTT PUBLISH Control Packet
The PUBLISH control packet is used to send messages from the client to the server. For instance, a fitness tracker (client) publishes the user's heart rate data to the health monitoring server. Additionally, four other packets are used to acknowledge QoS 1 and 2 messages, ensuring reliable delivery. Here’s a structured visualization of PUBLISH packet in tabular format:
MQTT Control Flags in PUBLISH and CONNECT Packets
Based on the MQTT specification, there are several important control flags that you can use under control field of the Fixed Header, as shown below:
DUP (Duplicate) flag: It is used in the PUBLISH packet. It indicates if the message is a duplicate or being republished. It’s set to 1 when the message is being re-delivered due to lack of acknowledgment.
QoS flag: It is used in the PUBLISH packet. It uses 2 bits and indicates the QoS level of the message. It can be 0 (At most once), 1 (At least once), or 2 (Exactly once).
RETAIN flag: It is used in PUBLISH packet. It indicates if the message should be retained by the broker. When set to 1, the broker will store the message and send it to future subscribers.
Clean Session flag: It is used in the CONNECT packet. It Indicates whether the client wants to establish a persistent session or not.
Will flag: It is used in the CONNECT packet. It indicates if a Will message should be stored on the server.
Username flag: It is used in the CONNECT packet and indicates if a username is present in the payload.
Password flag: It is used in the CONNECT packet and indicates if a password is present in the payload.
It's important to note that while the PUBLISH packet makes the most use of flags in its control field, other packet types like CONNECT also use flags in their Variable Headers to specify various connection options. The correct use and interpretation of these flags are crucial for proper MQTT communication between clients and brokers.
MQTT SUBSCRIBE, MQTT UNSUBSCRIBE, MQTT SUBACK, and MQTT UNSUBACK Control Packets
The SUBSCRIBE packet lets the client subscribe to specific topics. For example, a home automation controller (client) subscribes to the 'temperature/updates' topic. Conversely, the UNSUBSCRIBE packet is used to cancel these subscriptions. The server sends SUBACK and UNSUBACK packets to confirm the results of these subscription and unsubscription requests, respectively.
SUBACK is sent by the broker in response to a SUBSCRIBE packet, confirming the subscription and specifying the granted QoS levels for each topic.
UNSUBACK is sent by the broker in response to an UNSUBSCRIBE packet, confirming the unsubscription.
For example, after subscribing to temperature updates, the home automation controller receives a SUBACK packet to confirm the subscription. Later, if it unsubscribes, it receives an UNSUBACK packet.
The Fixed Header of a SUBSCRIBE packet is always 2 bytes. The Variable Header for a SUBSCRIBE packet consists of 3-4 bytes. The payload contains the list of topics and QoS levels to which the client wants to subscribe. Each topic filter is a UTF-8 encoded string, preceded by its length, followed by the QoS level. Here’s an example of how a SUBSCRIBE packet looks like at the byte level:
Byte 1: 1000 0010 (0x82) [Packet Type: SUBSCRIBE, Flags: 0010]
Byte 2: 0000 1100 (0x0C) [Remaining Length: 12]
Variable Header:
Byte 3: 0001 0010 (0x12) [Packet Identifier MSB]
Byte 4: 0011 0100 (0x34) [Packet Identifier LSB]
Payload:
Byte 5: 0000 0000 (0x00) [Topic Length MSB]
Byte 6: 0000 0110 (0x06) [Topic Length LSB]
Bytes 7-12: 0111 0100 0110 1111 0111 0000 0110 1001 0110 0011 (0x74 0x6F 0x70 0x69 0x63) [Topic: "topic"]
Byte 13: 0000 0001 (0x01) [QoS: 1]
Here’s a structured visualization of SUBACK packet in tabular format:
Byte Position | Hex Value | Binary Value | Description |
---|---|---|---|
0 | 0X90 | 1001 0000 | Fixed header: Packet Type (SUBACK)+ Flags |
1 | 0X03 | 0000 0011 | Remaining Length |
2 | 0X00 | 0000 0000 | Packet Identifier MSB |
3 | 0X01 | 0000 0001 | Packet Identifier LSB |
4 | 0X00 | 0000 0000 | Return Code |
Here’s a structured visualization of UNSUBACK packet in tabular format:
Byte Position | Hex Value | Binary Value | Description |
---|---|---|---|
0 | 0XB0 | 1011 0000 | Fixed Header: Packet Type (UNSUBACK) + Flags |
1 | 0X02 | 0000 0010 | Remaining Length |
2 | 0X00 | 0000 0000 | Packet Identifier MSB |
3 | 0X01 | 0000 0001 | Packet Identifier LSB |
Here’s a structured visualization of UNSUBSCRIBE MQTT packet in tabular format with an example where Topic 1 is sensor/temperature
and Topic 2 is sensor/humidity
.
Section | Description | Byte(s) | Value |
---|---|---|---|
Fixed header | Control Packet Type and Flags | 1 | A2 (UNSUBSCRIBE, QoS 1) |
Remaining Length | 1 | 22 (34 Bytes) | |
Variable Header | Packet Identifier | 2 | 00 10 (Packet Identifier 16) |
Payload | Topic 1 Length | 2 | 00 11 (17 Bytes) |
Topic 1 | 17 | 73 65 6E 73 6F 72 2F 74 65 6D 70 65 72 61 74 75 72 65 (sensor/temperature) | |
Topic 2 Length | 2 | 00 0E (14 Bytes) | |
Topic 2 | 14 | 73 65 6E 73 6F 72 2F 68 75 6D 69 64 69 74 79 (sensor/humidity) |
To learn how PUBLISH, SUBSCRIBE, and UNSUBSCRIBE work together, read our MQTT Essentials post here.
MQTT DISCONNECT Packets Explained
If communication needs to be terminated, either due to a routine shutdown or an error, the client and server can send a DISCONNECT packet to close the network connection. For example, a smart thermostat might send a DISCONNECT packet before shutting down for maintenance. The DISCONNECT packet is sent by either the client or the broker (in MQTT 5.0) to indicate that they are closing the connection. Here’s a structured visualization of DISCONNECT packet in tabular format:
Section | Field Name | Byte(s) | Description |
---|---|---|---|
Fixed Header | Control Packet Type | 1 | Contains the MQTT packet type (0XE0 for DISCONNECT) |
Remaining Length | 1 | Fixed value of 0 indicating no variable header or payload | |
Variable Header | NA | 0 | No variable header for DISCONNECT packet |
Payload | NA | 0 | No payload for DISCONNECT packet |
MQTT PUBACK, MQTT PUBREC, MQTT PUBREL, and MQTT PUBCOMP Control Packets
MQTT PUBACK, PUBREC, PUBREL, and PUBCOMP control packets are used for implementing Quality of Service (QoS) levels 1 and 2. Here’s a structured visualization of each packet in a tabular format:
Packet Type | Fixed Header | Variable Header | Payload |
---|---|---|---|
PUBACK | Control Packet Type (4 Bits): 0100 | Packet Identifier (2 Bytes) | None |
Flags (4 Bits): 0000 | |||
Remaining Length (1 Byte): 2 |
The purpose of PUBACK is to acknowledge the receipt of a PUBLISH packet (QoS 1).
Sent By: The subscriber (or the broker, depending on the implementation).
When: After receiving a PUBLISH packet with QoS 1.
Packet Structure:
Fixed Header
Variable Header (contains the Packet Identifier)
Packet Type | Fixed Header | Variable Header | Payload |
---|---|---|---|
PUBREC | Control Packet Type (4 Bits): 0101 | Packet Identifier (2 Bytes) | None |
Flags (4 Bits): 0000 | |||
Remaining Length (1 Byte): 2 |
The purpose of PUBREC is to acknowledge the receipt of a PUBLISH packet and inform the sender that the message has been received but not yet completely processed (QoS 2).
Sent By: The subscriber (or the broker, depending on the implementation).
When: After receiving a PUBLISH packet with QoS 2.
Packet Structure:
Fixed Header
Variable Header (contains the Packet Identifier)
Packet Type | Fixed Header | Variable Header | Payload |
---|---|---|---|
PUBREL | Control Packet Type (4 Bits): 0110 | Packet Identifier (2 Bytes) | None |
Flags (4 Bits): 0010 | |||
Remaining Length (1 Byte): 2 |
The purpose of PUBREL is to acknowledge the receipt of a PUBREC packet and to release the message to the subscriber (QoS 2).
Sent By: The publisher (or the broker, depending on the implementation).
When: After receiving a PUBREC packet in response to a PUBLISH packet with QoS 2.
Packet Structure:
Fixed Header
Variable Header (contains the Packet Identifier)
Packet Type | Fixed Header | Variable Header | Payload |
---|---|---|---|
PUBCOMP | Control Packet Type (4 Bits): 0111 | Packet Identifier (2 Bytes) | None |
Flags (4 Bits):0000 | |||
Remaining Length (1 Byte): 2 |
The purpose of PUBCOMP is to acknowledge the receipt of a PUBREL packet, indicating that the QoS 2 message flow is complete.
Sent By: The subscriber (or the broker, depending on the implementation).
When: After receiving a PUBREL packet.
Packet Structure:
Fixed Header
Variable Header (contains the Packet Identifier)
QoS 2 Message Flow Using PUBLISH, PUBREC, PUBREL, and PUBCOMP
The QoS 2 message flow, which guarantees exactly-once delivery, follows this sequence:
PUBLISH (QoS 2): The publisher sends a PUBLISH packet with QoS 2.
PUBREC: The subscriber (or broker) acknowledges receipt of the PUBLISH packet with a PUBREC packet.
PUBREL: The publisher sends a PUBREL packet to release the message for processing by the subscriber.
PUBCOMP: The subscriber (or broker) acknowledges receipt of the PUBREL packet with a PUBCOMP packet, completing the flow.
MQTT PINGREQ and PINGRESP Packets
PINGREQ is sent by the client to check if the connection is still alive. PINGRESP is the broker's response, confirming that the connection is active. In summary, to maintain the connection, the PINGREQ and PINGRESP packets are used for Keep Alive purposes. The client periodically sends a PINGREQ packet to the server to show it is still active. The server replies with a PINGRESP packet to confirm its status. For example, a remote weather station (client) sends a PINGREQ to the monitoring system (server) and awaits a PINGRESP in return.
Here’s visualization of PINGREQ packet in tabular format:
Component | Description | Size (Bytes) | Value |
---|---|---|---|
Fixed Header | |||
Byte 1 | Packet Type and Flags | 1 | 0XC0 (1100 0000) |
Byte 2 | Remaining Length | 1 | 0X00 |
Variable Header | None | ||
Payload | None |
Here’s visualization of PINGRESP MQTT packet in tabular format:
Component | Details | Size (Bytes) |
---|---|---|
Fixed Header | ||
Control Packet Type | 13 (PINGRESP) | 1 |
Remaining Length | 0 (since no variable header or payload) | 1 |
Variable Header | NA (not present in PINGRESP packets) | 0 |
Payload | NA (not present in PINGRESP packets) | 0 |
MQTT AUTH Packets Explained
In MQTT 5.0, the AUTH packet was introduced to enhance authentication. This packet ensures more secure verification between clients and servers. For instance, an industrial gateway might use an AUTH packet to authenticate itself securely to the central server. Here’s visualization of AUTH packets in tabular format:
Component | Field | Example Value | Description | Length |
---|---|---|---|---|
Fixed Header | Control Packet Type | 0XF0 | 15 (for AUTH) and Flags (0000) | 1 Byte |
Remaining Length | 0X05 | Length of Variable Header (5 Bytes) | 1 Byte | |
Variable Header | Reason Code | 0X00 | Success | 1 Byte |
Properties Length | 0X03 | Length of the Properties field (3 Bytes) | 1 Byte | |
Properties | 0X15 0X01 0X00 | Property Identifier, Length, Value | 3 Bytes | |
Payload | NA | NA | No payload for AUTH packets | NA |
Data Flow in MQTT Via Different Packets
Here's a quick overview of how information flow is transmitted and managed in the MQTT protocol using various types of packets:
Sending (Publishing): Here client publishes a message to a topic on the broker. Similar to how a sender drops a letter in a mailbox.
Processing (Broker): Here the broker receives the message and filters based on topic. Similar to how a post office sorts mail based on addresses.
Receiving (Subscribing): Here the subscribed clients receive messages from the broker. Similar to how recipients collect mail from their mailboxes.
Acknowledgment: Here depending on QoS, acknowledgments may be sent. Similar to how you get delivery confirmation for registered mail.
Security: This includes encryption and authentication mechanisms. Similar to how envelopes are secured with sealed envelopes, registered mail, signatures, etc.
Read our blog MQTT Publish, MQTT Subscribe & Unsubscribe to understand the data flow in detail.
Difference Between MQTT 3 and MQTT 5 Packets
MQTT 5.0 introduced several new features and improvements over MQTT 3.1.1:
1. Enhanced error reporting
4. Topic Alias
6. Subscription identifiers
7. Request/response pattern support
8. Server-side disconnect
These enhancements make MQTT 5.0 more flexible and powerful for IoT applications.
Here’s an overview:
Feature/ MQTT Control Packet Type | MQTT 3.1.1 | MQTT 5.0 |
---|---|---|
CONNECT Packet – Protocol Name | MQTT | MQTT |
CONNECT Packet – Protocol Level | 4 | 5 |
CONNECT Packet – Client Identifier | Yes | Yes |
CONNECT Packet – Clean Session | Yes | Clean Start |
CONNECT Packet – Will Message | Yes | Yes |
CONNECT Packet – Username/Password | Yes | Yes |
CONNECT Packet – Keep Alive | Yes | Yes |
CONNECT Packet – Properties | No | Yes (Session Expiry Interval, Max Packet Size, etc.) |
PUBLISH Packet – Topic Name | Yes | Yes |
PUBLISH Packet – Packet Identifier | Yes | Yes |
PUBLISH Packet – Payload | Yes | Yes |
PUBLISH Packet – QoS | 0,1,2 | 0,1,2 |
PUBLISH Packet – Retain Flag | Yes | Yes |
PUBLISH Packet – Properties | No | Yes (Message Expiry Interval, Content Type, etc.) |
SUBSCRIBE Packet – Packet Identifier | Yes | Yes |
SUBSCRIBE Packet – Topic Filters | Yes | Yes |
SUBSCRIBE Packet – QoS | 0,1,2 | 0,1,2 |
SUBSCRIBE Packet – Properties | No | Yes (Subscription identifier, User Property, etc.) |
Features – Error Reporting | Limited (CONNACK, PUBACK) | Detailed Reason Codes |
Features – Property Mechanism | No | Yes |
Features – Enhanced Authentication | No | Yes (Authentication exchange) |
Additional Properties – Reason Codes | No | Yes (Reason code in ACK packets) |
Additional Properties – User Properties | No | Yes (User-defined properties) |
What’s the Maximum Size of an MQTT Packet?
The maximum size of an MQTT packet is 256 MB (268,435,455 bytes). This limit is due to the 4-byte length field in the packet header, which can represent a maximum value of 2^28 - 1.
What’s the Maximum Length of MQTT Topic Name and Message ID?
The maximum length of an MQTT topic name is 65,535 bytes. This limit is because the topic name length is represented by a 2-byte field in the SUBSCRIBE and PUBLISH packets.
Common Errors Encountered in MQTT Message Formats
1. Incorrect packet type or flags
2. Invalid remaining length encoding
3. Malformed Variable Header or payload
4. Exceeding maximum packet size
5. Using reserved packet types or flags
Conclusion
The beauty of MQTT's packet structure lies in its flexibility and efficiency. By only including what's necessary for each packet type, MQTT keeps its messages lean and fast, making it ideal for IoT and other low-bandwidth scenarios. As the IoT landscape continues to evolve, MQTT remains a key protocol, and mastering its intricacies will undoubtedly be a valuable skill in your IoT toolkit.
Whether you're just starting out or looking to deepen your understanding of MQTT, read our MQTT Essentials eBook, a comprehensive guide that delves into the core of MQTT. Designed for developers, engineers, and IoT enthusiasts, this eBook not only helps you grasp the fundamental principles of MQTT but also guides you through real-world applications, enhancing your ability to implement efficient communication strategies in your projects.
HiveMQ Team
The HiveMQ team loves writing about MQTT, Sparkplug, Industrial IoT, protocols, how to deploy our platform, and more. We focus on industries ranging from energy, to transportation and logistics, to automotive manufacturing. Our experts are here to help, contact us with any questions.