Load Balancing With Shared Subscriptions - MQTT Client
MQTT is based on the publish/subscribe principle, which means messages can be distributed to any number of subscribers. (If this is news to you, check out the MQTT Essentials). Sometimes plain PubSub is not enough, though, and some kind of client load balancing is desired to cover some of the more advanced use cases that involve multiple MQTT subscribers. This blog post introduces the concept of Shared Subscriptions, a mechanism that allows to distribute the messages of a subscription group to the members of the subscription group.
Introducing MQTT Shared Subscriptions
Shared Subscriptions are a MQTT feature of HiveMQ that allows MQTT clients to share the same subscription on the broker. When using standard MQTT subscriptions, each subscribing client receives a copy of the message. If Shared Subscriptions are used, all clients that share the same subscription with a subscription group will receive messages in an alternating fashion. This mechanism is sometimes referred to as client load balancing, since the message load of a single topic is distributed amongst all subscribers.
MQTT clients can subscribe to a shared subscription with standard MQTT mechanisms, which means all common MQTT clients like Eclipse Paho can be used without modifying anything on the client side. Shared Subscriptions use a special topic syntax for subscribing, though.
The topic structure for shared subscriptions is the following:
$share/GROUPID/TOPIC
The shared subscription consists of 3 parts:
A static shared subscription identifier ($share)
A group identifier
The concrete topic subscriptions (may include wildcards)
A concrete example for such an subscriber would be $share/my-shared-subscriber-group/myhome/groundfloor/+/temperature.
Shared Subscriptions In-Depth
When using Shared Subscriptions, each subscription group can be conceptually imagined as a virtual client that acts as proxy for multiple real subscribers at once. HiveMQ selects one subscriber of the group and delivers the message to this client. By default a round-robin approach is used. The following picture demonstrates the principle:
There is no limitation on the number of Shared Subscription Groups in a HiveMQ deployment. So for example the following scenario would be possible:
In this example there are two different groups with 2 subscribing clients in each shared subscription group. Both groups have the same subscription but have different group identifiers. When a publisher sends a message with a matching topic, one (and only one) client of each group receives the message.
Shared Subscriptions Use Cases
There are many use cases for shared subscriptions, especially in high-scalability scenarios. Among the most popular use cases are:
Client Load Balancing for MQTT clients which can’t handle the load on subscribed topics on their own.
Worker (backend) applications which ingest MQTT streams and need to be scaled out horizontally.
HiveMQ intra-cluster node traffic should be relieved by optimizing subscriber node-locality for incoming publishes.
QoS 1 and 2 are used for their delivery semantics but Ordered Topic guarantees are not needed for the use cases.
There are hot topics with higher message rate than other topics in the system and these topics are the scalability bottleneck.
How to Subscribe With Shared Subscriptions?
Subscribing clients with Shared Subscriptions is straight forward. The following Java code (which uses the Java Eclipse Paho code shows two MQTT clients that subscribe to the same subscription group:
MqttClient client = new MqttClient(
"tcp://broker.mqttdashboard.com:1883", //URI
MqttClient.generateClientId(), //ClientId
new MemoryPersistence()); //Persistence
client.connect();
client.subscribe(“$share:my-group:#”, 1);
MqttClient client2 = new MqttClient(
"tcp://broker.mqttdashboard.com:1883", //URI
MqttClient.generateClientId(), //ClientId
new MemoryPersistence()); //Persistence
client2.connect();
client2.subscribe(“$share:my-group:#”, 1);Now both MQTT clients share a root wildcard subscription (they have the virtual group my-group) and would receive half of all MQTT messages that are sent over the MQTT broker due to the root wildcard subscription.
MQTT clients can join and leave the subscription group any time. If a third client would join the group, each client would receive 1/3 of all MQTT messages.
You can read about all semantics of shared subscriptions in the official HiveMQ Documentation.
Scaling MQTT Subscribers with Shared Subscriptions
The Shared Subscription mechanism is a fantastic approach to integrate backend systems via plain MQTT (for example if it’s not feasible to use HiveMQs plugin system and dynamic scaling is needed). Shared Subscribers can be added anytime as soon as they’re needed. An arbitrary number of “worker” applications can be used to consume the messages of the Shared Subscription Group.
So you get essentially work-distribution in a pushing fashion, since the messages are pushed by the broker to the Shared Subscription clients.
Shared Subscriptions are perfect for scaling and load balancing MQTT clients. When using a HiveMQ cluster, there are additional advantages in terms of latency and scalability since message routing is optimized internally. Learn more about Shared Subscriptions and HiveMQ clusters in the official documentation.
Conclusion
We have seen that Shared Subscriptions are a great way to distribute messages across different MQTT subscribers with standard MQTT mechanisms, which means you can add MQTT client load balancing without any proprietary additions to your MQTT clients. This is especially useful for backend systems or “hot-topics” that can quickly overwhelm a single MQTT client.
Do you already use or plan to use Shared Subscriptions? Do you need help? Contact us.
Oh and by the way: Shared Subscriptions are now part of the official MQTT version 5 specification , so Shared Subscriptions are definitely one of the more interesting concepts you should know about when deploying MQTT.
HiveMQ Team
The HiveMQ team loves writing about MQTT, Sparkplug, Industrial IoT, protocols, how to deploy our platform, and more. We focus on industries ranging from energy, to transportation and logistics, to automotive manufacturing. Our experts are here to help, contact us with any questions.
