SYS-Topics: Why you shouldn’t use SYS-Topics for Monitoring

Monitoring your systems is extremely important for production environments. When deploying an MQTT broker or a cluster of MQTT brokers, the operation team needs to know the health of the MQTT servers and uses monitoring data to circumvent foreseeable future problems. During the past few years, a feature of many MQTT brokers, the so called SYS-Topics, has gained popularity not only for debugging and developing MQTT systems but also for monitoring. This blog post discusses the use of SYS-Topics for monitoring and explains when to use - and more importantly - when NOT to use SYS-Topics.

SYS-Topics

Many MQTT brokers implement SYS-Topics. These topics are special meta topics that the broker can use to publish information about the broker itself and its MQTT client sessions. All SYS-Topics start with $SYS and are read-only for MQTT clients. The MQTT broker should prevent clients from using such topic names to publish messages. The use of SYS-Topics is mentioned in the MQTT 3.1.1 specification (4.7.2) as a non-standard feature that is available in some MQTT brokers.

An unofficial SYS-Topic specification is available on the MQTT Wiki, it is however subject to change and applications should not rely on this information.

Brokers publish SYS-Topics periodically, the default period on most brokers is 60 seconds. All MQTT clients that subscribe to one or more SYS-Topics receive the current value on the SYS topics as soon as they subscribe. After the subscription was successful, the client will receive metrics on periodical basis. Some static SYS-Topics (e.g. broker version) are only published upon subscription.

Debugging and Development with SYS-Topics

SYS-Topics can be used for developing and debugging MQTT applications. Developers can quickly monitor the current state of the broker and calculate and verify metrics such as message amplification rate or network metrics. Applications such as MQTT.fx even provide native support for SYS-Topics.

SYS-Topics for Production Monitoring

Although SYS-Topics have use-cases for development and debugging, SYS-Topics are not suitable for monitoring broker instances in production. If you need to rely on the availability guarantees of your MQTT broker, use an actual monitoring system, SYS-Topics do not replace monitoring applications.

We highly recommend that you disable SYS-Topics completely for production systems and use the available monitoring interfaces, e.g. Graphite or JMX.

5 reasons SYS-Topics are not suitable for production monitoring:

1. Metric resolution is not good enough

The metric publishing interval for most MQTT brokers is 60 seconds by default. Monitoring interfaces like JMX allow to poll the most current data whenever needed. So the broker does not dictate the monitoring system how the resolution should be. The monitoring system can decide how often new data should be collected, without modifying any configuration on the MQTT broker.

2. SYS-Topics provide only a subset of the available metrics

While the metrics provided by SYS-Topics can be useful, they are only a small subset of all available metrics. For example, HiveMQ exposes more than 100 different metrics (the number of exposed metrics also increase with every release). The SYS-Topic Metrics usually are more focused on MQTT session monitoring and thus don’t provide metrics that can give you an overview of the brokers health at a quick glance. The most interesting metrics for operation teams are usually product dependant, so it’s unlikely that a common SYS-Topic standard will cover these in the future.

3. SYS-Topics expose internal information to potential attackers

To make it more difficult for an attacker to use exploits that target a specific software version, it is a common best practice to conceal the actual software version that is used in a deployment. Frequently, you also want to hide the actual software that is used from attackers. While you should never rely on ‘Security by Obscurity’ as your primary security measure, it’s still important to hide this deployment information. SYS-Topics expose key information such as the Broker Software Used and the Version Number to every subscriber. This information can be valuable for attackers but is seldom useful to legitimate subscribers.

4. It’s hard to monitor broker clusters with SYS-Topics

If someone wants to build a custom monitoring solution based on SYS-Topics, MQTT connections to all cluster nodes need to be established. Typically MQTT clusters are behind a load balancer, so often it’s not even possible to connect to a specific cluster node. Monitoring integrations such as Graphite are built with clustering in mind and so it’s easy to get a quick health overview of the whole cluster. That’s much harder to achieve with SYS-Topics.

5. The monitored channel should be separated from the monitoring channel

A good monitoring practice is to use a dedicated monitoring channel instead of using the same channel you are monitoring. With MQTT brokers, this means that if you are monitoring the MQTT communication, you shouldn’t use MQTT (SYS-Topics) to monitor the MQTT communication. It’s the same as: Don’t use e-mail alerts for monitoring e-mail servers. If the MQTT communication is not available for any reason, you won’t get any monitoring data. The monitoring data may be most valuable especially when unexpected occurs. If you rely on this data for the alert, you may have a problem. If you’re using a dedicated monitoring channel such as JMX, Graphite, or Nagios, a problem with the MQTT communication should not affect your monitoring.

Conclusion

MQTT SYS-Topics are useful for debugging and developing MQTT applications but are not appropriate for monitoring MQTT deployments or alerting operation teams to possible problems. Dedicated monitoring interfaces like JMX do a better job at monitoring and provide greater flexibility.

What are your experiences with SYS-Topics? Do you use them regularly or do you even use them for production monitoring?