Securing MQTT Systems - MQTT Security Fundamentals

Welcome to the MQTT Security Fundamentals series. In our previous blogs, we focused on how to secure MQTT on a protocol level and shared best practices about how to implement security on the application level. In this blog, we’re focusing on the secure deployment of an MQTT system. We’ll examine different layers of security and how to strengthen your deployment to prevent and mitigate attacks.

There are different layers of security that we need to discuss, in particular, we are going to look at these layers:

The infrastructure
The operating system
The MQTT broker

Infrastructure

MQTT brokers are typically deployed on some kind of network infrastructure. It’s important to understand the network topology of the target system. It’s also important to lock out attackers as soon as possible to prevent damage and downtimes of downstream systems.

Firewall

Every connection to an MQTT broker should pass at least one firewall that implements sophisticated rules for accessing downstream components. If your are able to block attackers at the firewall level, they won’t be able to access other systems in your infrastructure. Unfortunately, there is no universally applicable solution. Firewall rules need to be configured individually for each use case. Many commercial and open source firewall solutions are currently available.

A solid rule of thumb is: only forward expected traffic to downstream systems. Block any traffic that you don’t expect in your downstream applications. For an MQTT broker the following traffic is probably worth blocking:

UDP: MQTT uses TCP, you can block all UDP datagram packets.
ICMP: While it may not be the smartest idea to block all ICMP traffic, ICMP traffic, ping and traceroute ICMP packets could be worth investigating as candidates to block.

It’s also a good idea to block traffic to any ports that are not needed for your MQTT system. Don’t block the following standard MQTT ports with your firewall:

1883: This is the default MQTT port. 1883 is defined at IANA as MQTT over TCP.
8883: This is the default MQTT port for MQTT over TLS. It’s registered at IANA for Secure MQTT.

If you are in full control of your MQTT system and know the IPs of your MQTT clients, only allow traffic for the needed IP ranges. This precaution locks out any clients that are not in the defined IP ranges.

Load Balancer

Load balancers are often used to distribute MQTT traffic to different MQTT brokers. For example, brokers that are operating in cluster mode. It’s important to deploy a load balancer that can handle the traffic and connection volume that you anticipate for your production system. Most commercial, hardware-based load balancers can handle huge amounts of connections and are designed for high-traffic scenarios. Most of these load balancers work straight out of the box.

Load balancers usually don’t add much additional security but are very useful to prevent overloading downstream systems by distributing traffic to multiple MQTT brokers. Typically, load balancers are also able to throttle traffic if the traffic is unusually high and you need to slow down MQTT clients to prevent overloading MQTT brokers.

DMZ

A DMZ is a demilitarized zone. Internet-facing services such as an MQTT broker often reside in this zone. A DMZ is a subnetwork. All access to downstream services such as databases and internal application services are protected by an additional firewall. If an attacker gets access to one of the services in the DMZ, he does not automatically gain access to other systems behind the additional firewall. In a dual-firewall scenario, firewalls from different vendors are frequently used. If an attacker is able to exploit a security hole of the first firewall, the second firewall cannot be breached by reusing of the same compromising technique. If you’re not familiar with the concept of a DMZ, this Wikipedia article is worth reading.

We strongly recommend deploying a DMZ, if you use HiveMQ plugins to access any internal business services (for example, web services or authentication services).

Operating System

MQTT Brokers are installed on servers (virtual and physical) that run various operating systems. Before we talk about MQTT broker application security, it’s important to understand that many attacks focus on security holes in the operating system or software and services typically installed with the operating system. While it’s out of scope to give a complete overview of how to harden servers in this blog post, we want to share some useful tips for making servers more secure.

Please note that the following tips are only valid for Linux-based servers.

Keep Libraries and Software Updated

No software is bug-free, and unfortunately, some bugs result in security holes. Keep your system and your software up-to-date. Recent famous security-related bugs such as the GHOST vulnerability were very easy to fix by updating specific libraries and software. In the case of the GHOST vulnerability, the version glibc had to be updated. If you rely on libraries that implement cryptographic algorithms such as openSSH or openSSL, it’s critical to update these immediately whenever new security holes are revealed. Insecure libraries can affect other software as well. Fortunately, it’s very easy to stay up-to-date since most Linux distributions have a dedicated package manager that can update all outdated software at once.

Disallow Root Access and Use SSH Keys for SSH

If you use SSH to connect to your server, always disallow root access via SSH, especially if your SSH port is open to the world. Another best practice is to disable password authentication via SSH and use SSH keys for authentication. This practice adds another layer of security because brute force attempts won’t succeed (in contrast to weak passwords if using password authentication).

Installing Fail2Ban

Fail2Ban is a neat software that scans different log files on your Linux box (such as the SSH log) and detects brute force attacks. It automatically updates your firewall (for example, iptables) to lock out attackers. If access to SSH ports is possible via the internet, this tool is invaluable.

Set Up iptables

If you don’t have any external firewall deployed, setting up iptables should be mandatory. Iptables is a software firewall, which is pre-installed on most Linux distributions. Even if you have external firewalls, iptables is highly recommended. Some people may find configuring iptables challenging. Fortunately, there are tools available which ease the burden of maintaining complex iptables rules. For example, ShoreWall or UFW. Other tools such as Fail2Ban rely on iptables and automatically update the rules if malicious clients are detected.

SELinux

Security Enhanced Linux s a kernel module which provides a mechanism to enforce security policies for access control. Having SELinux configured properly can significantly improve the overall security of the system. It has the reputation of being hard to understand and maintain, though. The time to learn and understand SELinux is well invested, since a well-configured SELinux installation can prevent attackers from doing malicious things, even if they gain access to your system. It also prevents applications from behaving differently after updating, so even if an attacker was successful in replacing a software with a malicious counterpart (or tricked you into installing that software), it won’t do too much harm if the SELinux policies were good enough.

Additional Resources

Hardening Linux systems is a vast topic and we just scratched the surface. Here are some good practical starting points for securing your Linux systems:

Securing MQTT Broker Deployment

Even if a reliable and secure network infrastructure is in place and your Linux system is hardened, some improvements are still possible for securing your MQTT broker deployment:

Authentication and Authorization

We talked about the importance of Authentication and Authorization in previous blog posts. These concepts and their implementation are important to make sure that only trusted MQTT clients can connect and don’t interfere with each other. For example, by separating topic namespaces. If it’s feasible for your use case, X509 client certificates add another layer of security and you should consider using them.

TLS

To ensure that eavesdropper can’t read and intercept the MQTT traffic from broker to clients and vice versa, most secure MQTT deployments use TLS for transport layer encryption. Although TLS adds extra bandwidth overhead to the communication, the added security is almost always worth it. If you haven’t read it yet, take a look at our post on securing MQTT at TLS and SSL level.

Throttling

If you know the bandwidth usage characteristics of your MQTT deployment beforehand, throttling MQTT clients can add additional protection from overloading MQTT brokers. Currently, bandwidth is expensive and limited. A few malicious MQTT clients with enough available bandwidth can overload your system very quickly. This situation can cause service degradation and could be very expensive.

HiveMQ allows you to throttle traffic on both a global and a per-client basis. You can limit the total incoming bytes per second and the total outgoing bytes per second independently. If you know you can only afford 20mbit of traffic (otherwise traffic would get too expensive) and your network and network interfaces can handle 100mbit, it may be wise to throttle the broker to this limit. You can adjust the limits while HiveMQ is up and running without a restart.

In addition to the global throttling, it’s possible to throttle specific clients with the HiveMQ plugin system after the client is successfully authenticated. For example, based on the client ID.

Message Size

MQTT defines a maximum message size of 256MB. In most MQTT deployment scenarios, messages are often smaller than a kilobyte. If you are familiar with your usage scenario and you know the maximum message size that can occur, it makes sense to decrease the maximum allowed message size to that limit. If no limit is set, it is possible for malicious MQTT clients to send large messages (which can result in excessive memory consumption and unneeded bandwidth usage).

With HiveMQ, you can limit the maximum message size on a global and per-client basis.

Summary

Securing MQTT systems (similar to any IT system) can be challenging and security needs to be considered at many levels. It’s important to know your network infrastructure and your servers very well and they need to be configured conscientiously.

If you have any comments or suggestions, please share your ideas with us. We would love to hear your feedback!