How to run a HiveMQ cluster with Docker and Kubernetes

by Florian Raschbichler

Nov 21, 2018 11 min read

The use of containerization and orchestration mechanisms like Docker and Kubernetes is ever growing among all IT deployments. Both the invention and spreading of the DevOps principle as well as the increasing market share and importance of cloud computing providers like Amazon Web Services or Google Cloud Platform are two of the main contributing factors behind this trend. This blog post shows that with the use of the HiveMQ DNS discovery Docker image it is simple and convenient to deploy a HiveMQ cluster with Docker and Kubernetes.

Docker and Kubernetes

Docker is a wide spread software project, providing software containers that aim to make application deployments more easy and portable. Kubernetes is an open-source container-orchestration system for automating deployment, scaling and management of containerized applications. It was originally designed by Google and is now maintained by the Cloud Native Computing Foundation. It aims to provide a “platform for automating deployment, scaling, and operations of application containers across clusters of hosts”. It works with a range of container tools, including Docker.

HiveMQ has released a Kubernetes Operator that allows you to run HiveMQ as a cloud-native system on Kubernetes. The HiveMQ Kubernetes Operator significantly simplifies the deployment and operation of HiveMQ clusters on any Kubernetes-based platform. For more information, see HiveMQ Kubernetes Operator.

DNS service discovery

DNS service discovery (DNS-SD) is a way of using standard DNS programming interfaces, servers and packet formats to browse the network for services. The HiveMQ DNS discovery plugin utilizes this technology to generate a discovery method for HiveMQ cluster nodes that is both very dynamic, which means that arbitrary numbers of HiveMQ broker nodes can be added and removed at runtime, and low maintenance. As long as a DNS discovery service, providing Round-robin A records is running and configured correctly, the configuration efforts on the broker side are very little and details about the logical network configuration are inconsequential.

Round-robin in this case means that the DNS service responds to a request for a service with a list of all available hosts (or pods in Kubernetes) for this service. A records are DNS records that point to an IP address directly.

$ kubectl exec hivemq-discovery-0 -- nslookup hivemq-discovery
Server: 10.0.0.12 
Address: 10.0.0.12#51 
 
Name: hivemq-discovery.default.svc.cluster.local 
Address: 172.17.0.1 
Name: hivemq-discovery.default.svc.cluster.local 
Address: 172.17.0.2 
Name: hivemq-discovery.default.svc.cluster.local 
Address: 172.17.0.3

The HiveMQ DNS Discovery Plugin will utilizes these A records for cluster discovery. DNS-SD is a perfect candidate for the use with orchestrated containerized environments or most cloud providers.

Installation and configuration

To achieve a dynamic HiveMQ cluster with Kubernetes, which uses the Kubernetes DNS-SD for cluster discovery, the following is required:

HiveMQ Docker image
HiveMQ DNS discovery plugin
Kubernetes cluster environment

HiveMQ Docker image with DNS cluster discovery

A Kubernetes cluster requires the existence of containerized services, to build a cluster. In this example we will be using the HiveMQ DNS Docker image from Dockerhub, which includes the HiveMQ DNS Discovery Plugin. If you want to create your own Docker images, please take a look at our HiveMQ Docker Images Repository.

Creating a Kubernetes headless cluster service

To enable the utilization of DNS-SD in a Kubernetes deployment, a headless service providing Round-robin A records is necessary. A headless Kubernetes service enables direct reaching of each individual pot. It is accomplished by setting clusterIP:none in the services specs. By creating this headless service without selectors for the pod, we also ensure that the DNS service of Kubernetes will return Round-robin A records.

An appropriate YAML configuration file for HiveMQ Kubernetes cluster looks like this:

apiVersion: v1
kind: ReplicationController
metadata:
  name: hivemq-replica
spec:
  replicas: 3
  selector:
    app: hivemq-cluster1
  template:
    metadata:
      name: hivemq-cluster1
      labels:
        app: hivemq-cluster1
    spec:
      containers:
      - name: hivemq-pods
        image: hivemq/hivemq3:dns-latest 
        ports:
        - containerPort: 8080
          protocol: TCP
          name: web-ui
        - containerPort: 1883
          protocol: TCP
          name: mqtt
        env:
        - name: HIVEMQ_DNS_DISCOVERY_ADDRESS
          value: "hivemq-discovery.default.svc.cluster.local." 
        - name: HIVEMQ_DNS_DISCOVERY_TIMEOUT
          value: "20"
        - name: HIVEMQ_DNS_DISCOVERY_INTERVAL
          value: "21"
        readinessProbe:
          tcpSocket:
            port: 1883
          initialDelaySeconds: 30
          periodSeconds: 60
          failureThreshold: 60
        livenessProbe:
          tcpSocket:
            port: 1883
          initialDelaySeconds: 30
          periodSeconds: 60
          failureThreshold: 60
---
kind: Service
apiVersion: v1
metadata:
  name: hivemq-discovery
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
  selector:
    app: hivemq-cluster1
  ports:
    - protocol: TCP
      port: 1883
      targetPort: 1883
  clusterIP: None

Line 17: We are using the official HiveMQ 3 DNS discovery image from Dockerhub Line 27: Make sure to set the value for HIVEMQ_DNS_DISCOVERY_ADDRESS according to your Kubernetes namespace and configured domain. Download this file. Line 58: We set clusterIP: None, making the hivemq-cluster1 service headless

Go to the Kubernetes Web UI, click on “+CREATE”, “Upload YAML or JSON file” and select the hivemq-k8s.yaml file and press “Upload”.

HiveMQ Docker Kubernetes Step 1 As soon as Kubernetes is finished with building the environment we have a working 3 node HiveMQ cluster. To verify the that the cluster discovery is working properly go to hivemq-replica replication controller and check the log file for one of the pods.

HiveMQ Docker Kubernetes Step 2 A correct configuration will result in a log line reading … INFO - Cluster size = 3, members … This is a sure sign that the cluster discovery is working properly.

Creating necessary load balancers

The next and last step to our functional HiveMQ cluster is creating 2 separate load balancers for the MQTT (1883) and Web UI (8080) port, so we enable external access to the HiveMQ cluster nodes.

For the MQTT load balancer use the following YAML configuration file and create the service the same as we did the HiveMQ replication controller.

kind: Service
apiVersion: v1
metadata:
  name: hivemq-mqtt
  annotations:
    service.spec.externalTrafficPolicy: Local
spec:
  selector:
    app: hivemq-cluster1
  ports:
    - protocol: TCP
      port: 1883
      targetPort: 1883
  type: LoadBalancer

Go to Services -> hivemq-mqtt to find the external end point of the load balancer that can be used to establish MQTT connections to our HiveMQ broker cluster.

HiveMQ Docker Kubernetes Step 3 The load balancer for the HiveMQ Web UI required the use of sticky sessions. Otherwise the login session information gets lost when browsing to the Web UI. Make sure your individual Kubernetes environment allows the use of sticky sessions The following YAML configuration file can be used to leverage the connecting client’s IP for sticky sessions.

kind: Service
apiVersion: v1
metadata:
  name: hivemq-web-ui
spec:
  selector:
    app: hivemq-cluster1
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
  sessionAffinity: ClientIP
  type: LoadBalancer

You can access the HiveMQ Web UI via this load balancer’s external endpoint with your web browser of choice.

That’s it! We now have fully functional, accessible HiveMQ cluster inside our Kubernetes environment, which we can dynamically scale with the help of our replication controller.

Adding a HiveMQ license to your Docker image

If you are a HiveMQ customer and want to use your license with Kubernetes and HiveMQ DNS Discovery Docker image you can base64-encode the license and set the resulting String as the environment variable HIVEMQ_LICENSE:

cat <path/to/license.lic> | base64

Summary and resource list

Choosing the example of Kubernetes this blog post showed that the new HiveMQ Docker image, including the HiveMQ DNS discovery plugin can be leveraged to create dynamic, auto scaling HiveMQ cluster. Likewise the HiveMQ DNS cluster plugin can be use with any other environment that supports DNS-SD like AWS and Route53 or Google Cloud DNS. If you’re looking to utilize the HiveMQ DNS cluster discovery in your specific environment and are facing difficulties, please do not hesitate to leave us a comment or contact us.

Here are some useful resources:

Florian Raschbichler

Florian serves as the head of the HiveMQ support team with years of first hand experience overcoming challenges in achieving reliable, scalable, and secure IoT messaging for enterprise customers.