Automating MQTT Broker Management on Kubernetes with IaC and GitOps: Part 2

by Sven Kobow

Dec 5, 2024 22 min read

In the first part of this article, we established the foundational elements of our GitOps-driven deployment. We set up Argo CD as the centerpiece for managing applications using the app-of-apps pattern. Using Terraform, we automated the deployment of these components and introduced best practices for handling sensitive data with SealedSecrets.

Now, in the second part, we’ll shift our focus to securing our MQTT broker cluster by covering essential security topics, including:

Enabling MQTT over TLS: Configuring the broker to secure MQTT communication channels with TLS encryption, ensuring data integrity and confidentiality.
Securing the HiveMQ Control Center with HTTPS: Configuring Kubernetes Ingress to use HTTPS for encrypted connections to the HiveMQ Control Center, preventing potential eavesdropping and man-in-the-middle attacks.
Certificate Management with cert-manager: Automating the issuance and management of certificates for both MQTT and HTTPS connections.

Security is a critical aspect of any production-grade system, especially when sensitive data is transmitted over networks. By implementing these measures, we’ll ensure that both communication channels and access points to the broker are properly secured and ready for production environments. Let’s dive into the details of these security enhancements.

Securing MQTT Traffic with TLS Encryption

Securing TCP-based traffic with TLS encryption is a widely adopted best practice for enhancing data security. TLS (Transport Layer Security) ensures that communication between a client and a server is encrypted, providing confidentiality, integrity, and protection against man-in-the-middle attacks. Since MQTT operates over TCP, the same TLS encryption principles can be applied to secure MQTT traffic.

To enable TLS encryption for MQTT, two key components are required:

A Certificate: The broker needs a certificate to establish encrypted communication. The HiveMQ Platform requires you to provide a Java Keystore which contains the necessary private key and certificate chain.
Modified Broker Configuration: The broker's configuration must be updated to include an additional TLS-enabled listener on the standard MQTT-over-TLS port, 8883. This ensures that secure connections are available alongside any existing non-TLS listeners.

For managing certificates, we’ll use cert-manager, which will handle the creation of certificates.

With these elements in place, we can securely enable MQTT-over-TLS and ensure that sensitive data transmitted between clients and the broker is protected.

Let’s Do a Short Recap of Service Deployment

Let’s recap the general repository and deployment structure we’ve established as discussed in Part 1 of this article. Our GitOps setup is organized to manage each service through two distinct Argo CD applications, allowing us to maintain a clear separation between core service deployments and any additional resource configurations they require.

Each service will usually have in the apps/<service> folder:

1. A primary Argo CD application focused on deploying the actual service, typically through a Helm chart.

2. A secondary Argo CD application pointing to the manifests/<service> folder, responsible for deploying any additional Kubernetes resources needed for the service. This may include configuration files, Secrets, ingress resources, and other custom manifests that aren’t directly managed by the Helm chart.

This approach ensures a modular and flexible structure, enabling us to configure and update each service independently while still tracking all related resources within version control. With this structure in place, we can proceed to install the necessary prerequisites for our setup: cert-manager for TLS certificate management and Traefik as our ingress controller. Each will have its own Argo CD application in the apps/ folder, following our established repository organization.

Setting Up Cert-Manager for Certificate Management

Cert-manager is essential for handling TLS certificates, which we’ll use to secure both MQTT connections (MQTT over TLS) and HTTPS access to the HiveMQ Control Center. By automating certificate provisioning and renewal, cert-manager reduces the complexity of maintaining secure connections, ensuring our broker cluster is always accessible securely. Cert-manager also supports generating certificates in the Java Key Store (JKS) format, which is required for configuring the TLS listener for the HiveMQ platform.

To keep the setup simple, we’ll use a self-signed issuer in this example to generate the required certificates. The setup can easily be adapted for production environments by using one of the different issuer types that cert-manager supports.

Using a self-signed issuer allows us to generate the necessary certificates for testing and development purposes without relying on an external Certificate Authority (CA). With cert-manager managing both certificate issuance and renewal, we ensure that our broker’s TLS configurations remain consistent and up-to-date.

For the cert-manager deployment we are using the official helm chart.

apps/certmanager/cert-manager.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.jetstack.io
    chart: cert-manager
    helm:
      valuesObject:
        crds:
          enabled: true
    targetRevision: 1.15.3 # check for current targetRevision
  destination:
    name: in-cluster
    namespace: cert-manager
  syncPolicy:
    automated:
      prune: false
      selfHeal: false
    syncOptions:
      - CreateNamespace=true

As we do not need to deploy any additional resources for cert-manager no cert-manager-manifests application is needed. But we are still missing the Issuer that is needed for actually issuing certificates. Cert-manager fundamentally distinguishes between Issuer and ClusterIssuer. Whereas ClusterIssuer are meant to be used cluster-wide, Issuers are local to the namespace they are created in. We only need an issuer for the HiveMQ Platform in this example and it will reside in the hivemq namespace. As it is therefore related to the HiveMQ deployment we will consequently add the manifests for creating the issuer into the manifests/hivemq folder.

manifests/hivemq/hivemq-issuer.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: hivemq-selfsigned-issuer
  namespace: hivemq
spec:
  selfSigned: {}

Using Traefik as Ingress Controller for Secure Control Center Access

An ingress controller is required to manage and route external traffic to HiveMQ Control Center within the Kubernetes cluster. In this setup, we’ll use Traefik as our ingress controller, a popular choice known for its flexibility and ease of integration with Kubernetes. Traefik will manage the ingress rules and TLS configuration, providing secure HTTPS access to the broker’s web UI.

Using Traefik allows us to:

Expose the HiveMQ Control Center securely over HTTPS.
Configure the ingress with TLS using certificates issued by cert-manager, ensuring that all Control Center traffic is encrypted.

With Traefik handling ingress routing and cert-manager managing certificate issuance, our setup provides a reliable, secure connection model that can easily scale to meet the needs of both development and production environments.

As an alternative to Traefik, you could also use the Nginx Ingress Controller, which basically provides the same functionality.

Installing Traefik

We will install Traefik using the official Helm Chart. If you need to configure the Helm chart you can specify certain values in the application as well. All possible values can be found here.

apps/traefik/traefik.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: traefik
  namespace: argocd # this is the namespace where Argo CD is installed
spec:
  project: default
  source:
    repoURL: https://traefik.github.io/charts
    chart: traefik
    targetRevision: 31.1.1 # check for current targetRevision
  destination:
    name: in-cluster
    namespace: kube-system # this is the namespace where Traefik is installed
  syncPolicy:
    automated:
      prune: true

The secondary application is responsible for deploying additional resources for Traefik such as an IngressRoute for accessing the Traefik dashboard and a Middleware to secure the access using basic authentication. A SealedSecret is used for providing a list of user credentials. These resources are optional and not needed for the HiveMQ Platform deployment. We use them here to further clarify the concept and familiarize you with it. Additionally you will learn how to create a SealedSecret.

apps/traefik/traefik-manifests.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: traefik-manifests
  namespace: argocd
spec:
  project: default
  source:
    repoURL: <YOUR_REPOSITORY_URL>
    targetRevision: HEAD
    path: manifests/traefik
  destination:
    name: in-cluster
  syncPolicy:
    automated:
      prune: true

manifests/traefik/traefik-dashboard.yaml

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: kube-system
spec:
  entryPoints:
    - websecure
  routes:
    - match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
      kind: Rule
      middlewares:
        - name: traefik-dashboard-auth
      services:
        - name: api@internal
          kind: TraefikService

---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: traefik-dashboard-auth
  namespace: kube-system
spec:
  basicAuth:
    secret: traefik-dashboard-auth

manifests/traefik/traefik-dashboard-auth-sealedsecret.yaml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: traefik-dashboard-auth
  namespace: kube-system
spec:
  encryptedData:
    users: AgAjfvRiTY9H9IK97t2tdcb87etSeLfYikBiA9qE7UvlBFWKqL9NcBZjWaNOR3rGllQrgY6KRer1MaoDNSUaA23+HC2YcNm3EaVD/LMeEvbfeuL/DcFOlCNzVdduJpxRdrQJuZyu8LQw1vswohkUL8+Mcy205xn3wRVM39Qq4UEkRWsR/uztY4EbJZErnZ4zRLnHwy7noNxOenGdoGR+ZQlMzNUIKC9hDIaOZYDhiO91PaZ22M0Gd06e1fDD0U6GxGdUx1soLXmgO+oVcVRQ152IaKzgfvK+CCnFY9HjkAlKGxSiIbnw4FVK8RSfumy5VwuKmGfeyzdV0RvQ4qhuvViZHfywCmY7BQE68Xz7hwLpBmMlzmVjRYcFjChyMNChpsnmW7Aq+e2888Rnswi13VejZxyHA4Ar4sNXgjbLdEy/a6bUjDphrXEOg7geRXFvsSArA/7OBeIpt2u7iWtLPKfrSxvbEUURVPTFo3nkNBHtbO0MwjriKv3RLTvutFN8JMV0VTGXZBbB7qK+GwnBvvjtxcTBaBk+4zU6plBOoIOE3QaTGULZX+x6/T9AsWv3qGDy9T+8bNe/xpSRozZiUzxDLhfKU/HXktgLLpznEEf7SQchpnnmrqGhIW0RQhNhlXsl7IFDbWJc9A54NWTW0WB3xFM7E3wxd9tP/m6DuHyV57/5ikAuUSEIbpJNLMZZ3BykQ24bciY8FRIqTYe0CWdO5jkOx1xuxUWLGdwCZbOHdwEiK+JS8ts9sZOMYg==
  template:
    metadata:
      creationTimestamp: null
      name: traefik-dashboard-auth
      namespace: kube-system

Excursions: Creating SealedSecrets with kubeseal

The spec.encryptedData section in the SealedSecret above contains encrypted secret data, which was generated with kubeseal using a standard Kubernetes Secret. Kubeseal is a tool designed to help securely manage and encrypt secrets for Kubernetes clusters in a GitOps workflow. With kubeseal, sensitive data is encrypted and stored as SealedSecrets, which can safely be stored in version control and only decrypted within the target Kubernetes cluster.

To create a SealedSecret, kubeseal uses a regular Kubernetes Secret as input. Here’s a brief guide on how to do this:

Create a Kubernetes Secret: First, create a regular Kubernetes Secret manifest with the data you want to encrypt.
Seal the Secret with kubeseal: Use kubeseal to encrypt the Secret, generating a SealedSecret resource. This is typically done using the following command:

kubeseal -f <YOUR>-secret.yaml -w <YOUR>-sealedsecret.yaml

This will convert the secret definition into a sealed secret which can safely be committed in Git.

For more detailed instructions and additional configuration options, refer to the official kubeseal documentation. This documentation provides extensive information on how to set up kubeseal, manage secrets, and ensure secure, GitOps-compatible workflows for handling sensitive data.

💡TIP:

Since you should prevent unencrypted secrets from being committed to Git under any circumstances, we recommend taking further measures, e.g. storing secrets in files with the postfix *-secret.yaml and adding this pattern to your .gitignore file.

Please refer to the Traefik documentation on how to create credentials and configure a middleware for basic authentication.

Configuring the HiveMQ MQTT Platform

With all the required services deployed, we can now finalize the setup by configuring the HiveMQ Platform and adding the remaining resources. These configurations are crucial to enable secure communication and proper access to the broker's services. Specifically, we need to configure the following:

Certificate for the TLS Listener:
- HiveMQ requires a Java KeyStore (JKS) certificate for its tls-tcp-listener.
- The certificate is issued by cert-manager and uses a corresponding secret that holds the password for the keystore. This secret ensures secure access to the private key and certificate chain and will be created as a sealed secret.
TLS Listener Configuration:
- The configuration for a Secure TCP Listener needs to be added to the HiveMQ Platform.
- The JKS certificate and the secret containing the password will be used.
IngressRoute for the HiveMQ Control Center:
- The HiveMQ Control Center is exposed through an IngressRoute to allow HTTPS access.
- We'll configure an IngressRoute resource to route external traffic securely to the Control Center, leveraging Traefik as the ingress controller.
Certificate for the IngressRoute:
- To secure the IngressRoute, we’ll issue another certificate using cert-manager.
- This certificate will be used for TLS termination, ensuring encrypted communication between clients and the web UI.

By completing these configurations, we enable the HiveMQ Platform to handle secure MQTT connections, expose its HiveMQ Control Center securely over HTTPS, and meet all requirements for a production-ready deployment. Let’s dive into the details of implementing these components.

Creating Certificates and Securing the MQTT Traffic

As a first step we will create a sealed secret for the password that is used to secure the keystore. Following the approach described above we create the secret manifest file making sure it is not committed to the Git repository.

hivemq-tls-keystore-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: hivemq-tls-keystore-password
  namespace: hivemq
type: Opaque
data:
  password: <YOUR_BASE64_ENCODED_PASSWORD>

Kubeseal will then be used to create the SealedSecret

   kubeseal -f hivemq-tls-keystore-secret.yaml -w hivemq-tls-keystore-sealedsecret.yaml

Once applied to your Kubernetes Cluster the sealed-secrets operator will pick up the definition and create the Kubernetes Secret for you.

Creating the Java Keystore Certificate for the Secure MQTT Listener is straightforward. Additionally to the properties of the certificate such as commonName or dnsNames we want cert-manager to create a Java Keystore. Therefore we add a keystores.jks configuration. The Secret containing the password we have defined above is referenced with keystores.jks.passwordSecretRef.

manifests/hivemq/hivemq-tls-keystore.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: hivemq-tls-keystore
  namespace: hivemq
spec:
  secretName: hivemq-tls-keystore
  commonName: HiveMQ GitOps Demo
  dnsNames:
    - mqtt.example.net # Use your FQDN here 
  issuerRef:
    name: hivemq-selfsigned-issuer
  keystores:
    jks:
      create: true
      passwordSecretRef: # Password used to secure the keystore
        key: password
        name: hivemq-tls-keystore-password
    pkcs12:
      create: true
      passwordSecretRef: # Password used to secure the keystore
        key: password
        name: hivemq-tls-keystore-password

With the Java Keystore and the required SealedSecret holding the keystore password in place we have everything we need to secure the MQTT traffic. We are still missing the HiveMQ Platform configuration though.

HiveMQ supports secure MQTT-over-TLS communication by using Secure TCP Listeners. Usually these tls-tcp-listeners are configured in the config.xml file for HiveMQ. In our case as we are using the HiveMQ Platform Operator for Kubernetes we will add the configuration to the valuesObject of the Argo CD Application we have created in Part 1. The HiveMQ Platform Operator will render a corresponding config.xml configuration file for the HiveMQ Platform.

          - type: mqtt
            exposed: true
            containerPort: 8883
            serviceType: LoadBalancer
            annotations:
              metallb.universe.tf/allow-shared-ip: hivemq-mqtt-port
            # The name of the Kubernetes secret that contains the keystore file.
            keystoreSecretName: "hivemq-tls-keystore"
            # The key of the Kubernetes secret that contains the keystore file. Defaults to `keystore`
            keystoreSecretKey: "keystore.jks"

            # The name of the Kubernetes secret which contains the keystore password and the private key password for the keystore.
            keystorePasswordSecretName: "hivemq-tls-keystore-password"
            # The optional entry key for the keystore password in the Kubernetes secret.
            keystorePasswordSecretKey: "password"
            # The optional entry key for the keystore private password in the Kubernetes secret.
            #keystorePrivatePasswordSecretKey: ""

The complete HiveMQ Platform Application then looks as follows:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hivemq-platform
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://hivemq.github.io/helm-charts
    chart: hivemq-platform
    helm:
      valuesObject:
        license:
          create: false
          name: "hivemq-licenses"
        services:
          - type: mqtt
            exposed: true
            containerPort: 1883
            serviceType: LoadBalancer
          - type: mqtt
            exposed: true
            containerPort: 8883
            serviceType: LoadBalancer
            keystoreSecretName: "hivemq-tls-keystore"
            keystoreSecretKey: "keystore.jks"
            keystorePasswordSecretName: "hivemq-tls-keystore-password"
            keystorePasswordSecretKey: "password"
          - type: control-center
            exposed: true
            containerPort: 8080
            serviceType: LoadBalancer
    targetRevision: 0.2.29
  destination:
    name: in-cluster
    namespace: hivemq
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true

After committing these changes to your repository and Argo CD synchronization you can check if the secure MQTT endpoint and a corresponding Kubernetes Service object was created.

❯ kubectl get services -n hivemq
NAME                                       TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
hivemq-hivemq-platform-mqtt-1883           LoadBalancer   10.98.158.67     172.16.2.4    1883:30203/TCP      11d
hivemq-hivemq-platform-mqtt-8883           LoadBalancer   10.107.179.211   172.16.2.1    8883:30778/TCP      8d

Getting the Server Certificate for Client Connections

As we are using a self-signed (untrusted) certificate for the MQTT communication now we will need to provide the server certificate to the MQTT client as the CA certificate in order to be able to connect. You can easily obtain the certificate using kubectl and jq:

❯ kubectl get secrets/hivemq-tls-keystore -n hivemq -o json | jq -r '.data."ca.crt"' | base64 --decode

-----BEGIN CERTIFICATE-----
MIIDATCCAemgAwIBAgIQVlsCyIB/p/qvRKVPXkoFpDANBgkqhkiG9w0BAQsFADAd
MRswGQYDVQQDExJIaXZlTVEgR2l0T3BzIERlbW8wHhcNMjQxMTExMTI0NDE0WhcN
MjUwMjA5MTI0NDE0WjAdMRswGQYDVQQDExJIaXZlTVEgR2l0T3BzIERlbW8wggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDJAhntX53jqjofDMnaNU95SfQB
WyqBb9HBrG3+fNuPOHoQEifMJGMaSq9oG8SH4u839lI8I8ZTM/3jVZG07/CBIdLo
C4IG4rh+FSkIQ0VqgIqntZUKf9s9dYR0f6uCVoF24mgCr4vX+uF3jHnVjYJ/csGO
Rab2ZMSxfMtOL9QexgX/AjpCffptjuBWv3968u0pMTEh4z4ndQ7P4A6pIcarsXLi
ZgkCBzuRVQXb3LQqWQTl1mji9p254Ag///TbXZ1ka0pqYg6k71hPUyxpFQ6y0Vse
FH8YWrLCcIDPcRq7w6+lNI/exdS5Aa0sqC9zutWnYaGqYAXL9+GGrDMw1Oo3AgMB
AAGjPTA7MA4GA1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMBsGA1UdEQQUMBKC
EG1xdHQuZXhhbXBsZS5uZXQwDQYJKoZIhvcNAQELBQADggEBAJPF7rcLCE9G9djB
DUs2I5mqd8xOUV+wgrPngZa3lqDtc7vCw+DfVHT165Pa4gxFeqkEitXSkaRR+8j8
7srs5LE3GZGqD5Dc7jL0Z+0qCnpzgQEwoBdQ8JvwonvSTRvZZhWBN+seXXvJOTzu
gbSqtElD6XenwBchCB/MGe2/dAwR7TgEh5qL0lPEQcrufQpX2dar9oJxdDMAL8Wn
KDDv4GLQCisqj00UIy+rHaWmXfxVo/Pc4UVfBp6S6GQKwWSjetgc8nIwt83vYSm5
48u8M+I/K7T+CnQU/ShxNsOsh5CTnkAZIqZ8cdB79FbBA53T9nRUq/mI0SDQT7/9
vrh0xvE=
-----END CERTIFICATE-----

Creating a Certificate and Secure Control Center Traffic

The final step for this Part of our blog series will be the TLS configuration for accessing the HiveMQ Control Center. You now already know all the steps that are necessary to add the required configuration: adding a Certificate and adding an additional IngressRoute resource to the HiveMQ manifests.

Let’s start with the Certificate:

manifests/hivemq/hivemq-cc-cert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: hivemq-cc-tls
  namespace: hivemq
spec:
  commonName: hivemq.example.net
  secretName: hivemq-cc-tls
  issuerRef:
    name: hivemq-selfsigned-issuer

As you can see we are again using our self-signed cert-manager Issuer for generating the certificate.

Even though securing the HiveMQ Control Center access is possible using the Helm chart with the respective configuration we will be using an IngressRoute resource in this example.

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: hivemq-cc-tls-ingress-route
  namespace: hivemq
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`hivemq.example.net`)
      kind: Rule
      services:
        - name: hivemq-hivemq-platform-cc-8080
          port: 8080
          sticky:
            cookie:
              httpOnly: true
              name: hivemq-cc-session-cookie
              secure: true
              sameSite: none
              maxAge: 86400
  tls:
    secretName: hivemq-cc-tls

Defining an IngressRoute is pretty easy. All you need to do is use the websecure entrypoint Traefik provides for TLS connections and the secretName of the certificate that we have defined above. Adjust the hostname in the route matcher according to your configuration.

The HiveMQ Control Center automatically ships with the HiveMQ Platform. To avoid problems with routing HTTP requests to different HiveMQ Cluster Nodes which would invalidate your login information we also need to configure sticky sessions for the Ingress. With Traefik cookies can be used for this purpose by using the sticky.cookie configuration.

In Part 1 of this blog series the type of the Control Center Service was set to LoadBalancer so that it was externally accessible using an assigned IP address. In case you want to limit the Control Center access to HTTPS only over the DNS name you can set the service type to ClusterIP to not expose the service externally.

          - type: control-center
            exposed: true
            containerPort: 8080
            serviceType: ClusterIP

Conclusion

In this series, we explored automating MQTT broker management on Kubernetes using Infrastructure as Code (IaC) and GitOps, combining these principles with robust security measures to create a scalable and secure deployment. After establishing a strong foundation with Argo CD, Terraform, and SealedSecrets, we enhanced security by enabling MQTT over TLS, securing the HiveMQ Control Center with HTTPS, and automating certificate management using cert-manager. These steps ensure data integrity, confidentiality, and resilience, making the deployment production-ready while addressing the critical need for secure communication and access. In Part 3 of this series, we will configure the Enterprise Security Extension.

Sven Kobow

Sven Kobow is part of the Professional Services team at HiveMQ with more than two decades of experience in IT and IIoT. In his role as IoT Solutions Architect, he is supporting customers and partners in successfully implementing their solutions and generating maximum value. Before joining HiveMQ, he worked in Automotive sector at a major OEM.