Deploy a HiveMQ cluster on Azure Kubernetes Service (AKS) as a StatefulSet with high-performance Azure storage disks
Today’s blog post focuses on deploying a HiveMQ cluster on Microsoft’s Azure Kubernetes Service (AKS) using a StatefulSet that provides built-in support for persistent volumes. To guarantee the disk performance you need for high-performance MQTT use cases, we show you how to specify premium Azure storage disks for your HiveMQ data directory.
Introduction
Kubernetes provides several built-in workload resource types such as Deployment, DaemonSet, and StatefulSet to make installing and maintaining containerized applications on Kubernetes much easier. Each workload resource configures a controller that creates and maintains a set of Kubernetes pods on your behalf so you don’t need to manage the pods directly. Different types of workload resources are ideal for different applications and use cases.
By default, the HiveMQ Kubernetes Operator uses the Deployment workload resource type to install and maintain a HiveMQ cluster. Since the Deployment type lacks built-in support for persistent volumes, ephemeral volumes are used for the HiveMQ data directory. Typically, the ephemeral volumes are mounted from the local storage of a Kubernetes node and are therefore shared between pods on the node. The use of shared ephemeral storage makes it difficult to guarantee the performance (IOPS) for individual pods or applications. Input and output per second IOPSs is the standard way to measure how quickly a given storage device can read and write data. Higher values mean a device is capable of handling more operations per second. For high-performance MQTT use-cases, it is important to exert control on which type of storage is used.
Since HiveMQ 4.7, the HiveMQ Kubernetes Operator can also be used to install HiveMQ clusters with the Kubernetes StatefulSet resource type. This is good news for MQTT use cases that require consistently high disk performance and can benefit from using a separate premium storage volume. For example, solutions that need to provide large numbers of retained messages.
Together with the HiveMQ operator, the StatefulSet controller maintains the deployment and scaling of pods for your HiveMQ cluster to make sure the number and type of pods you specified for your application are always running. Additionally, StatefulSets provide guarantees about the ordering and uniqueness of the pods and include specifications for persistent volumes. PersistentVolumes (PV) are storage volumes in the Kubernetes cluster that are created and managed independently of the lifecycle of a pod. AKS can select from various Azure disk storage offerings to provide persistent storage.
Prerequisites
Microsoft Azure account
A running AKS Kubernetes cluster with available compute resources for HiveMQ
The Kubernetes command-line tool, kubectl, installed locally
The Azure command-line tool, az, installed locally
The current version of Helm, installed locally
HiveMQ license (optional). If you don’t have a valid HiveMQ license, the HiveMQ cluster you deploy automatically uses a trial version that is restricted to non-production use and 25 concurrent connections. To test and evaluate HiveMQ with more connections, contact HiveMQ.
NOTE: This article does not go into the details on how to install an AKS cluster. If you need some help to get started with AKS, see Quickstart: Deploy an Azure Kubernetes Service and HiveMQ with Azure Kubernetes Service.
Step 1: Install the HiveMQ Kubernetes Operator and a HiveMQ cluster on AKS with Helm
In this step, you use the HiveMQ Helm chart to install the HiveMQ Kubernetes Operator and a HiveMQ cluster on AKS with the default Deployment resource type and you define a custom installation file.
Use kubectl to create a namespace for HiveMQ on your AKS cluster:
kubectl create namespace hivemqAdd the HiveMQ Helm Chart to your local Helm repository:
NOTE: If you have already installed the HiveMQ Helm Chart, use the following command to ensure that you are running the most current version:
Create a custom installation file named myCustomValues.yaml with the following values:
hivemq:
logLevel: INFO
nodeCount: 2
cpu: 4
memory: 4GiUse Helm to install the HiveMQ Kubernetes Operator and a HiveMQ cluster that uses the settings you specified in the myCustomValues.yaml file:
This command does the following:
Extends the Kubernetes API with the HiveMQ Cluster Custom Resource Definition (CRD).
Installs the HiveMQ Kubernetes Operator as a custom controller for HiveMQ clusters.
Installs a HiveMQ cluster with 2 nodes and the default HiveMQ configuration settings.
NOTE: This HiveMQ cluster requires 8 CPUs and 8 Gi of memory. If these resources are not available for your cluster, the HiveMQ pods remain in a pending state. For testing purposes, you can reduce the CPU and memory settings of the HiveMQ cluster.Saves the instance of the Helm chart with the release name myhivemq.
NOTE: A message on your command-line interface confirms that the Helm deployment was successful.
To verify that your HiveMQ cluster is installed, enter:
kubectl get hivemq-clustersAs soon as the displayed state shows RUNNING, your HiveMQ cluster is ready to go.
To access the HiveMQ Control Center of the HiveMQ cluster, forward the port of the Control Center service through kube-proxy:
kubectl get svc
kubectl port-forward svc/hivemq-myhivemq-cc 8080To open the HiveMQ control center for your HiveMQ cluster, navigate to http://localhost:8080 and enter the default control center credentials in the HiveMQ control center login dialog (username: admin, password: hivemq). The control center dashboard gives you detailed information for your HiveMQ cluster and MQTT clients. To learn more about the HiveMQ control center, see HiveMQ Control Center.
Step 2: Install a HiveMQ cluster with the StatefulSet template
In this step, you use Helm to delete and reinstall your HiveMQ cluster with the StatefulSet resource type and a PersistentVolume that uses the default Azure StorageClass.
Delete the previous HiveMQ cluster:
helm delete myhivemqEdit your myCustomValues.yaml installation file with the following values:
hivemq:
logLevel: INFO
nodeCount: 2
cpu: 4
memory: 4Gi
controllerTemplate: cluster-stateful-set.yamlUse Helm to install a new HiveMQ cluster with the StatefulSet template
helm install myhivemq hivemq/hivemq-operator -f myCustomValues.yamlThis command installs your HiveMQ cluster using the StatefulSet template. Since no custom persistent volume settings are specified, the HiveMQ operator installs a default persistent volume with 100Gi storage capacity and the default StorageClass of your AKS cluster.
To verify that the default persistent volume was successfully created, enter:
kubectl get persistentvolumeclaim
The result shows the created volume:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
data-myhivemq-0 Bound pvc-23b03202 100Gi RWO default
Step 3: Install a HiveMQ cluster with Azure managed-premium SSD storage
In the last step of this example, you use Helm to delete and reinstall your HiveMQ cluster so that you can specify a PersistentVolume that uses the Azure premium-managed StorageClass for high IOPS performance.
Delete the previous HiveMQ cluster:
helm delete myhivemqAdjust your myCustomValues.yaml installation file to specify a high-performance SSD-backed disk for your HiveMQ cluster:
hivemq:
logLevel: INFO
nodeCount: 2
cpu: 4
memory: 4Gi
controllerTemplate: cluster-stateful-set.yaml
volumeClaimTemplates:
- kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 300Gi
volumeMode: Filesystem
storageClassName: managed-premium
Use Helm to install your HiveMQ cluster with the new values:
helm install myhivemq hivemq/hivemq-operator -f myCustomValues.yamlThis command installs your HiveMQ cluster using the StatefulSet template and a VolumeClaimTemplate that specifies a disk with 1100 IOPS and 300Gi storage capacity that is suitable for production workloads.
Based on the PersistentVolumeClaims (PVC) that you specify in the volumeClaimTemplates section of your myCustomValues.yaml file, the Kubernetes StatefulSet controller provisions a PersistentVolume for each HiveMQ pod. The storageClassName that you specify in the volumeClaimTemplates setting identifies the type of Azure storage disk you want the StatefulSet controller to provide for the persistent volume. StorageClasses map Azure disk resources to Kubernetes. If you do not request a specific StorageClass, the default StorageClass is used.
Azure provides several StorageClasses that are installed and available for use in every AKS Kubernetes cluster.
To view the available Azure StorageClasses on your AKS cluster, enter:
kubectl get storageclassesThe result of this command lists the StorageClass name and key details for each available Azure storage disk type on the AKS cluster:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXP.
azurefile kubernetes.io/azure-file Delete Immediate true
azurefile-premium kubernetes.io/azure-file Delete Immediate true
default (default) kubernetes.io/azure-disk Delete WaitForFirstConsumer true
managed-premium kubernetes.io/azure-disk Delete WaitForFirstConsumer true
The default Azure StorageClass maps to the Azure Standard SSD storage offering. Azure Standard SSD storage is designed for medium IOPS workloads of up to 500 IOPS. Although Azure states higher max burst performance for the standard disk, the heightened performance is only available for brief periods. For your HiveMQ cluster on AKS, look for a fast and reasonably-priced disk that provides consistent performance of at least 900 IOPS.
Azure managed-premium storage offers much more suitable options. Starting at a size of 256Gi, the Azure Premium SSD disks in the managed-premium StorageClass deliver sustained performance of 1100 IOPS. For high-performance HiveMQ workloads, Azure Premium SSD disks are the recommended choice.
To verify that the persistent volume was successfully created as specified in your myCustomValues.yaml file, enter:
kubectl get persistentvolumeclaim The result shows the created volume:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
data-myhivemq-0 Bound pvc-23b03202 300Gi RWO managed-premium
For more information on Azure storage options for AKS, see
Storage options for applications in Azure Kubernetes Service (AKS), Best practices for storage and backups in AKS, and Azure Pricing - Managed Disks.
Clean up
Your Azure account is billed for all services you create. Be sure to delete storage disks that are no longer in use. Deletion of your HiveMQ pods does not automatically delete the associated storage volumes.
To view the persistent volume claims that are currently active, enter:
kubectl get persistentvolumeclaim
To delete a specific persistent volume claim and the associated persistent volume, enter the following command with the name of the PVC you want to delete. In our example, the name is
data-myhivemq-0:
kubectl delete persistentvolumeclaim data-myhivemq-0Conclusion
The latest version of the HiveMQ Kubernetes Operator makes it easy to deploy a HiveMQ cluster using a StatefulSet template. When you use a StatefulSet, you need to configure the type of persistent storage the StatefulSet controller provisions for your HiveMQ data directory. For HiveMQ production use cases, your persistent storage volumes need to deliver a sustained performance of at least 900 IOPS. On AKS, you can specify Azure managed-premium SSD disks to ensure high IOPS for your MQTT workloads on HiveMQ. In general, the same tools and principles can be used with other Kubernetes as a Service providers (keep in mind that the specific storage class names and performance levels for each cloud vendor vary).
Mary Brickenstein-Hofschen
Mary is a technical writer at HiveMQ. She enjoys providing precise information that helps HiveMQ users get things done.
