Securing HiveMQ Broker Deployments With Intermediate CA Certificates
In a standard security practice, the Root Certificate Authority (CA) is not utilized directly for signing server or client certificates. Instead, the Root CA's primary role is to establish one or more Intermediate Certificate Authorities. These Intermediate CAs are trusted entities, designated by the Root CA, to handle the task of signing certificates. Adopting this hierarchical approach serves a critical purpose: it enables the root key to remain offline and largely inactive, thereby enhancing overall security. If the intermediate key becomes compromised, the Root CA can respond effectively by revoking the compromised intermediate certificate. Following this revocation, the Root CA can then generate a new cryptographic key pair for a replacement Intermediate CA.
The root key is a highly sensitive asset; its exposure or compromise could have severe implications for the entire trust infrastructure. By minimizing the usage of the root key and employing Intermediate CAs for day-to-day certification tasks, the security of the digital certificate ecosystem is significantly bolstered.
An Intermediate Certificate Authority (CA) operates as a delegated entity, possessing the authority to issue and sign digital certificates on behalf of the Root CA. This delegation is established when the Root CA, which sits at the apex of the trust hierarchy, signs the Intermediate CA's certificate. This process effectively creates a 'chain of trust' extending the Root CA's trustworthiness to the certificates issued by the Intermediate CA.
In this guide, we set out to establish a private Certificate Authority (CA) that mirrors the structural framework of public CAs. At the core of this architecture will be a single Root CA, which serves as the foundational trust anchor. From this Root CA, we will have the ability to spawn various Intermediate CAs, thereby extending the trust chain. This setup becomes particularly relevant in the context of secure MQTT communications using HiveMQ Broker, where the implementation of certificates is crucial for ensuring the integrity and confidentiality of the data transmitted. However, it is important to note that we will not describe the certificate revocation process, including the use of Certificate Revocation Lists (CRLs) or Online Certificate Status Protocol (OCSP) responders. These critical aspects of certificate management deserve a thorough exploration and will be the focus of a future dedicated article.
Root CA Directory Structure
The first step is to create the directory structure and initialize some files used during the CA operation.
NOTE: Some of the following commands require root privileges, log in as root user, or use sudo to execute the commands.
Choose a directory (
/root/ca
) to store all keys, certificates, and two empty databasesindex.txt
andserial.txt
files.
NOTE: The default location to install certificates is /etc/ssl
but might be different on each operating system. This enables multiple services to use the same certificate without overly complicated file permissions.
Change the current directory to the Root CA directory.
Create all folder structures required.
NOTE: The "Keystores" and "Truststores" will be output to a directory named keystores one level above your working directory.
Protects "private" folders against any access from other users.
The database in index.txt
is a plaintext file that contains certificate information, one certificate per line.
When creating a new CA certificate, it’s important to initialize the certificate serial numbers with a random number generator. This is very useful if you ever end up creating and deploying multiple CA certificates with the same distinguished name (common if you make a mistake and need to start over); conflicts will be avoided because the certificates will have different serial numbers.
Add a crlnumber
file to the Intermediate CA directory tree. crlnumber
is used to keep track of certificate revocation lists.
In our current example, the intermediate
folder is located within /root/ca
for simplicity and ease of demonstration. However, it's important to underscore that this setup is not recommended for real-world applications. In practice, maintaining the Root CA and Intermediate CA on separate machines is a critical security measure.
Preparing OpenSSL Configuration Files
You must create configuration files for OpenSSL to use. Before we can create a CA, we need to prepare configuration files ca-openssl.cnf
and intermediate-openssl.cnf
that will tell OpenSSL exactly how we want things set up.
Create the OpenSSL Root CA Configuration File
Copy the following configuration into the OpenSSL Root CA configuration file. You must modify the components of the CA’s distinguished name. Change the [
req_distinguished_name
] section as per your company information.
Create the OpenSSL Intermediate CA Configuration File
Copy the following configuration into the OpenSSL Intermediate CA configuration file. Here again, you must modify the components of the CA’s distinguished name. Change the [
req_distinguished_name
] section as per your company information.
The initial arrangement of the folder structure, once all necessary files are in place, will appear as follows:
Create the Root Certificate
Acting as a certificate authority (CA) means dealing with cryptographic pairs of private keys and public certificates. The very first cryptographic pair we’ll create is the root pair. This consists of the root key (ca.key.pem
) and root certificate (ca.cert.pem
). This pair forms the identity of our CA.
First, we must create a Private Key and Root CA certificate.
Change the current directory to the Root CA directory.
Generate an RSA private key (
ca.key.pem
).
NOTE: You must enter the passphrase/password for ca.key.pem
. The parameter -passout pass:changeme
sets the password to encrypt the key. In this case, the password is changeme
. Replacing changeme
with a strong, unique password in a real-world scenario is important. As you progress through this guide, you will encounter various parameters like -passout
and -passin
. Please make sure to replace these with the password of your choice.
Ensure that only authorized users have the necessary access to the root key.
Use the root key (ca.key.pem
) to create a root certificate (ca.cert.pem
). Give the root certificate a long expiry date, such as ten years. Once the root certificate expires, all certificates signed by the CA become invalid.
Secure root certificate by making it universally readable while preventing any modifications.
NOTE: Whenever you use the req
tool, you must specify a configuration file with the -config
option; otherwise, OpenSSL will use the default configuration file /etc/ssl/openssl.cnf
.
Verify the root certificate.
The output shows:
the
Signature Algorithm
used (sha256WithRSAEncryption)the dates of certificate
Validity
the
Public-Key
bit length (4096 bit)the
Issuer
, which is the entity that signed the certificate (C = DE, ST = Bavaria, L = Landshut, O = HiveMQ, OU = Customer Services, CN = HiveMQ Root CA, emailAddress = email@hivemq.com)the
Subject
, which refers to the certificate itself
The Issuer
and Subject
are identical as the certificate is self-signed. Note that all root certificates are self-signed.
The output also shows the X509v3 extensions. We applied the v3_ca
extension, so the options from [ v3_ca
] should be reflected in the output.
Create the Intermediate Pair
As before, we take two steps to create the Intermediate CA. First, we create a Private Key and Intermediate CA certificate. Make sure to use a strong password.
Change the current directory to the Root CA directory.
Generate an RSA private key (
intermediate.key.pem
).
Ensure that only authorized users have the necessary access to the intermediate key.
Create the intermediate certificate. Use the intermediate key to create a certificate signing request (CSR). The details should generally match the Root CA. The Common Name, however, must be different.
Ensure you specify the Intermediate CA configuration file (intermediate/intermediate-openssl.cnf
).
To create an intermediate certificate, use the Root CA with the v3_intermediate_ca
extension to sign the intermediate CSR. The intermediate certificate should be valid for a shorter period than the root certificate. Ten years would be reasonable.
This time, specify the Root CA configuration file (/root/ca/ca-openssl.cnf
).
Secure intermediate certificate by making it universally readable while preventing any modifications.
The index.txt
file is where the OpenSSL CA tool stores the certificate database. Do not delete or edit this file by hand.
Now the
index.txt
file should contain only one line that refers to the intermediate certificate.
Output:
Each line contains six values separated by tabs:
Status flag (V for valid, R for revoked, E for expired)
Expiration date (in YYMMDDHHMMSSZ format)
Revocation date or empty if not revoked
Serial number (hexadecimal)
File location or unknown if not known
Distinguished name
As we did for the root certificate, check if the details of the intermediate certificate are correct.
Verify the intermediate certificate.
Verify the intermediate certificate against the root certificate. An OK indicates that the chain of trust is intact.
If Root and Intermediate certificates are OK, it is time to create the certificate chain. When an application tries to verify a certificate signed by the Intermediate CA, it must also verify the intermediate certificate against the root certificate. To complete the chain of trust, create a CA certificate chain to present to the application.
To create the CA certificate chain, concatenate the intermediate and root certificates together. We will use this file later to verify certificates signed by the Intermediate CA.
Secure the CA certificate chain by making it universally readable while preventing any modifications.
NOTE: The certificate chain file must include the root certificate because no client application knows about it yet. A better option is to install your root certificate on every client that needs to connect. In that case, the chain file need only contain your intermediate certificate.
Server Certificate (sign server)
Next, we will create a certificate and key for our server/broker, sign it, and generate the Keystore to be used by HiveMQ.
We will be signing certificates using our Intermediate CA. We will use these signed certificates to authenticate clients connecting to HiveMQ Broker.
Our root and intermediate pairs are 4096 bits. Server and client certificates normally expire after one year, so we can safely use 2048 bits instead. Although 4096 bits are slightly more secure than 2048 bits, they slow down TLS handshakes and significantly increase processor load during handshakes. For this reason, we will use 2048-bit pairs.
In the following examples, you must replace broker1.hivemq.com
/ broker2.hivemq.com
/ broker3.hivemq.com
with the FQDN of the individual nodes you are creating these for.
Generate the server’s private key, and use the private key to create a certificate signing request (CSR). The CSR details don’t need to match the Intermediate CA.
Change the current directory to the Root CA directory.
Generate an RSA private key (
broker.key.pem
).
Ensure that only authorized users have the necessary access to the server/broker key.
When configuring server and client certificates, it's essential to understand that the Common Name (CN) must be distinct from the names used in your Root or Intermediate certificates. To enhance versatility, we incorporate the Subject Alternative Name (SAN) into the server/broker certificate. This allows the certificate to remain valid for other broker nodes' Fully Qualified Domain Names (FQDNs). This strategy enables using a single certificate across multiple CNs, simplifying certificate management. It's important to note that when a certificate includes a Subject Alternative Name, the Common Name(s) are no longer considered, as SAN takes precedence.
An additional benefit of using Subject Alternative Name (SAN) in server certificates is the ability to include multiple domain names (DNS) under a single Common Name (For example, broker1.hivemq.com and broker1.hivemq.cloud)
Sign the server’s key and generate its certificate. To create a certificate, use the Intermediate CA to sign the CSR. Since the certificate will be used on a server, use the
server_cert
extension. Certificates are usually given a validity of one year.
Secure the server/broker certificate by making it universally readable while preventing any modifications.
Verify server/broker certificate.
Use the CA certificate chain file we created earlier (
ca-chain.cert.pem
) to verify that the new server certificate has a valid chain of trust.
We now have all the necessary parts to produce a Server Keystore.
Concatenate the certificate chain, be careful here to avoid "key values mismatch” error. Create the server certificate chain concatenated in the right order: SERVER CERTIFICATE > INTERMEDIATE CA CERTIFICATE > ROOT CA CERTIFICATE
Import the server/broker certificate chain and the private key into a PKCS12 container.
Import the contents of the PKCS12 container into a JKS container.
Verify the Broker Keystore.
Remove the PKCS12 container and the server concatenated certificate.
Visualizing certificate hierarchy in KeyStore Explorer and the SAN extension.
At this point, we can securely connect to the broker using the server/broker certificate. You can test by adding the following TLS listener configuration in the config.xml
file.
Testing Server/Broker TLS connection by using broker certificate file with HiveMQ MQTT CLI.
Now let's configure your cluster transport to use TLS as well. Because we use the same certificate for all of our HiveMQ Broker nodes, we set the same hivemq-keystore.jks
as our Cluster Server Keystore and Server Truststore.
NOTE: Note that this step is applicable only for deploying a cluster setup. If your deployment involves a single HiveMQ Broker, this step can be omitted as it is not necessary.
Client Certificates (sign client)
Now we can start creating certificates that our clients can present to the server while establishing a connection. You may replace mqtt-client with any desired name.
As before, our starting point is to generate the client's private key.
Change the current directory to the Root CA directory.
Generate an RSA private key (
mqtt-client.key.pem
).
Ensure that only authorized users have the necessary access to the client key.
Use the private key to create a certificate signing request (CSR). The CSR details don’t need to match the Intermediate CA. For client certificates, Common Name can be any unique identifier (eg. a client name).
Sign the client’s key and generate its certificate. To create a certificate, use the Intermediate CA to sign the CSR. Since the certificate will be used for client authentication, use the client_cert
extension. Certificates are usually given a validity of one year, though a CA will typically give a few days extra for convenience.
Secure the client certificate by making it universally readable while preventing any modifications.
Verify MQTT client certificate.
Use the CA certificate chain file we created earlier (
ca-chain.cert.pem
) to verify that the new client certificate has a valid chain of trust.
Time to Generate the Client’s Truststore
Concatenate the certificate chain, but be careful here to avoid "key values mismatch” error. Create the client certificate chain concatenated in the right order: CLIENT CERTIFICATE > INTERMEDIATE CA CERTIFICATE > ROOT CA CERTIFICATE
Import the client certificate chain and the private key into a PKCS12 container.
Import the contents of the PKCS12 container into a JKS container.
Given that clients present the entire certificate chain on connection, you only need to include the trusted Intermediate CA certificate in the HiveMQ Client Truststore.
Verify the MQTT Client Truststore.
Remove the PKCS12 container and the concatenated certificate.
At this point, we can set the communication between the client and server using mutual TLS encryption (use of both client and server certificate). You can test by adding the following TLS listener configuration in config.xml
file.
Notice that we set the <client-authentication-mode>
to REQUIRED. This setting ensures that only MQTT clients that use a certificate that is part of the configured trust store are allowed to connect.
Testing Mutual TLS connection by using certificates files with HiveMQ MQTT CLI
Here, we need to enter our mqtt-client.key.pem
private key password to access the key.
Testing Mutual TLS connection by using Keystore and Truststores files with HiveMQ MQTT CLI:
The MQTT CLI offers support through the direct usage of Keystores and Truststores files (.jks), enabling Mutual TLS connections to the broker, refer to the example provided below. Here, we need to enter the KeyStore and TrustStore passwords.
This demonstrates the correct usage of Certificates, KeyStore and TrustStore with the HiveMQ MQTT CLI.
Software/Tool Versions and Operating System Used in This Guide
HiveMQ 4.23
MQTT CLI 4.23
OpenSSL 3.0.2
KeyStore Explorer 5.5.3
Ubuntu 22.04.3 LTS
Conclusion
Securing HiveMQ Broker deployments with Intermediate CA Certificates is a fundamental step towards fortifying the trust infrastructure of your digital ecosystem. By adopting a hierarchical approach and minimizing the usage of the root key, this guide empowers you to establish a robust security framework that safeguards the integrity and confidentiality of your MQTT communications.
Diego Duarte
Diego Duarte is a Senior Support Engineer at HiveMQ. Over the years, he has collaborated with many companies worldwide. He is passionate about problem-solving, innovative solutions, and troubleshooting complex technical challenges. A smart home aficionado and tech enthusiast at heart, you'll often find Diego actively participating in various communities and channels dedicated to IoT and home automation, sharing his expertise and learning from fellow enthusiasts.