How to Build a File-based Protocol Adapter for HiveMQ Edge
With the recent release of HiveMQ Edge 2024.5, we made the Protocol Adapter SDK (Java Doc) publicly available to enable users to implement their protocol adapters. The Protocol Adapter SDK allows users to develop custom protocol adapters to resolve very specific and custom needs while still benefiting from the available features in HiveMQ Edge. This blog post demonstrates how to implement and use a new protocol adapter for HiveMQ Edge.
To do so, this post presents a sample file-based protocol adapter in a simple form that reads file content at a certain interval and publishes it as an MQTT message.
Once you have built your new protocol adapter and think we should integrate it into HiveMQ Edge’s protocol adapter catalog, please drop us a message.
NOTE: There will be a file-based protocol adapter soon available in HiveMQ Edge — an implementation distinct from the one described in this post.
Introduction to the Protocol Adapter
The HelloWorldProtocolAdapter
is used as a basis, which you can find at GitHub. It serves as a template for an initial implementation since it defines a common structure.
In the following, a walkthrough based on the Hello World example is described. The requirements for the file-based protocol adapter are the following:
A file located on the same file system as HiveMQ Edge must be read.
The file is read based on a configurable interval.
The content of the file is encoded as base64 and published as a common JSON-format.
Overview of Building the Protocol Adapter
The blog post is structured in the following steps:
Download the
HelloWorldProtocolAdapter
repository from GitHub.Open the project in an IDE and setup the project.
Check your setup by creating a jar from the repository.
Implement a simple polling logic from a file.
Try out the simple polling logic by adding it to HiveMQ Edge.
Expand the configuration to include a file path.
Update the polling logic and write tests for it.
Try out the improved polling logic by adding it to HiveMQ Edge.
Update the information on the adapter and rename classes.
Prerequisites
IDE of your choice
Java 11 or newer installed
For Testing: MQTT CLI or similar
File-based Protocol Adapter
The HelloWorldAdapter
used as the basis for this implementation is a simple but fully functional protocol adapter. We will incorporate logic to read from a file and encode the content in base64 encoding. This will ensure the file can contain any arbitrary binary format and still be represented as a string in JSON format.
To prevent potential memory issues, we will first check the file size against a specified limit. Subsequently, we will create a JAR file for our protocol adapter and install it on a local HiveMQ Edge deployment to verify that it functions as intended. Finally, we will show how to write automated tests for a protocol adapter.
How to Get Started
A hello world example can be used to easily bootstrap a new protocol adapter. This adapter can be either downloaded from GitHub or cloned via the following command into your local environment:
Open the Project in an IDE and Set Up a New Project
After downloading the repository, the next step is to open it in an IDE of your choice.
For this blog post, we use IntelliJ IDEA. You should see a similar environment as shown in the screenshot below.
Check Your Setup
You will likely need to select the Java SDK (at minimum, version 11) that you want to use to compile the project. The selected version should be the one you intend HiveMQ Edge to start with.
You also need to set up the build tool Gradle. Most IDEs recognize the build file build.gradle.kts
file automatically and open the project as a Gradle project or will suggest doing so. As a first step, we recommend validating whether your development environment correctly builds a protocol adapter JAR file. You may run the Gradle task shadowJar to build a JAR file. This task automatically downloads all dependencies of the projects, compiles them, and produces an artifact. The final JAR is located under the directory build/libs
.
You can also manually execute the tasks on the console in your project directory by running
./gradlew shadowJar
Eventually, a JAR file should be generated.
For testing, copy the JAR into HiveMQ’s Edge modules folder and start HiveMQ Edge. The newly built protocol adapter should be shown below.
If you have any issues, please check the log file of your HiveMQ Edge instance.
Structure of a Protocol Adapter
The following explains the most important Java classes for a Protocol Adapter. These files are located in:
hivemq-hello-world-protocol-adapter/src/main/java/com/hivemq/edge/adapters/helloworld/config
ProtocolAdapterFactory: This is the first class loaded by HiveMQ Edge to get the necessary information to create an instance of the protocol adapter and the information on it.
ProtocolAdapterInformation: This class contains all information about the adapter, such as its name, protocol, author, etc. HiveMQ Edge obtains an instance of this class via the ProtocolAdapterFactory.
PollingProtocolAdapter: The actual implementation of the protocol adapter, which handles the polling of data samples. The ProtocolAdapterFactory provides a means to construct an instance of this class.
ProtocolAdapterConfig: The implementation of this interface contains the configuration options for the protocol adapter. The config is parsed by HiveMQ Edge automatically and uses the Jackson annotations for mappings.
Polling from a File
As a next step, we want to show how to implement the basic file-based protocol adapter requirements stated in the “Introduction to the Protocol Adapter” section above. We start implementing the polling logic for a file-based protocol adapter.
As of now, we have no configuration available for the actual path to the file, so we just place the absolute path to the file as a constant value to keep it simple. This is good enough to test the file reading and provides the logic to add the content as the output of the polling process.
The polling process is in the HelloWorldProtocolAdapter
in the poll method:
The poll
function adds constant data points to the pollingOutput
data structure, which is later handled by HiveMQ Edge to publish to a destination topic.
For the file-based protocol adapter, we need to make the following changes:
Create a
String
containing the absolute path to the file.Check the size of the file against a predefined limit.
Load the content of the file as bytes and encode it to a Base64 string.
Add the content of the file to the output as a data point.
Tell HiveMQ Edge’s SDK that the polling is finished.
Handle exceptions by failing the output and returning from the method.
Note that the example reads a static file from path /tmp/sensor1.txt,
which may not work on Windows. Please use a different filename according to your environment.
Testing the Implementation
As we already introduced in the Check Your Setup section, we need to compile the source code into a JAR file by executing the shadowJar
gradle-task. Also copy the JAR into your HiveMQ deployment and restart HiveMQ Edge. Next, create a Protocol Adapter instance and check whether the first sample has been created using HiveMQ Edge’s Event Log, as shown in the screenshot. Remember — the HiveMQ Edge UI is accessible via http://localhost:8080 as the default configuration.
Next, you can also use an MQTT Client to subscribe and check whether the sample is correctly published. In this example, we use MQTT CLI as follows:
The output of the tools looks like this:
The value is base64 encoded and corresponds to “HiveMQ Edge.”
Make File Path Configurable
The current state reads the content from a statically defined file. We want to improve this to let the user configure the file's path, adding a configuration item to a protocol adapter.
In the poll method, we use the new configuration for the file path:
If you compile the project and copy it again into the HiveMQ Edge’s folder, you can see the file path down below in the subscriptions tab:
Unit Test
To ensure proper functionality, a unit test must be written and executed. In the following example, we use the test frameworks JUnit and Mockito:
UI Schema
The adapter created so far will integrate with Edge literally out-of-the-box. In particular, the JSON annotations will be used to generate a JSON Schema. This, in turn, is used by the UI to create and handle the configuration form. A default rendering of the properties is offered, and it might be the end of your concerns.
However, you can impact some aspects of this rendering's customization by defining a UiSchema specification.
It’s a JSON document whose content must abide by some rules regarding its structure and keywords. The full extent of the configuration can be explored in the documentation. Below is an example of UI configuration for our adapter.
Worth noting:
The
ui:tabs
section allows you to group properties that are originally under the root of the document, resulting in horizontally organized tabs on the UI.The
ui:batchMode
flag for thesubscriptions
property indicates that the adapter is able to support batch subscriptions import.
Final Tests
After the unit test is successful, we want to test the advanced polling logic with HiveMQ Edge on a real setup. The UI schema configuration is located under src/resources/helloworld-adapter-ui-schema.json
and can be tweaked to your preferences.
To test the final implementation, compile the latest stage, copy the JAR file into the modules folder, delete the config for the protocol adapter, and restart HiveMQ Edge. Congrats — you’ve just built a new protocol adapter!
Finalization
Eventually, you probably want to give the protocol adapter a proper name. Update the HelloWorldProtocolAdapterInformation
class.
Conclusion
In this blog post, we’ve guided you through how to build a protocol adapter for HiveMQ Edge. We took the HelloWorldProtocolAdapter and implemented a file-based protocol adapter that reads content from a file and publishes it as base64 encoded JSON format.
Enable interoperability between OT and IT systems by translating various protocols into the standardized MQTT format with HiveMQ Edge. Modernize IIoT infrastructure and achieve seamless edge-to-cloud integration with this software-based Edge MQTT gateway.
Stefan Frehse
Stefan Frehse is Senior Engineering Manager at HiveMQ. He earned a Ph.D. in Computer Science from the University of Bremen and has worked in software engineering and in c-level management positions for 10 years. He has written many academic papers and spoken on topics including formal verification of fault tolerant systems, debugging and synthesis of reversible logic.