How to Build a File-based Protocol Adapter for HiveMQ Edge
With the recent release of HiveMQ Edge 2024.5, we made the Protocol Adapter SDK (Java Doc) publicly available to enable users to implement their protocol adapters. The Protocol Adapter SDK allows users to develop custom protocol adapters to resolve very specific and custom needs while still benefiting from the available features in HiveMQ Edge. This blog post demonstrates how to implement and use a new protocol adapter for HiveMQ Edge.
To do so, this post presents a sample file-based protocol adapter in a simple form that reads file content at a certain interval and publishes it as an MQTT message.
Once you have built your new protocol adapter and think we should integrate it into HiveMQ Edge’s protocol adapter catalog, please drop us a message.
NOTE: There will be a file-based protocol adapter soon available in HiveMQ Edge — an implementation distinct from the one described in this post.
Introduction to the Protocol Adapter
The HelloWorldProtocolAdapter
is used as a basis, which you can find at GitHub. It serves as a template for an initial implementation since it defines a common structure.
In the following, a walkthrough based on the Hello World example is described. The requirements for the file-based protocol adapter are the following:
A file located on the same file system as HiveMQ Edge must be read.
The file is read based on a configurable interval.
The content of the file is encoded as base64 and published as a common JSON-format.
Overview of Building the Protocol Adapter
The blog post is structured in the following steps:
Download the
HelloWorldProtocolAdapter
repository from GitHub.Open the project in an IDE and setup the project.
Check your setup by creating a jar from the repository.
Implement a simple polling logic from a file.
Try out the simple polling logic by adding it to HiveMQ Edge.
Expand the configuration to include a file path.
Update the polling logic and write tests for it.
Try out the improved polling logic by adding it to HiveMQ Edge.
Update the information on the adapter and rename classes.
Prerequisites
IDE of your choice
Java 11 or newer installed
For Testing: MQTT CLI or similar
File-based Protocol Adapter
The HelloWorldAdapter
used as the basis for this implementation is a simple but fully functional protocol adapter. We will incorporate logic to read from a file and encode the content in base64 encoding. This will ensure the file can contain any arbitrary binary format and still be represented as a string in JSON format.
To prevent potential memory issues, we will first check the file size against a specified limit. Subsequently, we will create a JAR file for our protocol adapter and install it on a local HiveMQ Edge deployment to verify that it functions as intended. Finally, we will show how to write automated tests for a protocol adapter.
How to Get Started
A hello world example can be used to easily bootstrap a new protocol adapter. This adapter can be either downloaded from GitHub or cloned via the following command into your local environment:
git clone https://github.com/hivemq/hivemq-hello-world-protocol-adapter
Open the Project in an IDE and Set Up a New Project
After downloading the repository, the next step is to open it in an IDE of your choice.
For this blog post, we use IntelliJ IDEA. You should see a similar environment as shown in the screenshot below.
Check Your Setup
You will likely need to select the Java SDK (at minimum, version 11) that you want to use to compile the project. The selected version should be the one you intend HiveMQ Edge to start with.
You also need to set up the build tool Gradle. Most IDEs recognize the build file build.gradle.kts
file automatically and open the project as a Gradle project or will suggest doing so. As a first step, we recommend validating whether your development environment correctly builds a protocol adapter JAR file. You may run the Gradle task shadowJar to build a JAR file. This task automatically downloads all dependencies of the projects, compiles them, and produces an artifact. The final JAR is located under the directory build/libs
.
You can also manually execute the tasks on the console in your project directory by running
./gradlew shadowJar
Eventually, a JAR file should be generated.
For testing, copy the JAR into HiveMQ’s Edge modules folder and start HiveMQ Edge. The newly built protocol adapter should be shown below.
If you have any issues, please check the log file of your HiveMQ Edge instance.
Structure of a Protocol Adapter
The following explains the most important Java classes for a Protocol Adapter. These files are located in:
hivemq-hello-world-protocol-adapter/src/main/java/com/hivemq/edge/adapters/helloworld/config
ProtocolAdapterFactory: This is the first class loaded by HiveMQ Edge to get the necessary information to create an instance of the protocol adapter and the information on it.
ProtocolAdapterInformation: This class contains all information about the adapter, such as its name, protocol, author, etc. HiveMQ Edge obtains an instance of this class via the ProtocolAdapterFactory.
PollingProtocolAdapter: The actual implementation of the protocol adapter, which handles the polling of data samples. The ProtocolAdapterFactory provides a means to construct an instance of this class.
ProtocolAdapterConfig: The implementation of this interface contains the configuration options for the protocol adapter. The config is parsed by HiveMQ Edge automatically and uses the Jackson annotations for mappings.
Polling from a File
As a next step, we want to show how to implement the basic file-based protocol adapter requirements stated in the “Introduction to the Protocol Adapter” section above. We start implementing the polling logic for a file-based protocol adapter.
As of now, we have no configuration available for the actual path to the file, so we just place the absolute path to the file as a constant value to keep it simple. This is good enough to test the file reading and provides the logic to add the content as the output of the polling process.
The polling process is in the HelloWorldProtocolAdapter
in the poll method:
@Override
public void poll(@NotNull PollingInput pollingInput, @NotNull PollingOutput pollingOutput) {
// here the sampling must be done. F.e. sending a http request
pollingOutput.addDataPoint("dataPoint1", 42);
pollingOutput.addDataPoint("dataPoint1", 1337);
pollingOutput.finish();
}
The poll
function adds constant data points to the pollingOutput
data structure, which is later handled by HiveMQ Edge to publish to a destination topic.
For the file-based protocol adapter, we need to make the following changes:
Create a
String
containing the absolute path to the file.Check the size of the file against a predefined limit.
Load the content of the file as bytes and encode it to a Base64 string.
Add the content of the file to the output as a data point.
Tell HiveMQ Edge’s SDK that the polling is finished.
Handle exceptions by failing the output and returning from the method.
@Override
public void poll(final @NotNull PollingInput<HelloWorldPollingContext> pollingInput, final @NotNull PollingOutput pollingOutput) {
// absolute path to the file that contains the data. Magic string for now. Later it will be part of the config
final String absolutePathToFle = "/tmp/sensor1.txt";
try {
final Path path = Path.of(absolutePathToFle);
final long length = path.toFile().length();
final int limit = 64_000; // not a constant to have a more compact code example
if (length > limit) {
pollingOutput.fail(String.format("File '%s' of size '%d' exceeds the limit '%d'.", path.toAbsolutePath(), length, limit));
return;
}
// load the content of the file
byte[] fileContent = Files.readAllBytes(path);
// encode it as base64
final String encodedFileContent = Base64.getEncoder().encodeToString(fileContent);
// add the content of the file to the output
pollingOutput.addDataPoint("value", encodedFileContent);
} catch (IOException e) {
// in case something goes wrong while reading the file, an IOException will be thrown.
// we handle it by failing the poll process and returning from the poll method.
pollingOutput.fail(e, "An exception occurred while reading the file '" + absolutePathToFle + "'.");
return;
}
// we need to tell edge that the polling is done as edge also supports asynchronous polling.
pollingOutput.finish();
}
Note that the example reads a static file from path /tmp/sensor1.txt,
which may not work on Windows. Please use a different filename according to your environment.
Testing the Implementation
As we already introduced in the Check Your Setup section, we need to compile the source code into a JAR file by executing the shadowJar
gradle-task. Also copy the JAR into your HiveMQ deployment and restart HiveMQ Edge. Next, create a Protocol Adapter instance and check whether the first sample has been created using HiveMQ Edge’s Event Log, as shown in the screenshot. Remember — the HiveMQ Edge UI is accessible via http://localhost:8080 as the default configuration.
Next, you can also use an MQTT Client to subscribe and check whether the sample is correctly published. In this example, we use MQTT CLI as follows:
mqtt sub -t '#'
The output of the tools looks like this:
{
"timestamp" : 1719385976555,
"value" : "SGl2ZU1RIEVkZ2UK"
}
{
"timestamp" : 1719385977420,
"value" : "SGl2ZU1RIEVkZ2UK"
}
{
"timestamp" : 1719385977555,
"value" : "SGl2ZU1RIEVkZ2UK"
}
The value is base64 encoded and corresponds to “HiveMQ Edge.”
Make File Path Configurable
The current state reads the content from a statically defined file. We want to improve this to let the user configure the file's path, adding a configuration item to a protocol adapter.
@JsonProperty(value = "filePath", required = true)
@ModuleConfigField(title = "The file path",
description = "The path to the file that should be scraped.",
required = true)
protected @NotNull String filePath;
@JsonCreator
public HelloWorldPollingContext(
@JsonProperty("destination") @Nullable final String destination,
@JsonProperty("qos") final int qos,
@JsonProperty("userProperties") @Nullable List<UserProperty> userProperties,
@JsonProperty("filePath") @NotNull String filePath) {
this.destination = destination;
this.qos = qos;
if (userProperties != null) {
this.userProperties = userProperties;
}
this.filePath = filePath;
}
public @NotNull String getFilePath() {
return filePath;
}
In the poll method, we use the new configuration for the file path:
@Override
public void poll(final @NotNull PollingInput<HelloWorldPollingContext> pollingInput, final @NotNull PollingOutput pollingOutput) {
// absolute path to the file that contains the data
final String absolutePathToFle = pollingInput.getPollingContext().getFilePath();
try {
final Path path = Path.of(absolutePathToFle);
final long length = path.toFile().length();
final int limit = 64_000; // not a constant to have a more compact code example
if (length > limit) {
pollingOutput.fail(String.format("File '%s' of size '%d' exceeds the limit '%d'.", path.toAbsolutePath(), length, limit));
return;
}
// load the content of the file
byte[] fileContent = Files.readAllBytes(path);
// encode it as base64
final String encodedFileContent = Base64.getEncoder().encodeToString(fileContent);
// add the content of the file to the output
pollingOutput.addDataPoint("value", encodedFileContent);
} catch (IOException e) {
// in case something goes wrong while reading the file, an IOException will be thrown.
// we handle it by failing the poll process and returning from the poll method.
pollingOutput.fail(e, "An exception occurred while reading the file '" + absolutePathToFle + "'.");
return;
}
// we need to tell edge that the polling is done as edge also supports asynchronous polling.
pollingOutput.finish();
}
If you compile the project and copy it again into the HiveMQ Edge’s folder, you can see the file path down below in the subscriptions tab:
Unit Test
To ensure proper functionality, a unit test must be written and executed. In the following example, we use the test frameworks JUnit and Mockito:
@TempDir
@NotNull File temporaryDir; // @TempDir creates a temporary folder which will automatically be removed after the test.
// we mock these objects to easily control their behavior
private final @NotNull ProtocolAdapterInput<HelloWorldAdapterConfig> adapterInput = mock();
private final @NotNull HelloWorldAdapterConfig config = mock();
private final @NotNull PollingInput<HelloWorldPollingContext> pollingInput = mock();
@Test
void test_poll_whenFileIsPresent_thenFileContentsAreSetInOutput() throws IOException {
final File fileWithData = new File(temporaryDir, "data.txt"); // create a temporary file which content gets polled
Files.write(fileWithData.toPath(), "Hello World".getBytes(StandardCharsets.UTF_8)); // write "Hello World" into the file
when(adapterInput.getConfig()).thenReturn(config); // whenever the getConfig() is called on the input object, our mocked config is returned
when(pollingInput.getPollingContext()).thenReturn(new HelloWorldPollingContext("mqttTopic", 1, List.of(), fileWithData.getAbsolutePath()));
// whenever the pollContext is accessed on the pollingInput, a crafted context is returned, so that our file is polled and the correct mqtt topic is set
TestPollingOutput pollingOutput = new TestPollingOutput(); // test implementation for a pollingOutput
HelloWorldPollingProtocolAdapter adapter = new HelloWorldPollingProtocolAdapter(new HelloWorldProtocolAdapterInformation(), adapterInput);
// we create the adapter with the setup we want for the test
adapter.poll(pollingInput, pollingOutput); // make the adapter poll the file and insert the data into the output object.
// No further sync is needed here as the poll method is not asynchronous in the example
final Object value = pollingOutput.getDataPoints().get("value"); // get the data point from the output
assertNotNull(value); // check that there is a value present
assertTrue(value instanceof String); // check that it has the expected type String
String valueAsString = (String) value; // cast it to String
final byte[] decodedBytes = Base64.getDecoder().decode(valueAsString); // decode it as the value is encoded as Base64
final String actualDecodedValue = new String(decodedBytes, StandardCharsets.UTF_8); // create a UTF-8 String of it
assertEquals("Hello World", actualDecodedValue); // check that the string is correct
}
UI Schema
The adapter created so far will integrate with Edge literally out-of-the-box. In particular, the JSON annotations will be used to generate a JSON Schema. This, in turn, is used by the UI to create and handle the configuration form. A default rendering of the properties is offered, and it might be the end of your concerns.
However, you can impact some aspects of this rendering's customization by defining a UiSchema specification.
It’s a JSON document whose content must abide by some rules regarding its structure and keywords. The full extent of the configuration can be explored in the documentation. Below is an example of UI configuration for our adapter.
{
"ui:tabs": [
{
"id": "coreFields",
"title": "Core Fields",
"properties": [
"id"
]
},
{
"id": "subFields",
"title": "Subscription",
"properties": [
"subscriptions"
]
},
{
"id": "publishing",
"title": "protocolAdapter.uiSchema.groups.publishing",
"properties": [
"maxPollingErrorsBeforeRemoval",
"publishChangedDataOnly",
"pollingIntervalMillis"
]
}
],
"subscriptions": {
"ui:batchMode": true,
"items": {
"ui:order": [
"destination",
"*",
"userProperties"
],
"ui:collapsable": {
"titleKey": "destination"
}
}
}
}
Worth noting:
The
ui:tabs
section allows you to group properties that are originally under the root of the document, resulting in horizontally organized tabs on the UI.The
ui:batchMode
flag for thesubscriptions
property indicates that the adapter is able to support batch subscriptions import.
Final Tests
After the unit test is successful, we want to test the advanced polling logic with HiveMQ Edge on a real setup. The UI schema configuration is located under src/resources/helloworld-adapter-ui-schema.json
and can be tweaked to your preferences.
To test the final implementation, compile the latest stage, copy the JAR file into the modules folder, delete the config for the protocol adapter, and restart HiveMQ Edge. Congrats — you’ve just built a new protocol adapter!
Finalization
Eventually, you probably want to give the protocol adapter a proper name. Update the HelloWorldProtocolAdapterInformation
class.
public class HelloWorldProtocolAdapterInformation
implements ProtocolAdapterInformation {
public static final @NotNull ProtocolAdapterInformation INSTANCE = new HelloWorldProtocolAdapterInformation();
protected HelloWorldProtocolAdapterInformation() {
}
@Override
public @NotNull String getProtocolName() {
// the returned string will be used for logging information on the protocol adapter
return "PollingFileProtocol";
}
@Override
public @NotNull String getProtocolId() {
// this id is very important as this is how the adapters configurations in the config.xml are linked to the adapter implementations.
// any change here means you will need to edit the config.xml
return "Polling_File_Protocol";
}
@Override
public @NotNull String getDisplayName() {
// the name for this protocol adapter type that will be displayed within edge's ui
return "Polling File Protocol Adapter";
}
@Override
public @NotNull String getDescription() {
// the description that will be shown for this protocol adapter within edge's ui
return "This Protocol Adapter periodically polls and publishes the content of a given file.";
}
@Override
public @NotNull String getUrl() {
// this url will be displayed in the ui as a link to further documentation on this protocol adapter.
// e.g. this could be a link to the source code and a readme
return "https://www.hivemq.com/";
}
@Override
public @NotNull String getVersion() {
// the version of this protocol adapter, the usage of semantic versioning is advised.
return "0.1.0";
}
@Override
public @NotNull EnumSet<ProtocolAdapterCapability> getCapabilities() {
// this indicates what capabilities this protocol adapter has. E.g. READ/WRITE. See the ProtocolAdapterCapability enum for more information.
return EnumSet.of(ProtocolAdapterCapability.READ);
}
@Override
public @NotNull String getLogoUrl() {
// this is a default image that is always available.
return "/images/helloWorld.png";
}
@Override
public @NotNull String getAuthor() {
return "HiveMQ";
}
@Override
public @Nullable ProtocolAdapterCategory getCategory() {
// this indicates for which use cases this protocol adapter is intended. See the ProtocolAdapterConstants.CATEGORY enum for more information.
return ProtocolAdapterCategory.CONNECTIVITY;
}
@Override
public List<ProtocolAdapterTag> getTags() {
// here you can set which Tags should be applied to this protocol adapter
return List.of(ProtocolAdapterTag.AUTOMATION);
}
}
Conclusion
In this blog post, we’ve guided you through how to build a protocol adapter for HiveMQ Edge. We took the HelloWorldProtocolAdapter and implemented a file-based protocol adapter that reads content from a file and publishes it as base64 encoded JSON format.
Enable interoperability between OT and IT systems by translating various protocols into the standardized MQTT format with HiveMQ Edge. Modernize IIoT infrastructure and achieve seamless edge-to-cloud integration with this software-based Edge MQTT gateway.

Stefan Frehse
Stefan Frehse is Senior Engineering Manager at HiveMQ. He earned a Ph.D. in Computer Science from the University of Bremen and has worked in software engineering and in c-level management positions for 10 years. He has written many academic papers and spoken on topics including formal verification of fault tolerant systems, debugging and synthesis of reversible logic.
