Skip to content

How to Build a File-based Protocol Adapter for HiveMQ Edge

by Stefan Frehse, Daniel Krüger, Nicolas Van Labeke
17 min read

With the recent release of HiveMQ Edge 2024.5, we made the Protocol Adapter SDK (Java Doc) publicly available to enable users to implement their protocol adapters. The Protocol Adapter SDK allows users to develop custom protocol adapters to resolve very specific and custom needs while still benefiting from the available features in HiveMQ Edge. This blog post demonstrates how to implement and use a new protocol adapter for HiveMQ Edge.

To do so, this post presents a sample file-based protocol adapter in a simple form that reads file content at a certain interval and publishes it as an MQTT message. 

Once you have built your new protocol adapter and think we should integrate it into HiveMQ Edge’s protocol adapter catalog, please drop us a message.

NOTE: There will be a file-based protocol adapter soon available in HiveMQ Edge —  an implementation distinct from the one described in this post. 

Introduction to the Protocol Adapter

The HelloWorldProtocolAdapter is used as a basis, which you can find at GitHub. It serves as a template for an initial implementation since it defines a common structure. 

In the following, a walkthrough based on the Hello World example is described. The requirements for the file-based protocol adapter are the following:

  • A file located on the same file system as HiveMQ Edge must be read.

  • The file is read based on a configurable interval. 

  • The content of the file is encoded as base64 and published as a common JSON-format.

Overview of Building the Protocol Adapter

The blog post is structured in the following steps:

  1. Download the HelloWorldProtocolAdapter repository from GitHub.

  2. Open the project in an IDE and setup the project.

  3. Check your setup by creating a jar from the repository.

  4. Implement a simple polling logic from a file. 

  5. Try out the simple polling logic by adding it to HiveMQ Edge.

  6. Expand the configuration to include a file path.

  7. Update the polling logic and write tests for it.

  8. Try out the improved polling logic by adding it to HiveMQ Edge.

  9. Update the information on the adapter and rename classes.

Prerequisites

  1. IDE of your choice

  2. Java 11 or newer installed

  3. For Testing: MQTT CLI or similar

File-based Protocol Adapter

The HelloWorldAdapter used as the basis for this implementation is a simple but fully functional protocol adapter. We will incorporate logic to read from a file and encode the content in base64 encoding. This will ensure the file can contain any arbitrary binary format and still be represented as a string in JSON format. 

To prevent potential memory issues, we will first check the file size against a specified limit. Subsequently, we will create a JAR file for our protocol adapter and install it on a local HiveMQ Edge deployment to verify that it functions as intended. Finally, we will show how to write automated tests for a protocol adapter.

How to Get Started

A hello world example can be used to easily bootstrap a new protocol adapter. This adapter can be either downloaded from GitHub or cloned via the following command into your local environment:

git clone https://github.com/hivemq/hivemq-hello-world-protocol-adapter

Open the Project in an IDE and Set Up a New Project

After downloading the repository, the next step is to open it in an IDE of your choice. 

For this blog post, we use IntelliJ IDEA. You should see a similar environment as shown in the screenshot below.

Open the Project in an IDE and Set Up a New Project – How to Build a File-based Protocol Adapter for HiveMQ EdgeCheck Your Setup

You will likely need to select the Java SDK (at minimum, version 11) that you want to use to compile the project. The selected version should be the one you intend HiveMQ Edge to start with. 

You also need to set up the build tool Gradle. Most IDEs recognize the build file build.gradle.kts file automatically and open the project as a Gradle project or will suggest doing so. As a first step, we recommend validating whether your development environment correctly builds a protocol adapter JAR file. You may run the Gradle task shadowJar to build a JAR file. This task automatically downloads all dependencies of the projects, compiles them, and produces an artifact. The final JAR is located under the directory build/libs.

Check Your Setup – How to Build a File-based Protocol Adapter for HiveMQ EdgeYou can also manually execute the tasks on the console in your project directory by running 

./gradlew shadowJar

Eventually, a JAR file should be generated.

Generating a JAR file – How to Build a File-based Protocol Adapter for HiveMQ EdgeFor testing, copy the JAR into HiveMQ’s Edge modules folder and start HiveMQ Edge. The newly built protocol adapter should be shown below.

The newly built protocol adapter on HiveMQ EdgeIf you have any issues, please check the log file of your HiveMQ Edge instance.

Structure of a Protocol Adapter 

The following explains the most important Java classes for a Protocol Adapter. These files are located in:

hivemq-hello-world-protocol-adapter/src/main/java/com/hivemq/edge/adapters/helloworld/config

  • ProtocolAdapterFactory: This is the first class loaded by HiveMQ Edge to get the necessary information to create an instance of the protocol adapter and the information on it. 

  • ProtocolAdapterInformation: This class contains all information about the adapter, such as its name, protocol, author, etc. HiveMQ Edge obtains an instance of this class via the ProtocolAdapterFactory. 

  • PollingProtocolAdapter: The actual implementation of the protocol adapter, which handles the polling of data samples. The ProtocolAdapterFactory provides a means to construct an instance of this class. 

  • ProtocolAdapterConfig: The implementation of this interface contains the configuration options for the protocol adapter. The config is parsed by HiveMQ Edge automatically and uses the Jackson annotations for mappings.

Polling from a File

As a next step, we want to show how to implement the basic file-based protocol adapter requirements stated in the “Introduction to the Protocol Adapter” section above. We start implementing the polling logic for a file-based protocol adapter. 

As of now, we have no configuration available for the actual path to the file, so we just place the absolute path to the file as a constant value to keep it simple. This is good enough to test the file reading and provides the logic to add the content as the output of the polling process. 

The polling process is in the HelloWorldProtocolAdapter in the poll method:

@Override
    public void poll(@NotNull PollingInput pollingInput, @NotNull PollingOutput pollingOutput) {
        // here the sampling must be done. F.e. sending a http request
        pollingOutput.addDataPoint("dataPoint1", 42);
        pollingOutput.addDataPoint("dataPoint1", 1337);
        pollingOutput.finish();
    }

The poll function adds constant data points to the pollingOutput data structure, which is later handled by HiveMQ Edge to publish to a destination topic.

For the file-based protocol adapter, we need to make the following changes:

  1. Create a String containing the absolute path to the file.

  2. Check the size of the file against a predefined limit.

  3. Load the content of the file as bytes and encode it to a Base64 string.

  4. Add the content of the file to the output as a data point.

  5. Tell HiveMQ Edge’s SDK that the polling is finished.

  6. Handle exceptions by failing the output and returning from the method.

@Override
    public void poll(final @NotNull PollingInput<HelloWorldPollingContext> pollingInput, final @NotNull PollingOutput pollingOutput) {
        // absolute path to the file that contains the data. Magic string for now. Later it will be part of the config
        final String absolutePathToFle = "/tmp/sensor1.txt";
        try {
            final Path path = Path.of(absolutePathToFle);
            final long length = path.toFile().length();
            final int limit = 64_000; // not a constant to have a more compact code example
            if (length > limit) {
                pollingOutput.fail(String.format("File '%s' of size '%d' exceeds the limit '%d'.", path.toAbsolutePath(), length, limit));
                return;
            }
            // load the content of the file
            byte[] fileContent = Files.readAllBytes(path);
            // encode it as base64
            final String encodedFileContent = Base64.getEncoder().encodeToString(fileContent);
            // add the content of the file to the output
            pollingOutput.addDataPoint("value", encodedFileContent);
        } catch (IOException e) {
            // in case something goes wrong while reading the file, an IOException will be thrown.
            // we handle it by failing the poll process and returning from the poll method.
            pollingOutput.fail(e, "An exception occurred while reading the file '" + absolutePathToFle + "'.");
            return;
        }
        // we need to tell edge that the polling is done as edge also supports asynchronous polling.
        pollingOutput.finish();
    }

Note that the example reads a static file from path /tmp/sensor1.txt, which may not work on Windows. Please use a different filename according to your environment.

Testing the Implementation

As we already introduced in the Check Your Setup section, we need to compile the source code into a JAR file by executing the shadowJar gradle-task. Also copy the JAR into your HiveMQ deployment and restart HiveMQ Edge. Next, create a Protocol Adapter instance and check whether the first sample has been created using HiveMQ Edge’s Event Log, as shown in the screenshot. Remember — the HiveMQ Edge UI is accessible via http://localhost:8080 as the default configuration.

HiveMQ Edge UI via local hostNext, you can also use an MQTT Client to subscribe and check whether the sample is correctly published. In this example, we use MQTT CLI as follows:

mqtt sub -t '#'

The output of the tools looks like this:

{
  "timestamp" : 1719385976555,
  "value" : "SGl2ZU1RIEVkZ2UK"
}
{
  "timestamp" : 1719385977420,
  "value" : "SGl2ZU1RIEVkZ2UK"
}
{
  "timestamp" : 1719385977555,
  "value" : "SGl2ZU1RIEVkZ2UK"
}

 The value is base64 encoded and corresponds to “HiveMQ Edge.

Make File Path Configurable

The current state reads the content from a statically defined file. We want to improve this to let the user configure the file's path, adding a configuration item to a protocol adapter.

@JsonProperty(value = "filePath", required = true)
    @ModuleConfigField(title = "The file path",
            description = "The path to the file that should be scraped.",
            required = true)
    protected @NotNull String filePath;

    @JsonCreator
    public HelloWorldPollingContext(
            @JsonProperty("destination") @Nullable final String destination,
            @JsonProperty("qos") final int qos,
            @JsonProperty("userProperties") @Nullable List<UserProperty> userProperties,
            @JsonProperty("filePath") @NotNull String filePath) {
        this.destination = destination;
        this.qos = qos;
        if (userProperties != null) {
            this.userProperties = userProperties;
        }
        this.filePath = filePath;
    }


    public @NotNull String getFilePath() {
        return filePath;
    }

In the poll method, we use the new configuration for the file path:

@Override
    public void poll(final @NotNull PollingInput<HelloWorldPollingContext> pollingInput, final @NotNull PollingOutput pollingOutput) {
        // absolute path to the file that contains the data 
        final String absolutePathToFle = pollingInput.getPollingContext().getFilePath();
        try {
            final Path path = Path.of(absolutePathToFle);
            final long length = path.toFile().length();
            final int limit = 64_000; // not a constant to have a more compact code example
            if (length > limit) {
                pollingOutput.fail(String.format("File '%s' of size '%d' exceeds the limit '%d'.", path.toAbsolutePath(), length, limit));
                return;
            }
            // load the content of the file
            byte[] fileContent = Files.readAllBytes(path);
            // encode it as base64
            final String encodedFileContent = Base64.getEncoder().encodeToString(fileContent);
            // add the content of the file to the output
            pollingOutput.addDataPoint("value", encodedFileContent);
        } catch (IOException e) {
            // in case something goes wrong while reading the file, an IOException will be thrown.
            // we handle it by failing the poll process and returning from the poll method.
            pollingOutput.fail(e, "An exception occurred while reading the file '" + absolutePathToFle + "'.");
            return;
        }
        // we need to tell edge that the polling is done as edge also supports asynchronous polling.
        pollingOutput.finish();
    }

If you compile the project and copy it again into the HiveMQ Edge’s folder, you can see the file path down below in the subscriptions tab:

FIle path of the sensorUnit Test

To ensure proper functionality, a unit test must be written and executed. In the following example, we use the test frameworks JUnit and Mockito:

@TempDir
    @NotNull File temporaryDir;     // @TempDir creates a temporary folder which will automatically be removed after the test. 

    // we mock these objects to easily control their behavior 
    private final @NotNull ProtocolAdapterInput<HelloWorldAdapterConfig> adapterInput = mock();
    private final @NotNull HelloWorldAdapterConfig config = mock();
    private final @NotNull PollingInput<HelloWorldPollingContext> pollingInput = mock();

    @Test
    void test_poll_whenFileIsPresent_thenFileContentsAreSetInOutput() throws IOException {
        final File fileWithData = new File(temporaryDir, "data.txt"); // create a temporary file which content gets polled
        Files.write(fileWithData.toPath(), "Hello World".getBytes(StandardCharsets.UTF_8)); // write "Hello World" into the file
        when(adapterInput.getConfig()).thenReturn(config); // whenever the getConfig() is called on the input object, our mocked config is returned
        when(pollingInput.getPollingContext()).thenReturn(new HelloWorldPollingContext("mqttTopic", 1, List.of(), fileWithData.getAbsolutePath()));
        // whenever the pollContext is accessed on the pollingInput, a crafted context is returned, so that our file is polled and the correct mqtt topic is set
        TestPollingOutput pollingOutput = new TestPollingOutput(); // test implementation for a pollingOutput
        HelloWorldPollingProtocolAdapter adapter = new HelloWorldPollingProtocolAdapter(new HelloWorldProtocolAdapterInformation(), adapterInput);
        // we create the adapter with the setup we want for the test

        adapter.poll(pollingInput, pollingOutput); // make the adapter poll the file and insert the data into the output object. 
        // No further sync is needed here as the poll method is not asynchronous in the example

        final Object value = pollingOutput.getDataPoints().get("value"); // get the data point from the output
        assertNotNull(value); // check that there is a value present
        assertTrue(value instanceof String); // check that it has the expected type String
        String valueAsString = (String) value; // cast it to String
        final byte[] decodedBytes = Base64.getDecoder().decode(valueAsString); // decode it as the value is encoded as Base64
        final String actualDecodedValue = new String(decodedBytes, StandardCharsets.UTF_8); // create a UTF-8 String of it
        assertEquals("Hello World", actualDecodedValue); // check that the string is correct
    }

UI Schema

The adapter created so far will integrate with Edge literally out-of-the-box. In particular, the JSON annotations will be used to generate a JSON Schema. This, in turn, is used by the UI to create and handle the configuration form. A default rendering of the properties is offered, and it might be the end of your concerns. 

However, you can impact some aspects of this rendering's customization by defining a UiSchema specification.

It’s a JSON document whose content must abide by some rules regarding its structure and keywords. The full extent of the configuration can be explored in the documentation. Below is an example of UI configuration for our adapter

{
  "ui:tabs": [
    {
      "id": "coreFields",
      "title": "Core Fields",
      "properties": [
        "id"
      ]
    },
    {
      "id": "subFields",
      "title": "Subscription",
      "properties": [
        "subscriptions"
      ]
    },
    {
      "id": "publishing",
      "title": "protocolAdapter.uiSchema.groups.publishing",
      "properties": [
        "maxPollingErrorsBeforeRemoval",
        "publishChangedDataOnly",
        "pollingIntervalMillis"
      ]
    }
  ],
  "subscriptions": {
    "ui:batchMode": true,
    "items": {
      "ui:order": [
        "destination",
        "*",
        "userProperties"
      ],
      "ui:collapsable": {
        "titleKey": "destination"
      }
    }
  }
}

Worth noting:

  • The ui:tabs section allows you to group properties that are originally under the root of the document, resulting in horizontally organized tabs on the UI.

  • The ui:batchMode flag for the subscriptions property indicates that the adapter is able to support batch subscriptions import.

Final Tests

After the unit test is successful, we want to test the advanced polling logic with HiveMQ Edge on a real setup. The UI schema configuration is located under src/resources/helloworld-adapter-ui-schema.json and can be tweaked to your preferences.

To test the final implementation, compile the latest stage, copy the JAR file into the modules folder, delete the config for the protocol adapter, and restart HiveMQ Edge. Congrats — you’ve just built a new protocol adapter!

Finalization

Eventually, you probably want to give the protocol adapter a proper name. Update the HelloWorldProtocolAdapterInformation class.

public class HelloWorldProtocolAdapterInformation
 implements ProtocolAdapterInformation {

    public static final @NotNull ProtocolAdapterInformation INSTANCE = new HelloWorldProtocolAdapterInformation();

    protected HelloWorldProtocolAdapterInformation() {
    }

    @Override
    public @NotNull String getProtocolName() {
        // the returned string will be used for logging information on the protocol adapter
        return "PollingFileProtocol";
    }

    @Override
    public @NotNull String getProtocolId() {
        // this id is very important as this is how the adapters configurations in the config.xml are linked to the adapter implementations.
        // any change here means you will need to edit the config.xml
        return "Polling_File_Protocol";
    }

    @Override
    public @NotNull String getDisplayName() {
        // the name for this protocol adapter type that will be displayed within edge's ui
        return "Polling File Protocol Adapter";
    }

    @Override
    public @NotNull String getDescription() {
        // the description that will be shown for this protocol adapter within edge's ui
        return "This Protocol Adapter periodically polls and publishes the content of a given file.";
    }

    @Override
    public @NotNull String getUrl() {
        // this url will be displayed in the ui as a link to further documentation on this protocol adapter.
        // e.g. this could be a link to the source code and a readme
        return "https://www.hivemq.com/";
    }

    @Override
    public @NotNull String getVersion() {
        // the version of this protocol adapter, the usage of semantic versioning is advised.
        return "0.1.0";
    }

    @Override
    public @NotNull EnumSet<ProtocolAdapterCapability> getCapabilities() {
        // this indicates what capabilities this protocol adapter has. E.g. READ/WRITE. See the ProtocolAdapterCapability enum for more information.
        return EnumSet.of(ProtocolAdapterCapability.READ);
    }

    @Override
    public @NotNull String getLogoUrl() {
        // this is a default image that is always available.
        return "/images/helloWorld.png";
    }

    @Override
    public @NotNull String getAuthor() {
        return "HiveMQ";
    }

    @Override
    public @Nullable ProtocolAdapterCategory getCategory() {
        // this indicates for which use cases this protocol adapter is intended. See the ProtocolAdapterConstants.CATEGORY enum for more information.
        return ProtocolAdapterCategory.CONNECTIVITY;
    }

    @Override
    public List<ProtocolAdapterTag> getTags() {
        // here you can set which Tags should be applied to this protocol adapter
        return List.of(ProtocolAdapterTag.AUTOMATION);
    }
}

Conclusion

In this blog post, we’ve guided you through how to build a protocol adapter for HiveMQ Edge. We took the HelloWorldProtocolAdapter and implemented a file-based protocol adapter that reads content from a file and publishes it as base64 encoded JSON format.

Enable interoperability between OT and IT systems by translating various protocols into the standardized MQTT format with HiveMQ Edge. Modernize IIoT infrastructure and achieve seamless edge-to-cloud integration with this software-based Edge MQTT gateway. 

Get HiveMQ Edge FREE

Stefan Frehse

Stefan Frehse is Senior Engineering Manager at HiveMQ. He earned a Ph.D. in Computer Science from the University of Bremen and has worked in software engineering and in c-level management positions for 10 years. He has written many academic papers and spoken on topics including formal verification of fault tolerant systems, debugging and synthesis of reversible logic.

  • Contact Stefan Frehse via e-mail

Daniel Krüger

Daniel Krüger is a Senior Software Engineer in the HiveMQ Edge Team. He has a strong interest in messaging technologies and is focusing on expanding the functionality of HiveMQ Edge.

  • Contact Daniel Krüger via e-mail

Nicolas Van Labeke

Nicolas Van Labeke is Senior Frontend Engineer at HiveMQ. He is an experienced frontend engineer with a demonstrated history of working in R&D.

  • Contact Nicolas Van Labeke via e-mail
HiveMQ logo
Review HiveMQ on G2