Event Driven Microservices Architecture for IoT Solutions Using MQTT Protocol

Introduction

In this article, I’ll discuss an event-driven microservices architecture approach for IoT using MQTT with HiveMQ MQTT Broker as the central messaging component.

Here’s the reason why: Internet of Things (IoT) may be a planned priority for many organisations, but an overwhelming majority of IoT projects fail. In fact, a 2017 report by Cisco put this number at 74%. While there may be multiple reasons why IoT projects fail, it mostly comes down to companies underestimating the complexity of the mix of communication and computing technologies required to build a well-functioning IoT solution.

With so many moving parts, the lack of a well-thought-out IoT solution architecture could cause your project to fail to deliver on expected value. And this failure can occur at any stage, from project inception to execution and maintenance. Building an architectural foundation that enables the development of a real-time, responsive, scalable, extensible, and maintainable system is critical for the success of an IoT project. An event-driven microservices architecture approach using MQTT protocol is the way forward.

This is how it works!

Microservices Architectural Approach

A more traditional approach to build an IoT solution, or any software system for that matter, is to build an application in which the user interface, business logic, and database are combined into a single program residing on a single server.

While this approach may work in some cases, it has proven to be insufficient when it comes to building complex software systems such as a fully functional IoT solution. Mainly because it is difficult to scale and maintain, and when one part fails, the entire set of services goes down.

On the other hand, a Microservices Architectural Approach allows you to break down your IoT application and build it as a collection of granular functional services, with each service running in its own processes and communicating through APIs, as shown in the diagram below.

This approach increases flexibility, for example, the ability to update the code of a single function without having to redeploy the rest of the microservices architecture.

What’s more is that these collections of microservices can combine into large macro services, providing an even greater ability to quickly update the code of a single function in an overall service or larger end-user application.

The independent deployability of microservices also allows you to extend the functionality of your IoT application by simply adding more services to the system with no downtime, at scale.

Let’s consider, for example, the design of an IoT-based Asset Management system that is responsible for Remote Asset Monitoring, Asset Health Analysis, Generating Work Orders, and Scheduling Maintenance Checks. The system is cloud-based and incorporates IoT devices for its functions. And there are many client applications to this system, for administration staff, engineers, maintenance staff, and external clients.

This IoT system would be broken down into a set of services to serve each of its business functions. A Remote Asset Monitoring microservice, an Asset Health Analysis microservice, a Work Order Generation microservice, and a Maintenance Checks Scheduling microservice.

Additionally, microservices architectures have come into use along with Docker containers, allowing the code for each function (microservice) of your IoT solution to be deployed as a standalone software package that includes everything needed to run it. And that software package can exist anywhere in the public cloud or on-premise.

That means the failure points of your IoT system components are more independent of each other. Essentially, isolating the surface area of failure and creating a more stable overall application architecture.

This greatly accelerates software development time while making it easy to maintain, upgrade, and scale your IoT application.

So, for this example, you are writing your four new applications to serve your business functions in the microservices kind of style, and have them communicate with your existing IT systems using APIs.

Now, here’s the thing. The Internet of Things is a multi-level ecosystem, where devices equipped with sensors and embedded software collect telemetry data, transfer the data to an on-premise or cloud-based server, and therefore there is a lot of message exchange going on between IoT devices, microservices, existing IT systems and extra IoT layers that include databases, analytics solutions, visualisation dashboards, and mobile applications.

This calls for a well-thought-out messaging model on top of the Microservice architecture. This leads us to the topic of the Event-Driven Messaging Model.

Message Driven vs Event-Driven (Asynchronous Messaging in IoT)

Let’s take, for example, an Asset Health Analysis microservice that needs to notify the system when it detects an anomaly in the vibration telemetry data from a piece of equipment. Using a Message-Driven approach, IoT system components that are interested in knowing the equipment’s health status would have to repeatedly send a request to check if an anomaly condition has been raised. Simply stated, Message-Driven systems are based on synchronous request-response where you ask for something, and then you wait until you get your response back. In a nutshell, each of your IoT components poll each other for data in timed intervals, await messages, and react to them, HTTP works like that.

Now, imagine a system that is event-based, and everything is based on Action and Reaction, instead of the issuing of requests. As opposed to data being sent to a specific IoT component that is asking for it, in Event-Driven communication, data is emitted as an Event from an IoT component for anyone listening to consume. In other words, each IoT solution component produces items of data with a fixed sender and shares them with any consumer.

In this way, the system has a source of intelligence and is aware of everything that happens without having to continuously ask for it.

In reference to our earlier example, if the Asset Health Analysis service detects an anomaly in the vibration telemetry data, it gives out this information as an event without concern of who’s consuming it. Consequently, that same event could trigger a Work Order Generation service to generate a work order. And when the service is finished generating the order, it could produce an event back into the system for any other component that might be interested in that event, which makes it a responsive interaction.

Basic Concepts of Event-Driven Messaging Pattern

The first and most important component in an event-driven system is the Event. An event in an Event-Based system is a message that represents facts(data) or commands and is created when a change of an observed value happens, such as an action by a user, an increase or decrease of something, a condition that becomes true e.t.c. This message or event is generated or captured by what is referred to as an Event Generator which is an entity that detects the change and notifies the system.

The event generator could be a sensor, monitoring system or middleware, etc. A common design pattern that is used to implement this process is the Publish/Subscribe pattern. In this case, an event generator is called a publisher, and interested parties are called subscribers, or event handlers.

MQTT Broker for Event Delivery and Management

Now, the thing is, if you want to reliably pass events in your IoT solution then of course you need a messaging infrastructure that supports that.

MQTT is one of the most widely adopted IoT communication protocols that support an event-driven architecture, and it is based on the Publish/Subscribe pattern of communication using an MQTT Broker for coordinating the delivery of events. An MQTT broker would receive events from a producer and forward them to correspondent subscribers.

Any component can be an event producer, it could be an IoT device, Web app, or Mobile app e.t.c. And they are producing events into your system, while the MQTT Broker sits at the middle of your IoT Solution, instantaneous capturing facts and forwarding the information to interested parties, as events. This pattern provides a way of wiring together your IoT components whereby, messages are exchanged via a given information channel or topic.

When the broker receives a published message, it broadcasts it to all subscribers but only those who have subscribed to the given information channel can receive it. By avoiding direct communication between the publisher and the consumer, a highly efficient fan-out messaging pattern is created as there is no pausing to wait for any response to the generated messages.

Think about it, an IoT solution architecture that is highly scalable, even 10,000 components can subscribe to consume an event, and it wouldn’t affect how that event is produced.

As discussed earlier, the flexibility of integrating new functionalities is the primary feature to be considered during the design of your IoT solution cloud architecture. You may expand functionality by creating microservices for other business functions and simply plugging them onto the MQTT network.

Even better, for industrial applications you may want to take advantage of MQTT Sparkplug, an interoperability framework build on top of MQTT allowing events that use a standardised topic namespace, payload definition, and state management to be emitted from your IoT components as a single source of truth.

HiveMQ provides an MQTT broker and a client-based messaging platform designed for the fast, efficient and reliable movement of data to and from connected IoT devices and systems. It uses the MQTT protocol for instant, bi-directional messaging between your IoT device and enterprise systems. Further, because the MQTT protocol is so lightweight it can deployed on platforms with constrained resources as is typical of IoT devices. And it doesn’t end there, MQTT has security enabled so it works with TLS and common authentication protocols.

The enterprise version of the HiveMQTT broker is designed for horizontal scalability and the best-in-class throughput. In fact, we conducted a benchmark showing the connection of 10,000,000 real MQTT clients, which exchange more than 3 billion messages per hour.

Now, in our example scenario, your Asset Health Analysis microservice or other backend application might generate an “Anomaly Detected” Event and publish it to the MQTT broker under a specific topic, and because other microservices such as the Work Order Generation microservice is subscribed to that event topic through the same broker when it also receives the event, it will perform necessary operations and publish the new event under a different topic when finished.

But of course event-driven is not the answer to everything, Microservices also need data from each other and other backend systems. For example, the order Work Order Generation microservice might need additional details such as location, skills requirements, and tools needed from a system’s database. And for that you’d need to introduce an API interface to your IoT solution architecture as depicted below.

Conclusion: Adopt MQTT for IoT Event Driven Microservices Architecture

In conclusion, the event-driven architecture is a complementary solution for microservices and other types of architectural patterns. When combined with a reliable messaging infrastructure such as HiveMQ MQTT Broker, it enables you to build simple, resilient, maintainable, extensible and self-aware IoT Solutions.

Further, MQTT Quality of Service (QOS)-Level 2, which ensures the delivery of important messages exactly once, can be applied in a microservices pattern called SAGA. SAGA is used for transactional messaging across different services with separate databases.