mqtt-spy-daemon - HiveMQ MQTT Toolbox

Short Profile

General Information

mqtt-spy-daemon is an open source command line utility for publishing and monitoring activity on MQTT topics. It provides a subset of the functionality available in mqtt-spy, but because it doesn’t require a graphical user interface, it can be easily installed and used on any server or PC that has Java installed.

If you are after a command line utility for simple pub/sub, it’s probably best if you stick with the excellent mosquitto_pub & mosquitto_sub utilities. It’s worth considering mqtt-spy-daemon for more advanced use cases where high levels of automation are required.

This is the 2nd out of the 3 blog posts about the “mqtt-spy family”:

• the first post gave an overview of the mqtt-spy GUI utility

• this post covers mqtt-spy-daemon (the headless/console-based version of mqtt-spy) and its integration capabilities

• the last post will cover some of the advanced features of both mqtt-spy and mqtt-spy-daemon, including:

• advanced publications (scripting)

• subscriptions and advanced processing capabilities (formatting, scripting, advanced searching)

• message logging, replay and off-line browsing

• automated testing & test cases

Design Goals

The mqtt-spy-daemon sub-project was started in late 2014 with the aim of simplifying testing activities for MQTT-based systems where either no user interface is available or all publications/subscriptions need to be predefined.

Similarly to mqtt-spy, the daemon is particularly aimed at systems that generate a few hundred messages per second and use custom message envelopes with either encoded, signed and/or encrypted payloads. The daemon is also suitable for simpler environments where either headless operation and/or its scripting & logging functionality is required.

To support some more advanced modes of test automation, all of the mqtt-spy-daemon’s features can be also utilised via its Java API.

To help you with finding your way around it there is an extensive wiki available on the project’ home page.

MQTT Compatibility Overview

Unique Features

This brief section lists some of the unique features of mqtt-spy-daemon and what makes it different from other tools:

• Command line-based, driven by an XML configuration file

• Automatic reconnection and multiple server URIs

• Advanced TLS/SSL support with all common key store types: JKS, JCEKS, PKCS12 and BKS

• Message content formatting to support custom message envelopes (e.g. encoded, signed and/or encrypted payload)

• Message log – an audit file can be created for all received messages; this could be used for replay (and also off-line browsing with mqtt-spy)

• Scripted publications and subscriptions (with configurable handling of received messages, e.g. auto-reply)

• Automated testing with test cases

• Integration capabilities - can be embedded and used in your Java project; e.g. for unit/integration testing or continuous integration

Usage

Start Up

mqtt-spy-daemon is made available for download as an executable Jar file. This means it requires an appropriate version of the Java Runtime Environment to be installed on your operating system. Once downloaded, just run the following command and observe the output (it assumes you’ve got a broker running locally on the default port):

It is also possible to start mqtt-spy-daemon with a specific version of Java or run it with additional libraries - see Getting Started for more details.

Configuration File

In the example above, mqtt-spy-daemon with start with a built-in sample configuration file which will also run a sample publication script.

To run mqtt-spy-daemon with your own configuration file add its location as a parameter, e.g.:

For sample configuration files see Daemon Configuration.

The following sections will take you through most of the various configuration options so that you know where to look for the functionality you are after. Please always check the samples for the exact XML syntax.

Running Modes

mqtt-spy-daemon can be run in two modes:

• CONTINUOUS – this means the daemon will run until manually stopped by the user or another program

• SCRIPTS_ONLY – in this mode the daemon will only run for as long as background scripts are running (and for example they are publishing messages)

Protocol version

When no protocol version is specified, mqtt-spy-daemon will use the default and will try connecting with MQTT 3.1.1, and if that doesn’t work, it will then try to connect with MQTT 3.1. To configure the auto mode explicitly put “MQTT Default” into the Protocol tag.

To force a particular protocol version, either use “MQTT 3.1.1” or “MQTT 3.1”.

High Availability

There are two ways of achieving high availability with mqtt-spy-daemon:

• Multiple server URIs – see here for more details

• Automatic reconnection and re-subscription

User Credentials

To enable user authentication, populate the user credentials section with your MQTT username and password (Base64 encoded).

TLS / SSL

For more information on how to configure TLS/SSL certificates see the sample configuration (Secure connection to test.mosquitto.org) and the TLS/SSL wiki page.

Last Will and Testament

For LWT you can configure the publication topic, quality of service, retained flag and the message payload (Base64 encoded).

Subscriptions & Message Logging (Audit)

There can be any number of subscriptions configured. Each can define the subscription topic, QoS (optional) and the script file to run on each message (optional).

The main uses for defining subscriptions are:

• Script execution (e.g. auto-reply, writing to a database, custom logging)

• Message log (audit)

• Diagnostic logging

Script execution and message logging will be covered in greater detail in the next (3rd) blog post about the “mqtt-spy family”.

Formatting

Similarly to mqtt-spy, the daemon supports “message formatting”, or in other words, converting the received payload into more meaningful or useful format. This could be anything from decoding from Base64 or HEX, decrypting an encrypted payload or removing unnecessary tags, etc. This can then be logged in the message log file, in the diagnostic log, or used in a subscription script for further processing.

Formatters will be further explained in the next blog post.

Publications

The main mechanism for publishing messages is to define background scripts. Each definition contains a script file to run and optionally whether to repeat its execution.

A sample publication script file could look like this:

Scripted publications will also be covered in more detail in the following blog post.

Automated Testing & Test Cases

Similarly to mqtt-spy, the daemon also supports test cases with automatic test report creation. This will also be explained in the coming blog post about the mqtt-spy’s advanced features.

Diagnostic Logging

To get a better understanding of what mqtt-spy-daemon is doing at any given point in time, you can check its diagnostic log. It is available as a standard output and also being written to mqtt-spy-daemon.log (by default created in the current directory you start the daemon from). To increase/decrease the level of logging either edit the embedded log4j.properties file and restart the daemon, or supply a custom log4j configuration at start-up.

Integration / Embedding in your project

Apart from running mqtt-spy-daemon from the command line, you can also embed it in your project or unit/integration tests. This approach allows you to then trigger a set of predefined actions (see below).

To use the daemon in that mode:

• download the jar

• add it to the classpath of your project (either directly or e.g. via Maven)

• import the MqttSpyDaemon class from pl.baczkowicz.mqttspy.daemon package

• create a new instance of the daemon and you are ready to start!

Available actions:

• Start (takes the location of the XML configuration file)

• Can publish? (i.e. am I connected?)

• Run script

• Run script’s function

• Stop script

• Run test case

• Stop

For more information about the daemon’s API and samples of how to use see the wiki.

Eclipse Paho

For all its MQTT connectivity, mqtt-spy uses the Eclipse Paho Java client – a rock-solid library allowing applications to connect to any MQTT-compliant broker.

As of January 2016, mqtt-spy & mqtt-spy-daemon are also part of Eclipse Paho & Eclipse IoT.

Contributions and UNICEF donations

All contributions to the project are very welcome! Whether it is a bug report, a feature request, a bug fix, a feature implementation or just a suggestion – they are all very much appreciated. Get in touch via the GitHub page or Twitter.

There is also a suggestion to donate to UNICEF from time to time if you like and use mqtt-spy on a regular basis. You can do it via justgiving.com or directly on UNICEF websites - any donations count, big or small!

What’s next…

Next week the MQTT Toolbox will look at some of the advanced features of mqtt-spy and mqtt-spy-daemon – scripting, message logging (including replay) and automated testing.