Opportunities for IoT Development with MQTT v5.0

Summary: Message Queuing Telemetry Transport (MQTT) is among the top 10 most used protocols in the Internet of Things (IoT) industry. It has a vast legacy behind it, having been used for over two decades before. Even today, it receives significant attention thanks to its lightweight, open-source nature and accessibility. IoT devices can communicate through it despite spotty network conditions.

Even though the latest version, MQTT 5, officially became OASIS standard in 2019, not many users have transitioned from the previous v3.1.1 MQTT. The primary concern may be the limited documentation available regarding MQTT 5-compatible broker installation and configuration.

This article aims to fill you in on MQTT 5’s feature set. It will also go through a practical use case involving a Python-based system deployment. We will implement and configure an MQTT 5-compatible broker installation for an IoT system on LAN.

What Is MQTT?

 serves as a tool to connect many types of IoT devices in deployments of all magnitudes. It originally started in 1999 for oil and gas pipelines to communicate over remote satellites.

Operating above TCP/IP, MQTT is a network protocol that functions on a publisher-subscriber communication model. It’s lightweight enough to be used for a wide variety of IoT devices yet powerful enough to work through unstable network conditions.

HOW MQTT COMMUNICATION WORKS

A few entities work together to make up an MQTT protocol:

• Publisher: This device sends messages to subscribers via the Topic.
• Topic: Each resource has this unique identifier. The publisher sends a message to the Topic, which then passes it on to the Subscriber.
• Subscriber: The subscriber is the end device that receives the messages from the Publisher via the Topic.
• Broker: This is the server, a central hub responsible for organizational communication between publishers and subscribers.

The following diagram, which excludes the provided platform, shows how a simple MQTT communication works.If a cloud platform isn’t suitable for the development of a system using MQTT communication, we can use hbmqtt, gmqtt, paho-mqtt lib instead.

Quality of Service (QoS) level, a key feature of the MQTT protocol, is an agreement between the sender and receiver of a message. It defines how confidently the system can deliver a specific message.

QoS gives the client the power to choose a level of service that matches its network reliability and application logic. Because MQTT manages the re-transmission of messages and guarantees delivery (even when the underlying transport is not reliable), QoS makes communication in unreliable networks a lot safer and secure.

Why Is MQTT Used in IoT Development?

Thanks to its energy-efficient method of delivering data, MQTT is common for low-powered devices with limited CPU power or RAM. Some examples of use cases include:

• Efficient communication. MQTT’s low data and energy usage make it a prime candidate for a real-time, text-based messaging application.
• Security. In a home security system, for example, the QoS feature of MQTT systems can determine whether an important message has been delivered successfully, ensuring that the inhabitants are properly alerted of the danger.
• Intel gathering. MQTT allows organizations to collect data from multiple sources efficiently, such as smartphones or car sensors.
• Synchronizing sensors. For instance, multiple fire alarm detectors can communicate to determine whether a potential hazard is dangerous.
• Medical IoT applications. Multiple sensors can monitor the health parameters of patients leaving a hospital.
• Connected vehicles. Unlike HTTP, MQTT is capable of keeping up a persistent session between the client and broker. This feature is especially useful in connected cars. When a vehicle moves through a cellular network dead zone, the session can continue smoothly once reception returns. There is no need for a complicated HTTP handshake upon reconnection.

WHAT ARE THE BENEFITS OF MQTT?

1. Efficiency is the selling point of MQTT, as its architecture runs even on devices with low-power CPUs and low RAM. The result is smoother data transfers over competing protocols like AMQP. The lightweight architecture allows MQTT users to implement the protocol quickly and easily.
2. Fewer data packets being sent results in reduced network usage.
3. The reduced power usage makes it ideal for connected devices.
4. MQTT can achieve a more efficient distribution of data.
5. Implementation of remote sensing and control is easier through the protocol.

WHAT ARE THE DRAWBACKS OF MQTT?

MQTT might not be ideal in all circumstances, however. Some of the drawbacks to the protocol include:

1. Slower transmit cycles compared to Constrained Application Protocol (CoAP). Fast cycles are critical for systems with more than 250 devices.
2. Resource discovery. MQTT operates on a flexible topic subscription system. CoAP uses a stable resource discovery system.
3. Lack of security encryption. While MQTT uses Transport Layer Security/Secure Sockets Layer (TLS/SSL), it is primarily unencrypted.
4. Scalability. Creating a globally scalable network is more difficult with an MQTT protocol compared to other competitors.

MQTT is the ideal solution for many applications but consult with an  regarding its needs and preferences. The challenges are primarily related to security and interoperability.

MQTT v5.0 Features Overview

The new features of MQTT version 5.0 aim to achieve the following goals:

• Performance for large-scale systems. Communication among thousands or even millions of devices is more streamlined now. There are no protocol limitations, however, to orchestrate this number of devices, a proper architecture is required.
• Error reporting. The MQTT v5.0 protocol renames the return code to a reason code, which can indicate more types of errors.
• Implementation of common interactions. Repeated ways devices interacted with one another are now formalized in the current version of MQTT. Defining the capabilities of interacting devices and how they respond to requests is now built into the system.
• Addition of extensibility mechanisms. New functionalities include adding custom properties and specifying the content type or payload format.
• Better support, especially for smaller users who desire better productivity through MQTT.

MQTT 5.0 vs. MQTT 3.1.1

1. COMMUNICATION FEATURES

• Enhanced authentication, which can be implemented via Authentication Method and Authentication Data properties inside the payload.
• Session expiration, which can be implemented with the Session Expiry Interval property. For instance, the Topic could include the subscription time, the amount of time the message will be stored.
• Client and Server restrictions. Limitations can be built-in, including maximum packet size (number of bytes to transmit) and maximum to receive (number of messages to be sent simultaneously for a client or server).
• Will Delay Interval, a property to send messages on a delay.
• Server Reference or Server Redirect, properties that can help transfer packets to different brokers or servers.

2. POSTING FEATURES

• Message Expiry Interval, the time to keep messages stored.
• Payload format indicator and content type. This attribute defines the type of payload flags that can be used: bytes (binary), UTF-8, or MIME types.
• Topic Aliases. For instance, the topic/v1/device/ has an alias 1. This feature can minimize the number of data packages needed.
• Response Topics. With the use of Response Topics, the MQTT protocol can work similarly to an HTTP protocol with a response-request scheme.

3. SUBSCRIPTION FEATURES

• Non-local publishing. Users can optionally opt-out of receiving messages that the client published.
• Retained message control. This parameter controls message sorting.
• Subscription identifier, which is used for server identification of the subscription.
• Shared subscriptions to enable more flexible subscriptions with additional symbols and filtering features.

4. GENERAL FEATURES

• Reason codes on all ACK messages. Errors can occur at any stage. In MQTTv3.1.1, not much guidance from the server was available as to what went wrong at different stages, such as establishing communication, posting a message, or subscriptions to topics.
• Server disconnections. Unlike MQTT 3.1, packets regarding disconnections can be delivered from both client and server sides in MQTT 5.0.
• User properties. Keys can be used as value storage for different properties.

An Example of an MQTT 5.0 Small System Deployment

Let’s look at a case where we need to organize a local MQTT v5.0 network with Python-based clients. We will describe the challenges, issues, and pros and cons along the way. We will conclude by comparing it with an MQTT v3.1.1 network.

DESCRIPTION:

We have a building with several rooms containing a local area network (LAN). One room contains three standalone devices (e.g. activity standalone sensor, photo camera sensor, or audio sensor).The host device is located inside the LAN and connected to the router wirelessly or via cable. It must provide data gathering (and processing) functions from the standalone device in a period and must store this data locally in the database.For the current scope, an SQLLite database or simpler alternative is usable. The photo camera sensor and audio sensor must activate only after receiving the message from the activity sensor.

GOAL:

To ensure communication between the host device and standalone devices, and provide local database deployment and communication on the host side.

REQUIREMENTS:

1. All messages going from the sensors to the host device must be constrained with the additional properties of MQTT 5.0 (e.g. byte size of the message transferred to the topic).
2. Messages from topics must contain a MIME type for easier encoding on the host side.
3. Messages must be stored inside a database instance locally.

NOTATIONS:

• Standalone device: x86 or ARM-based (e.g. Raspberry Pi) with connected sensors and access to a local network.
• Host device: x86 or ARM-based (e.g. Raspberry Pi) which hosts MQTT broker and handles messages from standalone devices.

WHICH CLIENTS SUPPORT MQTT 5.0 AND PYTHON?

Currently, we have two options to work with:  and . However, these options do not have a built-in MQTT 5.0 broker and therefore are not preferable with a local deployment of a network. There is a non-Python implementation of the broker called Mosquitto that does support MQTT 5.0.Documentation can be found . It is able to support up to 50 000 devices per broker. Mosquitto has an “inflight queue” that can be configured in size (typical setting: 1000 messages) so that even under heavy load conditions like thousands of messages per sec or thousands of connected clients no connection or message gets lost.

HOW A PYTHON-BASED SYSTEM WORKS WITH MQTT 5.0

Not many libraries and documentation exist for the MQTT v5.0 protocol, especially from a Python developer’s point of view. The only current v5.0 clients for Python are gmqtt and paho-mqtt.

PROS AND CONS OF AN MQTT V5.0 LOCAL NETWORK

Pros:

1. Fully autonomous device interaction inside the local area network. There is no need for cloud providers like GCP or AWS and no need for a WAN connection for local IoT system functioning.
2. Network latency and data transfer speed. Transfer speeds depend only on the hardware capability of local devices. The disposition of devices within a LAN environment enables minimal latency.
3. Energy efficiency with MQTT compared to the competition.
4. Network security. Since the local network is not exposed to WAN, packets with messages cannot be captured or traced by an entity outside of the local network. The MQTT v5.0 protocol provides authentication of clients by servers and of servers by clients. MQTT also may use TLS certificates for secure connections and data transferring.
5. Packet restrictions can be applied to brokers inside the network.
6. Containerization. Easier containerization makes simulation and debugging much easier.

Cons:

1. Process and thread management for receiving messages and parallel works must be done beforehand. Threads for processing messages should be parallelized and managed properly for your devices to operate properly.
2. WAN connection. Developers must debug and troubleshoot regularly, and having a proper connection between the host and standalone devices must be organized first, usually with a secure SSH connection.
3. No support for stream transfer with the MQTT protocol. Look to other protocols if your organization needs it.
4. Transfer of large files not available on MQTT. Consider bucket uploads or the HTTP protocol.
5. Brokers cannot intelligently manage data. However, data can be stored for a limited time during a disconnection.

Major Practical Differences Between MQTT v3.1.1 and v5.0

• Properties for storing additional data
• Payload format indicators (bytes, UTF-8, or UTF-8 string pair)
• Request/Response PatternReason codes for client connection and disconnection
• Session expirations and control

The upgraded protocol version allows to simplify data payload processing and parsing. Brings the ability to separate and precise control over messages, connections, and sessions. And it allows the transfer of additional data through properties, which could lead to creating more sophisticated IoT solutions.

MQTT 5 Challenges

• Process/thread management for parallel publishing and listening for messages on standalone devices. You need to pay attention in a production environment.
• Limited documentation available and the implementation process of classes inside the package (paho-mqtt) is not obvious.
• Installation of a broker and its upgrade to MQTT v5.0 is difficult because of a lack of documentation.
• To recognize a device in the network, we need an IP discoverer to be added to the system.

MQTT v5.0 Is a Big Improvement

MQTT v5.0 is a suitable option for local IoT device communication if you have a central device that can host a message broker for communication between devices and/or the host. Despite its drawbacks (most of which were eliminated in MQTT v5.0), this protocol can be used for communication between small-to-medium sized networks of IoT devices.

The author - Daniil Liadov

Previously published at