what is the right messaging and data sharing standard for the internet of things
DESCRIPTION
This presentation considers what is the right messaging and data sharing standard for the Internet of Things. Different messaging and data sharing standards, such as AMQP, CoAP, DDS, MQTT have been proposed as candidates for addressing the data sharing challenges of the Internet of Things (IoT) and the Industrial Internet of Things (IIoT). In technical forums and social media there is no lack of passionate discussions that praise the merits of one standard over the other. Yet, to date, there is little or perhaps no analysis that looks at the details of the different standards and how they perform in an in depth, qualitative, analytic and empirical evaluation. This presentation, will (1) introduce the key standards that are being proposed for the IoT and the IIoT, such as AMQP, CoAP, DDS, MQTT, (2) present a qualitative comparison that highlights the different features provided by the various standards, (3) present an analytic comparison looking at the efficiency and scalability of the various protocols and (4) present the results of an empirical evaluation comparing the actual performances of the various standards.TRANSCRIPT
What is right messaging/data-sharing standard for the IoT?
Angelo Corsaro, PhD Chief Technology Officer
Internet of Things (IoT) Flavours
Cop
yrig
ht P
rism
Tech
, 201
4
Internet of Things Flavours
Wikipedia: Interconnection of
uniquely identifiable embedded
computing-like devices within the
existing Internet infrastructure
Internet of Things (IoT) was the term used to describe any kind of application that connected and made “things” interact through the internet
It is now clear that there are at least two kinds of IoT, Consumer IoT (CIoT) and Industrial IoT (IIoT)
The CIoT and IIoT follow the [Collect | Store | Analyse | Share] architecture, yet they have some key differences that is important to understand
Cop
yrig
ht P
rism
Tech
, 201
4
The Consumer Internet of Things (CIoT) represents the class of consumer-oriented applications where:
Devices are consumer devices, such as smart appliances, e.g. refrigerator, washer, dryer, personal gadgets such as, fitness sensors, google glasses, etc.
Data volumes and rates are relatively low
Applications are not mission or safety critical, e.g., the failure of fitness gadget will make you, at worse, upset, but won’t cause any harm
CIoT applications tend to be “consumer centric”
Consumer Internet of Things (CIoT)
Cop
yrig
ht P
rism
Tech
, 201
4
The Industrial Internet of Things (IIoT) represents industry-oriented applications where:
Devices are machines operating in industrial, transportation, energy or medical environment1
Data volumes and rates tend to be from sustained to relatively high
Applications are mission and or safety critical, e.g. the failure of a smart grid has severe impact on our life and economy, the misbehaving of a smart traffic system can threaten drivers
IIoT applications tend to be “system centric”
Industrial Internet of Things (IIoT)
1 The list of application domains is supposed to give examples and is not exhaustive
The IoT Data Pipeline
Cop
yrig
ht P
rism
Tech
, 201
4
Collect the data from the cyber-physical world
Depending on applications this could be:
- Sensor data
- Market Data
- Web page statistics
- ...
Collect | Store | Analyse | Share
Cop
yrig
ht P
rism
Tech
, 201
4
Organise , Store/Cache the data for on-line and off-line processing
Collect | Store | Analyse | Share
Cop
yrig
ht P
rism
Tech
, 201
4Make sense of the data
Detect short term / long term trends
...
Collect | Store | Analyse | Share
Cop
yrig
ht P
rism
Tech
, 201
4
Distribute Analytics -- or any other kind of clues about the data -- to applications that are supposed to act, display, publish, store, etc.
Collect | Store | Analyse | Share
Cop
yrig
ht P
rism
Tech
, 201
4
Efficient and scalable Data Sharing is a key requirement of practically any IoT system
The degree of performance required by the data sharing platform varies across Consumer and Industrial Internet on Things applications
Several standards have been proposed to address this key need of the IoT world
Data Sharing in IoT
Data Communication Standards
Cop
yrig
ht P
rism
Tech
, 201
4
The most discussed and promoted standards for the IoT are currently (in alphabetical order) AMQP, CoAP, DDS and MQTT
Top Contenders
Cop
yrig
ht P
rism
Tech
, 201
4
Originally defined by the AMQP consortium as a messaging standard addressing the Financial and Enterprise market
AMQP is an OASIS standard that defines an efficient, binary, peer-to-peer protocol for for transporting messages between two processes over a network. Above this, the messaging layer defines an abstract message format, with concrete standard encoding
https://www.oasis-open.org/committees/amqp/
Advanced Message Queueing Protocol (AMQP)
Cop
yrig
ht P
rism
Tech
, 201
4
CoAP is an IETF RFC defining a transfer protocol for constrained nodes and constrained (e.g., low-power, lossy) networks, e.g. 8-bit micro-controllers with small amounts of ROM and RAM, connected by Low-Power Wireless Personal Area Networks (6LoWPANs).
The protocol is designed for machine-to-machine (M2M) applications such as smart energy and building automation.
Constrained Application Protocol (CoAP)
CoAP provides a request/response interaction model between application endpoints, supports built-in discovery of services and resources, and includes key concepts of the Web such as URIs and Internet media types.
CoAP is designed to easily interface with HTTP for integration with the Web while meeting specialised requirements such as multicast support, very low overhead, and simplicity for constrained environments.
Cop
yrig
ht P
rism
Tech
, 201
4
DDS is the Object Management Group standard for data-centric publish subscribe
DDS is based on a fully distributed architecture and is equipped with a Dynamic Discovery Service that provide information about “who” and “what” is available
DDS comes with a rich set of QoS that control data availability, resource usage, e.g. network, memory, etc., and traffic prioritisation
The DDS standard family has recently been extended with RPC, Security, Web Integration
Data Distribution Service (DDS)
Cop
yrig
ht P
rism
Tech
, 201
4
MQTT was defined originally by IBM in the mid 90’s as a lightweight protocol for telemetry
MQTT supports a basic publish/subscribe abstraction with three different level of QoS
MQTT has recently gained much attention as a potential candidate for data sharing in the IoT
Message Queueing Telemetry Transport (MQTT)
Cop
yrig
ht P
rism
Tech
, 201
4
Main Characteristics
Transport Paradigm Discovery Content Awareness
Data!Encoding Security
AMQP TCP/IP Point-to-Point Message Exchange No None Defined TLS
CoAP UDP/IP Request/Reply (REST) Yes None Defined DTLS
DDS UDP/IP TCP/IP
Publish/Subscribe Request/Reply Yes Content-Based
Routing, Queries Defined TLS, DTLS, DDS Security
MQTT TCP/IP Publish/Subscribe No None Undefined TLS
Cop
yrig
ht P
rism
Tech
, 201
4
DDS and MQTT are the two protocol gaining more traction as candidate for the IoT
DDS applications are today mostly in IIoT while MQTT applications in the CIoT
The reminder of this presentation will focus on DDS and MQTT
Focus…
Protocol Analysis
Cop
yrig
ht P
rism
Tech
, 201
4
MQTT is a Client Server publish/subscribe messaging transport protocol
The protocol was designed for constrained environments and communication in Machine to Machine (M2M) and Internet of Things (IoT) contexts where a small code footprint is required and/or network bandwidth is at a premium
MQTT transports opaque data and does not define a mechanism to express the payload encoding
MQTT is a session oriented protocol — uses the transport connection to identify the session — that and runs over TCP/IP and WebSockets
MQTT v3.1.1
Cop
yrig
ht P
rism
Tech
, 201
4
MQTT provides three quality of service for message delivery:
- At most once: messages are delivered according to the best efforts of the operating environment. Message loss can occur. This level could be used, for example, with ambient sensor data where it does not matter if an individual reading is lost as the next one will be published soon after.
- At least once: messages are assured to arrive but duplicates can occur.
- Exactly once: messages are assured to arrive exactly once. This level could be used, for example, with billing systems where duplicate or lost messages could lead to incorrect charges being applied.
MQTT v3.1.1
Cop
yrig
ht P
rism
Tech
, 201
4
The family of DDS standards define an API for peer-to-peer, data-centric, publish/subscribe and an efficient interoperability wire protocol (DDSI) for disseminating data
The DDS standard was designed to address the requirement of a wide range of applications ranging from resource constrained embedded systems to large scale Consumer and Industrial Internet of Things (IoT) Systems
DDS provide a rich set of QoS providing control on:
- Data Availability: reliability and availability (for late joiners) of published data
- Resource Usage: memory and bandwidth utilisation
- Timeliness: data prioritisation and end-to-end traffic differentiation
DDS v1.2
Cop
yrig
ht P
rism
Tech
, 201
4
DDS is data centric, as such:
- Supports an extensible payload encoding scheme and supports natively multiple encodings, such as CDR (binary and efficient), PCDR (extensible, binary and efficient), JSON and XML
- Supports content filtering and queries. Where content filters are used to route information while queries are used to efficiently select received data
DDS can operate on both best effort transports such as UDP/IP as well as reliable transports such as TCP/IP
DDS v1.2
Cop
yrig
ht P
rism
Tech
, 201
4
(*) Assumptions: - The network can temporarily becoming unavailable - No application crashes
Reliability in DDS and MQTT
At Most Once At Least Once Exactly Once* Eventual!
Exactly Once*
MQTT QoS = 0 QoS = 1 QoS = 2 N.A.
DDS Reliability = BEST_EFFORT History = Any
DDS doen’t deliver
duplicates
Reliability = RELIABLE History = KEEP_ALL
Reliability = RELIABLE History = N
Message Structure
Cop
yrig
ht P
rism
Tech
, 201
4
MQTT messages are composed by:
Fixed header: included in every message
Variable Header: Included in some packets. Its content depends on the packet kind
Payload: Included in some packets
MQTT Message Structure
| 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+ | Fixed Header | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ ~ Variable Header ~ +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ ~ Payload ~ +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+
Cop
yrig
ht P
rism
Tech
, 201
4
The MQTT fixed header has a length that can vary from 2 to 5 bytes
The first two nibbles represents the message type and packet specific control flags
From the second byte starts the remaining packet length. This uses a “variable-length-encoding” that can take up to 4 bytes
MQTT Fixed Header
| 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+ | message type | Control Flags | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐+ ~ Remaining Length (1-‐4 bytes) ~ +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+
Cop
yrig
ht P
rism
Tech
, 201
4
A DDSI message is composed by a Header and a sequence of Sub-Messages
The structure of DDSI message make the protocol extensible as new sub message can be added w/o jeopardising backward compatibility
Each receiver only interprets the sub-messages it understands
DDSI Message Structure
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | Header | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | Submessage | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ ~ … ~ +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | Submessage | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+
Cop
yrig
ht P
rism
Tech
, 201
4
The header is composed by
4 bytes magic
Protocol version
Vendor identifier
Identifier of the participant that has produced the message (GUID-Prefix)
- A participant may have multiple communication end-points (e.g Data Reader and Data Writers)
- Each endpoint is identified by a 4 bytes ID, leading to 16 bytes GUID for identifying any communicating entity in the system
DDSI Header
1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | 'R’ | 'T' | 'P' | 'S' | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ | ProtocolVersion version | VendorId vendorId | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ | | + + | GuidPrefix guidPrefix | + + | | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+
Sending Data…
Cop
yrig
ht P
rism
Tech
, 201
4
Fixed Header DF: Duplication Flag QoS = 0, 1, 2 R: Retain
Topic Name (Variable Header) The length depends on the string chosen by the application It is likely, that to avoid collisions, users will give relatively long names such as: -‐ com/acme/myapp/mytopic
Message ID Only present for QoS=1,2 and used to uniquely identify messages
Payload Contains the data, but no information on the serialisation format
MQTT Publish Message | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+ | message type | DF| QoS | R | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐+ ~ Remaining Length (1-‐4 bytes) ~ +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ | Topic Name Length MSB | +-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+ | Topic Name Length LSB | +-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+ | | ~ Topic Name ~ | | +-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+ | Message ID MSB | +-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+ | Message ID LSB | +-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+ | | ~ Payload ~ | | +-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+-‐-‐-‐+
Cop
yrig
ht P
rism
Tech
, 201
4
INFO_TS Submessage This is required when source time-stamp is used to order samples coming from different sources
DATA Submessage Provides information on the origin and possibly the destination of the message Sequence number Potentially some inline QoS and parameters
DDS Data Message 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | 'R’ | 'T' | 'P' | 'S' | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ | ProtocolVersion version | VendorId vendorId | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ | | + + | GuidPrefix guidPrefix | + + | | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | INFO_TS |X|X|X|X|X|X|0|E| octetsToNextHeader | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ | | + Timestamp timestamp + | | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ | DATA |X|X|X|X|X|D|Q|E| octetsToNextHeader | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ | Flags extraFlags | octetsToInlineQos | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ | EntityId readerId | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ | EntityId writerId | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ | | + SequenceNumber writerSN + | | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ ~ ParameterList inlineQos [only if Q==1] ~ +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ ~ SerializedPayload serializedPayload [only if D==1 || K==1] ~ +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+
Overhead Analysis
Cop
yrig
ht P
rism
Tech
, 201
4
Message Size
Proto QoS Data Message Size (bytes)Transport Overhead
(bytes)
MQTT 0(2 to 5) + 2 + length(TopicName) +
length(Payload) TCP/IP(v4): 20-40
MQTT 1,2 (2 to 5) + 2 + length(TopicName) + 2 length(Payload) TCP/IP(v4): 20-40
DDS DestinationOrder = DestinationTimestamp 44 + length(Payload) UDP/IP(v4): 8
TCP/IP(v4): 20-40
DDS DestinationOrder = SourceTimestamp 56 + length(Payload) UDP/IP(v4): 8
TCP/IP(v4): 20-40
Cop
yrig
ht P
rism
Tech
, 201
4
The MQTT topic name is included in every PUBLISH message, this it is important to understand its structure
In general Topic name have the format:
- com/mycompany/app/deviceKind/deviceID
Notice that the device or appliance ID is useful to include to to be able to subscribe to the flow coming from a specific device, e.g. the refrigerator, as opposed to all instance of a given device
As such the length of the topic name, in real applications is on the order of tens of bytes
MQTT Topic Name
Cop
yrig
ht P
rism
Tech
, 201
4
20 Character MQTT Topic Name
Proto QoS Data Message Size (bytes) IP (v4) Headers Total Overhead (bytes)
MQTT 0(2 to 5) + 2 + 20 +
length(Payload) IP: 20-40 TCP: 20-40
min = 20 + 20 + 24 = 64 [TCP/IP] max = 40 + 40 + 27 = 107 [TCP/IP]
MQTT 1,2 (2 to 5) + 2 + 20 + 2 length(Payload)
IP: 20-40 TCP: 20-40
min = 20 + 20 + 26 = 66[TCP/IP] max = 40 + 40 + 29 = 109 [TCP/IP]
DDSDestinationOrder = DestinationTimesta
mp 44 + length(Payload)
IP: 20-40 UDP: 8
TCP: 20-40
min = 20 + 8 + 44 = 72 [UDP/IP] max = 40 + 8 + 44 = 94 [UDP/IP] min = 20 + 20 + 44 = 84 [TCP/IP] max = 40 + 40+ 44 = 124 [TCP/IP]
DDS DestinationOrder = SourceTimestamp 56 + length(Payload)
IP: 20-40 UDP: 8
TCP: 20-40
min = 20 + 8 + 56 = 84 [UDP/IP] max = 40 + 8 + 56 = 106 [UDP/IP] min = 20 + 20 + 56 = 96 [TCP/IP] max = 40 + 40+ 56 = 136 [TCP/IP]
Cop
yrig
ht P
rism
Tech
, 201
4
When running DDSI over TCP/IP one can transform a stream of DDSI messages into a single streamed message
This allows to drop the headers…
DDSI StreamingTCP/IP
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | Header | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | Submessage | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | Header | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | Submessage | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | Header | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | Submessage | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | Header | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | Submessage | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+
+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | Submessage | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+
+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+ | Submessage | +-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+-‐+
Cop
yrig
ht P
rism
Tech
, 201
4
DDSI Streaming Over TCP/IP
Proto QoS Data Message Size (bytes) IP (v4) Headers Total Overhead (bytes)
MQTT 0(2 to 5) + 2 + 20 +
length(Payload) IP: 20-40 TCP: 20-40
min = 20 + 20 + 24 = 64 [TCP/IP] max = 40 + 40 + 27 = 107 [TCP/IP]
MQTT 1,2 (2 to 5) + 2 + 20 + 2 length(Payload)
IP: 20-40 TCP: 20-40
min = 20 + 20 + 26 = 66[TCP/IP] max = 40 + 40 + 29 = 109 [TCP/IP]
DDSDestinationOrder = DestinationTimesta
mp 24 + length(Payload) IP: 20-40
TCP: 20-40min = 20 + 20 + 24 = 64 [TCP/IP] max = 40 + 40+ 24 = 104 [TCP/IP]
DDS DestinationOrder = SourceTimestamp 36 + length(Payload) IP: 20-40
TCP: 20-40min = 20 + 20 + 36 = 76 [TCP/IP] max = 40 + 40+ 36 = 116 [TCP/IP]
Cop
yrig
ht P
rism
Tech
, 201
4
MQTT makes a lot of effort to maintain header very compact, yet the Topic-Name included in the Variable Header of every PUBLISH introduces back a lot of overhead
In real applications the overhead of DDS when running on UDP/IP is very close to the MQTT overhead
When DDS runs over TCP/IP in streaming mode has lower overhead than MQTT, e.g. those using real topic names
Thus in summary, MQTT gives away loads of protocol features not to gain so much in efficiency.
Summary
Performance Evaluation
Cop
yrig
ht P
rism
Tech
, 201
4
The protocol analysis has already revealed that DDSI wire efficiency, for data transfer, is comparable to that of MQTT
Yet, the structure of the two protocols is very different and that may impact the performance over the network due to protocol processing
This session, will focus mostly on measuring efficiency, by looking at the end-to-end latency of both protocols
Performance Evaluation
Cop
yrig
ht P
rism
Tech
, 201
4
To evaluate the efficiency of both DDS and MQTT we will use the traditional latency test for data sizes going from 32 to 16K bytes
For evaluating MQTT we have used Mosquitto (www.mosquitto.org) in combination with the Eclipse PAHO MQTT Java Client Library
As a DDS implementations we have used the Vortex platform
The payload had the same time for both the DDS and the MQTT test. For MQTT the payload was serialised using Google Protocol Buffers
All tests were written in Java/Scala, in addition the various products were ran in their “out-the-box” configuration
General Info
Cop
yrig
ht P
rism
Tech
, 201
4
Open Source (BSD licensed) message broker that implements the MQTT v3.1 and v3.1.1
www.mosquitto.org
Mosquitto
Mosquitto Broker
Cop
yrig
ht P
rism
Tech
, 201
4
The Paho project provides open-source client implementations of open and standard messaging protocols aimed at new, existing, and emerging applications for Machine-to-Machine (M2M) and Internet of Things (IoT)
http://www.eclipse.org/paho/
Eclipse paho
Cop
yrig
ht P
rism
Tech
, 201
4
The Vortex Platform
Vortex enable seamless, ubiquitous, efficient and timely data sharing across mobile, embedded, desktop, cloud and web applications
OpenSpliceEnterprise
Cop
yrig
ht P
rism
Tech
, 201
4
One Standard, One set of Tools, One Goal — Ubiquitous Data Sharing
The Vortex Platform
VORTEXWeb
VORTEXLite
VORTEXGateway
VORTEXCloud
PrivateClouds
VORTEXTools
• Insight • Record/Replay • Tuner • Tester • Configurator
OpenSpliceEnterprise
VORTEXCafé
Cop
yrig
ht P
rism
Tech
, 201
4
Vortex Café Peer-to-Peer Latency
Measure the roundtrip for sending data from ping to pong and back and divide by two to obtain the latency
DDS Test Scenario
Ping Pong
Cop
yrig
ht P
rism
Tech
, 201
4
Vortex OpenSplice Peer-to-Peer Latency
Measure the roundtrip for sending data from ping to pong and back and divide by two to obtain the latency
DDS Test Scenario
Ping Pong
Cop
yrig
ht P
rism
Tech
, 201
4
Vortex Café Latency when brokered by Vortex Cloud
Measure the roundtrip for sending data from ping to pong and back and divide by two to obtain the latency
DDS Test Scenario
Ping Pong
Cop
yrig
ht P
rism
Tech
, 201
4
MQTT Latency for QoS = 0
Measure the roundtrip for sending data from ping to pong and back and divide by two to obtain the latency
MQTT Test Scenario
Mosquitto Broker
Ping Pong
MQTT Java Client MQTT Java Client
Cop
yrig
ht P
rism
Tech
, 201
4
MQTT Latency for QoS = 1
Measure the roundtrip for sending data from ping to pong and back and divide by two to obtain the latency
MQTT Test Scenario
Mosquitto Broker
Ping Pong
MQTT Java Client MQTT Java Client
Cop
yrig
ht P
rism
Tech
, 201
4
Linux Workstations running on i7 Intel Processor
Gbps Ethernet Network
Oracle JVM 1.8.0u20
Testbed
Cop
yrig
ht P
rism
Tech
, 201
4
Latency
This graph shows:
MQTT latency is for QoS = 0
DDS latency is for RELIABILITY=Reliable
Cop
yrig
ht P
rism
Tech
, 201
4
Latency Analysis
Peer-to-Peer DDS latency is always better than MQTT. The difference increases with payload size, to exceed 2x
DDS latency through Vortex Cloud is quite close to that of Mosquitto for small data and better for larger payloads (starting from ~2Kbytes)
Cop
yrig
ht P
rism
Tech
, 201
4
For small data, DDS is obviously faster then MQTT when non-brokered via Vortex Cloud
Vortex Cloud has a slightly higher latency than Mosquitto, but:
- Vortex Cloud is at its 1.0 release, still space for performance improvements
- Vortex Cloud has more complex logic, as it deals with instances, data-filtering, etc.
Small Data Latency
Cop
yrig
ht P
rism
Tech
, 201
4
With QoS=1 (or 2) setting for published data, MQTT latency goes up to the sky
This behaviour has been reproduced with both Mosquitto and Moquette and apparently depends on the disk access
The performance of QoS=1,2 really pose the question of their applicability and scalability
MQTT Latency for QoS=1
Ease of Use
Cop
yrig
ht P
rism
Tech
, 201
4
What’s Simpler?
val msg = new MqttMessage() builder.setSeqn(count) builder.setPayload(buf) builder.setTimestamp(System.nanoTime()) val sample = builder.build() sample.writeTo(os) val bytes = os.toByteArray msg.setPayload(bytes) msg.setQos(qos) msg.setRetained(false) broker.publish(pingTopic, msg)
MQTT DDS!val sample = new PingData(count, buf, System.nanoTime()) dw.write(sample)
Beside simple hello world examples that use Strings, when writing actual data with MQTT, you need to take care of serialisation… And ensure you use the right deserialisation routine on the receiving side… DDS, completely hides away this issues to you BTW… Not to mention the ClientID management….
Online Discussions
Cop
yrig
ht P
rism
Tech
, 201
4
MQTT. An Implementer’s Perspective
Cop
yrig
ht P
rism
Tech
, 201
4
MQTT. An Implementer’s Perspective
Concluding Remarks
Cop
yrig
ht P
rism
Tech
, 201
4
Among the different protocols proposed for the IoT DDS and MQTT have gained quite some attention
MQTT has mostly drawn the attention of Consumer IoT community while DDS the attention of the Industrial IoT community
This presentation has shown that DDS can be as wire efficient as MQTT and that deliver overall better performance for both M-2-M communication as well as cloud mediated communication
Summing Up
Questions?