using the mosquitto implementation in an …...using the mosquitto implementation in an embedded...
TRANSCRIPT
Using the Mosquitto implementation in anembedded environment
Emil Vanneback
Emil VannebackVT 2018Examensarbete, 30 hpSupervisor: Jerry ErikssonExaminer: Henrik BjorklundCivilingenjor i teknisk datavetenskap, 300 hp
Abstract
An essential challenge of the Internet of Things is how to share data be-tween embedded systems. One way of doing this is with the protocolMQTT. MQTT is a message distribution protocol which uses a pub-lish and subscribe method to share data. The company Faltcom uses animplementation of MQTT called Mosquitto to distribute data betweentheir units. Faltcom has developed an embedded unit called MIIPS Cwhich uses the Mosquitto implementation to function as a message bro-ker. The broker is the center of communication and all other units sendtheir messages through the broker. The questions of this thesis regardshow well Mosquitto works in an embedded environment with hardwarelimitations and an unstable internet connection. To answer these ques-tions tests were run on a MIPPS while observing key metrics such asCPU usage and latency. The results are summarized to compare perfor-mance between different cases. The results show how Mosquitto workswell in an embedded environment if configured correctly. The configu-ration has to limit message size and how many messages the broker canstore, in order to ensure that the broker does not run out of memory.
Acknowledgements
Thank you to the company Faltcom for supporting this thesis and providing the requiredresources. A special thanks to my supervisor at Faltcom, Erik Tideman, for all his help,support and feedback.
Thank you Umea University and my supervisor, Jerry Eriksson, for valuable feedback onthe final report. A final thanks to the peer reviewers Anton Nordin, Alexander Holmstromand Joacim Kastberg.
Contents
1 Background 1
2 MQTT 32.1 Basic concepts 3
2.1.1 Publish/subscribe 32.1.2 Broker 32.1.3 Topics 42.1.4 Quality of Service 42.1.5 Retained messages 42.1.6 Clean session 42.1.7 Last Will/Testament 42.1.8 Benefits of MQTT 52.1.9 Disadvantages of MQTT 5
3 Mosquitto 73.1 Configuration 73.2 Broker status 73.3 Bridge mode 8
4 MIIPS 9
5 Problem formulation 115.1 Question 1 115.2 Question 2 115.3 Question 3 115.4 Question 4 12
6 Method 136.1 Testing method 13
6.1.1 Heap memory 136.1.2 CPU usage 136.1.3 Latency 146.1.4 Mosquitto configuration 14
6.2 Analysis methods 146.2.1 Heap memory 146.2.2 CPU usage 156.2.3 Latency 15
6.3 Implementation methods 156.3.1 Mosquitto wrapper 156.3.2 Client handler 15
6.3.3 Output handler 166.4 Interpreting results 16
6.4.1 Box-plot 166.4.2 T-test 17
7 Test scenarios 197.1 Test latency and CPU usage at different message loads 197.2 Test latency and CPU at different levels of QoS 197.3 Test latency and CPU with many stored messages 207.4 Testing bridge 20
7.4.1 Testing different bridge configurations 217.4.2 Comparing multiple topics with topic wild card 21
8 Results 238.1 Compare message loads 23
8.1.1 CPU usage 238.1.2 Latency 23
8.2 Compare QoS levels 258.2.1 Latency 258.2.2 CPU usage 25
8.3 Does stored messages impact performance? 268.3.1 Latency 278.3.2 CPU usage 27
8.4 Does uncontrolled disconnects impact performance? 298.5 Testing bridge configuration 298.6 Comparing multiple topics with topic wild cards 31
9 Analysis 339.1 How does Mosquitto store messages that can not be distributed to sub-
scribers due to lost connection? 339.1.1 Controlled disconnect 339.1.2 Uncontrolled disconnect 33
9.2 Will storing messages in the broker affect the MIIPS overall performance? 349.2.1 Controlled disconnect 349.2.2 Uncontrolled disconnect 34
9.3 Will the current configuration options be enough to limit the memory usageof the protocol? 35
9.4 Will using the broker in bridge mode affects the MIIPS performance? 359.5 Solutions 35
9.5.1 Limiting storage in configuration 369.5.2 Using a status topic 369.5.3 Restarting Mosquitto 36
9.6 Update 1.4.15 369.7 Other Interesting results 37
9.7.1 Difference in message loads 379.7.2 Quality of Service 37
10 Conclusion 39
Bibliography 41
A Appendix 43A.1 Test scenarios 43
A.1.1 Test heap memory on subscribers and clients 43A.1.2 Test latency and CPU usage at different message loads 43A.1.3 Test latency and CPU with different number of subscribers 44
A.2 Results 44A.2.1 Test heap memory on subscribers and clients 44A.2.2 Test latency and CPU usage at different message loads 45A.2.3 Test latency and CPU with different number of subscribers 47
1(48)
1 Background
Internet of things (IoT) is connecting devices all across the world by enabling them to gatherand share data through network connections. This helps devices that is being used everyday to be smarter and more efficient. Today ”smart” devices occur in many places such assmart vehicles[2, 241-246] and smart health-care[3, 67-75]. These devices are in a lot ofcases small sensors with limited computational power. For these devices to communicateefficiently, a lightweight protocol can be used. One such protocol is Message QueuingTelemetry Transport (MQTT). This protocol uses a publish and subscribe method to shareinformation.
The purpose of this paper is to test an implementation of the MQTT protocol called Mosquittoon the specific hardware called the MIIPS C. The MIIPS is a hardware developed and usedby the company Faltcom. The goal is testing to what extent the MIIPS can use the Mosquittoimplementation. Currently MQTT is mostly used within the MIIPS to communicate be-tween different applications. However it would be beneficial to use it to a bigger extentbetween multiple MIIPS units.
The limits of using the MQTT between different MIIPS units are the limited hardware of theMIIPS. MQTT uses a message broker to share messages between units. The broker wouldin this case be responsible for distributing messages between a lot of units. These unitscould malfunction or be offline. This would entail the broker have to store the messageswhich has not yet been delivered. Usually the broker is on a server with a lot of memoryand lot of available CPU. In this case the broker is on a MIIPS unit with limited memory,CPU and often an unstable internet connection. This could cause unexpected behaviour.These problems are the foundation of this project.
2(48)
3(48)
2 MQTT
Message Queuing Telemetry Transport (MQTT) is a lightweight protocol for publish/subscribemessaging. It is based on a broker to distribute messages between publishers and sub-scribers. It is designed to be open, simple and lightweight. MQTT was first invented anddeveloped by IBM in 1999. In late 2014, it officially became an OASIS open standard.[4]
2.1 Basic concepts
This section includes the basic concepts of MQTT.
Figure 1: An example of how the MQTT protocol works. The publishers to the left pub-lisher messages to the broker. The subscribers gets the messages in the topic they havesubscribed to.
2.1.1 Publish/subscribe
The MQTT protocol is based on a publish/subscribe method. Clients can subscribe to inter-esting topics and will receive all messages later published on that topic. [10, 6–7]
2.1.2 Broker
The broker, sometimes referred to as server, is responsible for distributing the messages.The broker is the center of communication and all other clients send their messages throughthe broker. When a client connected to the broker publishes a message the broker sends themessage to any subscriber of that topic.
4(48)
2.1.3 Topics
Topics in MQTT can be thought of as subject areas. A subject can be divided into sub areasas well. For example a topic could be /vehicles/cars/volvo. [10, 6–7]
Topics include two wild cards (#, +). + can be used for a single level hierarchy for example/vehicles/+/volvo. Which mean all vehicle types of the brand Volvo. The wild card # is usedfor all remaining levels in the hierarchy. For example /vehicles/# mean all the sub types ofvehicles. [9]
2.1.4 Quality of Service
MQTT delivers messages divided into three different Quality of Service (QoS) levels.
QoS 0 - At most once delivery. This level does not guarantee messages being distributedto all subscribers. The broker only send the message once and does not expect aresponse.
QoS 1 - At least once delivery. This level does guarantee the message to be delivered butcould deliver the same message multiple times. The broker sends a message and thenwaits a certain time for a response. If no response is received it will send the messageagain until it receives a response.
QoS 2 - Exactly once delivery. This level guarantees each subscriber will receive the mes-sage exactly once. This is achieved by attaching an ID. The broker expects an re-sponse with the same message ID. If the client receives a message with an id alreadyreceived, the message is discarded.[4]
2.1.5 Retained messages
A message published can use a retain flag. The most recent message with a retain flag ineach topic is saved in the server. If a new client subscribes to the topic it will receive theretained message. [10, 6–7].
2.1.6 Clean session
When a client connects to a broker it can set the clean session flag. If the flag is set to true allthe clients subscriptions will be removed when disconnecting. If set to false the session willbe treated as durable thus the clients subscriptions remain active even after it disconnects.In this case any messages sent with QoS 1 or higher needs to be stored in the broker untilthey can be delivered to the disconnected client. [10, 6–7]
2.1.7 Last Will/Testament
When a client connects to a server it can specify a will. This is a message that will be sentout if the client disconnects. In some cases this is called a testament. [10, 6–7]
5(48)
2.1.8 Benefits of MQTT
• Uses publish/subscribe model which decouples the sender and receiver. A client onlyneed to know the broker not other subscribers or publishers.
• It’s completely open for easy integration with other solutions.
• The protocol keeps overhead at a minimum.
• Through the different levels of QoS, important messages can be persistent while lessimportant messages have minimum overhead.
• The content of messages can be of any type. [10, 12–13]
2.1.9 Disadvantages of MQTT
• Publishers are unaware of its subscribers, there is no natural way for the publisher toreceive any response that a subscriber received its message.
• One way communication (publisher will not receive any response from subscriber).This implies action/response implementations are not suitable.
• Single point of failure. All clients connected to a broker will not be able to share dataif the broker looses connection.
• MQTT uses a TCP connection to the broker. This implies that connection is alwayson, which limits the time devices can sleep. [5]
• MQTT lacks encryption since it was designed to be lightweight and encryption wouldincrease overhead substantially. However most implementations has added supportfor encryption with SSL and TSL. [5]
6(48)
7(48)
3 Mosquitto
Mosquitto is the MQTT broker implementation tested in this paper. Mosquitto is opensource under the EPL/EDL license and is developed by Eclipse. Mosquitto is a lightweightimplementation of the MQTT message broker and supports the MQTT protocol versions3.1 and 3.1.1. This makes it suitable for IoT where limited hardware is an issue. [6]
3.1 Configuration
A Mosquitto broker can be configured using a configuration file. There are a lot o config-uration which can be found in the mosquitto.conf man page[8]. The configuration optionsessential in this thesis are:
max inflight messages count - The maximum number of QoS 1 or QoS 2 messages thatcan be in the process of being transmitted simultaneously. This includes messagescurrently going through handshakes and messages that are being resent. Default is20. Set to 0 for no maximum. If set to 1, this will guarantee in-order delivery ofmessages.[8]
max queued messages count - The maximum number of QoS 1 or QoS 2 messages tohold in the queue above those messages that are currently in flight. Default is 100.Set to 0 for no maximum (not recommended).[8]
message size limit limit - This option sets the maximum publish payload size that the bro-ker will allow. Received messages that exceed this size will not be accepted by thebroker. The default value is 0, which means that all valid MQTT messages are ac-cepted. MQTT imposes a maximum payload size of 268 Megabytes.[8]
These messages all handle the size of queued messages in a single broker. This is essentialfor the purpose of this paper since its goal is to see how useful the MIIPS hardware is whenrun as a broker.
3.2 Broker status
A client can collect status from the broker by subscribing to the $SYS topic. This waythe client can get information from the broker, for example number of connected clientsor bytes sent. A full description of topics under SYS can be found at the Mosquitto manpage.[7]
8(48)
3.3 Bridge mode
To connect multiple brokers to each other a bridge mode is used. This helps distributemessages across more topics and areas without connecting all of the publishers to the samebroker. This adds horizontal scalability which might prove important when using brokerswith limited hardware. An example of this is displayed in Figure 2
Figure 2: An example of how Bridge mode can work. Without the bridged brokers thesubscribers in this example would not receive their messages.
9(48)
4 MIIPS
MIIPS C is a Linux based system. It is used to connect and monitor devices such as build-ings, vehicles and machines. It can be connected with a wired solution or by 2G, 3G and4G. It has the ability to connect a sim card and is also connected with GPS. It is open tothird party developers if own developed applications is desired. For detailed hardware seeTable 1.
Table 1 Hardware specification for the MIIPS C [1]
Operating system LinuxCommunication GSM,GPS,CDMA,
GPRS/EDGE/HSDPA/LTEEncryption RSA/AESProcessor 800Mhz
RAM 1GBFlash memory 8GBDimenstions 45mm x 130mm x 175mm
Weight 580g
10(48)
11(48)
5 Problem formulation
This section poses the questions to be answered by this project.
5.1 Question 1
How does the Mosquitto store messages that can not be distributed to subscribers due tolost connection?
As explained in Section 2 all messages published with an QoS 1 or higher is guaranteedto reach its subscribers. This implies that if a message is not received by all current sub-scribers the message need to be stored in the broker. This could cause problems if a clientis disconnected during a time when a lot of messages are being published to a subscribedtopic.
In theory this question is easy to answer. The messages will be stored until they can bedelivered. The purpose of this question is to see how this works in the MIIPS and howmany messages will be stored in different cases.
5.2 Question 2
Will storing messages in the broker affect the MIIPS overall performance (high CPU usageand high latency) because of the hardware limitations?
This question is similar to Question 1 but the central part is how the MIIPS hardware createslimitations. In theory there will be a limit to how much can be stored in the MIIPS beforeit’s full. The interesting question is when this occurs and if this will be a problem in apractical use case.
5.3 Question 3
Will the current options in the protocol be enough to limit the memory usage of the protocol?Can this be used make sure a unit hosting the broker does not run out of memory? If notwhat other options are needed?
As explained in Section 3.1 there are three current options used to limit messages in thebroker. These are max inflight messaes, max queued message and message size limit. Thequestion refers to if these three options are enough to control the broker. And if they are notsufficient, can any options be added to improve the protocol.
12(48)
5.4 Question 4
Will using the broker in bridge mode affect the MIIPS overall performance (high CPU usageand high latency) because of the hardware limitations?
Using a broker in bridge mode can be an easy way of passing information on certain topicsbetween multiple brokers. If this is useful it is interesting to know how this will impactperformance of the broker.
13(48)
6 Method
This section describes the method used to acquire the results. The method section is dividedinto testing, analysis, implementation and interpreting results.
6.1 Testing method
To run most of the tests a single MIIPS unit and two Linux computers are used. The MIIPScomes with a Mosquitto broker installed. In order to run the tests the configuration of thebroker has to be changed. The configuration used for testing is described in Section 6.1.4.Three major metrics are studied in the tests. These are heap memory, CPU usage, andlatency. These metrics, in particular CPU and latency, are of major concern of how well theMosquitto handles certain areas.
6.1.1 Heap memory
The first metric is the amount of heap memory is used by the broker. The broker will storeall clients, retained messages, system messages and stored messages in its heap memory.This metric is mainly studied to test linearity and different relations. This metric is observedby subscribing to the topic ”$SYS/broker/heap/current”. This will return the current usedheap memory of the broker.
6.1.2 CPU usage
CPU usage is the second metric. In this context this is the total CPU usage of the broker.This is measured with the vmstat method call in Linux. Vmstat returns the statistics ofmemory, IO and CPU. CPU is measured to get a sense of how well the Mosquitto handlesdifferent areas. If the CPU usage runs very high it will slow down other applications run onthe same system. The important information from vmstat to measure CPU is:
Sy - Time running system kernel code (%).
Us - Time running user code (%).
The total CPU usage is calculated with:
CPU = sy+us
To get a sense of the CPU usage over time each test measuring CPU is run over at least 120seconds, collecting one sample of data each second.
14(48)
6.1.3 Latency
The latency is measured as the round trip time of a message. A second computer is usedwith a single subscribe to an own topic. The same computer sends one message each secondmeasuring the time it takes to return. This metric is used to simulate how other clients willbe affected by a busy broker.
To measure the latency tcpdump and wireshark are used. tcpdump is built into Linux Ubuntuand is used to monitor sent and received packages. tcpdump has plenty of options to spec-ify which packages should be observed. The option used to measure latency in this caseis:
tcpdump -w file name ’(dst port 1883 && dst host broker-address)
or (src port 1883 && src host host broker-address)
This will cause only messages sent to the broker or received from the broker to be observed.In order to increase the quality of the result this test is run over at least 120 seconds. Toanalyze the results from tcpdump, Wireshark is used. Wireshark takes a dump file to gatherand group the information. In this case Wireshark is used only to interpret the information inthe packages and export it as a csv-file for further analysis. The language R with R-studio isused to further analyze these results. This will be further described in Section 6.2 (Methodanalysis)
6.1.4 Mosquitto configuration
This is the configuration used when testing the broker. The rest of the options are set todefault values.
• max inflight messages 0 - sets maximum of in flight messages to unlimited.
• max queued messages 0 - sets maximum of queued messages to unlimited
• message size limit 0 - sets maximum size limit to max of MQTT (268435455 bytes)
• max connections -1 - sets maximum of connections to unlimited
• sys interval 1 - sets delay of system messages to 1 second
6.2 Analysis methods
Most tests are analyzed with the statistical language R. This language features easy andpowerful statistical tools. R is good for creating plots and handling large amount of data.For easy handling of data, all the results are either saved or converted into a csv-file. Thiscan then be opened by R and be converted to finished tables or figures.
6.2.1 Heap memory
The analysis of heap memory is very simple. The saved values are put into a line graph.The graph can then be observed or combined with other plots to compare relations betweenfor example subscribers and publishers.
15(48)
6.2.2 CPU usage
To analyze CPU usage can be more challenging. The CPU usage is not as reliable sinceit changes over time and is affected by more uncontrolled variables. To account for thisthe CPU is measured over an extended period of time. For these tests this was done over120 seconds. During this time one result was gathered each second. These values are thenstored into a table in R. The values are used for visualization in box-plots and to comparedifferences in a T-test. Box-plot is used because it is a good way to visualize both themedian and standard deviation of the data. The T-test is a good way to demonstrate if thereis a significant difference between two data sets.
6.2.3 Latency
The latency is saved from Wireshark to a csv-file. This file includes all the tcp-packagesfrom the tcpdump, but has now got the package information from Wireshark. To handlethis information R is used to create a script. This Script strips everything except the publishmessages away from the data. This information is then used to determine when a messagewas sent and when it was received. The subtraction of these times will be the total latencyof one message.
This test is also run over 120 or 300 seconds with one sample each second. These results arealso visualized in plots such as box-plots, line-plots and compared with the T-test.
6.3 Implementation methods
To run most of the tests a testing framework is needed. This framework was implementedfrom scratch. The implementation was created in C++. This was the preferred way sincethe MIIPS app implementation is mostly in C++ and the Mosquitto library had a frame-work available as well. The major parts of the implementation will be described in thissection.
6.3.1 Mosquitto wrapper
This part of the implementation is essential to communication with Mosquitto. The Mosquittolibrary has a wrapper of its own to the MQTT protocol. The Mosquitto wrapper implementsthis wrapper in an easy way to provide support for only the essential parts of the protocol.This include functions to for example connect, publish and subscribe to the broker.
6.3.2 Client handler
This part provides support to create and manage multiple clients in an easy way. For sometests a lot of clients will be created simultaneously and that is the purpose of this class. Theclass is also used to spam messages to a broker from multiple or a single client. This classis a central part of most tests.
16(48)
6.3.3 Output handler
To handle output format and location an output handler class is used. This simplifies savingand handling the data by automating as much of it as possible. The output handler can beused to save data to a csv-file. The class can be used if data is observed from the ’$SYS’topic of the broker, from a system call or to write custom lines to a file.
6.4 Interpreting results
This section provides an explanation of how to interpret the results presented in Section8.
6.4.1 Box-plot
The box-plot is an ideal way to represent data when a lot of samples are used. The box-plot shows the median while also displaying the distribution of the data. In Figure 3 is anexplanation of the different fields in a box-plot.
Figure 3: An explanation of how to interpret box-plots
Median - The line in the box is the median. The median is a common way of representingthe center of the data.
Interquartile range box - The box between the lower and upper quartile represents themiddle 50% of the data.
17(48)
Whiskers - The whiskers represent the lower and upper 25% of the data. This excludes theoutliers.
Outliers - The points outside of the whiskers are the outliers of the data.
6.4.2 T-test
The T-test is a statistical method to determine if there is any significant difference betweentwo data sets. A T-test is usually set up by two hypotheses:
H0 : No significant difference between the two data sets.
H1 : There is a significant difference between the two data sets.
The T-test is designed to reject the null hypothesis (H0) thus proving there is a large chancethe alternative hypothesis (H1) is correct. The value of most significance in the T-test isthe p-value. This value can be interpreted as the chance that the null hypothesis is correct.This implies that a low p-value increases the chance that the data sets are in fact different.Usually the threshold of the p-value is 0.05 or 5%. A value lower than this is statisticalproof that the data sets have a significant difference.
18(48)
19(48)
7 Test scenarios
This section presents the test scenarios used to gather results.
7.1 Test latency and CPU usage at different message loads
To run these tests two different computers and the broker are used. The first computer runsa message load while the second one subscribes and publishes messages to an own topic.One publish is sent once a second and then waits for the broker to send it back. While thisis done the broker runs vmstat (from Section 6.1.2) to gather CPU information. See Figure4 for further explanation. This test is run twice. Once with messages of 1kb and once withmessages of 50kb each.
Figure 4: An example of how the testing setup looks like. This setup is used for most testswhere latency and CPU usage are the monitored metrics.
7.2 Test latency and CPU at different levels of QoS
This test is run by simply publishing and subscribing to messages of different levels of QoS.Each test is observed over five minuets with one sample each second. The CPU is tested in
20(48)
the same way as in Figure 4. All QoS levels was tested with 10 messages a second at 50kbeach and with 500 messages a second at 1kb each.
7.3 Test latency and CPU with many stored messages
If a client disconnects unexpectedly or with the clean flag set to 0 the broker have to saveall messages with a QoS of at least 1. Assuming the client also subscribed with a QoS ofat least 1. This could cause the broker to use a lot of heap memory. This test connects aclient with clean flag set to 0 and subscribes to a topic with QoS set to 1. The client thendisconnects. Another client later publishes messages to the same topic with QoS set to 1.This will cause all the messages to be stored in the broker. 500 mega bytes is publishedto the broker and then CPU and latency are measured in the same way as the other tests.Figure 5 shows an visualization of this test. These results are then compared to the samemeasurements without any stored messages on the broker. This is important to see if thereis any negative effects of storing messages except for the increased memory usage.
This test is done in two different scenarios. The first is with the clean flag set 0 and the clientperforms a controlled disconnects. The second scenario is when the client unexpectedlydisconnects. This is performed by disconnecting it from the internet.
Figure 5: A visual example of the setup used to test stored messages effect on CPU andlatency.
7.4 Testing bridge
Bridge mode is when one broker is set up to distribute certain topics to or from anotherbroker. The bridge is configured with different rules of what topics are distributed both inand out from the bridge. In these tests two scenarios will be used.
One scenario is when the subscriber is connected to the bridge, and messages are published
21(48)
to the other broker which is not in bridge mode. This is displayed in the right side of Figure6. This will be referred to as subscribe on bridge in the rest of this paper.
The other scenario is when the subscription is on the broker not in bridge mode and mes-sages are published to the bridge, as in the left side of Figure 2. This will be referred to aspublish on bridge. In both these scenarios the messages will be forwarded between both thebrokers to reach the subscribers.
The bridge can also add prefixes and suffixes to its messages. This makes it possible to addcertain information for example about what broker the message was originally publishedto. Any added prefixes or suffixes will be referred to as ”mapped”. And if no prefixes orsuffixes are added or removed it will be referred to as ”not mapped”. Further explanation ofbridge mode and its configuration are found in Section 3.3.
Figure 6: A demonstration of how the bridge and broker are set up in the tests.
7.4.1 Testing different bridge configurations
The first test run on the bridge is to see if subscribe on bridge or publish on bridge is moresuitable. Another one is to determine if using mapping has any affect on the bridges effec-tiveness. Finally a test is done to compare a broker running in bridge mode to a broker notin bridge mode. This is done by having the broker in bridge but subscribing and publishingto a topic not shared to the other broker. All these tests are done to measure both latency andCPU. To measure the CPU, 500 messages of 1 kilobytes is sent each second while the CPUusage is observed. To measure latency a single message is sent each second. One sample istaken each second for 120 seconds each. The results are finally compared to each other toget a sense of which configurations works well and which ones does not.
7.4.2 Comparing multiple topics with topic wild card
The bridge configuration decides what topics are shared between the brokers. If multiplemessages are to be sent at different topics the configuration can either provide every topicor just use the wild card. An example of a wild card is ”topic/#”. This test compares theCPU usage if the bridge specifies 100 different topics, one wild card topic or just one topicwhere all the messages are sent. 500 message of 1 kilo byte is sent each second, distributedevenly to 100 different topics for the first two cases. For the case with just one topic, all 500messages are sent to that specific topic.
22(48)
23(48)
8 Results
This section presents all results from the test scenarios in Section 7. These results arevisualized in box-plots and line-plots.
8.1 Compare message loads
This test is used to determine if the broker works better when sending a lot of small messagesor a few larger ones.
8.1.1 CPU usage
Figure 7 displays these results in two box-plots. This shows how the CPU load is muchhigher when sending a lot of small messages compared to sending the same total size infewer messages.
Figure 7: Boxplot to compare CPU usage when sending a lot of smaller messages or a fewlarger ones. Both boxes send 500kb/second. The left one with 1kb messages and the rightwith 50kb messages. Each box is 120 samples, one each second.
8.1.2 Latency
Figures 8 and 9 show how the latency scale with different message loads. Figure 8 showsthis with messages of 50kb and Figure 9 is with messages of 1kb each.
24(48)
Figure 8: Graph displaying the latency of a message at different message loads whensending messages of 50kb each. The x-axis is the number of messages sent each second tothe broker. Each message is 1kb. The y-axis is the median of 120 samples of the latency.
Figure 9: Graph displaying the latency of a message at different message loads whensending messages of 1kb each. The x-axis is the number of messages sent each second tothe broker. Each message is 1kb. The y-axis is the median of 120 samples of the latency.
25(48)
8.2 Compare QoS levels
This test determines if there is difference between the different levels of QoS.
8.2.1 Latency
Figure 10 displays the different latency times in three box-plots. There is an obvious differ-ence between QoS 1 and QoS 2 but barely any difference between QoS 0 and QoS 1. Theboxes are thin which implies there are very little variation in the samples.
Figure 10: Box-plot displaying the latency of 500 messages for QoS 0, 1 and 2. Onesample is taken each second for 500 seconds.
8.2.2 CPU usage
Figure 11 and 12 show how the CPU is affected by different QoS levels. Both tests usea total of 500kb sent each second, one with 10 messages of 50kb and the other one with500 messages of 1kb. Figure 11 shows no significant difference while Figure 12 showsan increasing load with higher QoS level. A deduction can be made that the overhead ofthe QoS does not give an significant effect when only a few messages are sent but is verysignificant when a lot of messages are sent.
26(48)
Figure 11: Box-plot displaying the CPU usage at different QoS levels. 10 messages, of 50kb each, are sent every second. One sample is taken each second for 120 seconds.
Figure 12: Box-plot displaying the CPU usage at different QoS levels. 500 messages, of 1kb each, are sent every second. One sample is taken each second for 120 seconds.
8.3 Does stored messages impact performance?
These tests are designed to evaluate if stored messages on the broker, due to disconnectedclients or similar reasons, affects the performance of the broker. The results are based onthe scenario when a client is connected with clean flag set to 0 and performs a controlleddisconnect.
27(48)
8.3.1 Latency
Figure 13 shows the latency difference between an empty broker and a broker with 500mega bytes of stored messages. As showed in the box-plots there is barely any differencein latency between the two. A T-test was also run on the two data sets. The T-test provideda p-value of 0.54 which also proves there is no significant difference between the two datasets.
Figure 13: Box-plot displaying the latency of messages. The red box show the samples runon an empty broker while the blue box have 500mb stored messages. One sample is takeneach second for 300 seconds.
8.3.2 CPU usage
Figure 14 shows the result when there is a running load on the broker. There is barely anydifference between the boxes which implies there is no significant effect on the brokersperformance. However, Figure 15 displays the same test when no load is run on the broker.In these results there is a small but significant increase when the broker has stored messages.This implies that there is an increase of CPU usage when the broker is under very low stressbut not when it is under high stress.
28(48)
Figure 14: Box-plot displaying the CPU usage of the broker. The red boxes show thesamples run on an empty broker while the blue boxes have 500mb stored messages. TheCPU is measured while there is running load on the broker. The load is 20 messages of50kb each second. One sample is taken each second for 120 seconds.
Figure 15: Box-plot displaying the CPU usage of the broker. The red boxes show thesamples run on an empty broker while the blue boxes have 500mb stored messages. TheCPU is measured while no load is run on the broker. One sample is taken each second for120 seconds.
29(48)
8.4 Does uncontrolled disconnects impact performance?
This test is the same as the previous Section 8.3 but with an uncontrolled disconnect per-formed by unplugging the internet connection. In theory the results should be very similarto the results of the controlled disconnect. However, if Figure 16 is observed a strangebehaviour can be spotted. The figure plots the heap memory of the broker after an uncon-trolled disconnect. First around 100mb is sent to the broker with QoS 1. This is saved in thebroker. The red line indicates were messages are not longer being sent to the broker. Thisshould imply that the memory of the broker stops increasing. But as shown in the figure thememory keeps increasing until all messages are dropped after about 50 seconds. This is asevere bug in the broker implementation. This is further discussed in the analysis Section9.2.2.
Figure 16: Line-plot with time along the x-axis and heap memory of the broker along they-axis. The red line indicates where messages are no longer sent into the broker.
8.5 Testing bridge configuration
In Figure 17 the CPU usage is displayed for all the tested configuration settings. The boxeslook very similar at first but some conclusions can be made. Both samples with mappinghas a bit higher median then their counterparts. This implies there is some extra load onthe bridge if mapping is used. The sample with no bridge enabled is a bit lower then allthe other samples. This proves the bridge configuration does put some extra load on thebroker, even if its not actively used. However these differences are very small and shouldnot impact performance enough to make a significant difference.
Figure 18 displays the same test scenario but with latency as the measured unit. These re-sults are very similar to the CPU but even clearer since the samples have a smaller deviation.Worth noting is that the difference between the bridge mode samples and the one withoutbridge mode is a bit larger. Still the difference across all configurations is very small andshould not be noticeable in practical scenarios.
30(48)
Figure 17: Box-plot comparing CPU usage of different bridge configuration. Configu-rations from the left is; subscribe on broker without mapping, subscribe on broker withmapping, publish on broker without mapping, publish on broker with mapping, subscribeand publish on broker in bridge mode, subscribe and publish on broker not in bridge mode.
Figure 18: Box-plot comparing latency of different bridge configuration. Configurationsfrom the left is; subscribe on broker without mapping, subscribe on broker with mapping,publish on broker without mapping, publish on broker with mapping, subscribe and publishon broker in bridge mode, subscribe and publish on broker not in bridge mode.
31(48)
8.6 Comparing multiple topics with topic wild cards
This test shows if there is any point in using multiple topics, one topic or topic wild cardswhen redirecting message with a broker in bridge. Figure 19 shows the results of this ina box plot. The medians are very similar for all three methods. The deviation is howeverhighest for multiple topics. This is also the one with highest peaks. The wild card sampleshows lower peaks than the previous one but also bigger deviation then when just using onetopic. However the results are very similar and the method used should not really have anysignificant effect.
Figure 19: Box-plot comparing CPU usage between using one topic, multiple topics ortopic wild cards.
32(48)
33(48)
9 Analysis
This section provides deeper analysis of the results. It will also answer the major questionsfor this thesis with possible solutions and an analysis of bugs in the Mosquitto implementa-tion.
9.1 How does Mosquitto store messages that can not be distributed to sub-scribers due to lost connection?
The major questions for this thesis are how the Mosquitto behaves when clients disconnectand messages have to be stored in the broker. This behaviour is broken down into twoscenarios. For both scenarios the client has QoS set to at least 1.
9.1.1 Controlled disconnect
The first scenario is when a client has clean flag set to 0 and do a controlled disconnect.When this happens the broker is aware that the client has disconnected. Therefore it doesnot try to send it any messages until it has reconnected. All messages sent with a QoS 1 orQoS 2 will then be stored in the broker until the client reconnects. This can force the brokerto save messages for a very long time which will use up memory.
How many and how large messages can be stored are decided in the brokers configuration.The default values are 120 messages of 268 megabytes. This is more than 20 gigabytes ofdata. Since the broker used in the tests has a RAM of 1 gigabyte, the default values will notprevent the broker to run out of memory.
9.1.2 Uncontrolled disconnect
The second scenario is when a client is disconnected unexpectedly. This is usually causedby losing connection to the internet or an error in the client application. To emulate thisbehaviour the internet connection of the client is disabled after it has subscribed to thebroker. The broker will not receive any indication from the client of the disconnect, thus ittries to send messages to it. Any messages published to the topic of the clients subscription(with QoS 1 or QoS 2) will be placed in the inflight queue. This queue is by default limitedto 20 messages. The messages in the queue will be resent to the disconnected client until itis received or the clients connection times out. The messages will by default be sent onceevery 20 seconds. This option and the inflight queue limit can be set in the configurationby changing the retry interval and max inflight messages option. Any connected client willchoose a keep alive option to determine how long the broker will keep the connection untilit finally times out. This option is set to 60 by default, which most clients will use.
34(48)
9.2 Will storing messages in the broker affect the MIIPS overall performance?
Like in the previous section the performance of the MIIPS also depends on if the subscriberperforms a controlled or and uncontrolled disconnect.
9.2.1 Controlled disconnect
In figures 13, 14 and 15 the CPU and latency where measured with 500 megabytes of datasaved in the broker. The results were compared to an empty broker. The only time it seemsto have an effect is for the CPU usage on a broker without any running message load. Underhigher message loads the broker performs the same with stored messages as without. Thelatency is also the same for both cases.
This implies there is no problem with storing messages as long as there is free memoryavailable. However the memory is limited and becomes even more limited if there are manyother running applications on the system. If the system starts to run out of memory theoperating system (Linux) will look for processes using a lot of the memory. It will thenterminate those processes. This will cause the Mosquitto to be shut down. The MIIPS unitdoes not start up Mosquitto once shut down so it will stay terminated until manually startedor the unit is restarted. When the Mosquitto shuts down all the applications using it tocommunicate both internally and externally will stop working correctly.
This can and should be limited by limiting the stored messages in the configuration file.This way the broker will not use up too much memory and will not be forces to shut down.Instead it will not accept any more messages once the limit is reached.
9.2.2 Uncontrolled disconnect
With the behaviour explained in the previous question from Section 9.2.2 taken in mind, theresults of a controlled disconnect and an uncontrolled disconnect should be quite similar.The uncontrolled disconnect will perhaps use a little more CPU since it will try to resendthe data. This is not the case because of how the Mosquitto handles the memory when ittries to resend the data. As displayed in Figure 16 the memory in the broker should stopincreasing at the red line. However, it keeps growing rapidly until the client times out. Thisis probably a memory related bug. With some further observation of the memory it doesincrease every time the broker tries to resend the messages. The implementation most likelyallocates memory for the messages in the queue every time the broker tries to resend them.This memory is most likely not freed properly. The limits set by the broker does not preventthis bug. The configurations only limit messages stored but since no more messages getstored this will not help.
This is definitely a severe bug since most disconnects will happen this way. This is a sce-nario that will occur from time to time and could unfortunately cause the broker to use upall memory and terminate the Mosquitto application. If the clients are good willed they willusually keep the keep alive to the standard 60 seconds. If the broker also have a limit onmessage size this should usually be enough for the broker not to run out of memory. Themessages will only be resent about 3 times until the client times out.
Since the broker can not limit clients keep alive time bad willed clients can take advantageof this bug. One client can set the keep alive time very high, subscribe to a topic with QoS 1
35(48)
and then disconnect. Another client can then publish messages to this topic. This will causethe memory to grow indefinitely and eventually the broker will run out of memory and shutdown.
9.3 Will the current configuration options be enough to limit the memoryusage of the protocol?
As mentioned in the previous section there are a number of options used to limit the amountof memory stored in the broker. The key configuration options are max inflight messages,max queued messages and message size limit. These options are explained in the MosquittoSection 6.1.4. If the memory bug in the previous section is excluded, these options can beused to limit the total memory of the messages stored on the broker.
The total number of allowed stored messages is the max inflight messages plus themax queued messages. This total number of stored messages multiplied withmessage size limit is the total memory that can be stored by the broker.
The major concern with this is that just a few messages might be of max size while amajority is way smaller. Then the max queued messages have to be way lower than actuallynecessary. To make the options sufficient to prevent a overloaded broker by still using itsfull capacity a total size limit should be added. This would provide a way to limit the totalmemory usage of the broker but still maintaining flexibility.
As long as the memory bug is present an option to limit the keep alive time should be addedto the configuration. The broker should have full control of how long it will need to resendmessages which is not yet delivered. Even if the bug is fixed this option would be good toprevent the broker from storing messages for an unnecessary long time.
9.4 Will using the broker in bridge mode affects the MIIPS performance?
Figures 17 and 18 can be used to answer if bridge mode affects the performance of theMIIPS. These figures display how the difference between using the broker in bridge con-figuration or not is very minor. There is no significant difference regarding CPU usage inany of the configurations. There is a tiny difference in latency between a broker in bridgeconfiguration and a broker not in bridge. This does imply there is a slight disadvantage ofusing a broker in bridge, thus it would not be recommended to do this without purpose.However if there is a purpose for the broker to be in bridge this latency is at most 0.5mswhich in not that significant.
The figures also show if there are any negative effects of topic mapping. This difference iseven smaller then the bridge configurations but also show slightly higher results when usingtopic mapping compared to no mapping.
9.5 Solutions
To prevent the broker from using too much memory there are a few solutions.
36(48)
9.5.1 Limiting storage in configuration
As discussed previously the broker can set a couple of options to limit the amount of mem-ory the broker can use. This is highly recommended since it will not cause the Mosquittoapplication to be terminated if it uses to much memory. This will at least keep the brokerrunning so applications depending on it does not suffer.
The problem with this limit is that the broker has no way to prioritize the messages itreceives. Thus if the queues get full and it receives a very important message it shouldstore, it will not be able to do that. This could cause essential information to be lost becausethe broker is full with less important messages.
9.5.2 Using a status topic
To solve the problem of prioritizing messages a ”status topic” could be used. The brokershould still have a configuration to limit stored messages but also implement a way to com-municate its status. This would imply some extra logic on the broker unit. The unit hostingthe broker would also have a client publishing the status of the broker. Good willed clientscan then subscribe to this topic to get information about the brokers status.
The brokers own client would monitor the memory usage of the broker. If the broker haslots of memory available, it would publish a message saying everything is fine. If the brokeris almost out of memory, the client would publish an message telling other clients not topublish more messages. This way a client will know if the broker is getting full and notpublish unnecessary messages while still having the option to publish important ones. Thebroker can also publish a message saying the broker is completely full. This way clientscan hold on to important messages and not publish them until the broker publishes a newmessage in the status topic.
9.5.3 Restarting Mosquitto
The last part of a solution is to have an automatic restart for Mosquitto. This should not beused as a solution of its own but rather as a backup if the memory bug or other issues wouldoccur. The Mosquitto process will be terminated when it uses up all available memory of theunit. To prevent all applications dependent of the Mosquitto to malfunction, the Mosquittoshould be restarted when terminated. This would cause the other applications to simplywait for the Mosquitto to reboot and this would minimize the problems caused to otherapplications.
9.6 Update 1.4.15
A new update was added in 2018-02-28. In this update a new configuration option mem-ory limit was added. This option is not documented in the Mosquitto documentation [8],but is added in the implementation. This option does fix the issue of limiting message sizeor the size of the queue (described in Section 9.3). This is an important option to limit thememory usage without sacrificing unnecessary performance.
This update does not fix the memory bug. The limit probably rejects messages when the
37(48)
limit is reached. However the bug does not cause more messages to be received. The bugjust duplicates the memory of the messages already received. Since no more messages arereceived the limit will not have any effect on this problem.
9.7 Other Interesting results
There are a lot results in the result section that gives important knowledge about the Mosquittobut is not directly linked to the problems formulation of the thesis. This section will takeup some of the more interesting results that does not fit the description of the major ques-tions.
9.7.1 Difference in message loads
An interesting result for users of Mosquitto is the difference of sending a few large messagesor a lot of small ones. As seen in Figure 7 the CPU usage is a lot higher when sending alot of messages compared to a few ones, even if the total size is the same. This is expectedsince there is the same amount of overhead for each message whether it is big or small.The odd result is displayed in Figure 9 which shows the latency of messages at differentmessage loads. For small messages as in this figure the latency actually decreases whenmore messages are sent. The expected results would be the opposite that latency increaseswith the message load. This behaviour is hard to explain but could be the result of the brokernot being as responsive when idle. This could cause it to go into a shorter sleep or a similaridle state. This seems unlikely but is one plausible explanation.
9.7.2 Quality of Service
The most important thing with Quality of Service levels is the major difference betweenQoS 1 and QoS 2 (Figure 10). Since the latency is almost the same with QoS 0 and QoS 1but twice as high with QoS 2, QoS 2 should be avoided if there is no specific reason to useit. Since QoS 1 offers guaranteed delivery this should be used frequently instead of QoS 2.The reason to use QoS 2 is to ensure that a message only gets delivered once. This couldeasily be implemented in the application instead so the delay of QoS 2 does not affect theperformance.
38(48)
39(48)
10 Conclusion
In conclusion the Mosquitto implementation is a suitable way of communicating in an em-bedded system environment. The minimal overhead makes it a good option for sendingmessages between different parts or nodes in a system. With the latest update 1.4.15 de-scribed in Section 9.6 it is easier to limit the memory usage on the device using Mosquitto.The option added in the update is memory limit which gives the option to strictly limit howmuch memory the Mosquitto will store as messages.
Storing messages in the broker does not significantly impact its performance and is not anissue unless the broker runs out of available memory. The same is true for bridge config-uration which barely affects the brokers performance. If there is a good reason to use thebroker in bridge, performance should not be of any concern.
The limited memory of the MIIPS only limits how many messages and how large messagescan be stored. If this is taken to account when designing the systems using the MIIPS, thisshould not cause problems. If the system however uses all available memory it will causesome messages to not be queued. Thus some clients may not receive all the messages. TheMosquitto will continue working even if this does occur.
The major concern is the bug discussed in the analysis Section 9.2.2. This bug can causethe application to crash entirely which will cause all dependent applications and devices tonot receive their messages. The probability of this crash can be avoided by changing someoptions in the configuration. However adversary clients with knowledge of this bug canexploit it to cause the broker to crash. This bug is most likely a memory related issue andthe developers of Mosquitto should be able to fix it in upcoming releases.
40(48)
41(48)
Bibliography
[1] Faltcom. upptack-miips-iot-plattform.
[2] M. Gerla, E.K Lee, G. Pau, and U. Lee. Internet of Vehicles: From Intelligent Grid toAutonomous Cars and Vehicular Clouds. 2014.
[3] L. Hu, M. Qiu, J. Song, M.S. Hossain, and A. Ghoneim. Software defined healthcarenetworks, volume 22. IEEE Wireless Commun, 2015.
[4] Eurotech International Business Machines Corporation (IBM). Mqtt v3.1 protocolspecification.
[5] Mae Kowalke. The pros and cons of the major iot communications protocols.
[6] Light Roger. Mosquitto - an open source mqtt v3.1/v3.1.1 broker.
[7] Light Roger. mosquitto — an mqtt broker.
[8] Light Roger. mosquitto.conf — the configuration file for mosquitto.
[9] Light Roger. mqtt — mq telemetry transport.
[10] Lampkin Valerie, Leong Weng, Tat, Olivera Leonardo, Rawat Sweta, SubrahmanyamNagesh, and Xiang Rong. Building Smarter Planet Solutions with MQTT and IBMWebSphere MQ Telemetry. Redbooks, 2012.
42(48)
43(48)
A Appendix
This appendix contains some tests and results not essential for the thesis questions. Thetests still shows some interesting results of the behavior of Mosquitto.
A.1 Test scenarios
A.1.1 Test heap memory on subscribers and clients
This test observes if memory usage of clients and subscribers are linear and compares themto each other. This test was performed by running one subscriber to the system topics”$SYS/broker/clients/active”, ”$SYS/broker/subscriptions/count” and”$SYS/broker/heap/current”. The implemented program was first used to increase numberof subscribers and later the number of clients.
A.1.2 Test latency and CPU usage at different message loads
To run these tests two different computers and the broker are used. The first computer runsa message load while the second one subscribes and publishes messages to an own topic.One publish is sent once a second and then waits for the broker to send it back. While thisis done the broker runs vmstat (from Section 6.1.2) to gather CPU information. See Figure4 for further explanation. This test is run twice. Once with messages of 1kb and once withmessages of 50kb each.
44(48)
Figure 20: An example of how the testing setup looks like. This setup is used for mosttests where latency and CPU usage are the monitored metrics.
A.1.3 Test latency and CPU with different number of subscribers
This test is done similarly to the previous one testing message loads. This test uses the samesetup as displayed in Figure 4 but the rate of sent messages in constant. Instead the numberof subscribers on that topic is increased. This will cause the messages returned from thebroker to increase but the messages sent from the client will stay the same. This test wasdone by sending 10 messages each second at a size of 10kb each.
A.2 Results
A.2.1 Test heap memory on subscribers and clients
This graph (Figure 21) shows how both clients and subscribers memory usage scale linearly.It also displays how a client with a subscription uses more memory than one without anysubscriptions.
45(48)
Figure 21: A line graph displaying the connected number of clients and subscribers towardsthe used heap memory in the broker
A.2.2 Test latency and CPU usage at different message loads
Figure 22 and 23 show that the CPU-load is quite linear to the message load at first. Whenit reaches a certain threshold the usage almost becomes constant. The threshold seems to beat a higher CPU load with the 50kb messages then the 1kb messages.
The latency of the 1kb messages in Figure 9 decreases drastically when the message loadis increased. It decreases fast until about 1000 messages a second and then slows down. Itseems to start increasing again at a very high amount of sent messages each second. Whensending messages of 50kb as in Figure 8 the latency increases with the message load. Itshows a big peak at around 200 messages a second and later it seem to stabilize.
46(48)
Figure 22: Box-plot displaying the CPU usage on different message loads. The x-axis isthe number of messages sent each second to the broker. Each message is 1kb. The y-axis isthe total CPU-usage as a percentage. A sample is taken each second for 120 seconds.
Figure 23: Box-plot displaying the CPU usage on different message loads. The x-axis isthe number of messages sent each second to the broker. Each message is 50kb. The y-axisis the total CPU-usage as a percentage. A sample is taken each second for 120 seconds.
47(48)
A.2.3 Test latency and CPU with different number of subscribers
This test shows how Mosquitto scale with multiple subscribers at a single topic.
Latency
Figure 24 displays how latency increases when more subscribers are added. The latencyseems to be growing exponentially and is at a reasonable increase until around 75 sub-scribers. After this it increases very fast and becomes substantially slow.
Figure 24: Line-plot displaying the relationship between latency and number of sub-scribers. The latency is the median of 120 samples
CPU usage
Figure 25 shows the CPU usage in a box-plot. This seems to scale fairly linearly with thenumber of added subscribers.
48(48)
Figure 25: Box-plot showing CPU usage with different number of subscribers. 10 mes-sages of 10 kb is sent every second. One sample is taken each second for 120 seconds.