catching bugs in the internet of things - from art to science · tugraz institute of software...
TRANSCRIPT
tugrazInstitute of Software Technology
Catching Bugs in the Internet of Things -from Art to Science
Bernhard K. Aichernig
Institute of Software TechnologyGraz University of Technology, Austria
Bozen - Bolzano , 17 Nov 2016
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science1 / 24
tugrazInstitute of Software Technology
Dependability of the IoT
We need a science to make the IoT as dependable as the power grid.
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science2 / 24
tugrazInstitute of Software Technology
Dependable Things
Things of high quality come with a warranty.
Are we ready to provide a warranty on our connected things withrespect to
I security?I safety?I correctness?I reliability?I availability?I maintainability?
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science3 / 24
tugrazInstitute of Software Technology
Agenda
I Quality of Things in the IoTI Dependable Things ProjectI Model-based TestingI Learning-based TestingI Results on MQTT Brokers
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science4 / 24
tugrazInstitute of Software Technology
Limited Warranties
I Cisco: “... In no event does Cisco warrant that theSoftware is error free or that Customer will be able tooperate the Software without problems or interruptions....”(http://www.cisco.com/public/limited-warranty.html)
I Skydrop Sprinkler Controller: “This warranty does notcover consumable parts, including batteries, unlessdamage is due to defects in materials or workmanship ofthe Product, or software (even if packaged or sold withthe product).”(https://www.skydrop.com/warranty/)
Marketing withApp
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science5 / 24
tugrazInstitute of Software Technology
Pollution of the Internet
I Distributed denial-of-service attack from IoT (21 Oct 2016)I on domain name provider DYNI Twitter, Pinterest, Reddit and PayPal went down for most of a dayI DYN estimated 100,000 malicious thingsI Source of Attack: Mirai botnet
I Mirai is malware that attacks vulnerable IoT devicesI scans for standard factory default usernames and passwordsI infected things listen to control server of the botnet
Bruce Schneier. Your WiFi-connected thermostat can take down thewhole Internet. We need new regulations. The Washington Post, 3 Nov2016.
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science6 / 24
tugrazInstitute of Software Technology
Mirai Infection Map
Source: https://intel.malwaretech.com/botnet/mirai(14 Nov 2016, 13:18)
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science7 / 24
tugrazInstitute of Software Technology
1st LEAD Project of TU Graz
dependablethings.tugraz.at
I Excellence initiative funded by TU GrazI 2 faculties, 10 key researchers + 10 PhD studentsI Initial phase: 2016–2018I Research questions:
I Systematic design for dependability?I Which provable guarantees can be given?I Which practical assumptions can be made?I Models of environment and system?I Verifying models and assumptions?I Compensation across components?
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science8 / 24
tugrazInstitute of Software Technology
Multidisciplinary Team of Key Researchers
I SP1: Dependable Wireless
I Wolfgang Boesch: Microwave EngineeringI Kay Römer: Embedded NetworkingI Klaus Witrisal: Wireless Signal Processing
I SP2: Dependable Computing
I Marcel Baunach: RT Operating SystemsI Stefan Mangard: Embedded Security
I SP3: Dependable Composition
I Bernhard Aichernig: Model-Based TestingI Roderick Bloem: Formal VerificationI Franz Pernkopf: Machine Learning
I SP4: Dependable Networked Control
I Martin Horn: Control TheoryI Gernot Kubin: Information Theory
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science9 / 24
tugrazInstitute of Software Technology
Catching Bugs: Objective I
Writing good test cases is hard!
Don’t write test cases,
generate them!
(John Hughes)
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science10 / 24
tugrazInstitute of Software Technology
Model-based Testing
Requirements
Model
Test-CaseGenerator
AbstractTest Cases
Test DriverSystemUnder Test
formalise
testspass
satisfies
Automated tasks:
I model verification
I test-case generation
I test-case concretion
I test-case execution
I assignment of verdicts
Manual tasks:
I (requirements analysis)
I model creation
I model validation
I concretion implementation
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science11 / 24
tugrazInstitute of Software Technology
MoMuT ToolsMoMuT
I is a family of tools implementing Model-based Mutation Testing.
I is jointly developed and maintained by AIT and TU Graz
I has been applied in industry: AVL, Thales Railways, Infineon, Volvo.
I supports different modelling styles:
I MoMuT::UMLI MoMuT::OOASI MoMuT::TAI MoMuT::Reqs
www.momut.org
Bernhard K. Aichernig, Jakob Auer, Elisabeth Jöbstl, Robert Korosec, Willibald Krenn, RupertSchlick, Birgit Vera Schmidt: Model-Based Mutation Testing of an Industrial MeasurementDevice. TAP 2014: 1-19
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science12 / 24
tugrazInstitute of Software Technology
Modelling is Hard: AVL489 Particle CounterAVL489
isReady
isBusy
Pause_0
send SPAU state /entrysend StatusBusy; set Busy /exit
Standby_1
send STBY_state /entrysend StatusBusy; set Busy /exit
Active
Purging_Pause_12
send SPUL_state /entry
Purging_Standby_12
send SPUL_state /entry
Response_14
send SEGA_state /entry
Leakage_11
send SLEC_state /entry
Integral_9
send SINT_state /entrysend StatusBusy; set Busy /exit
Measurement_2
send SMGA_state /entrysend StatusBusy; set Busy /exit
ZeroGas_10
send SNGA_state /entrysend StatusBusy; set Busy /exit
Manual
set Manual /entry
Remote
unset Manual /entry
DilutionSelection [ not Manual and not Busy ] / set Dilution
LeakageTest, ResponseCheck [ not (oclIsInState(Standby_1)) and not Manual and not Busy ] / send RejectNA
SetPurge [ not (oclIsInState(Pause_0) or oclIsInState(Standby_1)) and not Manual and not Busy ] / send RejectNA
SetZeroPoint [ not oclIsInState(Active::Measurement_2) and not Manual and not Busy ] / send RejectNA
StopIntegralMeasurement [ not oclIsInState(Active::Integral_9) and not Manual and not Busy ] / send RejectNA
StartMeasurement [ not (oclIsInState(Standby_1) or oclIsInState(Active::Integral_9)) and not Manual and not Busy ] / send RejectNA
StartIntegralMeasurement [ not (oclIsInState(Active::Measurement_2) or oclIsInState(Active::Integral_9)) and not Manual and not Busy ] / send RejectNA
when Busy
30 [ not (oclIsInState(Active::Response_14) or oclIsInState(Active::Purging_Standby_12) or oclIsInState(Active::Leakage_11) or oclIsInState(Active::ZeroGas_10) or oclIsInState(Active::Purging_Pause_12)) ] / set not Busy - send StatusReady
LeakageTest, ResponseCheck, SetPurge, SetZeroPoint, StopIntegralMeasurement, SetStandby, StartMeasurement, StartIntegralMeasurement, SetPause, DilutionSelection [ not Manual ] / send RejectBusy
SetStandby [ not Busy and not Manual ]
SetPurge [ not Busy and not Manual ]
SetPause [ not Busy and not Manual ]
SetPause [ not Busy and not Manual ]
SetPause [ not Busy and not Manual ]
10
SetStandby [ not Busy and not Manual ]
SetPurge [ not Busy and not Manual ]
LeakageTest [ not Busy and not Manual ]
StartMeasurement [ not Busy and not Manual ]
ResponseCheck [ not Busy and not Manual ]
10
10
10
SetStandby [ not Busy and not Manual ]
StartIntegralMeasurement, StopIntegralMeasurement, StartMeasurement [ not Busy and not Manual ]StartIntegralMeasurement [ not Busy and not Manual ]
SetZeroPoint [ not Busy and not Manual ] 10
/ send Offline
SetRemote / send Online
DilutionSelection, LeakageTest, ResponseCheck, SetPurge, SetZeroPoint, StopIntegralMeasurement, SetPause, SetStandby, StartMeasurement, StartIntegralMeasurement / send RejectOF
SetManual
SetManual / send Offline
SetRemote
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science13 / 24
tugrazInstitute of Software Technology
Catching Bugs: Objective II
Don’t create models,
learn them!
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science14 / 24
tugrazInstitute of Software Technology
Learning of Models
Requirements
Requirements
Model
Test-CaseGenerator
AbstractTest Cases
Test DriverSystemUnder Test
ReferenceSystem
Learner
formalise
testspass
satisfies
satisfies /“defines”
conforms to
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science15 / 24
tugrazInstitute of Software Technology
Minimally-Adequate-Teacher Framework (1) –Learning a Regular Language
TeacherLearningAlgorithm
Equivalence Query (Hypothesis Model)
Yes / Counterexample
Membership Query
Query Answer
BuildHypothesis
Angluin’s L∗-AlgorithmDana Angluin. Learning regular sets from queries and counterexamples.Information and Computation, 75:2, 1987.
Example - Language L over alphabet {0, 1}
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science16 / 24
tugrazInstitute of Software Technology
Minimally-Adequate-Teacher Framework (1) –Learning a Regular Language
TeacherLearningAlgorithm
Equivalence Query (Hypothesis Model)
Yes / Counterexample
Membership Query
Query Answer
BuildHypothesis
Example - Language L over alphabet {0, 1}I L contains strings with even number of 0- and 1-symbols, i.e.
L = {ε, 00, 11, 0000, 0011, 1111, 0110, 1001, . . .}I Learn DFA accepting L in black-box setting
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science16 / 24
tugrazInstitute of Software Technology
Minimally-Adequate-Teacher Framework (1) –Learning a Regular Language
TeacherLearningAlgorithm
Equivalence Query (Hypothesis Model)
Yes / Counterexample
Membership Query
Query AnswerBuild
Hypothesis
Example - Language L over alphabet {0, 1}Learner Teacherε ∈ L ? yes0 ∈ L ? no1 ∈ L ? no00 ∈ L ? yes01 ∈ L ? no
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science16 / 24
tugrazInstitute of Software Technology
Minimally-Adequate-Teacher Framework (1) –Learning a Regular Language
TeacherLearningAlgorithm
Equivalence Query (Hypothesis Model)
Yes / Counterexample
Membership Query
Query AnswerBuild
Hypothesis
Example - Language L over alphabet {0, 1}
q0start q1
0
1
0
1
Counterexample: 11 ∈ L but not accepted
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science16 / 24
tugrazInstitute of Software Technology
Minimally-Adequate-Teacher Framework (1) –Learning a Regular Language
TeacherLearningAlgorithm
Equivalence Query (Hypothesis Model)
Yes / Counterexample
Membership Query
Query AnswerBuild
Hypothesis
Example - Language L over alphabet {0, 1}
q0start q1
q2 q3
0
1 0 1
0
1
0
1. . . after further queries . . . → Equivalent
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science16 / 24
tugrazInstitute of Software Technology
Minimally-Adequate-Teacher Framework (2) –Learning a Software System
Teacher
Model-BasedTestingTool
SystemUnder
Learning
LearningAlgorithm
Equivalence Query(Hypothesis Model)
Yes / Counterexample
Perform Tests
All Pass /Failed Test
Output Query
Query Output
Inputs
Outputs
Outputs Inputs
I Output queries replace membership queriesI Teacher wraps system under learningI Generate tests of hypothesis to falsify hypothesis
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science17 / 24
tugrazInstitute of Software Technology
Mealy Machines
I Deterministic finite automata with inputs and outputsI Example with inputs I = {Ping,Connect} and outputs
O = {Pong,ConnectionClosed ,ConnAck}
q0start q1
Ping/ConnectionClosed
Connect/ConnAck
Ping/Pong
Connect/ConnectionClosed
I No accepting statesI Suited for reactive systems
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science18 / 24
tugrazInstitute of Software Technology
Applicability of Learning
I Works nicely in theoryI Many extensions:
timed, parametrised, non-deterministic . . . systemsI Useful in practise? System size limited by
I Expensive testing→ Harsh abstraction is necessary
I Can we still catch bugs in the IoT?
→ Try using an existing learning tool: LearnLib1
1http://learnlib.deB.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science
19 / 24
tugrazInstitute of Software Technology
Finding Bugs in MQTT – BasicsI Publish/Subscribe IoT-protocolI Architecture:
MQTT BrokerMQTT Broker
MQTT-SNForwarder
MQTT-SNGateway
MQTT-SNGateway
sensor
sensor
sensor
sensor
encapsulatedMQTT-SN
MQTT-SN
MQTT-SN
MQTT-SN
MQTT-SN
MQTT
MQTT client
MQTT client
MQTT client
MQTT
MQTT
MQTT
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science20 / 24
tugrazInstitute of Software Technology
Finding Bugs in MQTT - Approach
Implementation I Implementation J
Abstract Model MI Abstract Model MJ
Differences
Checkout(MI ) = out(MJ)
Standards Document
Bugs
Learn Learn
Analyse Manually
Repeat for all Pairs of 5 Implementations
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science21 / 24
tugrazInstitute of Software Technology
A Simple Bug in MQTT
s0 + / Closed
s1
Con / C_Ack Con / Closed Discon / Closed
+ / +
s2
Sub / S_Ack
Con / Closed Discon / Closed
UnSub / US_Ack
+ / +
Mosquitto
s0
s1
Con / C_Ack
s2
+ / Empty
s3
Discon / Closed
+ / +Con / Empty
Discon / Closed s4
Sub / S_Ack + / Empty
Discon / ClosedCon / C_Ack
+ / Closed
UnSub / US_Ack
Discon / Closed
+ / +
HBMQTT
outmos(Connect · Connect) = ConnectAck · ConnectionClosed
outhbmqtt(Connect · Connect) = ConnectAck · Empty
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science22 / 24
tugrazInstitute of Software Technology
Finding Bugs in MQTT - Results
I Investigated five OS implementationsI Apache ActiveMQ, emqttd, HBMQTT, Mosquitto, VerneMQ.
I Found 18 bugs in four of them, despiteI necessary abstraction → model with less than 20 statesI ignoring time-dependent behaviourI partially non-deterministic behaviourI long test duration: thousands of tests, up to 600ms per input
I Solution: use more expressive models → more testsI Smarter testing first (ongoing work)
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science23 / 24
tugrazInstitute of Software Technology
Conclusions
I Pollution in the IoTI insecure, incorrect and unknown Things.
I Integration Testing is hard → automation neededI Modelling is hard → automata learningI Learning-based TestingI 4 out of 5 MQTT brokers were faultyI Future work
I better test selection = faster learningI non-functional properties: response time, energy consumptionI load testing, fuzzing (robustness testing)I industrial cooperation
B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science24 / 24