testing real-time embedded systems using uppaal-tron -tool...
TRANSCRIPT
Testing Real-Time Embedded Systems Using UppAal-TRON
-Tool and Application
Kim G. Larsen, Marius Mikucionis,Brian Nielsen, Arne Skou
{kgl | marius | bnielsen | ask}@cs.aau.dk
Aalborg University, DK
2
AgendaAutomated Model-based TestingTesting Framework
Timed AutomataEnvironment ModelingRelativized I/O conformance
Online Testing AlgorithmDanfoss EKCOther Issues
Monitoring and Environment EmulationCoverage Measurement
DemoConclusions & Future Work
3
Testing Embedded Software
Testing: Execute actual software (system) with controlled inputs and check responses
To find errorsTo determine risk of release
10-20 errors per 1000 LOC30-50 % of development time and costSoftware and complexity increases
4
TestGene-ratortool
TestGene-ratortool
click?x:=0
click?x<2
x>=2
DBLclick!
Automated Model-Based Testing
fail
pass
Testexecution
tool
Testexecution
toolEvent
mapping
Driver
Model Test suite
TestGenerator
tool
TestGenerator
tool
Implementation Relation
Selection &optimization
Does the behavior of the (blackbox) implementation comply to that of the specification?
ImplementationUnder Test
5
TestGene-ratortool
TestGene-ratortool
click?x:=0
click?x<2
x>=2
DBLclick!
input
Online Testing
fail
pass
Testexecution
tool
Testexecution
toolEvent
mapping
Driver
Model
TestGenerator
tool
TestGenerator
tool output
Implementation Relation
Selection &optimization
•Test generated and executedevent-by-event (randomly), reactively
•Long Running, deep testing, imaginative
ImplementationUnder Test
inputinputinput
outputoutputoutput
6
Real-Time Systems
Environment Controller
Real Time SystemA system where correctness not only depends on the logical order of events but also on their timing
sensors
actuators
TaskTask
TaskTask
SystemModel
EnvironmentModel
Output
Input
Σ
Modelling &Abstraction
7
Our Framework
•Complete and sound algorithm•Efficient symbolic reachability algorithms•UppAal-TRON: Testing Real-Time Systems Online•Release 1.3 http://www.cs.aau.dk/~marius/tron/
Correct system behavior•Test Oracle•Monitor
•Relevant input event sequences•Load model
”Formal Relativized i/o conformance” Relation
•UppAal Timed Automata Network: Env || IUT
8
Related WorkFormal Testing Frameworks
[Brinksma, Tretmans]
Real-Time Implementation Relations[Khoumsi’03, Briones’04]
Symbolic Reachability analysis of TimedAutomata
[Dill’89, Larsen’97,…]
Online state-set computation[Tripakis’02]
Online Testing[Tretmans’99, Peleska’02, Krichen’04]
10
Sample Test Runs
INFINITELY MANY SEQUENCES!!!!!!
highTemp!·3·compressorOn? ⇒ PASS
highTemp!·3·compressorOn?·123·lowTemp!·3·compressorOff? ⇒ PASS
highTemp!·3·compressorOff? ⇒ FAIL
highTemp!·13·compressorOn? ⇒ FAIL
highTemp!·3·compressorOn?·17·lowTemp!·3·compressorOff?·3.14·highTemp!·5·compressorOn?·177·lowTemp!·3·compressorOff? ⇒ PASS
11
Sample Cooling Controller
IUT-model Env-model
On!
Off!
Low?
Med?
High?
Cr
12
Env. ModelingRealism and Guiding
EL
EM
E1 E2
EL E2 E1 EM
Temp.
time
High!
Med!
Low!
EM Any action possible at any timeE1 Only realistic temperature variationsE2 Temperature never increases when coolingEL No inputs (completely passive)
13
Sample Cooling Controller
IUT Env-model
On!
Off!
Low?
Med?
High?
EM
Cr
C’r rt-ioco EM Cr
C’r
14
Sample Cooling Controller
IUT Env-model
On!
Off!
Low?
Med?
High?
C’r rt-ioco E1 Cr , iff 3d<r
d.Med?.d.High?.d.Med?.d.Low?.ε.On, ε≤r
E1
C’r
15
Sample Cooling Controller
IUT Env-model
On!
Off!
Low?
Med?
High?
C’r rt-ioco E2 Cr
E2
C’r
16
Non-Determinism
time
threshold±err
switchOn!
switchOff!
T
•Modeling Action uncertainty•A controller switches a relay when a control variable crosses ‘around’ threshold value
•Modeling Timing uncertainty•A controller switches a relay between 2 and 10 time units
17
Implementation relationRelativized real-time io-conformance
•i rt-iocoe s =def•∀σ ∈ TTr(e): Out((e,i) after σ) ⊆ Out((e,s) after σ)
•i rt-iocoe s iff TTr(i) ∩ TTr(e) ⊆ TTr(s) ∩ TTr(e)
•Intuition, for all relevant environment behaviors•never produces illegal output, and•always produces required output in time
•~timed trace inclusion
•Let P be a set of states•TTr(P): the set of timed traces from states in P•P after σ = the set of states reachable after timed trace σ•Out(P) = possible outputs and delays in P
SystemModel
Environmentassumptions ε0’,o0,ε1’,o1…
ε0,i0,ε1,i1…e
IUT
s i
18
Randomized Online AlgorithmAlgorithm TestGenExec (TestSpec) returns {pass, fail}
Z:={⟨l0,0⟩},While Z ≠∅ and #iterations≤T do choose randomly
1. if EnvOutput(Z) ≠∅ // Offer an inputchoose randomly a ∈ EnvOutput(Z) send i to SUTZ:=Z after a
2. choose randomly δ ∈ Delays(Z) // Delay and wait for outputWait(δ)
if o occurred after δ’ ≤ δ thenZ:=Z after δ’
if o ∉ ImpOutput(Z) then return failZ:=Z after o
else // no output within δ timeZ:=Z after δ
3. reset IUTZ:={⟨l0,0⟩}
if Z=∅ then return fail else return pass•Sound•Complete as T→∞
19
Sound & CompleteTestGenExec is sound
Fail verdict ⇒¬( I iocoe S)
complete ¬( I iocoe S) ⇒ Prob(Fail) → 1 as T→∞
(using only unit delays)Assuming
IUT can be modeled by an input enabled, deterministic, non-blocking IO-TLOTS with isolated outputsTime unit of IUT is knownTTr(IUT) and TTr(E) are closed under digitization
LTS induced by TA with only non-strict guardsTTr(S) closed under inverse digitization
LTS induced by TA with only strict guards
20
State-set computationCompute all potential states the model can occupy after the timed trace ε0,i0,ε1,o1,ε2,i2,o2,…
Let Z be a set of statesZ after a: possible states after executing a (and t*)Z after ε :possible states after t* and εi , totaling a delay of ε
o is a legal output from SUT iff O in ImpOutput(Z)a is a relevant input in Env iff I in EnvOutput(Z)
ε is a permitted delay iff Z after ε ≠∅ε is a relevant delay iff Delays (Z)
21
State-set ComputationCompute all potential states the model can occupy after the timed trace ε0,i0,ε1,o1,ε2,i2,o2,…Let Z be a set of states
Z after a: possible states after executing a (and τ*)
Z after ε :possible states after τ* and εi , totaling a delay of ε
l0
x≤7, a
a
l3
l2
l1
l4a,
x:=0
τ l0
τ, x:=0l1
{ ⟨l0,x=3⟩ } after a = { ⟨l2,x=3⟩, ⟨l4, x=3⟩, ⟨l3, x=0⟩ }
{ ⟨l0,x=0⟩} after 4 = { ⟨l0,x=4⟩, ⟨l1, 0 ≤ x ≤ 4⟩ }
Represent state sets as sets of symbolic states Use symbolic reachability(similar to model checkers like UppAal)
22
Symbolic Reachability
23
Real-time Online•Compute all states reachable after timed trace•Maintain a set of symbolic states in real time!
Z2
Z4
Z0
Z1Z3Z7
Z5
Z8
Z6Z9
Z11
Z14
Z12
Z15Z18
Z17
Z16
SpecificationTA-network
i!2.75O?
SystemUnderTest
Online Tester:
[Tripakis’02, Krichen’04]
25
Danfoss EKC Case Electronic Cooling Controller
Output Relays•compressor relay•defrost relay•alarm relay•(fan relay)Display Output•alarm / error indication•mode indication•current calculated temperature
Sensor Input•air temperature sensor•defrost temperature sensor•(door open sensor) Keypad Input•2 buttons (~40 user settableparameters)
•Optional real-time clock or LON network module
26
Industrial Cooling Plants
27
Project GoalsCan we model significant aspects and time constraints?Can we test in real-time? Is the tool fast enough?How do we control and observe target?
Existing product Documentation
requirements specificationusers manualsequipment and software for real test executionMeeting and e-mail with Danfoss Engineers
Continued collaborationTest of new generation controllers being developedImproved test interface
28
Basic Refrigeration Control
Time
setpoint
setpoint+differential
highAlarmDeviation
lowAlarmLimit
highAlarmLimit
lowAlarmDeviation
differential
start compressor
stopcompressor
start compressor
stop compressor
startalarm
normal min restart time not elapsed
min cooling time not elapsed alarm delay
30
EKC Adaptation 1
Hardware+Physical I/O
Device drivers+kernel
Parameter DB (shared variables)
Control Software
Test Interface
LON GW RS232
win32+OLE+VB
•AK-Online (PC SW)•configuration•supervision•logging
•Read and write parameter “database”•47 parameters
EKC Software Layering
31
Adaptor
EKC Adaptation 2
tcp/ipLON+rs232
win32+OLE+VB Solaris/Linux (C++)
TRON Engine
compressorOn
22.3 0 1 22.1 0 1
16.7 0 0 old copy
new copy
“continous” readout 2 readouts/s
setTemp(20)
“par#4=20.0”
Need better test interface!•Read-only parameters•Delay and synchronization
32
Modeling PrinciplesModel significant subset
Temperature regulationAlarm monitoringManual and periodic timer based defrosting
Modular modelCompute “calculatedTemperature” in model
derive output events from that could be monitored in adaptor
Environment temperature generatorsUse non-determinism
Timing tolerancesModel adapter delay and timing uncertainty
33
Temperature TrackingTemperature
Time
“periodic” weighted average:54*1 sampledn
n
TTT
+= −
EKC calculated temperature
Model calculated temperatureError/uncertainty envelope
tolerance in sampling time
tolerance in value computation
compressorOn!
34
Main Model Components 18 concurrent timed automata14 clocks, 14 integers
Output
Input
IUT-Model
alarmRelay
compressorRelay
tempMeasurement
compressor
newTempnewTemp
on/off on/off
Environment
TemperatureGenerator
defrostRelay
defrost
autoDefrost
on/off
defrostEventGen
alarmDisplay
on/off
highTempAlarm
35
Reverse EngineeringUnclear and incomplete specificationsMethod of Working
1. Formulate hypothesis model2. Test 3. FAIL-verdict ⇒ Refine model4. (PASS) ⇒ Confirm with Danfoss
Detects differences between actual and modeled behaviorIndicates promising error-detection capability4 examples
36
Ex1: Control PeriodControl actions issued when ”calculatedTemp” crosses thresholds
No requirements on period givenTested to be 1.2 seconds
“periodic” weighted average:54*1 sampledn
n
TTT
+= −
37
Ex2: High Alarm Monitor v1
Clearing the alarm do not switch off alarm state, only alarm relay
38
Ex2: High Alarm Monitor v2
•Add HighAlarmDisplay action •Add location for “noSound, but alarmDisplaying”•(Postpone alarms after defrosting)
39
Ex3: Defrosting and AlarmsWhen defrosting the temperature risesPostpone high temperature alarms during defrost System parameter alarmDelayAfterDefrostSeveral Interpretations
1. Postpone alarmDelayAfterDefrost+alarmDelay after defrost?
2. Postpone alarmDelayAfterDefrost+alarmDelay after highTemp detected?
3. Postpone alarmdelayAfterDefrost until temperature becomes low; then use alarmDelay
Option 3 applies!
40
Ex4: Defrost TimeToleranceDefrost relays engaged earlier and disengaged later than expectedAssumed 2 seconds toleranceDefrosting takes long timeImplementation uses a low resolution timer (10 seconds)
41
Example Test Run (log visualization)
150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800
0 100000 200000 300000 400000 500000 600000 700000 800000 900000
setTempmodelTempekcTempCONCOFFAONAOFFalarmRstHADOnHADOffDONDOFFmanDefrostOnmanDefrostOff
defrostOff?
alarmOn!alarmDisplayOn!
resetAlarm?AOFF!
HighAlarmDisplayOff!
manualDefrostOn?COFF!DON!
compressorOn!
//defrost completeDOFF!CON!
42
State-set EvolutionState set plot
0
200
400
600
800
1000
1200
0 100 200 300 400 500 600 700 800 900 1000time (sec)
Num
ber o
f sta
tes
0
5
10
15
20
25
degr
ees
State-setHigh Temp LimitTemperatureAlarm Limit
Correlation between state-sets and model behavior
43
Cost of state-set update
Number ofSymbolic states
µS
Testing=Environment emulation+monitoring
45
Testing
Correct system behavior•Test Oracle•Monitor
•Relevant input event sequences•Load model
”Formal Relativized i/o conformance” Relation
•Replace Systems Real Environment by Tester•Tester provides inputs •Tester observes outputs
i
o
46
Environment Emulation
”Formal Relativized i/o conformance” Relation
i
o
∅
Compute inputs from environment modelRelevant input event sequencesLoad model
Feedback or one-wayOutputs may go to real-system
o
47
Monitoring
”Formal Relativized i/o conformance” Relation
∅
Passively check communication between system and its real environment
check system behaviorPassive Testing
oi
Measuring Coverage
49
Coverage MeasurementsHow thorough has testing been??Idea:
Use a model checker, e.g. UppAalConvert timed trace observed during test run to a timed automata (trace automata)Replace Environment by trace automatonPerform Reachability Analysis on annotated model (according to coverage criteria)
50
Location Coverage
Test sequence traversing all locationsEncoding:
Enumerate locations l0,…,lnAdd an auxiliary variable li for each location Label each ingoing edge to location i li:=trueMark initial visited l0:=true
Check: EF( l0=true ∧ … ∧ ln=true )
ljlj:=true
lj:=true
51
Edge Coverage
Test sequence traversing all edgesEncoding:
Enumerate edges e0,…,enAdd auxiliary variable ei for each edge Label each edge ei:=true
Check: EF( e0=true ∧ … ∧ en=true )
l1
l4 l3
l2
a? x:=0 e0:=1
x≥2
a? e2:=1
x<2
b! e1:=1c!
e3:=1
e4:=1
52
Coverage of non-deterministic models
Edge i possible covered (is some run)Check: EF( ei=true ∧ t.end)
Edge i definitely covered (in all runs)Check: AF(t.end ⇒ ei=true)
Edge i definitely not covered (in no runs)Check: AF(t.end ⇒ ei=false)
Trace 10.a!.5.b?
53
Demo
54
Touch-Sensitive Light-Controller
•Patient user: Wait=∞•Impatient: Wait=15
56
Touch-sensitive Light-Controller Model
User/ENV Interfaceswitch
Dimmer
grasp!
release!
touch!
Level!
light controller model
hold!
endhold!
58
Mutants
synchronized public void handleTouch() {if(lightState==lightOff) {setLevel(oldLevel);lightState=lightOn;
}
else { //was missingif(lightState==lightOn){
oldLevel=level;setLevel(0);lightState=lightOff;
}
•M2 incorrect additional delay in dimmer as if x:=0 was on ActiveUP ↔ActiveDN transitions
X:=0
X:=0
•M1 incorrectly implements switch
60
ConclusionsCan accurately model EKC-like devicesCan create models suitable for online testingComplete and detailed model not required
Select aspectsAbstraction
MBT feasible even if specification is incomplete/unclearPromising error-detection capabilities
Differences between actual and specified behavior in industrial caseAcademic mutation studies
Excellent performanceVery non-deterministic models causes very large state-sets which can become a computational bottleneckReal-time synchronization of IUT and tester is problematic
61
Future WorkTool Improvements
Import test trace into UppAalEdge & location-coverage measurementsGraphical User-InterfaceSeparate environment simulation and monitoring
Further Danfoss CollaborationBetter test interfaceTest newly developed product
Coverage Guiding & RT-criteria)Automatic test adaptation abstractionTesting Hybrid and Stochastic Systems
62
Research ChallengesModelling
How to model real-time systems easily, and quickly, precisely, expressively, ...What is a good formal notion of correctness?
Tool implementationHow to analyze these models?How to compute state-set estimation in real-time?Real-time execution and clock synchronization with IUT!?!
RobustnessPartial observability and uncertainty
GuidingCan we improve obtained coverage of model?? Real-time coverage criteria??Is it efficient in finding errors?
How to apply in industrial practice?Extensions
Probabilistic performance testing?Hybrid systems
63
END