How Model-Checking Can Help Model Exploration
Marsha Chechik
Dept of Computer ScienceUniversity of Toronto
Joint work with Arie Gurfinkel, Benet Devereux
Overview of Automated Verification
Answer +
Counterexample
SW/HW
artifactCorrectnessproperties
Temporallogic
Model of
System
Model Extraction Translation
CheckerEngine
Correct?
Correctness properties: CTLpropositional temporal logicbranching-time logic, allowing explicit quantification over
possible futuresSyntax:
True and False are CTL formulas
if p and q are CTL formulae, then so are: p, p q, p qEX p - p is true in some next statesEF p - along some path, p is true in some future state; E[p U q] - along some path, p holds until q holds; EG p - along some path, p holds in every state
Universal quantification: AX p , AF p , A[p U q], AG p
ECTL: EX, EF, EU, EG, disjunction, negation atomic ACTL: AX, AF, AU, AG, conjunction, negation atomic
Conventional state machinesM = <S, A, s0, I , R>S is a (finite) set of statesA is a (finite) set of propositional variabless0 is a unique initial state (s0 S)I: S 2A is a labelling function that maps each state
to the set of propositional variables that hold in it R S S is a (total) transition relation
p q
r
pqr
pqr
s0
s2
s1
Models:Kripke Structures
Counterexamples and Witnesses Goal:
explain why the answer is as givencounterexample: why (s0) = F
i.e., why the property fails to holdwitness: why (s0) = T
i.e., why the property holds Counterexamples and mathematical proofs:
to disprove that holds on all elements of S, produce a single element s S s.t. holds on s.
counterexamples are restricted to universally-quantified formulas
counterexamples are paths (trees) from initial state illustrating the failure of property
Examples
WitnessesEG p
pqr
pqr
p q
r
s0
s2
s1
pqrs0
p q
rs2
CounterexamplesAG p
pqrs0
pqrs1
EX p EX qpqr
s0
p q
rs2
pqrs1
AX p AX qpqr
s0
p q
rs2
pqrs1
Witnesses/Counterexamples Counterexamples (SMV)
AG(Running AF Idle) is false long path leading to state Running followed by a counterexample for AF
EX p is falsenothing given even though it is false!
AG (Running EF Idle)no feedback given when the result is either true or false
Witnesses (SMV)similar -- just negate the properties!A witness to is a counterexample to
So, what do we want? Witnesses/counterexamples
Goodshort -- contain only as much information as necessary
correspond to the modelBad
not available in all cases
often hard to navigate to an “interesting part”
Proofs:Good:
complete
(available for all temporal properties)
all information is hereBad:
too verbose
not particularly intuitive
Where is the model?
Want proof-like counter-examples [TACAS’03]
Example: Cruise Control System (CCS)For keeping an automobile running at a certain speed
Driver accelerates to desired speed and then presses a button on steering wheel (Button = bCruise)
System maintains car speed until The driver presses the brake pedal (Break)
The driver presses the accelerator (Accel)
The driver turns the cruise control off (Button=bOff)
The driver turns the ignition off (Ignition)
The car’s speed becomes uncontrollable (Toofast)
The system can be reactivated by pressing a “resume” button (Button=bResume)
The controlled variable is Throttle
Model-Checking : Example
Proof View: Example
Proof-Like Counter-Example (part 1)
Property:
AG((CC=Cruise) -> Brake)
Proof-Like Counter-Example
(part 2)
Property:
AG((CC=Cruise) -> Brake)
Proof-Like Counter-Example (part 3)
Property:
AG((CC=Cruise) -> Brake)
Mixed Quantifier Formula: Example
Mixed Quantifier Formula: Proof View
Property: EF (AX Brake)
Mixed Quantifier Property: Example
Property:
EF (AX Brake)
The Framework
Model-checkerengine
KEGVis
SMV model(with fairness)
Temporal logic
property (CTL)
solution +proof-like witness
partial witness/counterexample
Navigation/explorationstrategies
Exploration Witness/Counterexample exploration:
dealing with large witnesses How?
how to fast-forward to “interesting” partsSpecify a starting condition, e.g., navigate by a formula
Example: AG(Running AF Idle)not interested in how to get to Running
Specify stopping conditionPick direction (forward/backward)
how to limit information given to user so that interesting cases are easier to find
Navigation Witness/Counterexample Navigation:
making the most interesting choice Sources of choices:
explicit (disjunction)which part of property to consider
Example: (EF p) (EG q)
implicit (via EX)which state to pick as a witness?
Example: EX p
p pp
s0
s1 s3
s2
Bounds in Property-Based Navigation Depth of exploration
Example: witness for AF feasible: (AF p)(s0) = (AF1 p)(s0)
unfeasible: when bound |S|
(witness is as big as the model!)
pq
pq
pq
s0
s2
s1
Can get partial knowledge using depth of explorationExample: (AF p)(s0), set depth to 3
p
p
s0
s1
s4
s2
s3what if depth is smaller than expected?
State-based navigation Pick successor in which (some propositional formula) holds
EX p
pq
p q
pq
s0
s1s3s2
Pick a state using number of successorsleast -- lineargreatest -- branching
ts
Attempt to maintain largest common prefixExample: (EX p) (EX q)
try to pick next state where p and q hold
greedy approximation
shortest witness not necessarily the most interesting!Example: (EF Good EF Error)(s0)
= (E[T U Good ]) (s0) (E[T U Error]) (s0)
= (E [T U7 Good ]) (s0) (E [T U3 Error]) (s0)
picking the shortest counter-exampleautomatically manually based on size or additional information
Strategy: A combination of navigation and exploration to ensure that some user-specified goal is met
Example: shortest counterexample
traditionally, counterexample generators always do shortest (greedy)
counterexample
Strategies Choices:
anything based on paths, states, etc.
including depth of expansion, history, longest common prefix, shortest counter-example...
decision procedure always greedy (and thus non-optimal)
if a strategy was not followed, does not mean that it could not be followed!
Examples:User has complete control.
whenever there is a choice, always ask user
Always attempt to go through Idle state.Always choose state s over t; otherwise, ask user
Part II
Finding Properties
AG(p q)
AG(p AX q)
Model-Checking Typically used for verification
Answer +
Counter-exampleCheckerEngine
Correct?
Where do the properties come from? What to do when they do not hold?So, goal is not just verification but discovery of
properties that the system should have!
Temporal
logic
property
Model of
System
(state-based)
Query Checking [Chan, CAV’00] Goal: speed-up design understanding
discover properties not known a priorip
qr
pqr
pqr
s0
s2
s1
(p q) r is strongest solution to AG ?x,
p q is strongest solution to AG ?x{p,q}
Temporal logic querytemporal logic formula with placeholders
(unknowns)e.g., AG ?x, AG (p ?x)
evaluates to strongest propositional formula that makes query true.
Some applicationsprovide partial explanation when property holds
e.g. instead of AG (a b), ask AG ?x{a, b}answer a b is stronger!
provide diagnostic information when property failse.g. if AG (req AF ack) fails - ask AG (req AF ?x)
Types of queries number of placeholders
e.g., AG ?x, AG (?x EX ?y)
positive vs negativepositive queries - placeholder under even # of negations
e.g. AG ?x
then look for strongest solutions
negative queries - placeholder under odd # of negationse.g. AG (?x p) = AG (?x p)
then look for weakest solutions
mixed queries - neither positive nor negative number of maximally strong solutions
“valid” queries - one strongest solutionarbitrary queries - several strongest solutions
p q
r
pqr
pqr
s0
s2
s1
two solutions to (EX ?x)(s0):p q r and p q r
Deciding TLQ Problem Related work on TLQ
Original definition [Chan, CAV’00]algorithm for “valid” (single strongest solution) subset of CTL
Extended by [Bruns & Godefroid, LICS’01] arbitrary temporal logic formulae
via extended alternating automata
Our work[Gurfinkel, Chechik, Devereux, FSE’02, TSE’03]
reduction to multi-valued model-checking
implementation that deals with arbitrary temporal logic formulae
… with arbitrary number of “unknowns”
answers include witnesses
Example: Cruise Control System (CCS)For keeping an automobile running at a certain speed
Driver accelerates to desired speed and then presses a button on steering wheel (Button = bCruise)
System maintains car speed until The driver presses the brake pedal (Break)
The driver presses the accelerator (Accel)
The driver turns the cruise control off (Button=bOff)
The driver turns the ignition off (Ignition)
The car’s speed becomes uncontrollable (Toofast)
The system can be reactivated by pressing a “resume” button (Button=bResume)
The controlled variable is Throttle
Specification of CCS Specified using SCR method
State changes – in response to events (changes in environment)
@T(a) WHEN bab a
@T(a) WHEN b = a b a’
Input: monitored variablesOutput: controlled variablesSystem state: modeclasses
Sets of states (modes) that partition the state space
The system is in exactly one mode of each modeclass at any point
Mode TransitionTable for CCS
Cruise@T(Button=bResume) WHEN Ignition AND Running AND NOT(Toofast) AND NOT(Brake) AND NOT(Accel)
Cruise@T(Button=bCruise) WHEN Ignition AND Running AND NOT(Toofast) AND NOT(Break) AND NOT(Accel)
Inactive@F(Running) WHEN IgnitionOff@F(Ignition)Override
Override@T(Brake) OR @T(Accel) OR @T(Button=bOff)
Inactive@T(Toofast) OR @F(Running)Off@F(Ignition)Cruise
Cruise@T(Button=bCruise) WHEN Ignition AND Running AND NOT(Toofast) AND NOT (Brake) AND NOT (Accel)
Off@F(Ignition)InactiveInactive@T(Ignition)OffNew ModeEventOld Mode
Initial Mode: Off WHEN NOT(Ignition)
Event Table for Throttle
Initial: Throttle=tOff
tAccelThrottle’=
@T(Inmode)when (Speed=slow)Cruise@T(Speed=slow)Cruise
EventsModes@T(Speed=ok)@T(Inmode)when (Speed=ok)
@T(Speed=fast)@T(Inmode)when (Speed=fast)
@F(Inmode)
@F(Inmode)
tMaintain tDecel tOff
Applications of TLQ
“find all possible values of Throttle EF ( (CC=Cruise) ?x {Throttle})when the system is in mode Cruise”
“what modes can follow Off” EF(CC=Off EX?x {CC})
“which pairs of modes can EF(?x{CC} EX?y {CC})follow each other”
Invariant Discovery“find the strongest invariant” AG ?x
“find invariant of mode Inactive AG ((CC=Inactive) ?x{Ignition,Running})w.r.t. Ignition and Running”
“find invariants of all modes w.r.t AG (?x{CC} ?y{Ignition,Running})Ignition and Running”
Reachability analysis “are all modes of modeclass CCreachable?”
EF(CC=Cruise)EF ?x{CC}
Query-Checking Witnesses A witness
a subtree explaining why ECTL formula holdsproduced automatically by model-checker
p q
r
pqr
pqr
s0
s2
s1
Example:query: (EX ?x{p})(s0)solutions: ?x = p
?x = ps0
s2 s1
?x = p ?x = pwitness:
Same strategies as for exploring model-checking witnesses available
TLQ Applications: Testing Current approach (Gargantini, Heitmeyer [FSE’99])
branch coverage for each mode in mode transition table, test each event at least once for each mode, test every no-change at least once
Off@F(Ignition)Cruise
Cruise@T(Button=bCruise) WHEN Ignition AND Running AND NOT(Toofast) AND NOT (Brake) AND NOT (Accel)
Off@F(Ignition)InactiveInactive@T(Ignition)OffNew ModeEventOld Mode
e.g. for mode Off:need @T(Ignition) and no-change
form CTL propertiesEF((CC=Off) EX(CC=Inactive))
EF((CC=Off) EX(CC=Off))
witness produced by model-checker is the test case!
Query-Checking for Testing our approach:
witness to single queryEF((CC=Off) EX ?x{CC})
Off@F(Ignition)Cruise
Cruise@T(Button=bCruise) WHEN Ignition AND Running AND NOT(Toofast) AND NOT (Brake) AND NOT (Accel)
Off@F(Ignition)InactiveInactive@T(Ignition)OffNew ModeEventOld Mode
Testing Transitions from Mode Off
Generated Witness
Property:
EF((CC=Off) EX ?x{CC})
Query Checking for Testing An even better approach:
coverage of the entire mode transition tableEF(?x{CC} EX ?y{CC})
Testing All Transitions from Table
Witness
Implementation Our framework is named XChek [CAV’02]
multi-valued model-checking engineTLQSolver [CAV’03]
query checker built on top of it KEGVis [TACAS’03,FME’03]
tool for visualization and exploration of count. ex. and witnesses
Input: XML models (based on GXL derived language)SMV-like input language (synchronous product of simple state
machines)+ fairness condition (e.g., p is true infinitely often)CTL property or query
For more infoTo obtain a prototype version
send e-mail to [email protected]://www.cs.toronto.edu/~chechik/publications.html
Questions?Comments?Concerns?
Suggestions? THANKS FOR YOUR ATTENTION!
Overview of Multi-Valued Model-Checking
Answer +
Counter-example
How Correct?
SW/HW
artifactCorrectnessproperties
MV-logic
CheckerEngine
Temporallogic
MV-Model
of System
Model Extraction Translation
Multi-Valued Algebras Use additional ‘truth values’ to represent levels of
contradiction, uncertainty or anything elsee.g., True, False, Maybe, Likely, etc.
Can be defined on a (finite) distributive lattice of truth values with True at the top and False at the bottom using lattice meet as conjunction… and lattice join as disjunction
Negation is defined to preserve involution, i.e.A = A
Then get preservation of associativity, idempotency, distributivity, and De Morgan’s laws
These algebras are called quasi-boolean (optional) may also add a refinement operator
T
F
T
F
M(Maybe)
Classical logicRepresentinguncertainty
Uses: reasoning
about abstractionand partial systems
Representingdisagreement and uncertainty
Multi-Valued Algebras: ExamplesTT
TM
TF
MT
MMFT
FM MF
FF
Multi-valued state machines: Xkripke
structures Extension of conventional state machines (Kripke
structures)variables take any value from the logictransitions between states take any value from the logic
False transitions are not shown (by convention)
Example:p=TTq=FTr=TT
p=FFq=TTr=TT
p=TTq=TFr=TT
s0
s2
s1
TT
TT
FT
TFTT
TT
FF
TF FT
Partial information Algebra:
use three-valued algebra (Kleene)intermediate value represents incomplete
information or uncertainty
T
F
M
compact representation for all possible refinements of this model
if a property is True/False on the partial model, it is True/False on a refined one
initial theory developed by Bruns & Godefroid, CAV’99
p=Tq=Fr=T
p=Mq=Mr=F
p=Tq=Mr=T
s0
s2
s1
T
T
M
MT
Reasoning about Abstractiona way to overcome the state-space explosion
problem in classical model-checkingcollapses sets of concrete states into a single
abstract state… thus indicating that any differences between the concrete
states within a single abstracts state are ignored
Goal: state-wise preservation if a formula evaluates to True (False) in an abstract state, it
evaluates to True (False) in the corresponding concrete state
T
F
M
p q
r
pqr
pq
r
s0
s2
s1
s0,1 s2
p=Mq=Mr=T
p=Tq=Tr=F
M
T
M
ComplexityRunning time of the model checker is O(|S| ||
k), where |S| - size of state space || - the size of the XCTL formulak - time to compute EX
Same as classical!!!!!
Solving Query-Checkingarbitrary temporal logic formulae
not necessarily “valid”
positive, negative, mixed queries
any number of placeholdersoutput includes “reason” why the answer is as givenextend language of queries
e.g. EF ?x{p, q, r}, where p and q are not true simultaneously
Some formalismA - set of prop. variables
e.g., {p}
set of prop. formulas over A forms lattice ordered by
true
pp
false
{p, p, true}
{p, true}{p, true}
{true}{}
{false, p, p, true}
{}
{p, p}
{p}{p}
{true}
{false}
B ={c | bB s.t. b c} {p, p} = {p, p, true}
X is an upset if X=X {p, p} not an upset, {p, p, true} is
result - upset lattice, ordered by set inclusion
each upset can be represented by a set of minimal elements
{false} can represent {p, p, true, false}
So, if X is a solution to a query, all elements of X are solutions
Reasoning with Colors Given a non-temporal formula with
colors, how to evaluate in a state?e.g.: = (pq red) (pq green)
(p q yellow) (p q blue) in state where pq holds
s0
s1 pq
pq s2
{}
{p, p}
{p}{p}
{true}
{false}
answer: red Given a temporal formula, how does it
evaluate?(EX ) (s) = t succ(s) (t) e.g., evaluate (EX ) (s0):
(pq red) (pq green) (p q yellow) (p q blue)
= red green
= yellow
Encoding TLQ Encoding non-temporal formula with ?x
if pq holds in s0, then ?x{p,q}(s0) = (pq)
{}
{p, p}
{p}{p}
{true}
{false}
s0
s1 pq
pq s2
in general,?x{p} = (pp) (p (p)) = (p green) (p red)
each color is strongest possible solutionalways get exactly one color per state!
Temporal formula(EX ?x )(s) = t succ(s) ?x (t)
e.g., (EX ?x{p}) (s0)= red green
= yellow
= {p, p}
Queries with Multiple Placeholders Solution
Li - lattice of propositional formulas over i th placeholdersolution - from upset lattice over L1 … Ln
Example: ?x (EX ?x AX ?y)solution in (B L2) ((C L2) (L1 D)) = (B L2) (C D)
i.e., {(x,y) | x B (x C y D)}
p q
r
pqr
pqr
s0
s2
s1
Another example:?x{p,q} EX ?y{p,q} in state s0
[[?x]](s0) = {p q}
[[EX?y]](s0) = {p q, p q}
[[?x]](s0) = {p q} {false}
[[EX?y]](s0) = {false} {p q, p q}
solution:{p q} {p q, p q}answers: {(p q, p q), (p q, p q)}
Negation All occurrences of placeholder are either negative or
positiveExample: AG ?x
solve for AG ?x, choose from it
AG and thus AG ( ) hold
so is in solution-set for AG ?x
A given placeholder (?x) appears in both negative and positive formsreplace each positive occurrence with ?x+
replace each negative occurrence with ?x-
solvethe set of all solutions to ?x is intersection of solutions
to ?x+ and ?x-
Multi-Valued Model-Checking Multi-Valued CTL (XCTL)
introduce new constants that interact with true and falsee.g., true red = red
allow to include these constants into CTL formulas Multi-Valued Models
include colors in transition relations and/or in states XChek: symbolic model-checker
receives a lattice of colors, describing how to compose them
multi-valued model + XCTL property
fairness requirements (e.g., assume p holds infinitely often)
returnsappropriate color, counter-example/witness
Running time Naïve query-checking algorithm O(|S| || 22n)
n - number of atomic propositions of interest|S| - size of state space|| - the size of the CTL formula
Query-checking for valid queries O(|S| || 2n) Running time of our model checker O(|S| || dd)
dd - time to compute EX symbolicallydepends on:
size of lattice (number of join-irreducible elements in it)
cost of performing union and intersections
Running time of Query Checker Preliminaries
CT(n) - complexity of performing an operation on terminal nodes of ADD
n -- number of propositions restricting the placeholder
complexity of performing any operation on entire ADD is linear in CT(n) and exponential in |A|
Theorem: Complexity of solving a query with one placeholder is linear in |S|, ||, CT(n) and exponential in |A|
Theorem: Complexity of solving a query with multiple placeholders (?1(n1)…?k(nk)) is same as with single placeholder, where CT(i=1
k ni)but… how many terminal nodes are there?
Running time (Cont’d)CT(n) is quadratic in the number of strongest solutions to !
How many solutions are there in a query? Queries about states
Example: AG (q AX ?x{p})Number of solutions: O(2n < |S|)Query-checking - same complexity class as model-checking
Queries about pathsExample: EG (?x)Number of solutions in worst case [hornus02]: O(22n < 2|S|) Query-checking can be infeasible even for small problems!… but not always!
Ex: EF EG ?x{CC} in a Cruise Control System (later in the talk)
Improvements to Running Time Query-checking and model-checking
query built of some query-checking and some model-checking parts
AG(?x AF p): CTL model-checking of AF p, result used for query-
checking
Theorem: if placeholder occurs in scope of V temporal quantifiers is V Q + (| | V) M
Q - worst case complexity of query checking a formula with one temporal quantifier
M - worst case complexity of query checking a formula with one temporal quantifier
Improvements (Cont’d) Heuristics:
constructing decision diagram for ?x is $$ …but can be avoided in most queries
Transition relation is boolean so cost of performing conjunction and disjunction on terminal nodes is O(1) and does not depend on CT.