ifm 2005, eindhoven generated path conditions for timed systems doron peled dept. of computer...

IFM 2005, Eindhoven

Generated Path Conditions for Timed Systems

Doron PeledDept. of Computer ScienceUniversity of WarwickUnited KingdomJoint work with Saddek Bensalem, Hongyang Qu,Stavros Tripakis

IFM 2005

IFM 2005, Eindhoven

Tester’s Goals Help in selecting test cases.

Visual, by clicking on a path in flow chart. According to intuition about potential

errors. According to some formal specification.

Performing tests Forcing an execution (even when

nondeterminism exists). Calculating the probability of a path.

IFM 2005, Eindhoven

Architecture

compilercode

Test Selector

Visual Selection

Model Checker

Calculate Weakest Precondition

SatSolver

transitions

Flow graph

Executor

Add Synchro.

Calculate Probability

Counter-exampletest case

IFM 2005, Eindhoven

Unit testing: Selection of test cases

(for white-box testing)

The main problem is to select a good coverage

criterion. Some standard options are: Cover all paths of the program. Execute every statement at least once. Each decision (diamond node on flow chart)

has a true or false value at least once. Each condition predicate is taking each truth

value at least once. Check all possible combinations of conditions

in each decision.

IFM 2005, Eindhoven

How to cover the executions?

if (A>1)&(B=0) then X=X/A; if (A=2)|(X>1) then X=X+1;

Choose values for A,B,X at the beginning that would force the right path/conditions/predicates.

Value of X may change, depending on A,B. What do we want to cover? Paths?

Statements? Conditions?

IFM 2005, Eindhoven

Statement coverageExecute every statement at least onceBy choosingA=2,B=0,X=3each statement will

be chosen.The case where the

tests fail is not checked!

if (A>1)&(B=0) then X=X/A;

if (A=2)|(X>1) then X=X+1;

Now x=1.5

IFM 2005, Eindhoven

Decision coverageEach decision (diamond node in flow graph) tested with true and false outcome at least once.

Can be achieved using A=3,B=0,X=3 A=2,B=1,X=1

Problem: Does not test individual predicates. E.g., when X>1 is erroneous in second decision.

if (A>1)&(B=0) then X=X/A;

if (A=2)|(X>1) then X=X+1;

IFM 2005, Eindhoven

Preliminary:Relativizing assertions

(B) : x1= y1 * x2 + y2 /\ y2 >= 0Relativize B) w.r.t. the assignment

becomes B) [Y\g(X,Y)]e(B) expressed w.r.t. variables

at A.) (B)A =x1=0 * x2 + x1 /\ x1>=0Think about two sets of variables,

before={x, y, z, …} after={x’,y’,z’…}.

Rewrite (B) using after, and the assignment as a relation between the set of variables. Then eliminate after by substitution.

Here: x1’=y1’ * x2’ + y2’ /\ y2’>=0 /\x1=x1’ /\ x2=x2’ /\ y1’=0 /\ y2’=x1now eliminate x1’, x2’, y1’, y2’.

(y1,y2)=(0,x1)

A

B

A

B

(y1,y2)=(0,x1)

Y=g(X,Y)

IFM 2005, Eindhoven

Verification conditions: tests

C) is transformed to B)= t(X,Y) /\ C)

D) is transformed to B)=t(X,Y) /\ D)

B)= D) /\ y2x2

y2>=x2

B

C

D

B

C

Dt(X,Y)

FT

FT

IFM 2005, Eindhoven

How to find values for coverage?

•Put true at end of path.

•Propagate path backwards.

•On assignment, relativize expression.

•On “yes” edge of decision node, add decision as conjunction.

•On “no” edge, add negation of decision as conjunction.

•Can be more specific when calculating condition with multiple condition coverage.

A>1/\B=0

A=2\/X>1

X=X+1

X=X/Ano

no

yes

yes

true

true

IFM 2005, Eindhoven

How to find values for coverage?

A>1/\B=0

A=2\/X>1

X=X+1

X=X/Ano

no

yes

yes

true

true

A 2/\X>1

(A2 /\ X/A>1) /\ (A>1 & B=0)

A2 /\X/A>1Need to find a satisfying assignment:

A=3, X=6, B=0

Can also calculate path condition forwards.

IFM 2005, Eindhoven

Some real life story An expert programmer inspects the code of

NASA MER. He observes through his experience and

intuition that some execution path is suspicious. He decides how to force this path to execute,

e.g., by figuring some inputs and initial values. He executes the path, showing his supervisor

the presence of an error. We want to build some tools to help him with

this process. We’ll use LTL to help with formalizing the

intuition on where the error is.

IFM 2005, Eindhoven

Learning from another technique: Model Checking

Automaton description of a system B. LTL formula . Translate into an automaton P. Check whether L(B) L(P)=. If so, S satisfies . Otherwise, the intersection

includes a counterexample. Repeat for different properties.

¬

IFM 2005, Eindhoven

Unit Testing Model Checking

Unit Checking

IFM 2005, Eindhoven

New: Test case generation based on LTL specification

CompilerModel

CheckerPath condition

calculation

First orderinstantiator

Testmonitoring

Transitions

Path

Flowchart

LTLAut

IFM 2005, Eindhoven

Path conditions Path in flow chart multiple executions following

path. First order formula. All executions of a path must start with initial

values satisfying the path condition. In deterministic code, there can be only one

execution starting with particular values, hence all executions starting with initial values satisfying the path condition will follow that path.

In nondeterministic code, each such initial value has an execution following a path. May need to insert synchronizing code.

IFM 2005, Eindhoven

Goals Verification of software. Compositional verification. Use only a unit of

code instead of the whole code. Parameterized verification. Verifies a

procedure with any value of parameters in “one shot”

Generating test cases via path conditions: A truth assignment satisfying the path condition. Helps derive the demonstration of errors.

Generating appropriate values to missing parameters.

IFM 2005, Eindhoven

Spec: ¬at l2U (at l2/\ xy /\

(¬at l2/\(¬at l2U at l2 /\ x2y ))) Automatic translation of

LTL formula into an automaton [Gerth et all]

LTL is interpreted over finite sequences.

Can use other (linear) specification.

Property specifies the path we want to find (SPIN: never claim),not the property that must hold for all paths (for this, take the negation).

¬at l2

at l2/\xy

¬at l2

at l2/\x2y

Observation:each node hasconjunctions of

predicates onprogram variables

and programcounters

IFM 2005, Eindhoven

Divide and Conquer Intersect property automatonproperty automaton with the

flow chartflow chart, regardless of the statements and program variables expressions.

Add assertions from the property automaton to further restrict the path condition.

Calculate path conditions for sequences found in the intersection.

Calculate path conditions on-the-fly. Backtrack when condition is false.Thus, advantage to forward calculation of path conditions (incrementally).

IFM 2005, Eindhoven

Spec: (only program counters here)

¬at l2U (at l2/\ ¬at l2/\(¬at l2U at l2))

¬at l2

at l2

¬at l2

at l2

l2:x:=x+z

l3:x<t

l1:…

l2:x:=x+z

l3:x<t

l2:x:=x+z

XX==

at l2

at l2

¬at l2

Either all executions of a path satisfy the formula or none.

Sifts away paths not satisfying formula. Then calculate path condition.

IFM 2005, Eindhoven

Spec: ¬at l2U (at l2/\ xy /\

(¬at l2/\(¬at l2U at l2 /\ x2y )))

¬at l2

at l2/\xy

¬at l2

at l2/\x2y

l2:x:=x+z

l3:x<t

l1:…

l2:x:=x+z

l3:x<t

l2:x:=x+z

XX==

xy

x2y

Only some executions of path may satisfy formula

Modify calculation of path condition to incorporate property

IFM 2005, Eindhoven

Calculating the intersection of the property automaton and flow graph (abstract variables away).

¬a

a a

a

as1 s2

s3q2

q1

s1,q

1

s1,q2 s3,q2

s2,q1Acceptance isdetermined by

propertyautomaton.

<>a

a a

¬a

IFM 2005, Eindhoven

How to generate test cases Take the intersection of an LTL automaton (for a

never claim) with the flow graph. Some paths would be eliminated for not satisfying the assertions on the program counters.

Seeing same flow chart node does not mean a loop: program variables may value. Use iterative deepening.

For each initial path calculate the path condition. Backtrack if condition simplifies to false.

Report path condition based on flow graph path+LTL assertions.

Always simplify conditions!

IFM 2005, Eindhoven

How the LTL formula directs the search

Consider (x=4)U (x=5/\o…)x=4

x=5x<5

x:=x+1y:=7

truefalse

IFM 2005, Eindhoven


Consider x=4U (x=5/\o…)x=4

x=5x<5

x:=x+1y:=7

truefalse

IFM 2005, Eindhoven



x=5x<5

x:=x+1y:=7

truefalse

X=4

IFM 2005, Eindhoven



x=5x<5

x:=x+1y:=7

truefalse

X=4

X=4

IFM 2005, Eindhoven



x=5x<5

x:=x+1y:=7

truefalse

X=4

X=4

x<5

X=4

true

This is acontradiction

IFM 2005, Eindhoven



x=5x<5

x:=x+1y:=7

truefalse

X=5

X=4

IFM 2005, Eindhoven

Example: GCD

l1:x:=a

l5:y:=z

l4:x:=y

l3:z:=x rem y

l2:y:=b

l6:z=0?yesn

o

l0

l7

IFM 2005, Eindhoven

Example: GCD

l1:x:=a

l5:x:=y

l4:y:=z

l3:z:=x rem y

l2:y:=b

l6:z=0?yesn

o

Oops…with an error (l4 and l5 were switched).

l0

l7

IFM 2005, Eindhoven

Why use Temporal specification

Temporal specification for sequential software?

Deadlock? Liveness? – No! Captures the tester’s intuitionintuition about the

location of an error:“I think a problem may occur when the program runs through the main while loop twice, then the if condition holds, while t>17.”

IFM 2005, Eindhoven

Example: GCD

l1:x:=a

l5:x:=y

l4:y:=z

l3:z:=x rem y

l2:y:=b

l6:z=0?yesn

o

l0

l7

a>0/\b>0/\at l0 /\at l7

at l0/\a>0/\b>0

at l7

IFM 2005, Eindhoven

Example: GCD

l1:x:=a

l5:x:=y

l4:y:=z

l3:z:=x rem y

l2:y:=b

l6:z=0?yesn

o

l0

l7

a>0/\b>0/\at l0/\at l7

Path 1: l0l1l2l3l4l5l6l7a>0/\b>0/\a rem b=0

Path 2: l0l1l2l3l4l5l6l3l4l5l6l7 a>0/\b>0/\a rem b0

IFM 2005, Eindhoven

Potential explosion

Bad point: potential explosion

Good point: may be chopped on-the-fly

IFM 2005, Eindhoven

Now we add time Detailed model, for each transition we

have 4 parameters [l, u, L, U]: l Needs to be enabled at least that much. u Cannot be enabled without taken longer

than that. L Least time for transformation to occur

(after been chosen). U Transformation cannot take more than

that.

IFM 2005, Eindhoven

Translation to timed automata

s1

at l

s3,at lx2<u2x1<u1

s4,at lx2<u2

s2,at lx1<u1

c1c2

x2:=0

c1c2

x1:=0

c1c2

x1:=0

c1c2

x2:=0c1c2

x1:=0c1c2

x2:=0

c1c2 c1c2

c1c2

x1,x2:=0 c1c2

c1c2 c1c2

Timing out the enabledness:Zero the counters,Cannot wait enabled too much.

IFM 2005, Eindhoven


s1

at l

s3,at lx2<u2x1<u1

s6x2<U2

s5x1<U1

s4,at lx2<u2

s2,at lx1<u1

x1l1x1:=

0

x1l1x1:=

0

x2l2x2:=

0

x2l2x2:=

0

c1c2

x2:=0

c1c2

x1:=0

c1c2

x1:=0

c1c2

x2:=0c1c2

x1:=0c1c2

x2:=0

c1c2 c1c2

c1c2

x1,x2:=0 c1c2

c1c2 c1c2

ac

ac bc bc

Can fire only if waited enough,Zero counters again.

IFM 2005, Eindhoven


s1

at l

s3,at lx2<u2x1<u1

s8s7

s6x2<U2

s5x1<U1

s4,at lx2<u2

s2,at lx1<u1

x1L1 x2L2

x1l1x1:=

0

x1l1x1:=

0

x2l2x2:=

0

x2l2x2:=

0

c1c2

x2:=0

c1c2

x1:=0

c1c2

x1:=0

c1c2

x2:=0c1c2

x1:=0c1c2

x2:=0

c1c2 c1c2

c1c2

x1,x2:=0 c1c2

c1c2 c1c2

ac

ac bc bc

af bf

Conditions on paths represented using (symbolic) DBMs.

IFM 2005, Eindhoven

Should we really look at paths?

Its easy to select an interleaved sequence.

But due to time limitations, it may execute in a different order.

Just the order on events from the same process and using same variables is to be considered.

a

b

c

d

a b

c d

Sameprocess

Samevariable

IFM 2005, Eindhoven

Generate an automaton for all consistent interleavings

a b

c d

a

a b

b

c

c

bd

dc

Intersect this automaton with automaton for system.Calculate “partial order” condition: start from leaves.When there is a choice, usedisjunct.

IFM 2005, Eindhoven

Generate an automaton for all consistent interleavings

a

a b

b

c

c

bd

dc

IFM 2005, Eindhoven

An example — a simple network protocol

IFM 2005, Eindhoven

The flow charts

IFM 2005, Eindhoven

Path — no timeout

IFM 2005, Eindhoven

Precondition

The simplified precondition: l >= 110

IFM 2005, Eindhoven

The diagrams

IFM 2005, Eindhoven

The PET tool Basic mode: interactive choice of a path, calculating

of path conditions. Model checking mode. Iterative model checking mode: apply model

checking recursively to find successive segments, control backtracking.

Unit checking mode. Calculating path condition: simplify, simplify,

simplify.Use SML and HOL for rewriting and deciding on Pressburger arithmetic. Plan using other tools!

Problem: US patent 6,408,430 belongs to Lucent!

IFM 2005, Eindhoven

IFM 2005, Eindhoven

Drivers and Stubs(skip)

Driver: represents the program or procedure that called our checked unit.

Stub: represents a procedure called by our checked unit.

In our approach: replace both of them with a formula representing the effect the missing code has on the program variables.

Integrate the driver and stub specification into the calculation of the path condition.

l1:x:=a

l5:x:=y

l4:y:=z

l3:z’=x rem y/\x’=x/\y’=x

l2:y:=b

l6:z=0?yesn

o

l0

l7

IFM 2005, Eindhoven

Conclusions

Model checking and testing have a lot in common. Can use ideas from model checking for generating test cases. Integrate Formal Methods!

Unit Testing: Model checking of infinite state spaces.But: semidecidable: Don’t know when to stop search (undecideable), Don’t know when condition equivalent false

(undecideable). Tools, visual user interface. Generalization to real time systems. More tools:

automatic addition of synchronization.Calculate probability of execution.

IFM 2005, Eindhoven

Some references Translating LTL into automata:

Gerth, Peled, Vardi, Wolper, Simple on-the fly automatic verification of temporal logic, PSTV 1995.

The PET tool:Gunter, Peled, Path Exploration Tool, Tacas 1999, LNCS 1579

Unit Checking:Gunter, Peled, Unit Checking: symbolic model checking for unit of code, LNCS 2772 (Z.M. birthday volume)

Forcing an execution under nondeterminism:Qu, Peled, Enforcing Concurrent Temporal Behavior, RV 2004

IFM 2005, Eindhoven

Enforcing Executions: Goals

Instrument a program in order to demonstrate counterexamples.

Inspect generated test cases. Studying the effect of added

synchronization/timing. Still allow other runs, selected runs

are enforced in a controlled way.

IFM 2005, Eindhoven

Dekker’s mutual exclusion algorithmP1::c1:=1; while true do begin c1:=0; while c2=0 do begin if turn=2 then begin c1:=1; while turn=2 do begin /* no-op */ end; c1:=0 end end; /* critical-section 1*/ c1:=1; turn:=2end

P2::c2:=1; while true do begin c2:=0; while c1=0 do begin if turn=1 then begin c2:=1; while turn=1 do begin /* no-op */ end; c2:=0 end end; /* critical-section 2*/ c2:=1; turn:=1end

IFM 2005, Eindhoven

0:start P1

1:c1:=1

12:true?

10:c1:=1

9:crit-1

6:c1:=04:no-op

3:c1:=1

2:c1:=0 13:end

8:c2=0?

7:turn=2?

5:turn=2?11:turn:=2

yes

yes

yes

yes

no

no

no

no

0:start P2

1:c2:=1

12:true?

10:c1:=1

9:crit-2

6:c2:=04:no-op

3:c2:=1

2:c2:=0 13:end

8:c1=0?

7:turn=1?

5:turn=1?11:turn:=1

yes

yes

yes

yes

no

no

no

no

IFM 2005, Eindhoven

Two scenarios from same initial state(P1(0):start) (P2(0):start) [P1(1):c1:=1] [P2(1):c2:=1]<p1(12):true?>yes[p1(2):c1:=0] [P2(2):c2:=0]<p1(8):c2=0?>yes <P2(8):c1=0?>yes<p1(7):turn=2?>no <p2(7):turn=1?>yes [p2(3):c2:=1]<p1(8):c2=0?>yes[P1(9):crit-1]

(p1(0):start) (p2(0):start)[p1(1):c1:=1] [P2(1):c2:=1] <P2(12):true?>yes<p1(12):true?>yes [P2(2):c2:=0] <P2(8):c1=0?>no [P2(9):crit-2]

Starting with same state, i.e., with turn=1 does not guarantee repeating the same run due to nondeterminism.

IFM 2005, Eindhoven

0:start P1

1:c1:=1

12:true?

10:c1:=1

9:crit-1

6:c1:=04:no-op

3:c1:=1

2:c1:=0 13:end

8:c2=0?

7:turn=2?

5:turn=2?11:turn:=2

yes

yes

yes

yes

no

no

no

no

0:start P2

1:c2:=1

12:true?

10:c1:=1

9:crit-2

6:c2:=04:no-op

3:c2:=1

2:c2:=0 13:end

8:c1=0?

7:turn=1?

5:turn=1?11:turn:=1

yes

yes

yes

yes

no

no

no

no

2nd scenario

IFM 2005, Eindhoven

0:start P1

1:c1:=1

12:true?

10:c1:=1

9:crit-1

6:c1:=04:no-op

3:c1:=1

2:c1:=0 13:end

8:c2=0?

7:turn=2?

5:turn=2?11:turn:=2

yes

yes

yes

yes

no

no

no

no

0:start P2

1:c2:=1

12:true?

10:c1:=1

9:crit-2

6:c2:=04:no-op

3:c2:=1

2:c2:=0 13:end

8:c1=0?

7:turn=1?

5:turn=1?11:turn:=1

yes

yes

yes

yes

no

no

no

no

IFM 2005, Eindhoven

0:start P1 0:start P2

1:c1:=1 1:c2:=1

12:true?12:true?

2:c2:=0

8:c1=0?

9:crit-2Events (occurrences ofactions) participating is 2nd scenario

IFM 2005, Eindhoven


1:c1:=1 1:c2:=1

12:true?12:true?

2:c2:=0

8:c1=0?

9:crit-2

IFM 2005, Eindhoven


1:c1:=1 1:c2:=1

12:true?12:true?

2:c2:=0

8:c1=0?

9:crit-2

Action e is dependent on event fif e and f use mutual variable (including program counter).

Event (occurrence of action) e precedes event f if

•e appears before f in run, and

•e is dependent on f.

IFM 2005, Eindhoven


1:c1:=1 1:c2:=1

12:true?12:true?

2:c2:=0

8:c1=0?

9:crit-2

Partial order semantics.

Equivalent to set of all linearizations. Can define trace equivalencetrace equivalence between linearizations of the same partial order.

IFM 2005, Eindhoven

0:start P1

0:start P2

1:c1:=1

1:c2:=1

12:true?

12:true?

2:c2:=0

8:c1=0?

9:crit-2

Traceequiv

0:start P1

0:start P2

1:c1:=1

1:c2:=1

12:true?

12:true?

2:c2:=0

8:c1=0?

9:crit-2

IFM 2005, Eindhoven

Program transformation (I)For each dependent pair of events e and f of different processes, where e precedes f in run:

Define a semaphore Vij

Add after e:

Freeij : Vij: = Vij + 1

Add before f:

Waitij : wait Vij > 0; Vij: = Vij – 1

(After e, we signal f that it can continue)

IFM 2005, Eindhoven

Program transformation (II)Add a counter counti for each process, counting up before each dependent event participating in previous transformation.

counti := counti + 1

Add after e:

If counti =#e then Freeij

Add before f:

If counti =#f then Freeij

Count also last event on run g and add:

If counti =#g then halt process.

IFM 2005, Eindhoven

Program transformation (III):To allow other executions when not tracing runs, add a variable checki.

Wrap transformed segments Code with

If checki then Code

Minimize synchronization. If we synchronized ef and fg (including the case of synchronization using process sequentiality), then we do not need to add synchronization fo eg (use Floyd-Warshall algorithm to calculate transitive closure of ).

IFM 2005, Eindhoven

Boolean c1, c2, check1, check2;boolean V12 initially 0;integel (1..2) turn;

P1::c1:=1; if check1 then V12:=1; while true do begin if check1 then halt P1; c1:=0; while c2=0 do begin if turn=2 then begin c1:=1; while turn=2 do begin /* no-op */ end; c1:=0 end end; /* critical-section 1*/ c1:=1; turn:=2 end

P2::c2:=1; while true do begin c2:=0; if check2 then begin wait V12>0; V12:=0 end while c1=0 do begin if turn=1 then begin c2:=1; while turn=1 do begin /* no-op */ end; c2:=0 end end; /* critical-section 2*/ if check2 then halt P2; c2:=1; turn:=1end

IFM 2005, Eindhoven

Ultimately periodic sequences (skip)Prefix:(P1(0):start) (P2(0):start)[P1(1):c1:=1] [P2(1):c2:=1] <P2(12):true>yes<P1(12):true>yes[P1(2):c1:=0] [P2(2):c2:=0]<P1(8):c2=0?>yes <P2(8):c1=0?>yes<P1(7):turn=2?>no <P2(7)>turn=1?>yes [P2(3):c2:=1]

Periodic part: <p2(5):turn=1?>yes [P2(4): /* no-op */]

Generate graph G(P,E) for periodic part:

P – processes.E – an edge occurs from Pi to Pj if

there is a dependency between even e of Pi and f of Pj occurring later in the run.

What are the consequents of synchronizing after each period?

IFM 2005, Eindhoven

There are three cases: (skip)

1. The graph G includes all the processes in one strongly connected component.Limited overtaking is not present.

2. The graph includes multiple components, including all processes.Unbounded overtaking is not present.

3. Not all processes are present.The run may be unfair to some processes.

IFM 2005, Eindhoven

0:start P1

1:c1:=1

12:true?

10:c1:=1

9:crit-1

6:c1:=04:no-op

3:c1:=1

2:c1:=0 13:end

8:c2=0?

7:turn=2?

5:turn=2?11:turn:=2

yes

yes

yes

yes

no

no

no

no

0:start P2

1:c2:=1

12:true?

10:c1:=1

9:crit-2

6:c2:=04:no-op

3:c2:=1

2:c2:=0 13:end

8:c1=0?

7:turn=1?

5:turn=1?11:turn:=1

yes

yes

yes

yes

no

no

no

no

IFM 2005, Eindhoven

Preserving the checked property (skip)

Sometimes not all the runs that are trace-equivalent to the original one preserve the checked property .

1. Use a specification formalism that is closed under trace equivalence, or check closeness [PWW98].

2. Add dependencies so that trace equivalence is refined.

• Add dependency between actions when switching an independent pair results in an equivalent run, but fails to satisfy the checked property.

• Or add dependencies between actions that may change propositions that appear in .

IFM 2005, Eindhoven

Calculating the probability of a path. Continuous uniform distribution. Transitions have lower and upper

bound for execution [l,u]. f(x)= 1/(u-l) when lxu,

| 0 otherwise. Joint probability:

f1(y1)f2(y2)…fn(yn)dy1dy2…dyn

on constraint area.

IFM 2005, Eindhoven

Example path ag.a[1,5]

b[2,5]

c[1,4]

g[2,6]

h[3,7]

a c

gb

h

1xa5 2xg6xg7 (because of h)xg-xg4 (because of c)Now integrate on area.

ifm 2005, eindhoven generated path conditions for timed systems doron peled dept. of computer...

Documents