1 today executions, paths, and states testing via model checking

1

Today

Executions, paths, and states

Testing via model checking

2

Executions, Paths, and StatesControl flow graph (CFG):structure of a program’s code An execution is simply a

run of the program – the resultof some set of commandsSome executions, with different operations:

A path is a particular waythrough the CFG – ignoringthe commands – where theprogram “goes”

Same path, different executions

A state is the program’s location+ memory contents at a point intime

Same state, different executions,different paths(in the executions, color only shows

the location in the CFG, not thecontents of memory, which may bedifferent even if colors are the same) Same paths, different states

3

A Question

Testing can focus on exploring lots of• Executions

• Random testing• Paths

• Concolic testing• States?

• Model checking• But first: does this gain us anything?

4

A Question

First question: does it make the problem more manageable• That is, are there cases where exploring all

states is more feasible than exploring all paths?

5

Paths vs. Statesint G = 0;

int foo (int x, int y) {if (x < y) G++;if (x == 10) G++;

}

int bar (int x, int y) {if (x < 50) G++;if (y > 8) G++;

}

int main () {foo(x, y);bar(x, y);

if (x + y == 40) G++;

}

Once more, how many paths through thisprogram?

2

4

8

16

32

6

Paths vs. Statesint G = 0;

int foo (int x, int y) {if (x < y) G++;if (x == 10) G++;

}

int bar (int x, int y) {if (x < 50) G++;if (y > 8) G++;

}

int main () {foo(x, y);bar(x, y);

if (x + y == 40) G++;

}

What if we consider the number of states?

If we include x and y’s values in the state,the number is much larger than the numberof paths… so it’s certainly not always animprovement!

7

Paths vs. States

pick(chunk, 0, 10);

pick(nbytes, 0, 10);

write(chunk, wbuf, nbytes);

pick(chunk, 0, 10);


read(chunk, rbuf, nbytes);

pick(chunk, 0, 10);


write(chunk, wbuf, nbytes);

Less clear which is better in thiscase: lots of states, but also lotsof paths

Paths lets us consider inputs thatproduce the same sequence of writesas “the same”

States means we don’t have to worryabout error returns in the past, if thaterror return doesn’t change the systemstate

8

Paths vs. States

So, considering states rather than paths seems to be a trade-off without a clear winner – often at least as un-manageable as complete path exploration

Which is better for error detection?• Hard to say, but for a class of errors

(reachability), we can say that complete state coverage would definitely expose them

9

Model Checking

A model checker is a tool for exploring a state space

Basic idea: generate every reachable state of a transition system• Think of states as nodes in a graph• Directed edges mean “from this state, this is

a possible next state of the program”• Multiple outgoing edges where there is

input/thread scheduling/other nondeterminism

10

Model Checking

Model checking is a huge research field• Clarke, Emerson, and Sifakis received this

year’s Turing Award for coming up with the basic idea

• Some major subfields:• Hardware and software• Symbolic vs. explicit-state

11

Model Checking

Symbolic model checkers “explore” a state space by manipulating logical expressions representing many states

In explicit-state model checking (part of) the graph is actually enumerated• Especially useful for software, where

complex data structures are hard to handle symbolically

• Closer to testingGerard’s SPIN is the most popularexplicit-state model checker

12

Model Checking

We will be looking at one particular kind of explicit-state model checking• Using SPIN• To explore the state spaces of C

programs

Other model checkers used for this approach to testing• Java PathFinder 2 (NASA Ames)• Bogor (U Kansas/Nebraska)

13

Model Checking: State-Based Testing

Model-checking by executing the program• Backtracking search for all states

State already visited!Backtrack and try adifferent operation Done with test!

Backtrack and try adifferent operation

CFG

State already visited!Backtrack and try adifferent operation

Will explore, as a side-effect,many executions and many paths,but the goal is to explore states

14

Many Software Model Checkers

CBMC

BLAST

SLAM

JPF2

SPIN

CMC

CRunner

MAGIC

VeriSoftBogor

15

Two Approaches

CBMC

BLAST

SLAM

JPF2

SPIN

CMC

CRunner

MAGIC

Analysis of derived transition system

Execution of actual code

VeriSoft

CRunner

Our focus in this class

(dynamic: like testing)

(static: like a complete analysis)

Bogor

16

SPIN and Model-Driven Verification

SPIN 4.0 introduced model-driven verification• Can embed C code in a PROMELA model• Code is executed as part of a transition during

depth-first search

SPIN compiles a PROMELA model into a C program: it’s a model checker generator• Embed C code in transitions by executing the

compiled C code – at native speed, with real compiled behavior (including optimizations)

• Take advantage of all SPIN features – hashing, multicore exploration, etc.

17


Used to “model check” significant pieces of implementation code at JPL

Very useful for checking rich properties that automated abstraction engines often fail to handle• Functional correctness (differential checking

with a reference implementation)• Invariants of rich data structures

Using SPIN for testing C programs

18


When SPIN backtracks, it uses information on how to restore the state of the C program:• Tracked memory is

restored on backtrack• Matched memory is also

used to determine if a state has been visited before

• All other memory is not restored on backtrack or stored in state table

Execute C codeuntil control

returns to SPIN

Push tracked & matchedstate on stack

Has state beenvisited before?

Store matchedstate in state

table

Backtrack:pop stack &

restore tracked &matched state

Y N

19


Use state that is tracked but not matched to introduce abstractions• E.g., symmetry reduction: a property that does

not depend on the order of items in a linked list• Track the original version of the list• Copy and sort the list after each transition• Match on the sorted list only• SPIN will backtrack when it visits a state with

same list contents, even in a different order

20

Testing via Model Checking

We do not expect to cover all states• There are far too many (trillions, at least)

Use abstraction (“consider this state and that state to be the same”) to limit the exploration – may miss errors

Use techniques that have a small probability of missing some states (hashing schemes)

21


Unsound abstractions:• For example, in the flight flash file

systems we’ve checked, the state of the flash device

Used page

Free page

Dirty page

Bad block

live pages? (0-1) x dirty pages (0-1) x block state (bad, free, current)

22


Does not guarantee that states (or errors) won’t be missed, but provides a better understanding of which states have been explored, based on the program’s purpose

Manages state space size

Used page

Free page

Dirty page

Bad block

live pages? (0-1) x dirty pages (0-1) x block state (bad, free, current)

23

Unsound Abstractions

“Abstraction” can mean a lot of things

What I’ll be talking about in this class:• One operation: from a concrete state, c, we

can compute an abstract state A(c)• Backtrack according to abstract state matching• Goal: we would like to explore every reachable

abstract state of a C program

24


For some abstractions, the goal is reached if we simply match on abstract state – if

c1, c2 . A(c1) = A(c2) c3 . T(c1, c3) c4 . T(c2, c4) A(c3) = A(c4)

25


Otherwise:

These states abstract the same

So do theseAnd these

These, however, do not abstract equivalently,and can only be reached through their parents(and are the only concrete representatives oftheir respective abstract states).

Whichever of these concrete states we explorefirst, we will miss one of the children.

26


We usually don’t have soundness for any abstractions that reduce the state-space to a manageable size

If not, unsound in our context (underapproximating abstract successors by abstracting concrete states)

That’s ok• It’s just testing, we don’t expect to be

complete

27


Requires the ability to restore a running program to an earlier execution state• Difficult engineering problem, handled by

applying automatic code instrumentation• Rewrite the program to check that the

backtracking works properly• As a side-benefit, can simulate flight

software resets and hardware faults

28

Simple PROMELA Codeint x;

int y;

active proctype main () {

if:: x = 1:: x = 2fi;

assert (x == y);

}

Start simple

This model has 7 states

What are they?

State = (PC, x, y)

1

2

3

5

7

SPIN’s nondeterministic choice construct

Picks any one of the choices that is enabled

How do we guard a choice?if:: (x < 10) -> y = 1:: (x < 5) -> y = 3:: (x > 1) -> y = 4fi;

Not mutuallyexclusive!

29


int y;


if:: x = 1:: x = 2fi;

if:: y = 1:: y = 2fi;

if:: x > y -> x = y:: y > x -> y = x:: else -> skipfi;

assert (x == y);

}

This model has 17 states

What are they?

State = (PC, x, y)

123

5

79

13

14

1517

Er…

Don’t worry about state-counting toomuch – SPIN has various automaticreductions and atomicity choices thatcan make that difficult

30



do

:: (x < 10) -> x++

:: break

od

Only a couple more PROMELAconstructs to learn for building testharnesses: the do loop

Like if, except it introducesa loop to the top – break choicecan exit the loop

This nondeterministically assigns xa value in the range 0…9

31


inline pick (var, MAX)

var = 0;

do

:: (var < MAX) -> var++

:: break

od

inline gives us a macro facility

As you can imagine, this is auseful macro for building a test harness!

32


Basic idea:• We’ll write a test harness in PROMELA• Use SPIN to backtrack and explore all

input choices• Call C code we’re testing• Use abstraction to limit the number of

states we consider

• We can even “trick” SPIN into doing pure random testing!

33


Rest of class:• We’ll look at the idea of “model-driven verification” that

drives all this• I’ll show how to write a simple PROMELA harness to

test C code

• Thursday: we’ll investigate automatic code instrumentation, some novel coverage-based code strategies, and see how model checking and random testing compare

• Download SPIN yourself, and play with it!• http://www.spinroot.com• Examples available on class website

1 today executions, paths, and states testing via model checking

Documents