Taming Concurrency:A Program Verification Perspective
Shaz Qadeer
Microsoft Research
Reliable concurrent software?• Correctness problem
– does program behave correctly for all inputs and all interleavings?
• Bugs due to concurrency are insidious – non-deterministic, timing dependent– data corruption, crashes– difficult to detect, reproduce, eliminate
P satisfies S
Undecidable problem!
Why is verification of concurrent programs more difficult than verification of sequential programs?
P satisfies S
Assertions: Provide contracts to decompose problem into a collection of decidable problems• pre-condition and post-condition for each procedure• loop invariant for each loop
Abstractions: Provide an abstraction of the program for which verification is decidable• Finite-state systems
• finite automata• Infinite-state systems
• pushdown automata, counter automata, timed automata, Petri nets
Interference
int t;
t := x;
t := t + 1;
x := t;
Correct
pre x = 0;
post x = 1;
Interference
int t;
t := x;
t := t + 1;
x := t;
Incorrect!
pre x = 0;
post x = 2;
int t;
t := x;
t := t + 1;
x := t;
A B
Controlling interference
int t;
acquire(l);
t := x;
t := t + 1;
x := t;
release(l);
Correct!
pre x = 0;
post x = 2;
int t;
acquire(l);
t := x;
t := t + 1;
x := t;
release(l);
A B
Interference makes program verification difficult
• Annotation explosion with the assertion approach
• State explosion with the abstraction approach
Annotation explosion
• For sequential programs– assertion for each loop – assertion refers only to variables in scope
• For concurrent programs– assertion for each control location– assertion may need to refer to private state of
other threads
State explosion
Sequential Concurrent
Finite-statesystems
Pushdownsystems
P.G
(P.G)3
P = # of program locationsG = # of global states
PSPACEcomplete
Undecidable
n = # of threads
Taming interference
• Atomicity for combating annotation explosion
• Interference-bounding for combating state explosion
Critical_Section l;/*# guarded_by l */int balance;
Bank account
/*# atomic */void deposit (int x) { acquire(l); int r = balance; balance = r + x; release(l);}
/*# atomic */int read( ) { int r; acquire(l); r = balance; release(l); return r;}
/*# atomic */void withdraw(int x) { int r = read(); acquire(l); balance = r – x; release(l);}
Atomicity violation in StringBuffer (Flanagan-Q, PLDI 03)
public final class StringBuffer { private int count; private char[ ] value; . . public synchronized StringBuffer append (StringBuffer sb) { if (sb == null) sb = NULL; int len = sb.length( ); int newcount = count + len; if (newcount > value.length) expandCapacity(newcount); sb.getChars(0, len, value, count); //use of stale len !! count = newcount; return this; }
public synchronized int length( ) { return count; }
public synchronized void getChars(. . .) { . . . }}
Inadequate atomicity is a good predictor ofundesirable interference!• compared to data races
Numerous tools for detecting atomicity violations• static (Type systems, ESPC, QED)• dynamic (Atomizer, AVIO, AtomAid, Velodrome, …)
Significant effort cutting across communities • architecture and operating systems• programming languages and compilers • testing and formal methods
Definition of atomicity
• deposit is atomic if for every non-serialized execution, there is a serialized execution with the same behavior
acq(l) r=bal bal=r+n rel(l)x y z
Serialized execution of deposit
acq(l) r=bal bal=r+n rel(l)x y z
acq(l) r=bal bal=r+n rel(l)x y z
Non-serialized executions of deposit
S0 S1 S2 S3 S4 S7S6S5
acq(l) r=bal bal=r+n rel(l)x y z
S0 T1 T2 T3 S4 S7T6S5
r=bal bal=r+n rel(l)x y zacq(l)
S0 T1 S2 T3 S4 S7S6S5
y r=bal bal=r+n rel(l)x acq(l) z
S0 T1 T2 T3 S4 S7S6S5
acq(l) r=bal bal=r+n rel(l)x y z
S0 S1 S2 T3 S4 S7S6S5
acq(l) r=bal bal=r+n rel(l)y zx
Reduction (Lipton, CACM 75)
B: both right + left commutes– variable access holding lock
A: atomic action, non-commuting – access unprotected variable
Four atomicities
L: left commutes– lock release
R: right commutes– lock acquire
S0. S5
R* A L*x Y. . .
S0. S5
R* A L*x Y. . .
Sequential composition Theorem: Sequence (R+B)*;(A+); (L+B)* is atomic
CCCCCCCCCAAACCCLLLCARARRCARLBBCARLB;
; A
R;A;L ; R;A;L
C
A
; A
R; B ; A; L
A
R
Critical_Section l;/*# guarded_by l */int balance;
Bank account
/*# atomic */void deposit (int x) { acquire(l); int r = balance; balance = r + x; release(l);}
/*# atomic */int read( ) { int r; acquire(l); r = balance; release(l); return r;}
/*# atomic */void withdraw(int x) { int r = read(); acquire(l); balance = r – x; release(l);}
RBBL
A RBLB
A
ARBL
C
Incorrect!
Critical_Section l;/*# guarded_by l */int balance;
Bank account
/*# atomic */void deposit (int x) { acquire(l); int r = balance; balance = r + x; release(l);}
/*# atomic */int read( ) { int r; acquire(l); r = balance; release(l); return r;}
/*# atomic */void withdraw(int x) { acquire(l); int r = balance; balance = r – x; release(l);}
RBBL
A RBLB
A
RBBL
A
Correct!
Taming interference
• Atomicity for combating annotation explosion
• Interference-bounding for combating state explosion
Interference bounding
• An approach to the state-explosion problem
• Explore all executions with a bounded amount of interference– increase the interference bound iteratively
• Good idea if low interference bound– can be exploited by algorithms– is enough to expose bugs
Context-bounded verification [Wu-Q, PLDI 04] [Rehof-Q, TACAS 05]
Context Context Context
Context switch Context switch
• Interference proportional to number of context switches
• Explore all executions with few context switches• Unbounded computation within each context
– Different from bounded model checking
Context-bounding today
• CHESS [Musuvathi-Q, PLDI 07]• JMoped [Suwimonteerabuth et al., SPIN
08]• SPIN for multithreaded C programs [Zaks-
Joshi, SPIN 08]• CBA [Lal-Reps, CAV 08]• Static Driver Verifier
Testing concurrent programs is HARD
• Bugs hidden in rare thread interleavings
• Today, concurrency testing = stress testing– Poor coverage of interleavings– Unpredictable coverage results in
“Heisenbugs”
• The mark of reliability of the system still remains its ability to withstand stress
CHESS in a nutshell
• Replace the OS scheduler with a demonic scheduler
• Systematically explore all scheduling choices
ConcurrentProgram
Win32 API
Kernel SchedulerDemonicScheduler
CHESS architecture
Kernel: Threads, Scheduler, Synchronization Objects
While(not done) { TestScenario()}
TestScenario() { …}
Program
CHESSCHESS runs the scenario in a loop • Each run takes different interleaving• Each run is repeatable
Win32 API
Intercept synchronization and threading calls• To control and introduce
nondeterminismDetect• Assertion violations• Deadlocks• Dataraces• Livelocks
CHESS methodology generalizes
• CHESS works for– Unmanaged programs (written in C, C++)– Managed programs (written in C#,…)– Singularity applications
• With appropriate wrappers, can work for Java, Linux applications
.NETProgram
.NET CLR
CHESSWin32
Program
Win32 / OS
CHESSSingularityProgram
Singularity
CHESS
CHESS: Systematic testing for concurrency
Kernel: Threads, Scheduler, Synchronization Objects
While(not done){ TestScenario()}
TestScenario(){ …}
Program
CHESS
Win32 API
CHESS runs the scenario in a loop • Each run is a different interleaving• Each run is repeatable
x = 1; … … … … … x = k;
State-space explosion
x = 1; … … … … …x = k;
…
n threads
k steps each
• Number of executions
= O( nnk )
• Exponential in both n and k– Typically: n < 10 k > 100
• Limits scalability to large programs
Goal: Scale CHESS to large programs (large k)
Thread 1 Thread n
x = 1;if (p != 0) { x = p->f;}
Preemption-boundingPrioritize executions with small number of preemptions
Two kinds of context switches: Preemptions – forced by the scheduler
e.g. Time-slice expiration
Non-preemptions – a thread voluntarily yieldse.g. Blocking on an unavailable lock, thread end
x = p->f;}
x = 1;if (p != 0) {
p = 0;
preemption
non-preemption
Thread 1 Thread 2
Preemption-bounding in CHESS• The scheduler has a budget of c preemptions
– Nondeterministically choose the preemption points
• Resort to non-preemptive scheduling after c preemptions
• Once all executions explored with c preemptions– Try with c+1 preemptions
x = 1; … … … …
x = 1; … … …
…x = k;
Property 1: Polynomial bound• Terminating program with fixed inputs and deterministic threads
– n threads, k steps each, c preemptions
• Number of executions <= nkCc . (n+c)!
= O( (n2k)c. n! )
Exponential in n and c, but not in k
x = 1; … … … … …x = k;
x = 1; … … … … … x = k;
… …
x = k;
• Choose c preemption points
• Permute n+c atomic blocks
Thread 1 Thread 2
Property 2: Simple error traces• Finds smallest number of preemptions to the
error
• Number of preemptions better metric of error complexity than execution length
Property 3: Coverage metric• If search terminates with preemption-
bound of c, then any remaining error must require at least c+1 preemptions
• Intuitive estimate for– The complexity of the bugs remaining in
the program– The chance of their occurrence in practice
Property 4: Many bugs with 2 preemptions
Program Lines of code
Bugs
Work Stealing Q 4K 4
CDS 6K 1
CCR 9K 3
ConcRT 16K 4
Dryad 18K 7
APE 19K 4
STM 20K 2
TPL 24K 9
PLINQ 24K 1
Singularity 175K 2
37 (total)
Acknowledgement: testers from PCP team
Coverage vs. Preemption-bound
CHESS status
• Being used by testers in many Microsoft product groups
• Demo and session at Professional Developer Conference 2008
• CHESS for Win32 – http://research.microsoft.com/projects/chess
• CHESS for .NET– Coming soon
So far …
• Interference makes program verification difficult
• Taming interference– atomicity – interference-bounding
Whither next?
• Concurrent programs as a composition of modules implementing atomic actions
• Inteference-bounding for general concurrent systems
• Symbolic context-bounded verification
Structuring concurrent programs• In addition to atomicity, we need
– behavioral abstraction analogous to pre/post-conditions for sequential code fragments
– intuitive and simple contract language
• A calculus of atomic actions (Elmas-Tasiran-Q, POPL 08)– QED verifier
Interference-bounding for general concurrent systems
• Task = unit-of-work• Shared-memory systems
– Task (usually) corresponds to a thread• Message-passing systems
– Task corresponds to a sequence of messages• Need linguistic support for the task
abstraction in software models and implementations
Context-bounded reachability is NP-complete
Unbounded Context-bounded
Finite-statesystems
Pushdownsystems
PSPACEcomplete
NP-complete
Undecidable NP-complete
P = # of program locationsG = # of global states
n = # of threadsc = # of context switches
Symbolic context-bounded verification
• The transition relations of tasks are usually encoded symbolically– reachability analysis is PSPACE-complete even for a
single task
• Scalable tools for single symbolic task– Bebop, Moped, Getafix
• Can we extend these techniques to deal with multiple tasks?– Lal and Reps (TACAS 2008, CAV 2008)– Suwimonteerabuth et al. (SPIN 2008)
Questions?