poirot – a concurrency sleuth

Poirot – A Concurrency Sleuth

Shaz QadeerResearch in Software Engineering

Microsoft Research

Concurrent programming is difficult

…if (irp->Cancel) { IoCompleteIrp(irp);} else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp);}…

…irp->Cancel = TRUE;fn = IoSetCancelRoutine(Irp, NULL);if (fn) { fn(irp);}…

void CancelRoutine(IRP *irp) { IoCompleteIrp(irp);}

Normal Cancellation

IO_REQUEST_PACKET *irp;

irp->Cancel = FALSE;irp->CancelRoutine = NULL;

…if (irp->Cancel) { IoCompleteIrp(irp);} else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp);}…

…irp->Cancel = TRUE;fn = IoSetCancelRoutine(Irp, NULL);if (fn) { fn(irp);}…

void CancelRoutine(IRP *irp) { IoCompleteIrp(irp);}

Normal Cancellation

IO_REQUEST_PACKET *irp;

irp->Cancel = FALSE;irp->CancelRoutine = NULL;

…if (irp->Cancel) { IoCompleteIrp(irp);} else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp);}…

…irp->Cancel = TRUE;fn = IoSetCancelRoutine(Irp, NULL);if (fn) { fn(irp);}…

void CancelRoutine(IRP *irp) { IoCompleteIrp(irp);}

Normal Cancellation

IO_REQUEST_PACKET *irp;

irp->Cancel = FALSE;irp->CancelRoutine = NULL;

…if (irp->Cancel) { IoCompleteIrp(irp);} else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp);}…

…irp->Cancel = TRUE;fn = IoSetCancelRoutine(Irp, NULL);if (fn) { fn(irp);}…

void CancelRoutine(IRP *irp) { IoCompleteIrp(irp);}

Normal Cancellation

IO_REQUEST_PACKET *irp;

irp->Cancel = FALSE;irp->CancelRoutine = NULL;

…if (irp->Cancel) { IoCompleteIrp(irp);} else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp);}…

…irp->Cancel = TRUE;fn = IoSetCancelRoutine(Irp, NULL);if (fn) { fn(irp);}…

void CancelRoutine(IRP *irp) { IoCompleteIrp(irp);}

Normal Cancellation

IO_REQUEST_PACKET *irp;

irp->Cancel = FALSE;irp->CancelRoutine = NULL;

…if (irp->Cancel) { IoCompleteIrp(irp);} else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp);}…

…irp->Cancel = TRUE;fn = IoSetCancelRoutine(Irp, NULL);if (fn) { fn(irp);}…

void CancelRoutine(IRP *irp) { IoCompleteIrp(irp);}

Normal Cancellation

IO_REQUEST_PACKET *irp;

irp->Cancel = FALSE;irp->CancelRoutine = NULL;

…if (irp->Cancel) { IoCompleteIrp(irp);} else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp);}…

…irp->Cancel = TRUE;fn = IoSetCancelRoutine(Irp, NULL);if (fn) { fn(irp);}…

void CancelRoutine(IRP *irp) { IoCompleteIrp(irp);}

Normal Cancellation

Fatal error!

IO_REQUEST_PACKET *irp;

irp->Cancel = FALSE;irp->CancelRoutine = NULL;

Concurrent programming is difficult

• Multiple loci of control resulting in non-local control flow

• Code difficult to understand and review

What about verification?• Assertion-based modular reasoning becomes complicated due

to non-local interactions– Floyd-Hoare morphs into Owicki-Gries

• Even with simple (finite) abstractions, the presence of concurrency makes the analysis computationally very expensive

Sequential Concurrent

Single Procedure

P-time PSPACE-complete

Multi Procedure

P-time Undecidable

What about testing?

• Number of executions = O( nnk ) • Exponential in both n and k

x = 1; … … … … … x = k;

x = 1; … … … … …x = k;

…

Thread 1 Thread nScheduling nondeterminism

• Uncontrollable

• Unobservable

• Exponential

Concurrency is important• More than ever before

• Increasing importance of communicating systems– networked devices– cyber-physical systems

• Distributed programs running on the cloud– EC2, Azure, AppEngine, …

• Parallel programs running on multicores and GPUs– TBB, TPL, CUDA, AMP, …

Concurrency testing with CHESS

• Deterministic scheduling– make scheduling choices observable and

controllable

• Search prioritization– combating the combinatorial explosion of possible

schedules

Deterministic scheduling

Kernel: Threads, Scheduler, Synchronization Objects

While(not done){ TestScenario()}

TestScenario(){ …}

Program

CHESS

Win32 API

Tester Provides a Test Scenario

CHESS runs the scenario in a loop • Each run is a different interleaving• Each run is repeatable

Search prioritization (I)

• Given p ≥ 0, generate all schedules with up to p preemptions

• Pseudo-polynomial number of schedules– polynomial in preemption bound and schedule

points– exponential in number of threads

• Many bugs with fewer than 2 preemptions • Simple error traces for easier debugging

Search prioritization (II)

• Given p ≥ 0 and deterministic schedulers S0, …, Sp-1, schedule according to S0, …, Sp-1 in sequence moving from one to next nondeterministically– e.g., round-robin non-preemptive scheduling with p

different round-robin orders• Polynomial number of schedules• Testers can innovate by designing domain-

specific deterministic schedulers

CHESS is available

• Used internally by Microsoft product groups and externally by Microsoft customers

• Binary and source code available at:– http://chesstool.codeplex.com

Limitations of CHESS

• Exposing and gaining control of scheduling choices is difficult– most implementation effort and user frustration due

to this problem

• Testing components that interact extensively with the environment is difficult

• Input coverage is not addressed

Static program exploration with Poirot

• Symbolic instead of concrete execution

• C: Source code for software component• E: Model for environment and scheduler

• Explore behaviors of C+E– for all symbolic inputs– for all scheduling choices

Disk

AsyncRead(…) {

}

DiskReader(…) {

}

DiskReader(…) {

}

headtail

Request queue In-memory cache

cachecacheSize

Demo: Asynchronous File I/O

Poirot architecture

Trace Viewer

Concurrent .NET Program

Concurrent Boogie Program

Coverage Report

.NET Boogie

Concurrent C Program

C Boogie

Corral

Sequentialization Stratified Search

Error Trace

Concurrent Boogie Program

SequentialBoogie Program

Coverage Report

Searching with Corral

Refinement

Abstraction

Concurrent Boogie Program

Abstraction

• Set of global variables G

• Set of tracked variables T

• Drop writes to variables in G-T

• Replace reads to variables in G-T with nondeterministic values

Sequentialization Stratified Search

Error Trace

Concurrent Boogie Program

SequentialBoogie Program

Coverage Report

Searching with Corral

Refinement

Abstraction

Concurrent Boogie Program

Refinement

• Path p– feasible if only variables in T are tracked– infeasible if all variables in G are tracked

• Expand tracked set T to U such that p infeasible while tracking only variables in U

• Naïve algorithm: linear scan of G-T• New divide-and-conquer algorithm

– best case log(|G-T|)– worst case 2*|G-T|

Sequentialization Stratified Search

Error Trace

Concurrent Boogie Program

SequentialBoogie Program

Coverage Report

Searching with Corral

Refinement

Abstraction

Concurrent Boogie Program

Sequentialization (I)

• Given a concurrent program P, construct a sequential program Q such that Q P

• Drop each occurrence of async-call

• Convert each occurrence of async-call to call

Sequentialization (II)

• Given a concurrent program P, construct a family of programs Qi such that – Q0 Q1 Q2 … P

– iQi = P

• Even better if interesting behaviors of P manifest in Qi for low values of i

Context-bounding

• Under-approximation parameterized by K ≥ 0– executions in which each thread gets at most K

contexts to execute

• As K , we get all behaviors

• Can we create sequentializations for context-bounding?

Sequentializing context switchesShared Memory

T1 T2

Local Memory Local Memory

Execution: T1 T2 T1 T2 T1

T1 T2 T1(s1, l1) (s2, l2) s2l2

Guess and verify

T1 T2T1(s1, l1) (s2, l2) (s3, l2)Guess the effect of T2 Verify the guess

• Make copies of global variables• Source-to-source translation

– linear in program size and K• Generalizes to dynamically-created threads

Sequentialization Stratified Search

Error Trace

Concurrent Boogie Program

SequentialBoogie Program

Coverage Report

Searching with Corral

Refinement

Abstraction

Concurrent Boogie Program

Stratified search

main

…

Call tree given recursion bound r

VC(T)

assert no bugSummaries(L)T

L

assert no bug

VC(p)

Convert loops to recursive calls

Poirot status

• Medium-sized C programs– up to 20K low-level systems code– reports precise traces at scale

• Small .NET programs– bytecode to Boogie translator in progress

• Try: http://www.rise4fun.com/Poirot • Download available

Why bounded search?Data: Boolean, Integers, ArraysControl: Sequencing, Choice, Iteration, Call, Async-Call

SequencingChoice

NP-complete

SequencingChoiceIterationCallAsync-call

Undecidable

SequencingChoiceIterationCallAsync-call

+ bound

DecidablePSPACE-hard

Advances in SAT/SMT-solvers have made this problem tractableHAVOC verifier deployed for security analysis in Windows/IE

Rationale: It is better to fail at the simpler problem!

Poirot collaborators

Akash Lal, MSR Bangalore

Shuvendu Lahiri, MSR Redmond

Questions