Guoliang Jin, Linhai Song, Wei Zhang, Shan Lu, and Ben Liblit

University of Wisconsin–Madison

Automated Atomicity-Violation Fixing

Focus on Single-Variable Atomicity Bugs Multicore era is coming already here

Programmers still struggle to reason about concurrency Many ways to find and replay concurrency bugs

Atomizer, CTrigger, CCI, Kivati, Pacer, Falcon, … But that’s still not a fix!

Automated fixer for single-variable atomicity bugs Leverage variety of bug-finding tools Static analyses & optimizations to patch one bug Heuristics to merge & simplify groups of patches Run-time validation during testing or post-deployment

Thread 2Thread 1

Collect Atomicity-Violation Triples

read x

read x

write x

Locate and name bad interleaving participants previous access current access remote access

Full stack trace for each

Four standard variations Don’t care which is

which {(p1, c1, r1), …, (pn,

cn, rn)} Starting point for our

tool

p: read x

c: read x

r: write x

pi

ci

ri

Static Analyses to Construct One Patch1. Ensure that p, c are in

same function Longest common

prefix of stack traces

2. Find nodes on p c paths not crossing p or c

Defines protected region

3. Lock on region entry; unlock on region exit

4. Lock before r;unlock after r

p

c

Lock Selection and Optimization Any potentially-blocking operations in critical

region? No: wait forever when acquiring lock Yes: time out when acquiring lock Conservative over-approximation finds ad hoc spin

loops Is recursion possible within critical region?

No: use non-reentrant locks Yes: use reentrant locks

Can we reach r while in the p–c critical region? No: retain lock operations before/after r Yes: remove lock operations before/after r

Try to reuse existing global lock (in practice, never)

Selective Fusion of Multiple Patches Can improve performance and readability

Depends on costs of lock/unlock, thread count, contention, …

No statically-optimal answer But some redundant operations are always good

to remove May be mandatory to avoid deadlocks Merge if p1, c1, or r1 are in any of patch 2’s

critical regions Heuristic, but works well in practicep1 p2 c2 c1p1 p2 c1 c2

p1

p2

c1

c2

Run-Time Monitoring and Feedback Never adds new interleavings, but may cause deadlock Choice of two run-time deadlock detectors

1. High-overhead, immediate notification Must monitor all synchronization operations at all times Always knows complete resource-allocation graph

2. Low-overhead, delayed notification Does nothing until after a lock timeout Eventually infers resource-allocation graph, then checks for cycles

If bug detector is incomplete, reapply to patched program May report additional (p, c, r) pairs, requiring further

patching Done fixing when bug detector can no longer find problems

Evaluation: Overall Patch QualityBug Naïve Unmerged Merged Manual

Apache - -

MySQL 1 -

MySQL 2 -

Mozilla 1 -

Mozilla 2 -

Cherokee -

FFT - -

PBZIP2 - - -

Patched failure rates: 0% (except PBZIP2 and FFT)

Patched overheads: <0.6% (except PBZIP2)

Readily understandable, with few mutexes after merging

Conclusions Patient says, “Doctor, doctor, it hurts when I do this.”

Doctor replies, “Then don’t do that!” ☺ Natural to apply this to concurrency But must be exceedingly careful in the details

Makes strong guarantees, but does not promise the world Never allows interleavings that were not already possible But may cause deadlocks by over-constraining execution

space Uses some heuristics, but excellent results in practice

Overheads too low to reliably measure Produces small, simple, understandable patches Completely eliminates detected bugs in targeted class