Scalable Dynamic Analysis for Automated Fault Location and

Avoidance

Rajiv Gupta

Funded by NSF grants from CPA, CSR, & CRI programsand grants from Microsoft Research

Motivation

Software bugs cost the U.S. economy about $59.5 billion each year [NIST 02].

Embedded Systems• Mission Critical / Safety Critical Tasks • A failure can lead to Loss of Mission/Life.

(Ariane 5) arithmetic overflow led to shutdown of guidance computer.(Mars Climate Orbiter) missed unit conversion led to faulty navigation data.(Mariner I) missing superscripted bar in the specification for the guidance

program led to its destruction 293 seconds after launch. (Mars Pathfinder) priority inversion error causing system reset.(Boeing 747-400) loss of engine & flight displays while in flight.(Toyota hybrid Prius) VSC, gasoline-powered engine shut off. (Therac-25) wrong dosage during radiation therapy.…….

Overview

Dynamic SlicingOffline

Environment FaultsOnline

Program Execution

Fault

Fault Location

Fault Avoidance

ScalabilityTracing + Logging

Long-runningMulti-threaded

Fault Location

Goal:

Assist the programmer in debugging by automatically narrowing the fault to a small section of the code.

Dynamic Information• Data dependences• Control dependences• Values

Execution Runs• One failed execution & • Its perturbations

Dynamic Information

……

ExecutionProgram Dynamic Dependence Graph

Data Control

Approach

Detect execution of statement s such that Faulty code Affects the value computed by s; or Faulty code is Affected-by the value computed by s

through a chain of dependences.

Estimate the set of potentially faulty statements from s:

Affects: statements from which s is reachable in the dynamic dependence graph. (Backward Slice)

Affected-by: statements that are reachable from s in the dynamic dependence graph. (Forward Slice)

Intersect slices to obtain a smaller fault candidate set.

Backward & Forward Slices

Erroneous Output

Failure inducingInputBackward Slice

[Korel&Laski,1988] Forward Slice [ASE-05]

Failure InducingInput

ErroneousOutput

[ASE-05]

For memory bugs the number of statements is very small (< 5).

Backward & Forward Slices

Bidirectional Slices

BackwardSlice of CP

ForwardSlice of CP

+ BidirectionalSliceCombined

Slice

[ICSE-06]

• Found critical predicates in 12 out of 15 bugs• Search for critical predicate: Brute force: 32 predicates to 155K predicates; After Filtering and Ordering: 1 to 7K predicates.

Critical Predicate:

An execution instance

of a predicate such

that changing its

outcome “repairs” the

program state.

Pruning Slices

Confidence in v

v

C(v): [0,1]

0 - all values of v produce same

1 - any change in v will change

How? Value profiles.

1 1

1

[PLDI-06]

Test Programs

Real Reported Bugs Injected Bugs

Nine logical bugs (incorrect ouput)

• Unix utilities grep 2.5, grep 2.5.1,

flex 2.5.31, make 3.80.

Six memory bugs (program crashes)

• Unix utilities gzip, ncompress,

polymorph, tar, bc, tidy.

Siemens Suite (numerous versions)

schedule, schedule2, replace, print_tokens..

• Unix utilities gzip, flex

Dynamic Slice Sizes

Buggy Runs BS FS BiS

flex 2.5.31(a) 695 605 225flex 2.5.31(b) 272 257 NA

flex 2.5.31(c) 50 1368 NA

grep 2.5 NA 731 88

grep 2.5.1(a) NA 32 111

grep 2.5.1(b) NA 599 NA

grep 2.5.1(c) NA 12 453

make 3.80(a) 981 1239 1372

make 3.80(b) 1290 1646 1436

gzip-1.2.4 34 3 39

ncompress-4.2.4 18 2 30

polymorph-0.4.0 21 3 22

tar 1.13.25 105 202 117

bc 1.06 204 188 267

tidy 554 367 541

Combined Slices

Buggy Runs BS BS^FS^BiS (%BS)

flex 2.5.31(a) 695 27 (3.9%)

flex 2.5.31(b) 272 102 (37.5%)

flex 2.5.31(c) 50 5 (10%)

grep 2.5 NA 86 (7.4%*EXEC)

grep 2.5.1(a) NA 25 (4.9%*EXEC)

grep 2.5.1(b) NA 599 (53.3%*EXEC)

grep 2.5.1(c) NA 12 (0.9%*EXEC)

make 3.80(a) 981 739 (81.4%)

make 3.80(b) 1290 1051 (75.3%)

gzip-1.2.4 34 3 (8.8%)

ncompress-4.2.4 18 2 (14.3%)

polymorph-0.4.0 21 3 (14.3%)

tar 1.13.25 105 45 (42.9%)

bc 1.06 204 102 (50%)

tidy 554 161 (29.1%)

Evaluation of Pruning

Program Description LOC Versions Tests

print_tokens Lexical analyzer 565 5 4072

print_tokens2 Lexical analyzer 510 5 4057replace Pattern replacement 563 8 5542

schedule Priority scheduler 412 3 2627schedule2 Priority scheduler 307 3 2683

gzip Unix utility 8009 1 1217flex Unix utility 12418 8 525

Siemen’s Suite

• Single error is injected in each version.• All the versions are not included:

No output or the very first output is wrong; Root cause is not contained in the BS (code missing error).

Program BS Pruned Slice Pruned Slice / BS

print_tokens 110 35 31.8%

print_tokens2 114 55 48.2%replace 131 60 45.8%

schedule 117 70 59.8%schedule2 90 58 64.4%

gzip 357 121 33.9%flex 727 27 3.7%

Evaluation of Pruning

Effectiveness

Backward

Slice

[AADEBUG-05]

≈ 31% of Executed

Statements

Combined Slice

[ASE-05,ICSE-06]

≈ 36% of Backward

Slice≈ 11% of Exec.

Erroneous outputFailure inducing inputCritical predicate

Pruned Slice

[PLDI-06]

≈ 41% of Backward

Slice≈ 13% of Exec.

ConfidenceAnalysis

Effectiveness

Slicing is effective in locating faults.

No more than 10 static statements had to be inspected.

Program-bug Inspected Stmts.

mutt – heap overflow 8

pine – stack overflow 3

pine – heap overflow 10

mc – stack overflow 2

squid – heap overflow 5

bc – heap overflow 3

Execution Omission Errors

X=

X=

=X

A =

A<0

X=

X=

=X

A =

A<0

X=

=X

A =

A<0

• Inspect pruned slice.• Dynamically detect an Implicit dependence.• Incrementally expand the pruned slice.

[PLDI-07]

Implicitdependence

Scalability of Tracing

Dynamic Information Needed• Dynamic Dependences

for all slicing• Values for Confidence Analysis

for pruning slices

annotates the static program representation

Whole Execution Trace (WET)

Trace Size• ≈ 15 Bytes / Instruction

Trace Sizes & Collection Overheads

Trace sizes are very large for even 10s of execution.

Program Running Time

Dep. Trace

Collection Time

mysql 13 s 21 GB 2886 s

prozilla 8 s 6 GB 2640 s

proxyC 10 s 456 MB 880 s

mc 10 s 55 GB 418 s

mutt 20 s 388 GB 3238 s

pine 14 s 156 GB 2088 s

squid 15 s 88 GB 1132 s

Compacting Whole Execution Traces

Explicitly remember dynamic control flow trace. Infer as many dynamic dependences as possible

from control flow (94%), remember the remaining dependences explicitly (≈ 6%). Specialized graph representation to enable inference.

Explicitly remember value trace. Use context-based method to compress dynamic

control flow, value, and address trace. Bidirectional traversal with equal ease

[MICRO-04, TACO-05]

Input: N=2

51: for I=1 to N do61: if (i%2==0) then

71: p=&a

81: a=a+191: z=2*(*p)

101: print(z)

11: z=021: a=031: b=2

41: p=&b

52: for I=1 to N do

62: if (i%2==0) then

82: a=a+192: z=2*(*p)

1: z=02: a=03: b=24: p=&b5: for i = 1 to N do6: if ( i %2 == 0) then7: p=&a endif endfor8: a=a+19: z=2*(*p)10: print(z)

Dependence Graph Representation

5:for i=1 to N

6:if (i%2==0) then

7: p=&a

8: a=a+1

9: z=2*(*p)

10: print(z)

T F

1: z=0

2: a=0

3: b=2

4: p=&b

T

Input: N=2

11: z=0

21: a=0

31: b=2

41: p=&b

51: for i = 1 to N do

61: if ( i %2 == 0) then

81: a=a+1

91: z=2*(*p)

52: for i = 1 to N do

62: if ( i %2 == 0) then

71: p=&a

82: a=a+1

92: z=2*(*p)

101: print(z)

T1

2

3

4

5

6

7

8

9

10

11

12

13

14

<3,8><2,7>

<7,12>

<11,13>

<13,14>

<4,8>

<12,13>

<5,6><9,10>

<10,11>

<5,7><9,12>

<5,8><9,13>

F

Dependence Graph Representation

1

2

3

4 5

6

7 8

9

10

1

2

3

4

5

6

7

2

1

10

34679

35679

34689

35689

1

2

3

2

1

10

34679

1

2

3

34 5

67 8

9

Transform: Traces of Blocks

Infer: Local Dependence Labels

X =

Y= X

X =

Y= X

(10,10)

(20,20)

(30,30)

10,20,30

X =

Y= X

=Y

21

(20,21)

...

(...,20)

...

Transform: Local Dep. Labels

X =

*P =

Y= X

X =

*P =

Y= X

(10,10)

(20,20)

10,20

X =

*P =

Y= X

(20,20)

Transform: Local Dep. Labels

X =

*P =

Y= X

X =

*P =

Y= X

(10,10)

(20,20) X =

*P =

Y= X

10,20

X =

*P =

Y= X

2010

11,21

(10,11) (20,21)

=Y

11,21

=Y

(20,21) (10,11)

Group: Non-Local Dep. Edges

X =

Y =

= Y

= X

X =

Y =

X =

Y =

= Y

= X

X =

Y =

(10,21)

(10,21)

(20,11)

(20,11)

X =

Y =

= Y

= X

X =

Y =

(20,11)

(10,21)

11,21

10

20

Compacted WET Sizes

ProgramStatementsExecuted(Millions)

WET Size (MB) Before /

After Before After

300.twolf

256.bzip2

255.vortex

197.parser

181.mcf

164.gzip

130.li

126.gcc

099.go

Average

690

751

609

615

715

650

740

365

685

647

10,666

11,921

8,748

8,730

10,541

9,688

10,399

5,238

10,369

9,589

647

221

105

188

416

538

203

89

575

331

16.49

54.02

83.63

46.34

25.33

18.02

51.22

58.84

18.04

41.33

≈ 4 Bits / Instruction

[PLDI-04]vs.

[ICSE-03]

Slicing Times

Dep. Graph Generation Times

Offline post-processing after collecting address and control flow traces ≈ 35x of execution time

Online techniques [ICSM 2007] Information Flow: 9x to18x slowdown Basic block Opt.: 6x to10x slowdown Trace level Opt.: 5.5x to 7.5x slowdown Dual Core: ≈1.5x slowdown

Online Filtering techniques Forward slice of all inputs User-guided bypassing of functions

Reducing Online Overhead

Record non-deterministic events online• Less than 2x overhead• Deterministic replay of executions

Trace faulty executions off-line• Replay the execution• Switch on tracing• Collect and inspect traces

Trace analysis is still a problem• The traces correspond to huge executions• Off-line overhead of trace collection is still significant

Reducing Trace Sizes

Checkpointing Schemes Trace from the most recent checkpoint

• Checkpoints are of the order of minutes.• Better but the trace sizes are still very large.

Exploiting Program Characteristics Multithreaded and server-like [ISSTA-07, FSE-06]

• Examples : mysql, apache.• Each request spawns a new thread.• Do not trace irrelevant threads.

Beyond Tracing

Checkpoint: capture memory image.

Execute and Record (log) Events.

Upon Crash, Rollback to checkpoint. Reduce log and Replay execution using reduced log. Turn on tracing during replay.

xCheckpoint log

x TraceReducedlog

Applicable to Multithreaded Programs

[ISSTA-07]

An Example

A mysql bug • “ load …” command will crash the server if database is not

specified

sql/mysql_load.cc:int mysql_load (THD *thd,...){

…150 if( …151 … +strlen(thd->db) + 3 <152 FN_REFLEN) ...}

Without typing “use database_name”, thd->db is Null.

Example – Execution and log file

open path=/etc/my.cnf

…

Wait for connection

Create Thread 1

Wait for command

Create Thread 2

Wait for command

Recv “show databases”

Handle command

Recv “load data …”

Handle -- (server crashes)

Recv “use test; select * from b”

Handle command

Run mysql server

User 1 connects to the server

User 2 connects to the server

User 1: “show databases”

User 2: “use test”

“ select * from b”

User 1: “load data into table1”

Time

Blue – T0

Red – T1

Green – T2

Gray - Scheduler

Execution Replay using Reduced log

open path=/etc/my.cnf

…

Wait for connection

Create Thread 1

Recv “load data …”

Handle -- (server crashes)

Run mysql server

User 1 connects to the server

User 2 connects to the server

User 1: “show databases”

User 2: “show databases”

“ select * from b”

User 1: “load data into table1”

Time

Execution Reduction

Effects of Reduction Irrelevant Threads Replay-only vs. Replay & Trace

How? By identifying Inter-thread Dependences Event Dependences - found using the log File Dependences - found using the log Shared-Memory Dependences - found using replay

Space requirement reduced by 4x

Time requirement reduced by 2x

Naïve approach requires thread id of last writer of each address Space and time efficient detection

o Memory Regions: Non-shared vs sharedo Locality of References to Regions

Experimental Results

Original OptimizedProgram-bug

Trace SizesNum. of

dependences

Experimental Results

Orig. OPT.Program-bug

Execution Times

(seconds)

Logging

Experimental Results

Debugging System

SlicingModule

WETSlicesApplication

binary

ExecutionEngineValgrind

Input Output

Traces

Instrumentcode

CompressedTrace

Static BinaryAnalyzer

Diablo

Control Dependence

ReducedLog

RecordReplayJockey

Checkpoint+

log

Fault Avoidance

Large number of faults in server programs are caused by the environment.

• 56 % of faults in Apache server.

Types of Faults Handled Atomicity Violation Faults.

• Try alternate scheduling decisions.

Heap Buffer Overflow Faults.• Pad memory requests.

Bad User Request Faults.• Drop bad requests.

Avoidance Strategy Recover first time, Prevent later.

• Record the change that avoided the fault.

Experiments

Program Type of Bug Env. Change # of Trials

Time taken (secs.)

mysql-1 Atomicity Violn. Scheduler 1 130

mysql-2 Atomicity Violn. Scheduler 1 65

mysql-3 Atomicity Violn. Scheduler 1 65

mysql-4 Buffer Overflow. Mem. Padding 1 700

pine-1 Buffer Overflow. Mem. Padding 1 325

pine-2 Buffer Overflow. Mem. Padding 1 270

mutt-1 Bad User Req. Drop Req. 3 205

bc-1 Bad User Req. Drop Req. 3 290

bc-2 Bad User Req. Drop Req. 3 195

Summary

Dynamic SlicingOffline

Environment FaultsOnline

Program Execution

Fault

Fault Location

Fault Avoidance

ScalabilityTracing + Logging

Long-runningMulti-threaded

Dissertations

Xiangyu Zhang, Purdue University Fault Location Via Precise Dynamic

Slicing, 2006. SIGPLAN Outstanding Doctoral Dissertation Award

Sriraman Tallam, Google

Fault Location and Avoidance in Long-Running Multithreaded Programs, 2007.

Ongoing Work

Monitoring Parallel Applications On Multicores: Vijay Nagarajan [ISCA’09]

Debugging Parallel Applications

State-based Approach: Dennis Jeffrey [ISSTA’08a]

Race-detection in Parallel Applications:

Chen Tian & Vijay Nagarajan [ISSTA’08b]

Optimistic Parallelization for Multicores Speculation: Min Feng & Chen Tian [MICRO’08]

Breakdowns of Different Optimizations

Infer

Transform

Group

Others

Explicit

Reducing Trace Sizes

Checkpointing Schemes Trace from the most recent checkpoint

• Checkpoints are of the order of minutes.• Better but the trace sizes are still very large.

Exploiting Program Characteristics Multithreaded and server-like [ISSTA-07]

• Examples : mysql, apache.• Each request spawns a new thread.• Do not trace irrelevant threads.

Single-threaded and event processing [FSE-06]• Examples : pine, mutt, squid.• Events are independent.• Do not trace irrelevant events.

Example of a Fault

Input:• > ./mysqld –log-bin= binlog• > ./mysql -u root -D test -e 'delete from b' &• > ./mysql -u root -D test -e 'insert into b values (1)' &

Output Observation• > ./mysqlbinlog binlog

set timestamp = 1152573474 insert into b values (1); set timestamp = 1152573470 delete from b

File : mysql_delete.cc

mysql_delete(THD *thd, ...)

...

152 error=generate_table(thd, ...);

...

generate_table(THD *thd, ...)

...

81 pthread_mutex_lock(...);

... // Critical Section

105 pthread_mutex_unlock(...);

108 ... // Logging not locked

109 mysql_update_log.write(thd,...);

...

Log of an execution

(scheduling decision)

TEI 1 Recv “delete …”

Handle it but hasn’t logged it

TEI 2 Recv “insert”

Handle it (logged insert)

TEI 3 Logging delete

Red – T1

Green – T2

Gray - Scheduler

Example of a Fault

System for Fault Avoidance

Put the server program into a 3-phase system• Log the original execution• Replay execution and apply the environmental changes• Use environment patch to prevent the fault from occurring again

Logging – jockey system. Apply environment changes – valgrind system.

Logging PhaseOrig. Execution

EventLog

Fault AvoidancePhase

(Env. Changes)Env. Fault

Patch

Prevention-LoggingPhase

Fault Avoided

FAULT REPLAY & AVOID

NEWFAULT

OLDFAULT

RECORDPATCH

[COMPSAC-08]