austin: a tool for search based software testing and its evaluation on deployed automotive systems...

AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTINGAnd its Evaluation on Deployed Automotive Systems

Kiran Lakhotia, Mark Harman and Hamilton Gross in

Publication Trend in SBSE

Spread of Activities in SBSE

Yet Another Search Based Testing Tool?

IGUANA - for C (Phil

McMinn)

eTOC – for JAVA (Paolo

Tonella)

Others?

Yet Another Search Based Testing Tool?

IGUANA - for C (Phil McMinn)

eTOC – for JAVA

(Paolo Tonella)

Others?

AUSTIN

Outline

AUSTIN •Introduction to Austin

ETF •Overview of ETF

EmpiricalStudy

•Hypotheses, Test Subjects•Results and Analysis

Conclusion •ETF or AUSTIN?

AUgmented Search based TestINg

CIL

AST, CFG

instrument

gcc -o sut.exe

runAUSTIN

gcc -P -Ec sourcefiles

preprocessed

c sourcefiles

instrumented

c source

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 87, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 88, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 86, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 84, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 80, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 80, key2 = 33, p = 0, q = 0

Fitness Function:approach_level + norm(branch_distance)

Search Method:Alternating Variable Method

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 33, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 33, key2 = 33, p = 0, q = 0

Symbolic Path Condition:key1 == key2 && p == 0

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 33, key2 = 33, p = 0, q = 0

Symbolic Path Condition:key1 == key2 && p == 0

Simplified Path Condition:p == 0

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 33, key2 = 33, p = 0, q = 0

Symbolic Path Condition:key1 == key2 && p == 0

Simplified Path Condition:p == 0

Solve: p ≠ 0

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 33, key2 = 33, p = 0, q = 0

Symbolic Path Condition:key1 == key2 && p == 0

Simplified Path Condition:p == 0

Solve: p ≠ 0

p 0

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0

Symbolic Path Condition:key1 == key2 && p ≠ 0 && p ≠ q

Simplified Path Condition:p ≠ 0 && p ≠ q

Solve: p ≠ 0 && p == qp 0

q

p,q

0

AUgmented Search based TestINg

void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}

Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0x...

Evolutionary Testing Framework (ETF)

rZSlJ3hb

ETF was developed as part of the EvoTest project Integrated into the Eclipse IDE Supports white & black box testing Uses Evolutionary Algorithms (through the

GUIDE, EO evolutionary library interface) to generate test data

ETF

Selection

Recombinatio

n

Mutation

ReinsertionGUIDE + EO

Send Fitness Values

Send Individuals

ETF cont.rZSlJ3hb

Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009

ETF cont.rZSlJ3hb

Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009

Fitness Function and Algorithm Both ETF and AUSTIN use the standard

fitness function for branch coverage in Search Based Testing apporach_level + norm(branch_distance)

normalization function: 1 - 1.001-branch_distance

ETF uses a Genetic Algorithm AUSTIN uses the Alternating Variable

Method

Pointers in ETF and AUSTIN

AUSTIN Construct an equivalence of memory locations based on =

and ≠ operators in symbolic path condition Initialize (pointer) input variables based on the graph

Assign new memory location via malloc or use existing memory location

On-demand pointer handling ETF

Generate an index and auxiliary variable for each pointer input

Assign all input variables of the same type (including globals) to a pool Optimize indices and values of auxiliary variables along with the

inputs. Indices are used to select variables from pools and assign their address to the pointer input

Pointers in ETF

pool of input parameters and auxiliary variables

pCell

globDbl

val_p, val_q,globX

struct cell{ int value; struct cell* next;};int globX;double globDbl;struct cell* pCell;

void testme(int* p, int* q){ ...}

p = &globX;q = &val_p;

Hypothesis

Null Hypotheses Alternate Hypotheses

AUSTIN is as effective as the ETF in achieving branch coverage.

AUSTIN is more effective than the ETF in achieving branch coverage.

AUSTIN is equally as efficient as the ETF in achieving branch coverage of a function.

AUSTIN is more efficient than the ETF in achieving branch coverage of a function.

Empirical Study

3 case studies (comprising 8 functions) Embedded Software Modules from

Automotive IndustryCase Study

Description LOC - Tested

Total Branches

Nr. Inputs

B Adaptive Headlight Control Software

1,236 598 132

C Door-Lock Control Software

383 315 71

D Electric Windows Control Software

339 248 31

Coverage ETF vs AUSTIN

Efficiency ETF vs AUSTIN

Coverage Random vs AUSTIN

Efficiency Random vs AUSTIN

ETF or AUSTIN?

Coverage is about the same AUSTIN is overall a little bit more

efficient than the ETF in terms of fitness evaluations. In terms of wall clock time the difference is greater (due to the overhead of the ETF-GUIDE setup)

AUSTIN does not construct random memory graphs

ETF is not able generate inputs such as

struct cell{ int mem;};void testme(int* p, struct cell* c) { if( p == &c->mem ) //target}

Summary

Summary

http://www.cs.ucl.ac.uk/staff/K.Lakhotia/software/austin.htm