korat: automated testing based on java predicates

Korat: Automated Testing Based on Java Predicates

Chandrasekhar Boyapati1, Sarfraz Khurshid2, and Darko Marinov3

1University of Michigan Ann Arbor2University of Texas at Austin

3University of Illinois at Urbana-Champaign

Examples of Structurally Complex Data

0

1

3

2

red-black tree

<library> <book year=2001> <title>T1</title> <author>A1</author> </book> <book year=2002> <title>T2</title> <author>A2</author> </book> <book year=2003> <title>T3</title> <author>A3</author> </book></library>

/library/book[@year<2003]/title

XML document

Testing Setup

• examples of code under test – abstract data type

• input/output: data structure

– XML processing program• input/output: XML document

codetest

generator

testing

oracle

0

1

3

2

0 3

2

0 3

2

0

3

pass

fail

inputs outputs

Manual Test Generation

• drawbacks of manual generation– labor-intensive and expensive– vulnerable to systematic bias that

eliminates certain kinds of inputs

codetest

generator• manual

testing

oracle

0

1

3

2

0 3

2

0 3

2

0

3

pass

fail

inputs outputs

Automated Test Generation

• challenges of automated test generation– describing test inputs– (efficient) test generation– checking output

code

testgenerator

• automated

testing

oracle

0

1

3

2

0 3

2

0 3

2

0

3

pass

fail

inputs outputs

Running Exampleclass BST { Node root; int size;

static class Node { Node left, right; int value; }

}

B0: 3

root

N0: 2

left right

N1: 1 N2: 3

void remove(int i) { … }…

Example Valid Inputs

• trees with exactly 3 nodes

right

right

N0: 1

N1: 2

N2: 3

B0: 3

root

right

left

N0: 1

N1: 3

N2: 2

B0: 3

root

left

right

N0: 3

N1: 1

N2: 2

B0: 3

root

left

left

N0: 3

N1: 2

N2: 1

B0: 3

root

left right

N0: 2

N1: 1 N2: 3

B0: 3

root

Example Invalid Inputs

• object graphs violating some validity property

left right

N0: 2

N1: 1 N2: 3

B0: 3

root

left right

N0: 2

N1: 1 N2: 3

B0: 2

root

left right

N0: 3

N1: 1 N2: 2

B0: 3

root

Validity Properties for Example• underlying graph is a tree

(no sharing between subtrees)

• correct number of nodes reachable from root

• node values orderedfor binary search

left right

N0: 2

N1: 1 N2: 3

B0: 3

root

Describing Validity Properties

• Korat uses a predicate written in standard implementation language (Java, C#…)– takes an input that can be valid or invalid– returns a boolean indicating validity

• advantages– familiar language– existing development tools– predicates often already present

• challenge: generate tests from predicates

Example Predicateboolean repOk(BST t) { return isTree(t) && hasCorrectSize(t) && isOrdered(t);}boolean isTree(BST t) { if (t.root == null) return true; // empty tree Set visited = new HashSet(); visited.add(t.root); List workList = new LinkedList(); workList.add(t.root); while (!workList.isEmpty()) { Node current = (Node)workList.removeFirst(); if (current.left != null) { if (!visited.add(current.left)) return false; // sharing workList.add(current.left); } if (current.right != null) { if (!visited.add(current.right)) return false; // sharing workList.add(current.right); } } return true; // no sharing}

Input Space

• all possible object graphs with a BST root(obeying type declarations)

rightleft

N0: 2

N1: 1 N2: 3

B0: 3

root

N0: 1

B0: 1

root

B0: 0

right

right

N0: 1

N1: 2

N2: 3

B0: 3

root

right

left

N0: 1

N1: 3

N2: 2

B0: 3

root

left

right

N0: 3

N1: 1

N2: 2

B0: 3

root

left

left

N0: 3

N1: 2

N2: 1

B0: 3

root

N0: 1

B0: 1

root

left

N0: 1

B0: 1

root

right

N0: 1

B0: 1

root

left right

left right

N0: 2

N1: 1 N2: 3

B0: 3

root

left right

N0: 2

N1: 1 N2: 3

B0: 2

root

left right

N0: 3

N1: 1 N2: 2

B0: 3

root

Bounded-Exhaustive Generation• finitization bounds input space

– number of objects– values of fields

• generate all valid inputs up to given bound– eliminates systematic bias– finds all errors detectable within bound

• avoid isomorphic inputs– reduces the number of inputs– preserves capability to find all errors

Example Finitization

• specifies number of objects for each class– 1 object for BST: { B0 }

– 3 objects for Node: { N0, N1, N2 }

• specifies set of values for each field– sets consist of objects and literals

– for root, left, right: { null, N0, N1, N2 }

– for value: { 1, 2, 3 }– for size: { 3 }

Example Input Space• 1 BST object, 3 Node objects: total 11 fields

B0 N0 N1 N2

root size left rightvalue right left rightvalueleft value

null

N0

N1

N2

null

N0

N1

N2

null

N0

N1

N2

1

2

3

null

N0

N1

N2

null

N0

N1

N2

1

2

3

null

N0

N1

N2

null

N0

N1

N2

1

2

3

4 * 1 * (4 * 4 * 3)3 > 218 inputs, only 5 valid

3

B0 N0 N1 N2


N0 N1 N1 null null null null3 2 31

Example Input

• each input is a valuation of fieldsB0 N0 N1 N2



left right

N0: 2

N1: 1 N2: 3

B0: 3

root

Generation Problem• given

– predicate– finitization

• generate– all nonisomorphic valid inputs within finitization

• simple “solution”– enumerate entire input space– run predicate on each input– generate input if predicate returns true

– infeasible for sparse input spaces (#valid << #total)

– must reason about behavior of predicate

field accesses:

[ ]

boolean repOk(BST t) { return isTree(t) && …;}boolean isTree(BST t) { if (t.root == null) return true; Set visited = new HashSet(); visited.add(t.root); List workList = new LinkedList(); workList.add(t.root); while (!workList.isEmpty()) { Node current = (Node)workList.removeFirst(); if (current.left != null) { if (!visited.add(current.left)) return false; workList.add(current.left); } if (current.right != null) { if (!visited.add(current.right)) return false; workList.add(current.right); } } return true;}

Example Execution

left right

N0: 2

N1: 1 N2: 3

B0: 3

root

[ B0.root ][ B0.root, N0.left ][ B0.root, N0.left, N0.right ]

Failed Execution

• failed after few accesses for a concrete input

• would fail for all inputs with partial valuation

B0 N0 N1 N2


N0 N1 N1 - - - -- - --

B0 N0 N1 N2



1 * 3 * 4 * 4 * 3 * 4 * 4 * 3 > 212

Key Idea

• observe execution of predicate• record field accesses• prune large chunks of input space for

each failed execution• use backtracking to efficiently

enumerate valid inputs

Search Step

• backtracking on [ B0.root, N0.left, N0.right ]

• produces next candidate input

B0 N0 N1 N2



B0 N0 N1 N2



Next Input

• execution returns true, generate this input B0 N0 N1 N2



left rightN0: 2

N1: 1 N2: 3

B0: 3

root

Korat• solver for executable predicates

• Korat systematically searches input space– encodes inputs as vectors of integers– creates candidate object-graph inputs– monitors execution of predicate– prunes search by backtracking on field

accesses– avoids isomorphic inputs

Koratall

nonisomorphicvalid inputs

repOk

finitization

• equivalent for all code and all properties• example for trees: permutation of nodes

• n! isomorphic trees => generation infeasible• removing isomorphic inputs from test suites

– significantly reduces the number of tests– does not reduce quality

left right

N1: 2

N2: 1 N0: 3

B0: 3

root

N0: 2

N1: 1 N2: 3

B0: 3

left right

root

Isomorphic Inputs

• inputs: rooted and labeled (object) graphs– edges have fields– nodes have object identities

• isomorphism with respect to object identitites

• definition: inputs I and I’ consisting of objects from a set O are isomorphic iff : O->O. o,o’ reachable in I. ffields(o). o.f = o’ in I => (o).f = (o’) in I’ and o.f = p in I => (o).f = p in I’

Isomorphism Definition

Example Nonisomorphic Trees

• different edges or primitives, not only nodes

right left

N0: 2

N1: 1 N2: 3

B0: 3

root

N0: 2

N1: 1 N2: 3

B0: 3

left right

root

left right

N0: 3

N1: 1 N2: 2

B0: 3

root

Nonisomorphic Generation

• simple “solution”– generate all inputs– filter out isomorphic inputs

• Korat does not require filtering– generate only one (candidate) input from

each isomorphism class– only the lexicographically smallest input– search increments some field values for >1

Correctness

• Korat’s input generation– sound

• no invalid input

– complete• at least one valid input from each isomorphism

class

– optimal• at most one (valid) input from each isomorphism

class

Example Specificationclass BST {

Node root; int size; static class Node { Node left, right; int value; }

//@ invariant repOk();

//@ precondition true; //@ postcondition !contains(i) && …; void remove(int i) { … }

boolean contains(int i) { … } …}

Testing Scenario

removeKoratand

pass

fail

right

N0

: 1 N

1

: 2

B0: 3

root

right

right

N0

: 1 N

1

: 2 N

2

: 3

B0: 3

root

, 3

invariantrepOk()

precondition

true

postcondition!contains(i) &&

…

and

finitization

linked data structures and array-based data structures

Korat Structure Generation

benchmarkBST

HeapArray

java.util.LinkedList

java.util.TreeMap

java.util.HashSet

IntentionalName


very large input spaces

benchmark size

inputspac

eBST 8

12253

292

HeapArray 68

220

229


812

291

2150

java.util.TreeMap

79

292

2130

java.util.HashSet

711

2119

2215

IntentionalName

5 250


pruning based on filed accesses very effective

benchmark size

inputspac

e

candidate

inputsBST 8

12253

292

5441812284830

HeapArray 68

220

229

645335231385


812

291

2150

54555034894

java.util.TreeMap

79

292

2130

25676350209400

java.util.HashSet

711

2119

2215

19320039075006

IntentionalName

5 250 1330628


correct number of nonisomorphic structures (Sloane’s)

benchmark size

inputspac

e

candidate

inputs

validinput

sBST 8

12253

292

5441812284830

1430208012

HeapArray 68

220

229

645335231385

13139100507

5


812

291

2150

54555034894

4140421359

7

java.util.TreeMap

79

292

2130

25676350209400

35122

java.util.HashSet

711

2119

2215

19320039075006

2386277387

IntentionalName

5 250 1330628 598358


800Mhz Pentium III Sun’s Java 2 SDK 1.3.1 JVM

benchmark size

inputspac

e

candidate

inputs

validinput

s

time[sec]

BST 812

253

292

5441812284830

1430208012

2234

HeapArray 68

220

229

645335231385

13139100507

5

243


812

291

2150

54555034894

4140421359

7

2690

java.util.TreeMap

79

292

2130

25676350209400

35122

92149

java.util.HashSet

711

2119

2215

19320039075006

2386277387

4927

IntentionalName

5 250 1330628 598358 63

Korat Method Testing

several operations for each data structure

benchmark bound

inputsgenerate

d

gen.[sec

]

test[sec

]BST 7 41300 9 1

HeapArray 7 1175620 7 18

LinkedList 7 58175 1 2

TreeMap 7 12754 3 1

HashSet 7 54844 3 2

IntentionalName

5 417878 87 135

SortedList 7 1047608 23 38

DisjSet 5 1246380 11 20

BinomialHeap 7 2577984 36 76

FibonacciHeap 5 941058 14 23

Conclusions

• Korat automates specification-based testing– uses method precondition to generate all

nonisomorphic test inputs• prunes search space using field accesses

– invokes the method on each input and uses method postcondition as a test oracle

• Korat prototype uses JML specifications• Korat efficiently generates complex data

structures including some from JCF

korat: automated testing based on java predicates

Documents