Automatic test case generation for programs that are coded against interfaces and

annotations or use native code

Mainul IslamSupervisor: Dr. Christoph Csallner

December 12th, 2012

The University of Texas at Arlington

PhD Research Proposal

2

Outline

• Problem Description• Motivation• Limitation of Current Approaches (with example)• Thesis Statement• Background• Solution Approach• Experimental Results• Plan of Action

3

Problem Description

• Current state of the art tools are not very good at generating test cases when the code under test:

– requires additional pieces of code that are not yet part of the program.

– uses multiple interfaces, annotations or reflection.– imposes complex (type) constraints.– uses native code.

4

Motivation

• Automatic test case generation is important• At the initial stage of any development the

implementation of some code may not be available• Existing techniques (such as Pex and Moles*) are

not very good at generating test cases when the code under test uses interfaces/annotations, multiple inheritance and native code.

• Significant number of Java programs use native code

* http://research.microsoft.com/en-us/projects/pex/

5

Limitation of Current Techniques: Motivating Example 1

public @interface A { /* … */ }public interface I { public int m1(); /* … */ }public interface J { public int m2(); /* … */ }

public class C { public static void foo(I i) { int x = i.m1(); if ( i instanceof J ) { J j = (J) i; int y = j.m2(); } if ( i.getClass().isAnnotationPresent(A.class) ) { // .. } }}

To reach this block of code ‘i’ must be an

instance of I, as well as an instance of J

To reach this block of code ‘i’ must be an

instance of I, as well as annotated with A

6

Motivating Example 2 (Dependency on Native Code)

public class C { public native boolean isDivisible(int x, int y); public static void NativeTest(int a, int b) { boolean divisible = new C().isDivisible(a, b); if (divisible) { // … } }}

bool isDivisible(int x, int y) { if ( y*(x/y) == x ) return true; return false;}

Java code C++ code

To reach this block of code ‘a’ must be divisible by ‘b’

7

Motivation: Patterns in Real World Program

8

Pattern 1

R m(…, T t, …) { // … if ( … (t instanceof X) ) // …}

• ‘T’ an ‘X’ are non-compatible• ‘m’ is user-defined• At least one of {T, X} is an

interface• None of {T, X} is final• At least one of {T, X} is a user-type

9

Thesis Statement

• We can generate automatic test inputs and systematically increase code coverage compared to existing techniques specially when the code under test:– requires additional pieces of code that are not yet part of

the program.– uses multiple interfaces, annotations or reflection.– imposes complex (type) constraints.– uses native code.

10

Background: Symbolic Execution

• Systematically explore all feasible execution paths

• Initialize the input with symbolic values and execute the program over symbolic values

• At conditional statements check if either of the branches can be taken

• For each path get an accumulated path condition

C` = C S⋀ C` = C ⋀ ⌝S

S

true false

C

If (S) then … else …

Dynamic Symbolic Execution by Exampletaken with permission from Nikolai Tillmann (Microsoft Research)

11

public static void TestMe(int a[]) { if (a == null) return; if (a.length > 0) if (a[0] == 123) throw new Exception(“Error”);}

a == null

a[0] == 123

a.length > 0

F T

F T

F T

Constraints to Solve Input (a) Observed Constraints

null a == null

a != null {} a != null && !(a.length > 0)

a != null && a.length > 0 {0}

a != null && a.length > 0 && a[0] != 123

a != null && a.length > 0&& a[0] == 123

{123} a != null && a.length > 0 && a[0] == 123

Solve Execute

Choose Next Path

Done: No Path Left

12

• Java defines a binary sub-type relation• If type B implements/extends type A then,– A is a direct super-type of B– B is a direct sub-type of A

Reflexive: A is also a subtype of itselfTransitive: if B is a subtype of A and C is a

subtype of B then C is also a subtype of A

Sub-/Supertype relation in Java

13

• A class has one direct class super type and arbitrarily many interface super types.

• Exceptions:type object – has no direct super typetype null – has no direct sub type

Sub-/Supertype relation in Java

14

• Introduce Dynamic Symbolic Mock Classes

Technique implemented on: Dsc1 – Dynamic Symbolic Execution Engine for Java Z32 – SMT solver (from Microsoft Research)

Our Solution Approach

1 - http://ranger.uta.edu/~csallner/dsc/index.html2 - http://z3.codeplex.com/

15

Solution WorkflowDefault Input

values (0, null, …)

New Test Cases (and corresponding

mock classes )

Invoke DSE on given input values and collect path constraints

Map the constraint solver model to new test cases (and mock classes)

Invert one of the collected path constraints

(Map each reference type to a constraint literal and encode their properties to build

a constraint system)

More Paths

?

Add mock classes and map each of them to a constraint literal

Encode properties (e.g., subtype relation) of mock

classes in the constraint system

Stop

Constraint System

Satisfiable?

No

No

Yes

Yes

Constraint System

Satisfiable?

Yes

No

16

Subtype Constraints

public @interface A { /* … */ }public interface I { public int m1(); /* … */ }public interface J { public int m2(); /* … */ }

public class C { public static void foo(I i) { int x = i.m1(); if ( i instanceof J ) { J j = (J) i; int y = j.m2(); } // .. } }

A desired solution with new Type: M, to reach the code block

Object

I J C

null

Initial Types in the system

A

Annotation

M

Constraints:type(i) subtypeof Itype(i) != null typetype(i) subtypeof J

17

Subtype Constraints

public @interface A { /* … */ }public interface I { public int m1(); /* … */ }public interface J { public int m2(); /* … */ }

public class C { public static void foo(I i) { int x = i.m1(); // .. if ( i.getClass(). isAnnotationPresent(A.class) ) { // .. } }

A desired solution with new Type: M1, to reach the code block

Object

I J C

null

Initial Types in the system

A

Annotation

M1

Constraints:type(i) subtypeof I

type(i) subtypeof Atype(i) != null type

18

Subtype Relation Matrix

public @interface A { /* … */ }public interface I { public int m1(); /* … */ }public interface J { public int m2(); /* … */ }

public class C { public static void foo(I i) { int x = i.m1(); if ( i instanceof J ) { J j = (J) i; int y = j.m2(); } // .. } }

null Object An I J C A M

0 null x x x x x x x x

1 Object x

2 An x x

3 I x x

4 J x x

5 C x x

6 A x x x

7 M x mAn mI mJ mC mA x

Solution: mI = true mJ = true mC = false mA = mAn = false

null Object An I J C A

0 null x x x x x x x

1 Object x

2 An x x

3 I x x

4 J x x

5 C x x

6 A x x x

null Object An

0 null x x x

1 Object x

2 An x x

19

Experimental Results (1)

• Experiments are done on simplified version of real world code

• C# codes are translated manually

20

Experimental Results (2)

• Experiments are done on original code• Randoop has several side effects

21

Workflow: to handle Native CodeDefault Input

values (0, null, …)

New Test Cases

Invoke DSE on given input values and collect path constraints

Map the constraint solver model to new test cases

Invert one of the collected path constraints and build constraint system

More Paths

?

Native Code

Invoked?

Collect the constraints from the native code for current input

values

Convert the native code constraints to Java constraints

Stop

Constraint System

Satisfiable?

No

No

No

YesYes

Yes

22

Technique to handle Native Code

In each iteration of the dynamic symbolic execution of a Java program:• Check if a native code call is invoked. If yes, start executing

the native code on current input values• Use any existing tool (such as Klee* for C++ code) to collect

the path constraints of the native code • Convert the constraints collected from the native code and

merge them with the constraints previously collected from the Java program

• Solve the whole constraint system using a Constraint solver and generate test case.

• http://klee.llvm.org/

23

Initial Experiments

In JDK 1.6:• Total # types: 23799• Total # of types (have at least one native

method): 381• Total # of native methods: 2019

• http://klee.llvm.org/

24

Plan of Actions

Time Action Plan (to do)Spring 2013 Implement and evaluate the

technique to handle native code.

Submit a short paper

Summer 2013 Internship (currently interviewing)

Join back to UTA in Fall 2013

Fall 2013 Detail experiments of the technique to handle native

code against the existing techniques.

Submit the experiment results to a journal

paper

Spring 2014 Write up and submit dissertation.

Ph.D. defense

25

List of publications

1. Mainul Islam and Christoph Csallner. Generating Test Cases for Programs that are Coded Against Interfaces and Annotations (submitted)

2. Mainul Islam and Christoph Csallner. Dsc+Mock: A test case + mock class generator in support of coding against interfaces (In Proc. 8th International Workshop on Dynamic Analysis (WODA), co-located with International Symposium on Software Testing and Analysis (ISSTA), 2010)

26

Thank You!

27

Backup Slides

28

Pattern 2

class P {T t, …}

class Q { M m(…, P p, …) { // … if ( … (p.t instanceof X) ) // …}

29

Pattern 3

M m(…) { // … if ( … (..).foo() instanceof X ) // …}

30

Related work

Mock classes inferred from Programmer-Written specification:

• EasyMock, jMock, Mockito• NMock• Google Mock• SynthiaMock

31

Related work: example with EasyMock

public interface K extends I, J { /* … */ }public class Test { public void testFooIsInstanceOfJ() { K mock = createMock(K.class); expect(mock.m1()).andReturn(0); expect(mock.m2()).andReturn(0); replay(mock); C.foo(mock); verify(mock); }}

32

0.5

u1

0.2

u2

0.6

u3

0.4

u4

0.8

u5

0.3 0.3

0.4

0.4

0.2

0.2

0.9

0.9

0.5

0.5

Visual similarity

Geo similarity