a framework for automated parallel random unit testing of sequential programs

A Framework For Automated Parallel Random Unit Testing Of Sequential

Programs

by Martin Brown

A Master’s Thesis

Presented to the Faculty of the Computer and Information Sciences Department

1. Introduction

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

Background

Introduction

1.1 Background

Definition of parallel computing

Definition of random testing

Definition of parallel software testing

Background

1.1.1 Parallel Computing Parallel computing is the discipline of

employing multiple processors running all at once to

solve the same problem in less time

to solve a larger problem in the same time

Background

1.1.2 Random Testing Random testing is a black box testing

technique that uses random number generators to produce random data to the inputs of a program

Background

1.1.3 Parallel Software Testing

Parallel software testing is the testing of sequential software on multiple processors

2. Thesis Problem Statement The power of parallel computing needs to be

exploited in software testing.

Two proposed strategies

1. Data diversity

2. Technique diversity

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

Thesis Problem Statement

2.1 Data Diversity

Using a combination of data sources when generating random numbers

Thesis Problem Statement

2.2 Technique Diversity

Applying multiple testing techniques in parallel, e.g. boundary testing and random testing

2.3 Thesis Problem Statement, cont’d

Can parallelism help to improve the quality of testing (test effectiveness)?

YESA Framework for Automated Parallel Random Unit Testing of Sequential Programs

Thesis Problem Statement

2.4 Scope of this Thesis1. Develop a framework for automated parallel

random unit testing of sequential programs

2. Implement data diversity through random testing

3. Evaluate the effectiveness of data diversity

4. Technique diversity – future work

3. Research Contribution

Framework for parallel random unit testing

Demonstration of the effectiveness of parallel testing

Introduction of Parallel Java to the CIS Department

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

Research Contribution

3.1 Features of the Framework

1. An API for parallel programming in Java

2. A library for random number generation

3. A wizard that guides the tester in creating a test specification

3.1 Features of the Framework, cont’d

Research Contribution

4. The ability to save and load test specifications

5. The ability to store generated random test data.

4. Related Work

Random unit testing using JML

Random testing of OO applications over time

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

4.1 Random Unit Testing using JML

Related Work

Leavens et al.

JML is a behavioral interface specification language for Java that uses pre- and post-conditions and invariants

JML is often used as a partial test oracle to random testing

4.2 Random Testing of OO Applications

Related Work

Ciupa, et al

Confirms the existence of data diversity in random testing

the seed used for an random number generator (RNG) can make an important difference in the number of bugs found over time

different seeds find different bugs

5. Test Specification Wizard

guides the tester through the creation of a test specification

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

Test Specification Wizard

Test SpecificationSystem

Under Test

Test Specification

System Under TestClasses Under Test

Class A Class B Class C

Test Specification Wizard

Test SpecificationSystem

Under Test

Test Specification Wizard

Test SpecificationSystem

Under TestClass

Instance Variables

Test Specification

Class Instance Variables

each selected constructor is associated with an instance variable

a, b, … z

Test Specification Wizard

Test SpecificationSystem

Under TestClass

Instance Variables

Test Sequences

Test Specification

Test Sequenceinstantiation

method calls

assertion

1 or more

0 or more

1, at the end

Test Specification Wizard

Test SpecificationSystem

Under TestClass

Instance Variables

Test Sequences

Random Variables

Test Specification

Random Variablesr1

type: intrange: (0, 100)

r2type: doublerange: (0.0,

32.4)

p1type: int

p2type: int

p3type:

double

Test Specification Wizard

Test SpecificationSystem

Under TestClass

Instance Variables

Test Sequences

Random Variables

Test Specification Wizard

Test SpecificationSystem

Under TestClass

Instance Variables

Test Sequences

Random Variables

Assertions

Test Specification

Assertionspublic class MyClass {

public int myField = 0;} MyClass m = new MyClass();

m.myField == 0

Test Specification

Assertionspublic class MyClass {

public int myField = 0;

public int getMyInt(){

…}

} MyClass m = new MyClass();

a.getMyInt() >= 0

Test Specification

Assertions

a.myField == \old a.myField + 1

public class MyClass {

public int myField = 0;

public int getMyInt(){

…}

} MyClass m = new MyClass();

Test Specification Wizard

Test SpecificationSystem

Under TestClass

Instance Variables

Test Sequences

Random Variables

Assertions

Test Specification Wizard

Test SpecificationSystem

Under TestClass

Instance Variables

Test Sequences

Random Variables

AssertionsTest

Execution Limit

Test Execution Limit

5000 (5 seconds)

Test Specification

Test Execution Limit

100 tests

Test Specification

Test Specification Wizard

Test SpecificationSystem

Under TestClass

Instance Variables

Test Sequences

Random Variables

AssertionsTest

Execution Limit

Test Specification Wizard

Test SpecificationSystem

Under TestClass

Instance Variables

Test Sequences

Random Variables

AssertionsTest

Execution Limit

Random Number

Generators

Random Number Generators

package umontreal.iro.lecuyer.

rng1 32 4

5 76 8

9 1110 12

RNG Pool

Test Specification

Random Number Generators

package umontreal.iro.lecuyer.

rng1 32 4

5 76 8

9 1110 12

2x 3x

RNG Pool

Test Specification

Random Number Generatorsstream

1

Thread-1 RNG collection

stream2stream

1stream

2stream

3

stream1

Thread-2 RNG collection

stream2stream

1stream

2stream

3

stream1

Thread-n RNG collection

stream2stream

1stream

2stream

3

Test Specification

Random Number Generators

stream1

Thread-n RNG collection

stream2stream

1stream

2stream

3

Java.util.Random

nextInt(collection_size – 1)

Random Number Generators

stream1

Thread-n RNG collection

stream2stream

1stream

2stream

3

Java.util.Random

r1

nextInt(collection_size – 1)

nextInt(r_min, r_max)

Test Specification Wizard

Test SpecificationSystem

Under TestClass

Instance Variables

Test Sequences

Random Variables

AssertionsTest

Execution Limit

Random Number

GeneratorsNumber of

Threads

Test SpecificationSystem

Under TestClass

Instance Variables

Test Sequences

Random Variables

AssertionsTest

Execution Limit

Random Number

GeneratorsNumber of

Threads

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

What is done with each component?

6. The Test Driver

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

Executes each test sequence sequentially

Each sequence is executed multiple times in parallel

Test DriverParallel Java Library RNG Library

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

Test SpecificationSystem

Under TestClass

Instance Variables

Test Sequences

Random Variables

AssertionsTest

Execution Limit

Random Number

GeneratorsNumber of

Threads

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

Test Specification

Test Driver

Test Results

IN

OUT

?

Test Resultspass /

fail results

Total execution time

Generated test data

Thread execution times

Sequence

execution times

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

7. Design of the Framework

1. Compile source

files

2. Parse source

files

6. Execute test driver

5. Compile test driver

3. Read test specificatio

n4. Create test

driver

7. Store test specification

Store test data

Oracle

Parallel operation

Sequential operation

optional

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

8. Challenges in Developing the Framework

Need for thread-safe subject programs

Need for parallel RNG Java libraries

Mutation testing is even more compute-intensive

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

9. Evaluation

Operating System Windows 7 Home Premium

Processor Intel® Core™ i3 CPU M 330 @ 2.13 GHz

RAM 4.00GB

System type 64-bit

Cores/threads 2/4

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

9 Evaluation, cont’d

Each subject program was

tested 10 times using different test configurations

10 seconds for each execution

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

9 Evaluation, cont’d

Name Types

Triangle1 int, int, int

Triangle2 int, int, int

Tax double, boolean

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

9 Evaluation, cont’d

n-m homogeneous test configuration n threads and m streams of one RNG

n-m heterogeneous test configuration n threads and m streams of different RNGs

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

Thesis Problem Statement

9 Evaluation, cont’d

1. Coverage analysis to determine if the more computing power increases code coverage

2. Mutation analysis to measure the ability to detect bugs

Evaluation

9.1 Total Percent Coverage The total coverage percentage of a class tells

how well the class is covered

TPC = (BT + BF + SC + MC) / (2 * B + S + M)whereBT - branches that evaluated to "true" at least onceBF - branches that evaluated to "false" at least onceSC - statements coveredMC - methods enteredB - total number of branchesS - total number of statementsM – total number of methods

9.1.1 Homogeneous Test Configurations

Code Coverage Analysys

9.1.1 Homogeneous Test Configurations

Code Coverage Analysys

9.1.1 Homogeneous Test Configurations

Code Coverage Analysys

9.1.2 Heterogeneous Test Configurations

Code Coverage Analysys

9.1.2 Heterogeneous Test Configurations

Code Coverage Analysys

9.1.2 Heterogeneous Test Configurations

Code Coverage Analysys

Evaluation

9.2 Mutation Analysis

Measures the ability for test data to detect bugs

methods

Evaluation

9.2 Mutation Analysis

Name # of Mutants

Triangle1 67

Triangle2 297

TaxReturn 280

Mutation Analysis

9.2.1 Homogeneous Test Configurations TPC = (BT + BF + SC + MC) / (2 * B + S + M) where BT - branches that evaluated to "true" at least once BF - branches that evaluated to "false" at least once SC - statements covered MC - methods entered B - total number of branches S - total number of statements M – total number of methods

Mutation Analysis

9.2.1 Homogeneous Test Configurations TPC = (BT + BF + SC + MC) / (2 * B + S + M) where BT - branches that evaluated to "true" at least once BF - branches that evaluated to "false" at least once SC - statements covered MC - methods entered B - total number of branches S - total number of statements M – total number of methods

Mutation Analysis

9.2.1 Homogeneous Test Configurations TPC = (BT + BF + SC + MC) / (2 * B + S + M) where BT - branches that evaluated to "true" at least once BF - branches that evaluated to "false" at least once SC - statements covered MC - methods entered B - total number of branches S - total number of statements M – total number of methods

9.2.2 Heterogeneous Test Configurations

Mutation Analysis

TPC = (BT + BF + SC + MC) / (2 * B + S + M) where BT - branches that evaluated to "true" at least once BF - branches that evaluated to "false" at least once SC - statements covered MC - methods entered B - total number of branches S - total number of statements M – total number of methods

9.2.2 Heterogeneous Test Configurations

Mutation Analysis

TPC = (BT + BF + SC + MC) / (2 * B + S + M) where BT - branches that evaluated to "true" at least once BF - branches that evaluated to "false" at least once SC - statements covered MC - methods entered B - total number of branches S - total number of statements M – total number of methods

9.2.2 Heterogeneous Test Configurations

Mutation Analysis

TPC = (BT + BF + SC + MC) / (2 * B + S + M) where BT - branches that evaluated to "true" at least once BF - branches that evaluated to "false" at least once SC - statements covered MC - methods entered B - total number of branches S - total number of statements M – total number of methods

10. Conclusion

It is difficult to predict which test configuration will produce optimal results for a given test

There is some luck in achieving data diversity

It is best to use many RNGs (RNG streams) for a fixed amount of time

Combine some degree of parallelism with multiple RNGs

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

10. Conclusion, cont’d This project has met its goals

1. Design, and implement a framework for automated parallel random unit testing of sequential programs

2. Employ the framework to implement testing based on data diversity

3. Evaluate the effectiveness of data-diversity using accepted software testing experimental methodology: coverage analysis and fault detectionA Framework for Automated Parallel Random Unit Testing of Sequential Programs

10. Conclusion, cont’d

Can parallelism help to improve the quality of testing (test effectiveness)?

YES Parallelism enables diversity data diversity

increases test effectiveness

A Framework for Automated Parallel Random Unit Testing of Sequential Programs

10. Conclusion

A Framework for Automated Parallel Random Unit Testing of Sequential Programs