a framework for automated parallel random unit testing of sequential programs
DESCRIPTION
A Framework For Automated Parallel Random Unit Testing Of Sequential Programs. by Martin Brown A Master’s Thesis Presented to the Faculty of the Computer and Information Sciences Department. 1. Introduction. Background. 1.1 Background. Definition of parallel computing - PowerPoint PPT PresentationTRANSCRIPT
A Framework For Automated Parallel Random Unit Testing Of Sequential
Programs
by Martin Brown
A Master’s Thesis
Presented to the Faculty of the Computer and Information Sciences Department
1. Introduction
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
Background
Introduction
1.1 Background
Definition of parallel computing
Definition of random testing
Definition of parallel software testing
Background
1.1.1 Parallel Computing Parallel computing is the discipline of
employing multiple processors running all at once to
solve the same problem in less time
to solve a larger problem in the same time
Background
1.1.2 Random Testing Random testing is a black box testing
technique that uses random number generators to produce random data to the inputs of a program
Background
1.1.3 Parallel Software Testing
Parallel software testing is the testing of sequential software on multiple processors
2. Thesis Problem Statement The power of parallel computing needs to be
exploited in software testing.
Two proposed strategies
1. Data diversity
2. Technique diversity
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
Thesis Problem Statement
2.1 Data Diversity
Using a combination of data sources when generating random numbers
Thesis Problem Statement
2.2 Technique Diversity
Applying multiple testing techniques in parallel, e.g. boundary testing and random testing
2.3 Thesis Problem Statement, cont’d
Can parallelism help to improve the quality of testing (test effectiveness)?
YESA Framework for Automated Parallel Random Unit Testing of Sequential Programs
Thesis Problem Statement
2.4 Scope of this Thesis1. Develop a framework for automated parallel
random unit testing of sequential programs
2. Implement data diversity through random testing
3. Evaluate the effectiveness of data diversity
4. Technique diversity – future work
3. Research Contribution
Framework for parallel random unit testing
Demonstration of the effectiveness of parallel testing
Introduction of Parallel Java to the CIS Department
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
Research Contribution
3.1 Features of the Framework
1. An API for parallel programming in Java
2. A library for random number generation
3. A wizard that guides the tester in creating a test specification
3.1 Features of the Framework, cont’d
Research Contribution
4. The ability to save and load test specifications
5. The ability to store generated random test data.
4. Related Work
Random unit testing using JML
Random testing of OO applications over time
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
4.1 Random Unit Testing using JML
Related Work
Leavens et al.
JML is a behavioral interface specification language for Java that uses pre- and post-conditions and invariants
JML is often used as a partial test oracle to random testing
4.2 Random Testing of OO Applications
Related Work
Ciupa, et al
Confirms the existence of data diversity in random testing
the seed used for an random number generator (RNG) can make an important difference in the number of bugs found over time
different seeds find different bugs
5. Test Specification Wizard
guides the tester through the creation of a test specification
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
Test Specification Wizard
Test SpecificationSystem
Under Test
Test Specification
System Under TestClasses Under Test
Class A Class B Class C
Test Specification Wizard
Test SpecificationSystem
Under Test
Test Specification Wizard
Test SpecificationSystem
Under TestClass
Instance Variables
Test Specification
Class Instance Variables
each selected constructor is associated with an instance variable
a, b, … z
Test Specification Wizard
Test SpecificationSystem
Under TestClass
Instance Variables
Test Sequences
Test Specification
Test Sequenceinstantiation
method calls
assertion
1 or more
0 or more
1, at the end
Test Specification Wizard
Test SpecificationSystem
Under TestClass
Instance Variables
Test Sequences
Random Variables
Test Specification
Random Variablesr1
type: intrange: (0, 100)
r2type: doublerange: (0.0,
32.4)
p1type: int
p2type: int
p3type:
double
Test Specification Wizard
Test SpecificationSystem
Under TestClass
Instance Variables
Test Sequences
Random Variables
Test Specification Wizard
Test SpecificationSystem
Under TestClass
Instance Variables
Test Sequences
Random Variables
Assertions
Test Specification
Assertionspublic class MyClass {
public int myField = 0;} MyClass m = new MyClass();
m.myField == 0
Test Specification
Assertionspublic class MyClass {
public int myField = 0;
public int getMyInt(){
…}
} MyClass m = new MyClass();
a.getMyInt() >= 0
Test Specification
Assertions
a.myField == \old a.myField + 1
public class MyClass {
public int myField = 0;
public int getMyInt(){
…}
} MyClass m = new MyClass();
Test Specification Wizard
Test SpecificationSystem
Under TestClass
Instance Variables
Test Sequences
Random Variables
Assertions
Test Specification Wizard
Test SpecificationSystem
Under TestClass
Instance Variables
Test Sequences
Random Variables
AssertionsTest
Execution Limit
Test Execution Limit
5000 (5 seconds)
Test Specification
Test Execution Limit
100 tests
Test Specification
Test Specification Wizard
Test SpecificationSystem
Under TestClass
Instance Variables
Test Sequences
Random Variables
AssertionsTest
Execution Limit
Test Specification Wizard
Test SpecificationSystem
Under TestClass
Instance Variables
Test Sequences
Random Variables
AssertionsTest
Execution Limit
Random Number
Generators
Random Number Generators
package umontreal.iro.lecuyer.
rng1 32 4
5 76 8
9 1110 12
RNG Pool
Test Specification
Random Number Generators
package umontreal.iro.lecuyer.
rng1 32 4
5 76 8
9 1110 12
2x 3x
RNG Pool
Test Specification
Random Number Generatorsstream
1
Thread-1 RNG collection
stream2stream
1stream
2stream
3
stream1
Thread-2 RNG collection
stream2stream
1stream
2stream
3
stream1
Thread-n RNG collection
stream2stream
1stream
2stream
3
Test Specification
Random Number Generators
stream1
Thread-n RNG collection
stream2stream
1stream
2stream
3
Java.util.Random
nextInt(collection_size – 1)
Random Number Generators
stream1
Thread-n RNG collection
stream2stream
1stream
2stream
3
Java.util.Random
r1
nextInt(collection_size – 1)
nextInt(r_min, r_max)
Test Specification Wizard
Test SpecificationSystem
Under TestClass
Instance Variables
Test Sequences
Random Variables
AssertionsTest
Execution Limit
Random Number
GeneratorsNumber of
Threads
Test SpecificationSystem
Under TestClass
Instance Variables
Test Sequences
Random Variables
AssertionsTest
Execution Limit
Random Number
GeneratorsNumber of
Threads
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
What is done with each component?
6. The Test Driver
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
Executes each test sequence sequentially
Each sequence is executed multiple times in parallel
Test DriverParallel Java Library RNG Library
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
Test SpecificationSystem
Under TestClass
Instance Variables
Test Sequences
Random Variables
AssertionsTest
Execution Limit
Random Number
GeneratorsNumber of
Threads
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
Test Specification
Test Driver
Test Results
IN
OUT
?
Test Resultspass /
fail results
Total execution time
Generated test data
Thread execution times
Sequence
execution times
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
7. Design of the Framework
1. Compile source
files
2. Parse source
files
6. Execute test driver
5. Compile test driver
3. Read test specificatio
n4. Create test
driver
7. Store test specification
Store test data
Oracle
Parallel operation
Sequential operation
optional
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
8. Challenges in Developing the Framework
Need for thread-safe subject programs
Need for parallel RNG Java libraries
Mutation testing is even more compute-intensive
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
9. Evaluation
Operating System Windows 7 Home Premium
Processor Intel® Core™ i3 CPU M 330 @ 2.13 GHz
RAM 4.00GB
System type 64-bit
Cores/threads 2/4
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
9 Evaluation, cont’d
Each subject program was
tested 10 times using different test configurations
10 seconds for each execution
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
9 Evaluation, cont’d
Name Types
Triangle1 int, int, int
Triangle2 int, int, int
Tax double, boolean
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
9 Evaluation, cont’d
n-m homogeneous test configuration n threads and m streams of one RNG
n-m heterogeneous test configuration n threads and m streams of different RNGs
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
Thesis Problem Statement
9 Evaluation, cont’d
1. Coverage analysis to determine if the more computing power increases code coverage
2. Mutation analysis to measure the ability to detect bugs
Evaluation
9.1 Total Percent Coverage The total coverage percentage of a class tells
how well the class is covered
TPC = (BT + BF + SC + MC) / (2 * B + S + M)whereBT - branches that evaluated to "true" at least onceBF - branches that evaluated to "false" at least onceSC - statements coveredMC - methods enteredB - total number of branchesS - total number of statementsM – total number of methods
9.1.1 Homogeneous Test Configurations
Code Coverage Analysys
9.1.1 Homogeneous Test Configurations
Code Coverage Analysys
9.1.1 Homogeneous Test Configurations
Code Coverage Analysys
9.1.2 Heterogeneous Test Configurations
Code Coverage Analysys
9.1.2 Heterogeneous Test Configurations
Code Coverage Analysys
9.1.2 Heterogeneous Test Configurations
Code Coverage Analysys
Evaluation
9.2 Mutation Analysis
Measures the ability for test data to detect bugs
methods
Evaluation
9.2 Mutation Analysis
Name # of Mutants
Triangle1 67
Triangle2 297
TaxReturn 280
Mutation Analysis
9.2.1 Homogeneous Test Configurations TPC = (BT + BF + SC + MC) / (2 * B + S + M) where BT - branches that evaluated to "true" at least once BF - branches that evaluated to "false" at least once SC - statements covered MC - methods entered B - total number of branches S - total number of statements M – total number of methods
Mutation Analysis
9.2.1 Homogeneous Test Configurations TPC = (BT + BF + SC + MC) / (2 * B + S + M) where BT - branches that evaluated to "true" at least once BF - branches that evaluated to "false" at least once SC - statements covered MC - methods entered B - total number of branches S - total number of statements M – total number of methods
Mutation Analysis
9.2.1 Homogeneous Test Configurations TPC = (BT + BF + SC + MC) / (2 * B + S + M) where BT - branches that evaluated to "true" at least once BF - branches that evaluated to "false" at least once SC - statements covered MC - methods entered B - total number of branches S - total number of statements M – total number of methods
9.2.2 Heterogeneous Test Configurations
Mutation Analysis
TPC = (BT + BF + SC + MC) / (2 * B + S + M) where BT - branches that evaluated to "true" at least once BF - branches that evaluated to "false" at least once SC - statements covered MC - methods entered B - total number of branches S - total number of statements M – total number of methods
9.2.2 Heterogeneous Test Configurations
Mutation Analysis
TPC = (BT + BF + SC + MC) / (2 * B + S + M) where BT - branches that evaluated to "true" at least once BF - branches that evaluated to "false" at least once SC - statements covered MC - methods entered B - total number of branches S - total number of statements M – total number of methods
9.2.2 Heterogeneous Test Configurations
Mutation Analysis
TPC = (BT + BF + SC + MC) / (2 * B + S + M) where BT - branches that evaluated to "true" at least once BF - branches that evaluated to "false" at least once SC - statements covered MC - methods entered B - total number of branches S - total number of statements M – total number of methods
10. Conclusion
It is difficult to predict which test configuration will produce optimal results for a given test
There is some luck in achieving data diversity
It is best to use many RNGs (RNG streams) for a fixed amount of time
Combine some degree of parallelism with multiple RNGs
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
10. Conclusion, cont’d This project has met its goals
1. Design, and implement a framework for automated parallel random unit testing of sequential programs
2. Employ the framework to implement testing based on data diversity
3. Evaluate the effectiveness of data-diversity using accepted software testing experimental methodology: coverage analysis and fault detectionA Framework for Automated Parallel Random Unit Testing of Sequential Programs
10. Conclusion, cont’d
Can parallelism help to improve the quality of testing (test effectiveness)?
YES Parallelism enables diversity data diversity
increases test effectiveness
A Framework for Automated Parallel Random Unit Testing of Sequential Programs
10. Conclusion
A Framework for Automated Parallel Random Unit Testing of Sequential Programs