OCAT: Object Capture based Automated Testing
Hojun Jaygarl, Carl K. ChangIowa State University
Sunghun KimThe Hong Kong University of Science and Technology
Tao XieNorth Carolina State University
ISSTA 2010
Problem
2
Generating object inputs is hard
in Object-Oriented (OO) unit testing
Automated Test Generation (ATG)
3
Automatically generate test inputs for a unit.
Reduce manual efforts in OO unit testing.
Two Main Types of ATG Techniques
Direct object construction Directly assign values to object fields of the object
instance E.g., Korat [ISSTA'02]
Method sequence generation Generate method sequences that can produce an
object instance under construction E.g., Randoop [ICSE’07], Pex [TAP'08]
Results of the State of the Arts in ATG
5
Pex Pex automatically generates test inputs.
Dynamic Symbolic Execution
21% of branch coverage [Thummalapenta et al. FSE '09] for QuickGraph, a C# graph library.
Randoop Randoop automatically generates method sequences
Random but feedback directed
58% of branch coverage [Thummalapenta et al. FSE '09]
45% according to our evaluation for Apache Common Collections 3.2
Cause of not-covered branches
# of branches
Explanation
Insufficient object 135 (46.3%) unable to generate desirable object instances required to cover certain branches.
String comparison 61 (20.9%) Difficult to randomly find a desirable string to satisfy such constraints, since the input space of string type values is huge.
Container object access
39 (13.4%) Not easy to create a certain size of a container with necessary elements.
Array comparison 25 (8.6%) Not easy to create a certain size of an array with necessary elements.
Exception branches 18 (6.1%) These branches have a particularity of exception handling code that handles run-time errors
Environmental setting
9 (3.1%) Hard to get environment variables and file-system structure.
Non-deterministic branch
4 (1.3%) Hard to handle multi-threading and user interactions
Case of not-covered branches
6
Almost 90%
Run Randoop for three projects Apache Commons, XML Security, and JSAPWe randomly selected 10 source code files from each subject and investigated the causes of uncovered branches.
Not-Covered Branch Example
7
Checks the
validity of “doc”
Korat Require manual efforts for writing class invariants
and value domains.
Pex Pex uses built-in simple heuristics for generating
fixed sequences, which are often ineffective.
Randoop Random approach cannot generate relevant
sequences that produce desirable object instances.
Limitations of Current Approaches
8
Practical approach (high coverage)
Reflect real usage
Easier process
Idea
9
Object capture based Automated Testing
OCAT Overview
10
Capturing Process
Instrument the target program
Capture objects From program execution From system test execution
11
Instrumentation
12
Byte code instrumentation
Serialization
13
Get a type and a concrete state of captured
objects.
Prune redundant objects.
A concrete state representation for the state-
equality checking [Xie et al. ASE’04]
Serialized Object Example
14
A list that has bank accounts
Object Generation
15
Feed captured objects to an existing automated
method sequence generation technique.
Object Capturin
g
Object Capturin
g
Method Sequenc
esGenerati
on
Method Sequenc
esGenerati
on
Execution
ExecutionCaptured
objects
Sequences
Generated objects
Usage of Captured Objects
16
Captured objects are de-serialized and used as
test inputs.
Evolved Objects
Indirect usage
Captured instance of
A
foo(A) returns A
Captured instance of
A
bar(A)returns B
New instance
of B
New instance of
A
Method Sequence with Captured Objects
17
Evaluation - Setup
18
Evaluation - Captured Objects
19
Q1 How much can OCAT improve code coverage through captured object instances?
19.0% improved from 45.2%
28.5% and 17.3% improved from 29.6% and 54.1%
Evaluation- Captured Objects
20
Object Mutation
Approach – Object Mutation (MTT)
22
AbstractReferenceMap
Evaluation- Mutated Objects
23
Q2 How much can mutated object instances further improve code coverage?
Evaluation- Total
24
25.5% improved on average, with maximum 32.7%22.9
%
32.7%
20.9%
Why is It a Feasible Idea?
25
Reflect real usage
Potential for being desirable inputs in achieving new branch coverage
Capturing objects is easy
Discussion Object Capturing Process
Problem: OCAT’s coverage depends on captured objects
Capturing objects is an easy process (but we still need to capture “good-enough” objects).
Captured Objects and Software Evolution Problem: software evolves and objects are changing. Captured object instances may be obsolete and not
be valid anymore.
26
Discussion
Branches to Cover Problem: Still not-covered branches are more
than 20%. Cross-system object capturing
objects can be captured from system A and used for system B.
Static analysis currently we use a simple static analysis.
Iterative process two phases, object generation and object mutation,
can be iteratively applied
27
Threats to Validity
Software under test might not be representative Our three subjects may yield better or worse OCAT
coverage than that of other software projects.
Our object capturing relies on the existing tests OCAT test coverage reported in this paper
depends on the quality and quantity of existing tests.
28
Conclusions Problem
Hard to generate desirable object instances in OO unit testing.
OCAT approach Capture objects from program execution. Generate more objects with method sequence
generation. Mutate directly the captured objects to try to
cover those not-yet-covered branches.
29
Conclusions Results
OCAT helps Randoop to achieve high branch coverage averagely 68.5%
25.5% improved (with maximum 32.7%) from only 43.0% achieved by Randoop alone.
Future work Enhance static analysis part Apply captured objects to other state-of-the-art
techniques (e.g., parameterized unit testing, concolic testing, and model checking approaches).
Release the tool.30
31
Thank you!