A Specification Language and Test Planner for Software TestingAolat A. Adedeji1 Mary Lou Soffa1

1DEPARTMENT OF COMPUTER SCIENCE, UNIVERSITY OF VIRGINIA

Juliya L. Reed, Bruce R. Childers

[1] B. Childers, M. L. Soffa, J. Beaver et al., “SoftTest: Aframework for software testing of Java programs”,Eclipse Technology eXchange Workshop, 2003

[2] J. Misurda, J. Clause, J. Reed, B. R. Childers, and M. L.Soffa, “Jazz: A Tool for Demand-Driven Structural Testing”, International Conference on Compiler Construction, April 2005.

[3] J. Misurda, J. Clause, J. Reed, B. R. Childers and M. L. Soffa, “Demand-Driven Structural Testing with Dynamic Instrumentation”, ACM SIGSOFT International Conference on Software Engineering (ICSE), May 2005.

[4] J. Reed, J. Misurda, B. R. Childers and M. L. Soffa, “An Integrated Code Coverage System for Software Test and Analysis”, University of Pittsburgh, 2005.

One of the most important software development concerns in recent years has been producing reliable and robust software. Testing, a process by which software quality can be assured through the collection of information, can improve software reliability.

A number of techniques can be applied to test software. One class of techniques that is commonly used is Structural testing, which checks that a given coverage criterion (i.e. branch, def-use, or node testing) is satisfied. For example, def-use testing checks which pairs of variable definitions and uses are covered throughout the execution of a software program.

Software testing tools typically are inflexible and have high overhead. However, a new scalable and flexible framework for testing programs has been developed to improve reliable software production. This prototype tool, called Jazz, has major components as shown below:

Figure 1. Jazz Tool Framework

Jazz instruments a program along an execution path. This tool 1) Uses dynamic instrumentation on the binary code that can be inserted and removed on-the-fly, 2) Performs branch, node and def-use coverage testing over multiple code regions in a Java program and 3) Includes a mechanism for specifying a software test process.

ACKNOWLEDGMENTS

INTRODUCTION PROBLEM STATEMENT

Testing tools typically target only one type of structural test, and are often not flexible in mapping out a testing plan. They do not offer the user the fine granularity for selecting parts of code to be tested. Also, often the user can only specify one test process at a time.

For each type of structural testing, there is a testing “plan”. To create the testing plan a specification language that describes what tests to apply and under what condition must be in place. Also, a test planner that generates a test plan from a test specification has to exist.

Problem:Jazz currently lacks a specification language that defines what part of code is to be tested in addition to defining the type of testing one may choose to perform on a selected code segment. For generating the test plan from the specification language, Jazz also requires a test planner to determine the actions necessary to carry out tests.

REFERENCES

In our specification language, the user can control the granularity of the code area that needs to be tested. They can define the region to be an entire file or class, or it can be as specific as a particular range of lines in the source code.

Figure 3. Example test specification for file Counter.java

The test planner then generates a test plan for the example specification above.

Figure 4. Test Plan for the example specification

The test planner determines when to insert probes, where to instrument a test region, and what to do at a probe. It automatically generates the probe location table (PLT) to encode probes and their locations, determines global and local storage, and links payloads to probes. Also integration of different tests into a single plan can be achieved by the test planner.

APPROACH

The front-end of this framework tool consists of a test GUI, implemented as a plug-in for the Eclipse IDE. The back-end coverage analysis system consists of the test dynamic instrumenter, test analyzer, and IBM’s Jikes Java RVM.

Figure 2. System Architecture

Focusing on the front-end of this tool we are implementing a specification language from which a test plan can be automatically generated by a test planner and developing the test planner that generates a test plan for testing a program from a test specification.

public class Counter { private int mycount; private int inc;

public Counter () { mycount = 0; inc = 1; count(); }

public void count () { if (mycount > 100) { reverse(); } else { mycount += inc; } }

public void reverse () { System.out.println("done\n"); }}

DEFINITIONS { NAME: c_method, REGION_D, LOCATION: FILE Counter.java { CLASS Counter, METHOD count }} BODY { DO NODE_TEST ON REGION c_method UNTIL: 85% }

Front-End(Eclipse Plug-in)

Back-End(Coverage Analysis

Engine)

Program Source

Test Request

Graphical Reporting

Code-Under-Test

Test Plan

Report & Results

Test Planner