Combinatorial Testing Using Covering Arrays

- Going Beyond Pairwise Testing

Raghu Kacker, NIST

Yu Lei, UTA

5/15/06

Combinatorial Testing Using Covering Arrays 2

Outline

Introduction

The IPO Strategy

FireEye: An N-Way Testing Tool

Related Work

Conclusion

Combinatorial Testing Using Covering Arrays 3

Why Software Testing?

Modern society is increasingly dependent on the quality of software systems.

A NIST study reports that software failure costs the US economy billions of dollars every year.

Testing (or dynamic analysis) is the most widely used approach to ensuring software quality

Other approaches like static analysis and formal verification are more difficult to apply and do not seem to scale

Combinatorial Testing Using Covering Arrays 4

Testing Process

The testing process consists of three stages:

Test Generation: Generate test data For model-based testing, a model (or abstraction) of

the system has to be built.

Test Execution: Test setup and the actual test runs

Test Evaluation: Check if the output is in line with expectations

Combinatorial Testing Using Covering Arrays 5

All About Trade-Off

Testing is labor intensive and can be very costly

estimated to often consume more than 50% of the development cost

Exhaustive testing is often impractical due to resource constraints

From a certain perspective, testing is basically about making a good trade-off between test effort and quality assurance.

Combinatorial Testing Using Covering Arrays 6

Combinatorial Testing

Generates tests from an input parameter model by combining the values of the parameters.

Requires lightweight specification and no knowledge about the implementation structure

Can be virtually applied to any software and at different levels of abstraction

Can be implemented as a push-button feature

Usually performed to achieve t-way coverage, i.e. guarantees to cover every t-way interaction

Motivation: Not every parameter contributes to every fault

Can dramatically reduce the number of tests while still preserving important fault detection capabilities.

Combinatorial Testing Using Covering Arrays 7

The ASTUCA Project

Develop new methods and tools for efficient t-way testing for up to 6-way testing

Create new algorithms for efficient test set construction

Provide adequate support for parameter relations and constraints

Explore integration with other software testing tools Conduct empirical evaluation of t-way testing in an

industrial setting

A collaborative effort among the following institutions:

The US National Institute of Standards and Technology George Mason University The University of Texas at Arlington

Combinatorial Testing Using Covering Arrays 8

State-of-the-Art

Existing work has mainly focused on pairwise testing

Many failures are caused by the interaction involving more than two parameters

For certain software, pairwise testing discovers a relatively low percentage of faults

• e.g., For the RAX in NASA Deep Space 1 mission, pairwise testing only discovers 54 percent of interface faults, and 47 percent of engine faults.

Increased coverage leads to a higher level of assurance

Many applications, e.g., security protocols, have strict requirements on test coverage

Combinatorial Testing Using Covering Arrays 9

Technical Challenges

The computational complexity for t-way testing grows rapidly as the value of t increases

New algorithms must strike a balance between the time and space requirements and the optimality of the resulting test sets

The number of tests also grows rapidly as the value of t increases

Impractical to manually execute and inspect the results of a large number of test runs

Test generation, test execution, and test evaluation must integrate together to enable test automation

Combinatorial Testing Using Covering Arrays 10

Terminology

N-Way Test set -> N-Way Covering array

Tests -> Rows

Parameters -> Factors or Columns

Values -> Levels

Combinatorial Testing Using Covering Arrays 11

Outline

Introduction

The IPO Strategy

FireEye: An N-Way Testing Tool

Related Work

Conclusion

Combinatorial Testing Using Covering Arrays 12

The Framework (1)

Builds a t-way test set in an incremental manner

A t-way test set is first constructed for the first t parameters,

Then, the test set is extended to generate a t-way test set for the first t + 1 parameters

The test set is repeatedly extended for each additional parameter.

Two steps involved in each extension for a new parameter:

Horizontal growth: extends each existing test by adding one value of the new parameter

Vertical growth: adds new tests, if necessary

Combinatorial Testing Using Covering Arrays 13

The Framework (2)

Strategy In-Parameter-Orderbegin /* for the first t parameters p1, p2 , …, pt*/ T := {(v1, v2, …, vt) | v1, v2, …, vt are values of p1, p2, …, Pk, respectively} if n = t then stop; /* for the remaining parameters */ for parameter pi, i = t + 1, …, n do begin /* horizontal growth */ for each test (v1, v2, …, vi-1) in T do replace it with (v1, v2, …, vi-1, vi), where vi is a value of pi

/* vertical growth */ while T does not cover all the interactions between pi and each of p1, p2, …, pi-1 do add a new test for p1, p2, …, pi to T; endend

Combinatorial Testing Using Covering Arrays 14

Example (1)

Consider a system with the following parameters and values:

parameter A has values A1 and A2

parameter B has values B1 and B2, and

parameter C has values C1, C2, and C3

Combinatorial Testing Using Covering Arrays 15

Example (2)

A BA1 B1A1 B2A2 B1A2 B2

A B CA1 B1 C1A1 B2 C2A2 B1 C3A2 B2 C1

A B CA1 B1 C1A1 B2 C2A2 B1 C3A2 B2 C1A2 B1 C2A1 B2 C3

Horizontal Growth Vertical Growth

Combinatorial Testing Using Covering Arrays 16

Comparison to AETG (1)

A commercial tool developed by Telcordia, and protected by a US patent

Starts with an empty set and adds one (complete) test at a time

Each test is locally optimized to cover the most number of missing pairs:

Generate a random order of the parameters Use a greedy algorithm to construct a test that

covers the most uncovered pairs Repeat the above two steps for a given number of

times (suggested 50), and select the best one

Combinatorial Testing Using Covering Arrays 17

Comparison to AETG (2)

A B CA1 B1 C1

A B CA1 B1 C1A1 B2 C2

A B CA1 B1 C1A1 B2 C2A2 B1 C3A2 B2 C1A2 B1 C2A1 B2 C3

Adds the 1st test Adds the 2nd test Adds the last test

A B C

Combinatorial Testing Using Covering Arrays 18

IPO vs AETG

IPO is deterministic, whereas AETG is inherently non-deterministic

IPO has a lower order of complexity, both in terms of time and space, than AETG

IPO constructs a test set one parameter at a time and in a more incremental nature.

The results generated by IPO are still competitive to those generated by AETG.

IPO is more flexible than AETG IPO can take a test set for a subsystem (i.e., for a

subset of parameters) and then extend it to a complete set for the entire system

Combinatorial Testing Using Covering Arrays 19

Outline

Introduction

The IPO Strategy

FireEye: A Prototype Tool

Related Work

Conclusion

Combinatorial Testing Using Covering Arrays 20

Major Features

Uses Java as the programming language Relatively easier to program, leading to reduced

development time and ease of maintenance Supports the concept of “write-once-run-

everywhere”

Data structures are carefully designed to optimize the runtime performance

A hierarchical structure is used to manage the possible interactions

Allows an incomplete test set to be extended to a complete one

Add new tests or parameters, if necessary, to achieve t-way coverage

Combinatorial Testing Using Covering Arrays 21

Initial Results (1)

[System]

Name: Test Configuration for TCAS

[Parameter]

-- only compare with MINSEP and MAXALTDIFF

Cur_Vertical_Sep : 299, 300, 601

High_Confidence : TRUE, FALSE

Two_of_Three_Reports_Valid : TRUE, FALSE

-- Low and High, only compare with Other_Tracked_Alt

Own_Tracked_Alt : 1, 2

Other_Tracked_Alt : 1, 2

-- only compare with OLEV

Own_Tracked_Alt_Rate : 600, 601

Alt_Layer_Value : 0, 1, 2, 3

-- compare with each other (also see NOZCROSS) and with ALIM

Up_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840

Down_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840

Other_RAC : NO_INTENT, DO_NOT_CLIMB, DO_NOT_DESCEND

Other_Capability : TCAS_TA, OTHER

Climb_Inhibit : TRUE, FALSE

Combinatorial Testing Using Covering Arrays 22

Initial Results (2)

All the experiments are performed on a desktop with 1.2GHZ CPU and 1GB memory.

Combinatorial Testing Using Covering Arrays 23

Outline

Introduction

The IPO Strategy

FireEye: An N-Way Testing Tool

Related Work

Conclusion

Combinatorial Testing Using Covering Arrays 24

Classification

Search-Based methods that are mainly developed by computer scientists

AETG (from Telcordia), TCG (from JPL/NASA), DDA (from ASU), PairTest

Algebraic methods that are mainly developed by mathematicians

Orthogonal Arrays Recursive Construction

Combinatorial Testing Using Covering Arrays 25

Orthogonal Arrays

Orthogonal arrays can be constructed very fast and are always optimal

Any extra test will cause a pair to be covered for more than once

However, there are several limitations: Orthogonal arrays do not always exist Every parameter must have the same number v of

values Existing methods often require v be a prime power. Every t-way interaction must be covered at the same

number of times

Combinatorial Testing Using Covering Arrays 26

Recursive Construction

Covering arrays are a more general structure, which requires every t-way interaction be covered at least once

Constructing a covering array from one or more covering arrays with smaller parameter sets

Recursive construction can be fast, but it also has restrictions on the number of parameters and the domain sizes

Combinatorial Testing Using Covering Arrays 27

Search-Based vs Algebraic Methods

Search-based methods: Advantages: no restrictions on the input model, and

very flexible, e.g., relatively easier to support parameter relations and constraints

Disadvantages: explicit search takes time, the resulting test sets are not optimal

Algebraic methods: Advantages: very fast, and often produces optimal

results Disadvantages: limited applicability, difficult to

support parameter relations and constraints

The advantages and disadvantages of the two types of methods seem to complement with each other

Combinatorial Testing Using Covering Arrays 28

Outline

Introduction

The IPO Strategy

FireEye: An N-Way Testing Tool

Related Work

Conclusion

Combinatorial Testing Using Covering Arrays 29

Conclusion

Combinatorial testing is a well-defined problem and has been used widely in practice.

The IPO strategy has a lower order of complexity than AETG, and still produces competitive results.

Algebraic methods, if applicable, are fast and can be optimal, whereas search-based algorithm are very flexible.

Going beyond 2-way testing presents challenges and opportunities to the area of combinatorial testing.

Combinatorial Testing Using Covering Arrays 30

References

1. Boroday S. Y. and Grunskii I. S., “Recursive generation of locally complete tests,” Cybernetics and Systems Analysis 28 (1992), 20-25.

2. K. A., Bush, “Orthogonal arrays of index unity,” Annals of Mathematical Statistics, 23 (1952), 426-434.

3. D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, “The AETG System: An Approach to Testing Based on Combinatorial Design,” IEEE Transactions on Software Engineering, 1997, Vol. 23, No. 7.

4. M. B. Cohen, C. J. Colbourn, P. B. Gibbons and W. B. Mugridge, “Constructing test suites for interaction testing,” In Proc. of the Intl. Conf. on Software Engineering, (ICSE 2003), 2003, pp. 38-48, Portland.

5. R. Kuhn, D. Wallace, A. Gallo, “Software Fault Interactions and Implications for Software Testing,” IEEE Transactions on Software Engineering, June 2004, Vol. 30, No. 6.

6. Alan Hartman, Leonid Raskin, “Problems and algorithms for covering arrays,” Discrete Mathematics 284(1-3): 149-156 (2004)

7. Y. Lei and K. C. Tai , “In-parameter-order: a test generation strategy for pairwise testing,” Proceedings Third IEEE Intl. High-Assurance Systems Engineering Symosium., 1998, pp. 254-261.

8. K. C. Tai and Y. Lei, “A Test Generation Strategy for Pairwise Testing,” IEEE Transactions on Software Engineering, 2002, Vol. 28, No. 1.