1

Applications of Optimization to Logic Testing

Gary Kaminski and Paul AmmannICST 2010

CSTVA Workshop

2

OutlineMotivation

Test Sets and Coverage CriteriaShould We Apply Well Known Optimization Techniques?

Testing DNF Predicates For Logic FaultsMinimal DNF SummaryDNF Faults and Fault HierarchyCoverage Criteria

Optimizing DNF Criteria

ResultsOptimized Test Set Sizes

Benefit, Computational CostOptimizing Individual Tests

Implications For Regression Testing

Thoughts On Where to Go from Here

3

MotivationTest Sets and Coverage Criteria

Problem 1: Find Test Set that Satisfies CriterionVery Common ApproachOften Tests Are Redundant

Problem 2: Find Minimal Test Set that Satisfies CriterionGiven A Test Set, Throw Out Redundant TestsNow Every Test Is There For a Reason!

Problem 3: Find Minimum Test Set that Satisfies CriterionGenerally Much HarderCan’t Be Found With “Local” InformationNeed To Make Globally Optimal Choices

Question:Is It Useful To Recast Coverage As Standard Optimization Problem?

Gain?Cost?

Here We Address This Question In Context Of DNF Logic Testing

4

Minimal DNF SummaryTerms separated by OR, literals by AND

ab + a!c vs. a(b + !c)

• Make each term true and other terms false ab + ac vs. ab + abc

• Impossible to remove a literal without changing the predicate semanticsab vs. abc + ab!c

Green – in minimal DNF Red – not in minimal DNF

5

Nine Minimal DNF Logic FaultsOriginal: ab + bc

• Expression Negation Fault (ENF) !(ab + bc)• Operator Reference Fault (ORF) abbc• Operator Reference Fault (ORF) a + b + bc• Term Negation Fault (TNF) !(ab) + bc• Term Omission Fault (TOF) bc• Literal Negation Fault (LNF) !ab + bc• Literal Insertion Fault (LIF) abc + bc• Literal Reference Fault (LRF) ac + bc • Literal Omission Fault (LOF) a + bc

Fault Detection Hierarchy

Expression Negation Fault

ENF

Literal Insertion FaultLIF

Term Omission FaultTOF

Literal Reference Fault

LRF

Literal Negation FaultLNF

Operator Reference Fault

ORF+

Literal Omission FaultLOF

Term Negation FaultTNF

Operator Reference Fault

ORF*

6

Key Fact:Detection Relations Depend on Whether Fault is Feasible

7

Coverage Criteria Preamble : Unique True Points and Near False Points

• UTP: An assignment of values such that only one term evaluates to true.

ab + bc: TTF is a UTP for term ab

• NFP: An assignment of values such that the predicate evaluates to false but when a literal is omitted, it evaluates to true.

ab + bc: TFF is an NFP for b in term abClosely Related to MCDC Notion

• For each implicant find unique true points (UTPs) so that– Literals not in term take on values T and F

• Consider the DNF predicate: – f = ab + cd

• For term ab– Choose TTFT, TTTF

• For term cd– Choose FTTT, TFTT

• MUTP test set– {TTFT, TTTF, FTTT, TFTT}

MUTP Criterion (Chen, Lau, Yu)

01

00

10110100 ab cd

t

t

tt11

10

t

tt

CUTPNFP Criterion (Chen, Lau, Yu)• Consider f = ab + cd• For term ab

– For a, choose UTP, NFP pair• TTFF, FTFF

– For b, choose UTP, NFP pair• TTFT, TFFT

• For term cd– For c, choose UTP, NFP pair

• FFTT, FFFT– For d, choose UTP, NFP pair

• FFTT, FFTF• Possible CUTPNFP test set

– {TTFF, TTFT, FFTT //UTPs FTFF, TFFT, FFFT, FFTF} //NFPs

01

00

10110100 ab cd

t

t

tt11

10

t

tt

• Find NFP tests for each literal such that all literals not in the term attain F and T

• Consider the DNF predicate: – f = ab + cd

• For implicant ab– Choose FTFT, FTTF for a– Choose TFFT, TFTF for b

• For implicant cd– Choose FTFT, TFFT for c– Choose FTTF, TFTF for d

• MNFP test set– {TFTF, TFFT, FTTF, TFTF}

• Example is small, but generally MNFP is large

MNFP Criterion (Chen, Lau, Yu)

01

00

10110100 ab cd

t

t

tt11

10

t

tt

11

MUMCUT/Minimal-MUMCUT Criteria• MUMCUT is simply MUTP + CUTPNFP + MNFP• Minimal-MUMCUT omits tests via low level criterion feasibility analysis

– Adds CUTPNFP and MNFP tests only when necessary• Minimal-MUMCUT guarantees detecting LIF, LRF, LOF

– And thus all 9 faults in the hierarchy, even if some faults infeasible

Key Point: Minimal != MinimumLocal vs. Global Perspective

CUTPNFPfeasible? MNFP

Test Set =MUTP + MNFP

For Each

Literal In Term

Test Set =MUTP + CUTPNFP

MUTP feasible?

Test Set =MUTP + NFP

For Each Term

12

MUTP Optimization • Minimize test set size subject to detecting LRFs with UTPs

– LIFs Guaranteed Be Detected When LRFs are Feasible• Each UTP is a 0-1 variable (integer programming)• Each constraint represents the UTPs that detect a LRF• Can reduce number of variables and constraints by focusing on a fault

called the TRF-LIF instead of the LRF (see paper for details)

ab + cdMinimize x1100 + x1101 + x1110 + x0011 + x0111 + x1011Subject tox1100 + x1101 >= 1 (cb + cd)x1110 >= 1 (!cb + cd)x1100 + x1110 >= 1 (db + cd)x1101 >= 1 (!db + cd)…….

13

CUTPNFP Optimization• Minimize test set size subject to finding UTP – NFP corresponding pairs• Each point is a 0-1 variable• Constraints represent an implication that the UTP and NFP be corresponding• Additional constraint to reflect that an UTP must be selected

ab + cdMinimize x1100 + x0100 + x1000 + x1101 + …Subject To• x1100 <= M1; -- if 1100 is selected as an UTP then M1 must = 1• x0100 >= M1; -- if M1 = 1, 0100 must be chosen as a NFP • x1000 >= M1; -- if M1 = 1, 1000 must be chosen as a NFP • x1101 <= M2; -- if 1101 is selected as an UTP then M2 must = 1 • x0101 >= M2; -- if M2 = 1, 0101 must be chosen as a NFP• x1001 >= M2; -- if M2 = 1,1001 must be chosen as a NFP • x1110 <= M3; -- if 1110 is selected as an UTP then M3 must = 1 • x0110 >= M3; -- if M3 = 1, 0110 must be chosen as a NFP • x1010 >= M3; -- if M3 = 1, 1010 must be chosen as a NFP • x1100 + x1101 + x1110 >= 1; -- an UTP must be selected

14

MNFP Optimization

• Minimize test set size subject to detecting LRFs with NFPs• Each NFP is a 0-1 variable• Each constraint represents the NFPs that detect a LRF

ab + cdMinimize x0100 + x0101 + x0110 + x1000 + x1001 + x1010 ...Subject tox0110 >= 1 (cb + cd)x0100 + x0101 >= 1 (!cb + cd)x1001 >= 1 (db + cd)x1000 + x1010 >= 1 (!db + cd)…….

15

Minimal-MUMCUT Optimization

• Applies optimization techniques for MUTP, CUTPNFP, and MNFP above based on criterion feasibility

• When MUTP is feasible, can select any NFP• When MUTP is not feasible, but CUTPNFP is feasible, apply

CUTPNFP optimization• Otherwise, apply MNFP optimization

ab + cdMinimize x0100 + x0101 + x0110 + x1000 + x1001 + x1010 + …Subject tox0100 + x0101 + x0110 >= 1 (any NFP for a works as MUTP is feasible)x1000 + x1001 + x1010 >= 1 (any NFP for b works as MUTP is feasible)

…….

16

Case Study• Analyzed 19 Boolean predicates in an avionics software

system (Weyuker, Chen, Lau, and Yu)

• Number of unique literals range: 5 to 13

• Built a tool in Java to automatically:- generate MUTP, CUTPNFP, MNFP, Minimal-MUMCUT

tests based on heuristics- generate an optimization model that when solved by

CPLEX yields minimized test sets

• Examined test set size and computation time difference between heuristics and optimization

17

Average Test Set Size Results

Criterion Minimized Test Set Size

Heuristic Test Set Size

Percentage

MUTP 15.00 15.00 100.00%

CUTPNFP 41.16 44.79 91.90%

MNFP 37.89 46.16 82.08%

Minimal-MUMCUT

40.37 43.95 91.85%

18

Average Computation Time Results (sec)

Criterion Heuristics Optimization(Model Generation

and Solver)

MUTP 0.07 0.23

CUTPNFP 0.11 1.33

MNFP 0.22 5.95

Minimal-MUMCUT

5.15 6.09

19

Average Optimization Model Size

Criterion Number of Variables

Number of Constraints

MUTP 15 15

CUTPNFP 561 336

MNFP 414 245

Minimal-MUMCUT

392 106

20

Interpretation of Results

• For a small difference in computation time, optimization produces smaller test sets

• Running tests is time-consuming and expensive, so reducing test set size by one can be significant

• Choose optimization over heuristics

21

Fault Detection Maximization• Testers have limitations on the number of tests they can run and thus it

may be impossible to satisfy a criterion. In this case, it would help to know what tests maximize fault detection subject to a given test set size.

• Each point (test) is a 0-1 variable - x1100 represents the point (test) where a is T, b is T, c is F, d is F - x1100 will be 0 if it is not in the test set and 1 if it is in the test set

• Each fault is a 0-1 variable - LIF_1_c represents a fault where literal c is inserted into term 1 - LIF_1_c will be 0 if it is not detected and 1 if it is detected - objective function is to maximize the sum of all the fault variables - a fault variable exists for each fault based on the 9 DNF fault types

22

Fault Detection MaximizationConsider ab + cd

Maximize …. + LIF_1_c + ...

Subject to• x1101 + x1100 >= M_LIF_1_c; (1)• LIF_1_c <= M_LIF_1_c; (2)……

LIF_1_c represents a fault where c is inserted into term 1 (ab).

If neither 1101 nor 1100 is selected, the x1101 and x1100 variables will be 0. For constraint (1) to hold, the variable M_LIF_1_c must then be 0.Thus, variable LIF_1_c must be 0 as well and the fault goes undetected.

If 1101 or 1100 is selected, either x1101 and x1100 will be 1. This allows the variable M_LIF_1_c to be 1, which in turn allows LIF_1_c to be 1 and thus the fault is detected.

23

Case Study• Used same predicates as in the first study

• Built a tool in Java to automatically:- generate an optimization model that when

solved yields a test set that maximizes fault detection subject to a given test set size

- generate an optimization model that when solved yields a test set that minimizes fault detection subject to a given test set size (by flipping constraints in the model)

• Used the tool to compare test set size between a best, random, and worst case scenario

24

Case Study Results

Amongst all 19 predicates:• As many as 288 times more faults are detected in the

best case than the worst case for a single test.• As many as 7 times more faults are detected in the best

case than the random case for a single test.Implication for regression testing

Scenario Fault Detection Percentage for 1 test

Worst 0.35%

Random 6.13%

Best 15.97%

25

Case Study Results

0102030405060708090

100

Percentage of Faults Detected

1 5 10 15 MINTest Set Size

RandomBest

26

Fraser and Gargantini Study• Found fault detecting tests for the same 19 predicates by formalizing

the problem as a logic satisfiability problem which can be solved by a SAT algorithm (not optimization)

• 3 approaches1) Minimal-MUMCUT heuristic every test needed for fault detection2) Satisfiability Algorithm improves on heuristic3) Optimization guaranteed smallest test set size

Minimal-MUMCUT

Satisfiability Algorithm

Optimization

Avg Test Set Size 43.95 42.37 40.37

27

Conclusion• Applied optimization techniques in the Minimal DNF logic

testing domain to:1) Minimize test set size subject to guaranteeing fault

detection (by satisfying a given criterion) 2) Maximize fault detection subject to a certain test set size

• Case study showed for a small sacrifice in computation time, optimization produced smaller test sets

• Case study showed large difference in fault detection between a best, random, and worst case scenario

• Applications in testing of programs with large predicates

28

Thoughts On Where To Go Next• Reformulating Model To Handle Infeasible Tests

– Crucial for Handling Predicates Embedded In Code– Requires Removal Of Some Constraints

• Some Faults Become May Become Equivalent• Hence Constraint To Trigger Fault is Infeasible

– Also Requires Adding Additional Constraints• Fault Hierarchy Only Applies For Feasible Faults

• Applying Optimization To Other Coverage Criteria– Eg: Branch Coverage

• Building Optimal “Test Databases”– Given a Query With A Particular “Where” Clause– What Minimal Database Guarantees Detecting All 9 Faults in

Query?• Tool Available at Ammann/Offutt Book Web Site

– http://cs.gmu.edu/~offutt/softwaretest