Software Logic Mutation Testing

Presented by

Gary Kaminski

Overview

• Introduction and Motivation

• Logic Faults and Detection

• Case Study Comparison

• Context

• Conclusion

Logic Mutation Testing Process

1) Systematically create program variations (mutants) each containing a single typical logic fault

if (a && b) if (a || b)

2) For each mutant, find an input that distinguishes the original program from the mutant program (killing)

- a && b and a || b evaluate to different truth values

• Focus is on step 1)

Logic Mutation Testing

• Logic mutants are created by applying mutation operators to Boolean predicates

• Boolean predicates may come from:

1) UML Diagrams

2) Security Requirements

3) Looping constructs (while / do-while / for)

4) Preconditions for methods (if - then)

MotivationCurrent mutation systems:

– generate excess logic mutants– do not detect certain classes of logic faults

Goal:

- generate fewer logic mutants

- detect more logic faults

Assumption:

- formally restrict attention to minimal Disjunctive Normal Form (DNF) Boolean predicates

Minimal DNF

• Terms separated by OR, literals by AND

ab + a!c vs. a(b + !c)

• Make each term true and other terms false

ab + ac vs. ab + abc

• Impossible to remove a literal or term without changing the predicate

ab vs. abc + ab!c

Excess Logic Mutants - Example

Original: ab + b!c + !bc

1) mutant generated multiple times: ab + _ + !bc

2) equivalent mutants: ac + b!c + !bc

3) mutants guaranteed to be killed by an input that kills some other mutant:

!ab + b!c + !bc and cb + b!c + !bc

4) highly effective mutation operators not used: abc + b!c + !bc

Minimal DNF Logic Faults

Original: ab + b!c + !bc

• Expression Negation Fault: !(ab + b!c + !bc)• Expression Stuck At Fault 0: FALSE• Expression Stuck At Fault 1: TRUE• Term Negation Fault: !(ab) + b!c + !bc• Term Omission Fault: ab + _ + !bc• Scalar Variable Replacement TOF: ab + c!c + !bc

• TOF generates same mutant twice: ab + _ + !bc

Minimal DNF Logic Faults

Original: ab + b!c + !bc

• Operator Reference Fault+: abb!c + !bc• Operator Reference Fault .: a + b + b!c + !bc• Operator Reference Fault+0: FALSE + !bc• Operator Reference Fault+1: TRUE + !bc• Operator Reference Fault.0: FALSE + b!c + !bc• Operator Reference Fault.1: TRUE + b!c + !bc

Minimal DNF Logic Faults

Original: ab + b!c + !bc

• Literal Negation Fault: !ab + b!c + !bc• Literal Stuck At Fault 0: FALSEb + b!c + !bc• Literal Stuck At Fault 1: TRUEb + b!c + !bc• Literal Omission Fault: _b + b!c + !bc• SVR LOF: bb + b!c + !bc

• LOF generates same mutant twice original: abc _bc, a_c, a_c, ab_

Minimal DNF Logic Faults

Original: ab + b!c + !bc

• Literal Reference Fault: ac + b!c + !bc • Literal Reference Fault: a!c + b!c + !bc• Literal Insertion Fault: abc + b!c + !bc• Literal Insertion Fault: ab!c + b!c + !bc

• LRF and LIF generate equivalent mutants

Logic Fault Minimal DNF Detection

Green Arrow – Detection may not holdBlack Arrow – Detection always holds

Red Box – Mutation Operator does not existBlue Box – Mutation Operator partially existsBlack Box – Mutation Operator exists

LOF

ORF.

LRF

LNF

TNF

LIF

TOF

LSTF0

ORF.0

ESTF0

ORF+

ORF+0

ENF

LSTF1

ORF.1

ORF+1

ESTF1

SVRTOF

SVRLOF

Logic Mutation Comparison

Original: ab + b!c + !bc• 61 mutants vs. 4 LIFs, 3 LOFs, 3 LRFs

• Equivalent LIF: abc + b!c + !bc• LRF: cb + b!c + !bc & ac + b!c + !bc• LOF: a_ + b!c + !bc

• Equivalent LIF: ab + b!c!a + !bc• LRFs: ab + !a!c + !bc & ab + b!a + !bc

• No Equivalent LIFs for term !bc• LOFs: ab + b!c + _c & ab + b!c + !b_

Mutation Efficiency Measures Mutation Efficiency Score = Faults Detected - Mutants GeneratedMutation Efficiency Ratio = Faults Detected / Mutants Generated• FD assumes an input is found to kill the mutant• FD includes equivalent mutants detected automatically

ab + b!c + !bc

Current Mutation Systems:• MES: 60 FD - 61 MG = -1• MER: 60 FD / 61 MG = .98

LIF-LRF-LOF• MES: 73 FD – 10 MG = 63 • MER: 73 FD / 10 MG = 7.30

Case Study

• Analyzed 19 Boolean predicates in an avionics software system

• Number of unique literals range: 5 to 13

• Compared MES and MER for current mutation systems vs. LIF-LRF-LOF

• Examined how MES and MER vary based on number of unique literals

Case Study Results

• LIF-LRF-LOF generated 18% of the mutants current mutation systems would (150 vs. 819)

• Current mutation systems guarantee detecting 75% of faults LIF-LRF-LOF does (819 vs. 1082)

• Number of literals matters for MES but not for MER• MES = FD – MG and MER = FD / MG

Current LIF-LRF-LOF

MES 0 932MER 1.00 7.22

Mutation Efficiency Score (FD – MG) vs.

Number of Unique Literals

0

500

1000

1500

2000

2500

5 6 7 8 9 10 11 12 13

LIF-LRF-LOF

CurrentMutation

Mutation Efficiency Ratio (FD/MG) vs.

Number of Unique Literals

0

2

4

6

8

10

12

5 6 7 8 9 10 11 12 13

LIF-LRF-LOF

CurrentMutation

Context: Internal Variable Problem

• What input values kill a mutant?

• Mutation may be buried deep in the code

• Must reach predicate and have mutant predicate and original predicate evaluate to different truth values

• Partial solutions using constraints exist

Context: Minimal DNF in Practice

1) 95% of 20,256 Boolean predicates in avionics software were in minimal DNF*

2) Detected > 99% of corresponding faults in non-minimal DNF Boolean predicates*

*Source: Y.T Yu and M.F. Lau. Comparing Several Coverage Criteria for Detecting Faults in Logical Decisions. In Proceedings QSIC 2004: 4th International Conference on Quality Software, Pages 14-21.

Context: Industry Audience

• LIF-LRF-LOF for software with Boolean predicates having >= 5 unique literals

• Exhaustive coverage for < 5 unique literals

• ab + b!c + bc (8 tests vs. 10 mutants)

• Avionics software*

*Source: J.J Chilenski and S.P. Miller. Applicability of modified condition/decision coverage to software testing. IEE/BCS Software Engineering Journal, 9(5): 193-200, September 1994.

Related Work1) J.J. Chilenski. An Investigation of Three Forms of the Modified Condition

Decision Coverage (MCDC) Criterion. Final Technical Report, DOT/FAA/AR-01/18, April 2001.

• Non-Minimal DNF based logic coverage criteria

2) T.Y. Chen, M.F. Lau, and Y.T. Yu. MUMCUT: A Fault-Based Strategy for Testing Boolean Specifications. Software Engineering Conference, 1999. (APSEC ’99) Proceedings. Sixth Asia Pacific. Pages 606-613.

• Minimal DNF based logic coverage criteria and Double Faults

3) D. Richard Kuhn. Fault Classes and Error Detection Capability of Specification Based Testing. ACM Transactions on Software Engineering and Methodology, 8(4): 411-424, October 1999.

• Seminal work in logic fault hierarchy

Conclusion

• Current logic mutation testing is inefficient• Excess mutants generated and faults are missed • Fault hierarchy approach based on minimal

DNF: detect more faults with less mutants• Applications in avionics software

Other Research and Publications

Logic Coverage Criteria:• Reduce test set size and maintain fault detection • Use feasibility of smaller, less expensive component criteria

Internal variable problem• CoJava and constraint programming

Publications:G. Kaminski, G. Williams, and P. Ammann. Reconciling Perspectives of Logic Testing for Software. Submitted to Journal of Software Testing, Verification, and Reliability, January 2008. Online.