mutatis mutandis evaluating dbms test adequacy with mutation testing
DESCRIPTION
Mutatis Mutandis Evaluating DBMS Test Adequacy with Mutation Testing. Ivan T. Bowman, HANA Product Engineering June 24, 2013. Public. Agenda. Test adequacy and why we want to evaluate it Mutation testing and how to apply it to database systems Conclusions and future work. Test adequacy. - PowerPoint PPT PresentationTRANSCRIPT
Mutatis MutandisEvaluating DBMS Test Adequacy with Mutation TestingIvan T. Bowman, HANA Product EngineeringJune 24, 2013 Public
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 2Public
Agenda
Test adequacy and why we want to evaluate it
Mutation testing and how to apply it to database systems
Conclusions and future work
Test adequacy
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 4Public
Challenges of testing database servers
Self-management featuresAdapting to current conditions leads to increased internal state that needs to be tested
Relational equivalenceDML statements can be transformed into different equivalent formsThe correct answer may be returned even if a desirable optimization was not considered
Concurrent executionDatabase systems need to work well with concurrent connections and intra-query parallelismConcurrency leads to more primitives that need testing and more interactions to test
Performance requirement are not clearly specifiedTests of performance need to control many factors, limiting their sensitivity
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 5Public
Goals of measuring test adequacy
We want to make sure that software is “thoroughly tested”
We want to compare advanced test generation techniques:Random query generation (RAGS)Genetic algorithms
We want to minimize test suites by removing expensive tests that contribute little
We want to prioritize test suitesFor example, require all developers to run a minimum-acceptance test before submission
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 6Public
Measuring test adequacy
Coverage techniques are commonly usedMeasure how many statements / basic blocks / code combinations are executed under testsThe metric is simple to compute and understandCoverage metrics are necessary but not sufficient for computing test adequacy
A direct approach to evaluating adequacy considers how many faults the tests findWe could try to use “natural” faults that result from developer errors found before or after shippingWe could manually “seed” faults in the code to see how many are detected For example, disable the effect of specific types of mutex
We could automatically introduce faults using patterns: mutation testing
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 7Public
Mutation testing
Mutation testing evaluates the effectiveness of a test suite for detecting incorrect programsEvaluation focuses on those that are “close” to the correct version
Mutation operators are defined to alter source codeFor example, change logical and (&&) to or (||)Each operator creates a “mutant” programA test suite “kills” the mutant if it passes with the original and fails with the mutant
Mutation adequacy score is the ratio of killed mutants to total mutantsA particular problem arises with mutants that don’t affect program semantics (equivalence)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 8Public
Does statement coverage predict ability to kill mutants?
0% 5% 10% 15% 20% 25% 30% 35%0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
R² = 0.169599259716198
Percent of Statements Executed
Perc
ent o
f Mut
ants
Kill
ed
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 9Public
General approach of mutation testing
Generate Mutants P’
Input Program P Generated
Mutants P’
Input Test Set T
P’ Fails T?
Yes: P’ Killed No: P’ Lives
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 10Public
Mutation operators we evaluated
FunctionApplies to all methods and C-style functions. Skip the contents of the function.
ConditionApplies to the condition in for, while, and if statements. Evaluate a mutated condition.
SwitchApplies to all switch statements. Add one to the expression.
CaseApplies to all case statements within a switch. Skip the contents of the case.
DefaultApplies to default statements within a switch statement. Skip the default statements.
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 11Public
Mutation types and their outcomes (400K of query processing code)
Count Covered Killed Lethal PassFunction 13,563 75.6% 69.4% 41.1% 30.6%
Condition 35,680 74.8% 65.6% 34.9% 34.4%
Switch 1,176 78.1% 69.0% 34.1% 31.0%
Case 7,528 55.7% 46.1% 27.4% 53.9%
Default 955 24.2% 18.8% 7.9% 81.2%
Total 58,902 71.8% 63.3% 34.9% 36.7%
Practical issues
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 13Public
Large systems generate many mutants
In a 400K line subset of SQL Anywhere, about 59K mutants identifiedCompiling these separately would take too much time and space
Running the general mutant algorithm would take 432 days
Generate a single meta-mutant where mutants can be enabled at run-timeA:if( len > 0 )B:if(mutation_on(123) ? (len>=0) : (len>0) )
Use the same code editing to track mutation coverageIdentify which tests cover mutated code linesNo need to evaluate whether a test kills a mutant it does not execute
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 14Public
Simplifying assumptions
Independence of mutationsIf M is a set of mutants then test kills iff it kills or (or both)
Test failures do not corrupt stateTest may terminate on input with one of Crash, Timeout, Fail or Pass
Tests deterministically find faults
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 15Public
Proposed Improvements
1. Test independent mutations simultaneouslyIndividual tests do not kill many tests On average, a test kills 12% of the mutants it covers
Guess test will not kill any of the currently living mutants it coversUse binary search if it does fail with enabled If test fails on a single mutant, it kills the mutant
From 432 days to 17.3 with this improvement
2. Identify “lethal” mutations in a first passExecute each mutant with the cheapest test that kills it
3. Order tests so that cheaper tests run firstMutants that are easily killed are removed quickly
2.Lethal Mutation Step
3.Ordering
No Yes
No 415h 54h
Yes 49h 34h
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 16Public
Reminder: general approach of mutation testing
Generate Mutants P’
Input Program P Generated
Mutants P’
Input Test Set T
P’ Fails T?
Yes: P’ Killed No: P’ Lives
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 17Public
Adjusted approach of mutation testing
Input Program P
Input Test Set T
Meta-Mutant P’
Living Mutants
Prepare Meta-Mutant
Measure test timeand mutant coverage
Find lethal mutants
For each ordered by durationGuess that passes on
Recursively divide and conquer
Delete lethal
Cheapest
Delete
k
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 18Public
Coverage and mutation adequacy as tests execute
0 100 200 300 400 500 6000%
20%
40%
60%
80%
100%
Statements Covered Killed Mutants
Number of Tests Executed
Perc
ent o
f Max
imum
Conclusions and Future Work
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 20Public
Conclusions
Mutation testing is practical for large systems such as DBMSs
Mutation testing gives a “harder” adequacy metric than code coverage
Our approach required simplifying assumptions that are not generally trueSmall-scale testing shows these hold sufficiently for some purposes
Future work is needed to address simplifying assumptionsTests do not deterministically find faults; in particular, some mutations are non-deterministicMutants may interact; can we characterize them as independent by analysis?
Future work includes comparing test generation frameworks on mutation adequacy
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 21Public
Acknowledgements
Feedback from the SQL Anywhere Query Processing group was invaluableIn particular, detailed suggestions from Daniel J. Farrar helped shape this paperIntern J. Devin Papineau provided significant motivation and early discussion for this work
The anonymous reviewers provided helpful and insightful suggestions
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Thank you
Contact information:
Ivan T. BowmanResearch Manager of SQL Anywhere Query ProcessingIvan.Bowman (at) sap.com