better code through making bugs

Tuesday, 22 April 14

Be#er code through making bugs

Seb Rose

Claysnow Limited

@sebrose


Massive thanks to:

3

Henry Colespitest

Filip van Laenenmutant


How do you assure the quality of your test suite?


Only employ coding gods




TDD will protect us




TDD will protect us

Peer review




TDD will protect us

Peer review

QA will catch anything that we miss




TDD will protect us

Peer review

QA will catch anything that we miss

Good code coverage



Code coverage?



Code coverage?

Line coverage?



Code coverage?

Line coverage?Branch coverage?



Code coverage?

Line coverage?Branch coverage?

Statement coverage?



Does good coverage guarantee that:


• I can safely refactor my tests?




• I can trust a test suite I inherited?





• My team are writing effective tests?






• I've retrofitted enough tests to protect my legacy code?






• I've retrofitted enough tests to protect my legacy code?


NO, IT

DOESN’T


Coverage tells us nothing about the quality of our tests



It tells us which statements have been run by our tests



It tells us which statements have NOT been tested



... but it is very useful for something else


public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}


@Test public void shouldStartWithEmptyCount() { assertEquals(0,testee.currentCount()); }

@Test public void shouldCountIntegersAboveTen() { testee.count(11); assertEquals(1,testee.currentCount()); } @Test public void shouldNotCountIntegersBelowTen() { testee.count(9); assertEquals(0,testee.currentCount()); }


Lipton, “Fault diagnosis in computer programs”,1971

“If we want to know if a test suite has properly checked some code

deliberately introduce a bug”


The deliberate introduction of a bug by changing the code under test

Mutation:


A version of the code under test that has had a single mutation

Mutant:


Mutation test:


Run your test suite

Mutation test:


Run your test suite

If any test fails, the mutant has been killed

Mutation test:


Run your test suite

If any test fails, the mutant has been killed

If no test fails, the mutant has survived

Mutation test:


Mutation testing:


Generate lots of mutants

Mutation testing:



Run each one through your test suite

Mutation testing:



Run each one through your test suite

Record and interpret the results

Mutation testing:



if ( i > 10 ) {


http://pitest.org

Demo time


http://pitest.org

http://pitest.org

Mutation operators:

• Conditionals Boundary Mutator• Negate Conditionals Mutator• Remove Conditionals Mutator• Math Mutator• Increments Mutator• Invert Negatives Mutator• Inline Constant Mutator• Return Values Mutator• Void Method Calls Mutator• Non Void Method Calls Mutator• Constructor Calls Mutator


“Generated mutants are similar to real faults”

Andrews, Briand, Labiche, ICSE 2005


“In practice, if the software contains a fault, there will usually be a set of mutants that can only be killed by a test case that also detects that fault.”

Geist et. al., “Estimation and Enhancement of Real-time Software Reliability through Mutation Analysis,” 1992


“Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults.”

K. Wah, “Fault Coupling in Finite Bijective Functions,” 1995


Poor performance

Equivalent mutations

Why isn’t mutation testing widely used?


How bad is performance?


How bad is performance?

Joda Time, consider, let us


Joda Time is a ...


small library for dealing with dates and times

Joda Time is a ...



68k lines of code

Joda Time is a ...



68k lines of code70k lines of test code

Joda Time is a ...




Takes about 10 seconds to compile

Joda Time is a ...




Takes about 10 seconds to compileTakes about 16 seconds to run the unit tests

Joda Time is a ...


Let’s use 10 mutation operators


Let’s use 10 mutation operatorsassume about 10k mutations



If it takes 1 second to compile each one



If it takes 1 second to compile each one 2.5 hours to generate the mutants




Run the test suite for each mutant




Run the test suite for each mutant10k x 16 seconds = 44.5 hours


pitest manipulates the byte code directly

10k mutants generated < 1 second

Don’t compile!


Run fewer tests!


Stop when a test fails

Run fewer tests!


Stop when a test failscan easily halve the run time

Run fewer tests!



Choose your tests carefully

Run fewer tests!



Choose your tests carefullynot every test can kill every mutant

Run fewer tests!




Parallelise the test runner

Run fewer tests!




Parallelise the test runneryour tests are unit tests, right?

Run fewer tests!


Choosing your tests well is critical


Each mutant can only ever be killed




by a subset of the tests




by a subset of the tests

Every other test run is waste



How can we know which tests might kill

any given mutant?



any given mutant?

Naming conventions?



any given mutant?

Naming conventions?Static analysis?



any given mutant?

Naming conventions?Static analysis?

COVERAGE DATA!



shouldCountIntegersAboveTenshouldNotCountIntegersBelowTenshouldCountIntegersOfExactlyTen




shouldCountIntegersAboveTenshouldCountIntegersOfExactlyTen





not covered by any test





not covered by any test

shouldStartWithEmptyCount does not cover any of the lines shown


Joda Time on my machine, with 2 threads:

- Timings> scan classpath : < 1 second> coverage and dependency analysis : 59 seconds> build mutation tests : 1 seconds> run mutation analysis : 8 minutes and 21 seconds> Total : 9 minutes and 22 seconds

- Statistics>> Generated 9922 mutations Killed 7833 (79%)>> Ran 117579 tests (11.85 tests per mutation)


Why only 79% killed?


Missing test cases



Missing test cases

Time outs



Missing test cases

Time outs

Equivalent mutations



public void someLogic(int i) { if (i <= 100) { throw new IllegalArgumentException(); }

if (i >= 100) { doSomething(); }}

Equivalent mutant:




if (i > 100) {

Equivalent mutant:




if (i > 100) {

i can never be 100 here

Equivalent mutant:


http://pitest.org

Demo time


http://pitest.org

http://pitest.org

Maybe we should have written:


doSomething();}


Some causes of equivalence are:


Dead/useless code



Dead/useless codeNon-functional modifications



Dead/useless codeNon-functional modificationsUnsatisfiable guards



Dead/useless codeNon-functional modificationsUnsatisfiable guardsInternal state



Dead/useless codeNon-functional modificationsUnsatisfiable guardsInternal state


This can help us improve our code


Mutants that survive need to bechecked manually

to determine if they are equivalent


Is this a show-stopper?



Not all that common in practice




TDD helps prevent equivalents




TDD helps prevent equivalents

Tolerate a few survivors


What about really large code bases?



Mutation testing can take a long time:



Mutation testing can take a long time:Use fewer mutants



Mutation testing can take a long time:Use fewer mutantsFilter the candidates



Mutation testing can take a long time:Use fewer mutantsFilter the candidatesRun infrequently


Get the developers to mutation test



Only on code they are working on




Faster feedback




Faster feedback

Improves design of the code


Get your CI server to run the mutation test



Over whole codebase if quick enough



Over whole codebase if quick enough

Use SCM integration to identify subset


Experiences and recommendations

Filip van Laenen


Use mutation testing from day 1


Filip van Laenen


Use mutation testing from day 1 Start on a small code base


Filip van Laenen


Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class low


Filip van Laenen


Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class lowHave small classes


Filip van Laenen


Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class lowHave small classesSelect a good tool:


Filip van Laenen



Configurable


Filip van Laenen



ConfigurableFlexible


Filip van Laenen



ConfigurableFlexibleIdentifies surviving the mutant


Filip van Laenen



Filip van Laenen


Believe the tool


Filip van Laenen


Believe the toolFix the problem


Filip van Laenen


Believe the toolFix the problemDon't turn mutation testing off


Filip van Laenen


Believe the toolFix the problemDon't turn mutation testing offLess code


Filip van Laenen


Believe the toolFix the problemDon't turn mutation testing offLess codeMore unit tests


Filip van Laenen


Believe the toolFix the problemDon't turn mutation testing offLess codeMore unit tests

More intelligent unit tests


Filip van Laenen


Available tools

http://en.wikipedia.org/wiki/Mutation_testing#External_links






Ruby: Heckle, Mutant

Available tools







Ruby: Heckle, MutantJava: pitest, Jumble, Jester

Available tools







Ruby: Heckle, MutantJava: pitest, Jumble, JesterC#: Nester, NinjaTurtle, Cream

Available tools







Ruby: Heckle, MutantJava: pitest, Jumble, JesterC#: Nester, NinjaTurtle, CreamPython: Pester

Available tools







Open sourceWorks with Java 5, 6, 7Works with all mocking frameworks including PowerMockIntegrates with Maven, Ant, Gradle & SBTPlugins for Eclipse & IntelliJPlugins for Jenkins & SonarQubeReleases every 3 months or so since 2011


• The Ladders - New York• Sky - Livingston• Insurance companies• Investment banks• Biotech companies• Norway's e-voting system


-JVM

Seb Rose,


-JVM

Seb Rose,

Availa

ble 20

14

(hopef

ully)


Seb Rose

Twi#er: @sebrose

Blog: www.claysnow.co.uk

E-‐mail: [email protected]


better code through making bugs

Technology