mutation testing
DESCRIPTION
Brown Bag lightning talk I gave on 31st May 2012TRANSCRIPT
Mutation Testing
Chris Sinjakli
Testing is a good thing
But how do we know our tests are good?
Code coverage is a start
But it can give a “good” score with really dreadful tests
Really dreadful testspublic int addTwoNumbers(int a, int b) {
return a – b;}
...
@Testpublic void shouldAddTwoNumbers() {
int result = addTwoNumbers(1, 1);assertTrue(true);
}Coverage: 100%Usefulness: 0
A contrived example
But how could we detect it?
Mutation Testing!
“Who watches the watchmen?”
If you can change the code, and a test doesn’t fail, either the code is never run or the tests are
wrong.
Going with our previous examplepublic int addTwoNumbers(int a, int b) {
return a – b;}
...
@Testpublic void shouldAddTwoNumbers() {
int result = addTwoNumbers(1, 1);assertTrue(true);
}
Let’s change something
Going with our previous examplepublic int addTwoNumbers(int a, int b) {
return a + b;}
...
@Testpublic void shouldAddTwoNumbers() {
int result = addTwoNumbers(1, 1);assertTrue(true);
}
This still passes
So it caught a really rubbish test
How about something slightly less obvious?
Slightly less obvious (and I mean slightly)
public int checkConditions(boolean a, boolean b) { if (a && b) {
return 42;}else {
return 0;}
}
@Testpublic void testBothFalse() {
int result = checkConditions(false, false);assertEquals(0, result);
}@Testpublic void testBothTrue () {
int result = checkConditions(true, true);assertEquals(42, result);
}
Coverage: 100%Usefulness: >0But still wrong
Slightly less obvious (and I mean slightly)
public int checkConditions(boolean a, boolean b) { if (a && b) {
return 42;}else {
return 0;}
}
@Testpublic void testBothFalse() {
int result = checkConditions(false, false);assertEquals(0, result);
}@Testpublic void testBothTrue () {
int result = checkConditions(true, true);assertEquals(42, result);
}
Mutate
Slightly less obvious (and I mean slightly)
public int checkConditions(boolean a, boolean b) { if (a || b) {
return 42;}else {
return 0;}
}
@Testpublic void testBothFalse() {
int result = checkConditions(false, false);assertEquals(0, result);
}@Testpublic void testBothTrue () {
int result = checkConditions(true, true);assertEquals(42, result);
}
Passing tests
Mutation testing caught our mistake
:D
Useful technique
But still has its flaws
The downfall of mutation(Equivalent Mutants)
int index = 0
while (someCondition) {doStuff();index++;if (index == 100) {
break;}
}
int index = 0
while (someCondition) {doStuff();index++;if (index >= 100) {
break;}
}
Mutates to
But the programs are equivalent, so no test will fail
Tools
Some Java, then some Ruby
Java
• Loads of tools to choose from• Bytecode vs source mutation• Will look at PIT (seems like one of the better
ones)
PIT - pitest.org
• Works with “everything”– Command line– Ant– Maven
• Bytecode level mutations (faster)• Very customisable
– Exclude classes/packages from mutation– Choose which mutations you want– Timeouts
• Makes pretty HTML reports (line/mutation coverage)
Ruby
Ruby
• Mutant seems to be the new favourite• Runs in Rubinius (1.8 or 1.9 mode)• Only supports RSpec• Easy to set up
rvm install rbx-headrvm use rbx-headgem install mutant
• And easy to usemutate “ClassName#method_to_test” spec
Summary
• Seems like it could identify areas of weakness in our tests
• At the same time, could be very noisy• Might be worth just trying it against an
existing project and seeing what happens
Questions?