what test coverage mean to us
DESCRIPTION
Mutation test is a way to put a "mutation" into your code, run the test and then see if the test fails or not. A mutation is a change to production code which should make it behave in a different way. The idea is, if the code is just enough to pass the test, any mutation should make the test failed due to the behavior change. Although this statement is not always true, it should be correct in most cases. By TDD, we should get just enough and the simplest code that passing all the test. But, how do we know this? Sometimes, since writing test and code is so interactive in TDD, we may not choose the smallest baby step to pass the test. This means we may write some code which is not driven by current test. On the other side, regardless doing TDD or not, we hope those unit tests can avoid future regression when we changing the code, right? To make it more clear, our expectation is that any "bad" code change (aka. mutation) can be detected by those tests.TRANSCRIPT
![Page 1: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/1.jpg)
What Test Coverage Mean to us?
Joseph Yao
Tuesday, July 9, 13
![Page 2: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/2.jpg)
Who I am?
n Agile Coach at Odd-e
n Father, Husband...
n Software Craftsman
n Basketball, Board Game...
Tuesday, July 9, 13
![Page 3: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/3.jpg)
Agenda
n Bias and Truth of Test Coverage
n Classical Test Coverage
n Mutation Test Coverage
n TDD and Test Coverage
Tuesday, July 9, 13
![Page 4: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/4.jpg)
Bias of Test Coverage
Tuesday, July 9, 13
![Page 5: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/5.jpg)
Truth of Test Coverage
I expect a high level of coverage. Sometimes managers require one. There's a subtle difference.-- Brian Marick
http://martinfowler.com/bliki/TestCoverage.html
Tuesday, July 9, 13
![Page 6: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/6.jpg)
Truth of Test Coverage
Tuesday, July 9, 13
![Page 7: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/7.jpg)
Classical Test Coverage
Tuesday, July 9, 13
![Page 8: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/8.jpg)
Function Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}
Satisfied Test: foo(2, 2)Tuesday, July 9, 13
![Page 9: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/9.jpg)
Function Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}
Tuesday, July 9, 13
![Page 10: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/10.jpg)
Statement Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}
Satisfied Test: foo(2, 2)Tuesday, July 9, 13
![Page 11: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/11.jpg)
Statement Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}
Unsatisfied Test: foo(2, -1)Tuesday, July 9, 13
![Page 12: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/12.jpg)
Decision Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}
T/F
Satisfied Test: foo(2, 2), foo(-1, 2)Tuesday, July 9, 13
![Page 13: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/13.jpg)
Decision Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}
T/F
Unsatisfied Test: foo(2, 2)Tuesday, July 9, 13
![Page 14: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/14.jpg)
Condition Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}
T/F
Satisfied Test: foo(2, 2), foo(2, -1), foo(-1, -1)
T/F
Tuesday, July 9, 13
![Page 15: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/15.jpg)
Condition Coverageint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}
T/F
Unsatisfied Test: foo(2, 2), foo(-1, 2)
T/F
Tuesday, July 9, 13
![Page 16: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/16.jpg)
Classical Test Coverage
Tuesday, July 9, 13
![Page 17: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/17.jpg)
Classical Test Coverage
Tuesday, July 9, 13
![Page 18: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/18.jpg)
Expectation of Automated Test
Tuesday, July 9, 13
![Page 19: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/19.jpg)
Expectation of Automated Test
Any Possible “Bad” Code
Change can be Detected
Tuesday, July 9, 13
![Page 20: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/20.jpg)
Mutation Test Coverage
Tuesday, July 9, 13
![Page 21: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/21.jpg)
Why Mutation Test
Tuesday, July 9, 13
![Page 22: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/22.jpg)
Redundant Code?int foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z;}
Tuesday, July 9, 13
![Page 23: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/23.jpg)
Redundant Code?int foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z;}
int foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z;}
Tuesday, July 9, 13
![Page 24: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/24.jpg)
Redundant Code?int foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z;}
int foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } sideEffect(z); return z;}
n Missing test to prove the sideEffect call is needed
n If code is redundant, it can’t be learnt via any classical test coverage
Tuesday, July 9, 13
![Page 25: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/25.jpg)
Conditional Boundary Mutatorint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}
Condition Coverage Satisfied Test:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, -1))
assertEquals(0, foo(-1, 2))
Tuesday, July 9, 13
![Page 26: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/26.jpg)
Conditional Boundary Mutatorint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}
int foo (int x, int y){ int z = 0; if ((x>0) && (y>=0)) { z = x; } return z;}
Condition Coverage Satisfied Test:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, -1))
assertEquals(0, foo(-1, 2))
Tuesday, July 9, 13
![Page 27: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/27.jpg)
Conditional Boundary Mutatorint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}
int foo (int x, int y){ int z = 0; if ((x>0) && (y>=0)) { z = x; } return z;}
Condition Coverage Satisfied Test:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, -1))
assertEquals(0, foo(-1, 2))
Test to cover this Mutator:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, 0))
assertEquals(0, foo(-1, 2))
Tuesday, July 9, 13
![Page 28: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/28.jpg)
Conditional Boundary Mutatorint foo (int x, int y){ int z = 0; if ((x>0) && (y>0)) { z = x; } return z;}
int foo (int x, int y){ int z = 0; if ((x>0) && (y>=0)) { z = x; } return z;}
Condition Coverage Satisfied Test:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, -1))
assertEquals(0, foo(-1, 2))
n Test not prevent “bad” code change to happen
Test to cover this Mutator:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, 0))
assertEquals(0, foo(-1, 2))
Tuesday, July 9, 13
![Page 29: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/29.jpg)
Other Mutatorsn Negate Conditionals Mutator
n Math Mutator
n Increments Mutator
n Invert Negatives Mutator
n Inline Constant Mutator
n Return Values Mutator
n Void Method Calls Mutator
n Non Void Method Calls Mutator
n Constructor Calls Mutator
Tuesday, July 9, 13
![Page 30: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/30.jpg)
Test Driven Development
Tuesday, July 9, 13
![Page 31: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/31.jpg)
TDD and Test Coverage
n You can most likely ignore Classical Test Coverage
n You can’t ignore Mutation Test Coverage
n If not TDD, then you can learn a lot from both
Classical and Mutation Test Coverage
Tuesday, July 9, 13
![Page 32: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/32.jpg)
PokerHands Kata
Tuesday, July 9, 13
![Page 33: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/33.jpg)
True Mutation
Tuesday, July 9, 13
![Page 34: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/34.jpg)
Problematic Test
Tuesday, July 9, 13
![Page 35: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/35.jpg)
False Mutation
Tuesday, July 9, 13
![Page 36: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/36.jpg)
Reference
n Martin Fowler’s blog http://martinfowler.com/bliki/TestCoverage.html
n Mutation Test Wikipedia http://en.wikipedia.org/wiki/Mutation_testing
n Mutation Test Analysis Report http://crestweb.cs.ucl.ac.uk/resources/mutation_testing_repository/TR-09-06.pdf
Tuesday, July 9, 13
![Page 37: What test coverage mean to us](https://reader034.vdocuments.net/reader034/viewer/2022051609/5463ad29b4af9f493f8b4608/html5/thumbnails/37.jpg)
Q & A
新浪微博:姚若舟微信:yaoruozhou
土豆(Kata视频):姚若舟Github: JosephYao
Tuesday, July 9, 13