lessons learned breaking the - british computer society › events › 2015 ›...
TRANSCRIPT
![Page 1: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/1.jpg)
Lessons LearnedBreaking theTDD Rules
Nat Prycehttp://www.natpryce.com
[email protected]@natpryce
github.com/npryce
![Page 2: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/2.jpg)
“You are not allowed to write any production
code unless it is to make a failing unit test pass.
You are not allowed to write any more of a unit
test than is sufficient to fail; and compilation
failures are failures.
You are not allowed to write any more
production code than is sufficient to pass the
one failing unit test.”
Bob Martin
![Page 3: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/3.jpg)
![Page 4: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/4.jpg)
Fake it ‘til you make it
Always watch the test fail
Tests must berepeatable
Tests must beisolated
Reset persistent state before the test, not
afterwards
One assertion per test
One behaviour per test
Don’t mock types you don’t own
Only mock out-of process resources
Manage dependencies in test code the same way as in production code
Given, When, Then
Avoid “When” Steps
Integrated tests are a scam
Hide incidental detailsOne domain
at a time
Test public API, not private
implementation
Allow queries; expect commands
Test for information, not representation
Listen to the tests
![Page 5: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/5.jpg)
Digital TV PVR
![Page 6: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/6.jpg)
PVR Platform Stack
Electronic Programme Guide
Third-Party Digital TV Middleware
Linux
Clean-Room JVM + JNI Platform Adaptors
MIPS or ARM + TV & PVR hardware
Java
C
![Page 7: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/7.jpg)
A More Realistic View
Electronic Programme Guide
Linux
Clean-Room JVM + JNI Platform Adaptors Broad, async API
Most of the product functionality
Continually changing as product evolves
Stabilised towards end of product cycle
Valuable legacy
MIPS or ARM + TV & PVR hardware
Third-Party Digital TV Middleware
![Page 8: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/8.jpg)
Shock! Testing with Live Data
![Page 9: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/9.jpg)
We know the TV schedule
![Page 10: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/10.jpg)
Functional Test Strategy
EPG
TV Middleware
Linux
JVM + JNI
Hardware
Control Service
TestQuery UI & Middleware state
(TCP)
User input (Infrared)
TV Guide Database
UPNP
Set TopBox User TV GuideUI
![Page 11: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/11.jpg)
(Idealised) Functional Test
@Test public void can_record_free_to_air_programme_from_guide_screen() { Showing showing = tvGuide.find(aShowing( onAFreeToAirChannel(), onAir(now()), withDuration(greaterThan(minutes(5)))));
Activity recordAndPlayShowing = on(Guide.SCREEN, Guide.record(showing)).then( on(Recordings.SCREEN, Recordings.findAndPlay(showing)));
SetTopBoxUser user = startUsingSetTopBox(); user.perform(recordAndPlayShowing); user.assertIsOn(FullScreenVideo.SCREEN); user.assertThat(FullScreenVideo.isPlaying(showing));}
![Page 12: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/12.jpg)
Unit-Level Fuzz Testing
JsonResponseParser parser = new JsonResponseParser();
@Test public void parsesResponseSuccessfullyOrThrowsIOException() { Mutator<String> mutator = new JsonMutator().forStrings(); for (String validResponse : validResponses()) for (String mutant : mutator.mutate(validResponse, 100)) assertParsesSuccessfullyOrThrowsIOException(mutant);}
void assertParsesSuccessfullyOrThrowsIOException(String json) { try { parser.parse(json); } catch (IOException _) { // allowed } catch (Exception e) { fail("unexpected exception for JSON input: " + json, e); }}
http://github.com/npryce/snodge
![Page 13: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/13.jpg)
Both Tests have the Same Structure
∀x∈X P(x)
![Page 14: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/14.jpg)
![Page 15: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/15.jpg)
Lesson
Repeatable failurerather thanrepeated success
![Page 16: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/16.jpg)
Lesson
Test automation is a search problem
CC-BY 2.0 Les Chatfield
![Page 17: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/17.jpg)
Optimising Search-Based Testing
Input Generator Code under Test
Instrumentation
Test
E.g. AFL http://lcamtuf.coredump.cx/afl/
![Page 18: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/18.jpg)
A. Causevic, R. Shukla, S. Punnekkat & D. Sundmark. Effects of Negative Testing on TDD: An Industrial Experiment. In Proc. XP2013, June 2013.
“...it is evident that positive test bias (i.e. lack of negative test cases) is present when [a] test driven development approach is being followed. …
When measuring defect detecting effectiveness and quality of test cases … negative test cases were above 70% while positive test cases contributed only by 30%”
![Page 19: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/19.jpg)
N. Nagappan, B. Murphy, and V. Basili. The Influence of Organizational Structure on Software Quality: an Empirical Case Study. 2008
“Organizational metrics are better predictors of failure-proneness than the traditional [software] metrics used so far.”
![Page 20: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/20.jpg)
more people touch the code → lower quality
loss of team members → loss of knowledge → lower quality
more edits to components → higher instability → lower quality
lower level of ownership (organizationally) → higher quality
more cohesive contributors (organizationally) → higher quality
more cohesive is the contributions (edits) → higher quality
more diffused contribution to a binary → lower quality
more diffused organizations contributing code → lower quality
Organisational Measures
![Page 21: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/21.jpg)
N. Nagappan, A. Zeller, T. Zimmermann, K. Herzig, and B. Murphy. Change Bursts as Defect Predictors. 2010
“What happens if code changes again and again in some period of time? … Such change bursts have the highest predictive power for defect-prone components [and] significantly improve upon earlier predictors such as complexity metrics, code churn, or organizational structure.”
![Page 22: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/22.jpg)
What About Specification by Example?
CC-BY 2.0 Mitch Huang
![Page 23: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/23.jpg)
Lesson - Separate Concerns
Testing
Living documentation
Understanding through examples
![Page 24: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/24.jpg)
Specification by Example Tools
![Page 25: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/25.jpg)
Approval Testing Tools
![Page 26: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/26.jpg)
Generate Documentation from Test Log
![Page 27: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/27.jpg)
Very few rules define TDD
![Page 28: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/28.jpg)
Very few rules define TDD
The rest are made to be broken
![Page 29: Lessons Learned Breaking the - British Computer Society › events › 2015 › lessons_learned_breaking_… · Manage dependencies in test code the same way as in production code](https://reader035.vdocuments.net/reader035/viewer/2022070812/5f0bb02b7e708231d431baf6/html5/thumbnails/29.jpg)
Very few rules define TDD
Nat Prycehttp://www.natpryce.com
[email protected]@natpryce
github.com/nprycespeakerdeck.com/npryce
The rest are made to be broken!