maintainable automated acceptance tests - agile alliance · use executable acceptance criteria as a...

http://thoughtworks-studios.com/

Maintainable AutomatedAcceptance Tests

Jez Humble

Wednesday, January 18, 12

http://thoughtworks-studios.com

http://thoughtworks-studios.com

support programming critique the product

technology facingbusiness facing

or

or

Brian Marick

http://www.exampler.com/old-blog/2003/08/21/#agile-testing-project-1Wednesday, January 18, 12

http://www.exampler.com/old-blog/2003/08/21/#

http://www.exampler.com/old-blog/2003/08/21/#

critique the product

technology facing

business facing

unit tests

functional tests acceptance tests

showcasesexploratory testing

usability testing

performance testssecurity tests

component testssystem testssu

ppor

t pro

gram

min

g


UI

Service

Unit

Mike Cohn: Succeeding with Agile


End to EndBusiness Facing

Localized Technology Facing


“BDD is a second-generation, outside-in, pull-based, multiple-stakeholder, multiple-scale, high-automation, agile

methodology.” - Dan North

behavior-driven development


describes system behaviour

uses plain English, with business vocabulary

Given... , When... , Then...

helps ‘business’ and ‘technology’ communicate

automated tests


Acceptance Criteria"Given ...When ...Then ..."

Test Implementation LayerCode uses domain language; no

reference to UI elements.

Application Driver LayerUnderstands how to interact with the application to perform actions

and return results.


Acceptance Criteria"Given ...When ...Then ..."

Test Implementation LayerCode uses domain language; no

reference to UI elements.

Application Driver LayerUnderstands how to interact with the application to perform actions

and return results.

Customer Tester

Developer Tester

Developer


Add a guard to prevent it happening again

Triage to find root cause1. There was an environmental problem2. There is a bug with the test3. An assumption changed4. The test actually caught a bug

Fix the problem

Optimise your process for time to fix tests

Optimise your test suite to detect failures fast

When acceptance tests break


#1301As a... I want... So that...

Given... Given...When... When...Then... Then...




































Buy Product

Search product catalogueAdd product to cartCheck outCreate new accountProvide address detailsProvide credit card detailsComplete orderVerify order createdVerify credit card debitedVerify email sent

#1612As a customerI want a gift wrapping optionSo that I don’t have to wrap them and post them myself Buy Product

Search product catalogueAdd product to cartCheck outCreate new accountProvide address detailsProvide credit card detailsSelect gift wrapping optionComplete orderVerify order createdVerify gift wrapping optionVerify credit card debitedVerify email sent

Scenario tests


Intermittent

Flakey

Flakey

Non-Deterministic


Flakey

Non-Deterministic

Useless=Wednesday, January 18, 12

Flakey

Non-Deterministic

Useless=worse than


Quarantine

Limit!


• Interference caused by dependencies• Asynchrony• Remote services• Time• Resource leaks

Causes

http://martinfowler.com/articles/nonDeterminism.html


Application reference data

Test-specific data

Test reference data

Don’t use production data dumps (except for performance testing and staging)

Ensure tests are independent from each other

Test data


Don't separate test and dev teams

People with testing capability specify criteria; people with programming capability build test fixture

Use record/playback only for limited cases

Version control tests together with production code

Use mainstream programming environments for your tests

Put time and effort into test data management - don’t use production data

Treat failures seriously

Handle new features iteratively not incrementally

Treat test code as codeUse executable acceptance criteria as a communication mechanism with business

Devs can run a single test or a suite on their dev machine

Build your own abstractions for acceptance criteria

Choosing between UI and subcutaneous tests depends on the architecture of your application

Build tests for parallelizationFix flaky tests


maintainable automated acceptance tests - agile alliance · use executable acceptance criteria as a...

Documents