maintainable automated acceptance tests - agile alliance · use executable acceptance criteria as a...

http://thoughtworks-studios.com/

Maintainable AutomatedAcceptance Tests

Jez Humble

Wednesday, January 18, 12

support programming critique the product

technology facingbusiness facing

or

or

Brian Marick

http://www.exampler.com/old-blog/2003/08/21/#agile-testing-project-1Wednesday, January 18, 12

critique the product

technology facing

business facing

unit tests

functional tests acceptance tests

showcasesexploratory testing

usability testing

performance testssecurity tests

component testssystem testssu

ppor

t pro

gram

min

g

Wednesday, January 18, 12

UI

Service

Unit

Mike Cohn: Succeeding with Agile

Wednesday, January 18, 12

End to EndBusiness Facing

Localized Technology Facing

Wednesday, January 18, 12

“BDD is a second-generation, outside-in, pull-based, multiple-stakeholder, multiple-scale, high-automation, agile

methodology.” - Dan North

behavior-driven development

Wednesday, January 18, 12

describes system behaviour

uses plain English, with business vocabulary

Given... , When... , Then...

helps ‘business’ and ‘technology’ communicate

automated tests

Wednesday, January 18, 12

Acceptance Criteria"Given ...When ...Then ..."

Test Implementation LayerCode uses domain language; no

reference to UI elements.

Application Driver LayerUnderstands how to interact with the application to perform actions

and return results.

Wednesday, January 18, 12

Wednesday, January 18, 12

Wednesday, January 18, 12

Wednesday, January 18, 12

Acceptance Criteria"Given ...When ...Then ..."

Test Implementation LayerCode uses domain language; no

reference to UI elements.

Application Driver LayerUnderstands how to interact with the application to perform actions

and return results.

Customer Tester

Developer Tester

Developer

Wednesday, January 18, 12

Add a guard to prevent it happening again

Triage to find root cause1. There was an environmental problem2. There is a bug with the test3. An assumption changed4. The test actually caught a bug

Fix the problem

Optimise your process for time to fix tests

Optimise your test suite to detect failures fast

When acceptance tests break

Wednesday, January 18, 12

#1301As a... I want... So that...

Given... Given...When... When...Then... Then...

#1302As a... I want... So that...

Given... Given...When... When...Then... Then...

#1303As a... I want... So that...

Given... Given...When... When...Then... Then...

#1304As a... I want... So that...

Given... Given...When... When...Then... Then...

#1305As a... I want... So that...

Given... Given...When... When...Then... Then...

#1306As a... I want... So that...

Given... Given...When... When...Then... Then...

#1307As a... I want... So that...

Given... Given...When... When...Then... Then...

#1308As a... I want... So that...

Given... Given...When... When...Then... Then...

#1309As a... I want... So that...

Given... Given...When... When...Then... Then...

#1310As a... I want... So that...

Given... Given...When... When...Then... Then...

#1311As a... I want... So that...

Given... Given...When... When...Then... Then...

#1312As a... I want... So that...

Given... Given...When... When...Then... Then...

#1313As a... I want... So that...

Given... Given...When... When...Then... Then...

#1314As a... I want... So that...

Given... Given...When... When...Then... Then...

#1315As a... I want... So that...

Given... Given...When... When...Then... Then...

#1316As a... I want... So that...

Given... Given...When... When...Then... Then...

#1317As a... I want... So that...

Given... Given...When... When...Then... Then...

#1318As a... I want... So that...

Given... Given...When... When...Then... Then...

Wednesday, January 18, 12

Buy Product

Search product catalogueAdd product to cartCheck outCreate new accountProvide address detailsProvide credit card detailsComplete orderVerify order createdVerify credit card debitedVerify email sent

#1612As a customerI want a gift wrapping optionSo that I don’t have to wrap them and post them myself Buy Product

Search product catalogueAdd product to cartCheck outCreate new accountProvide address detailsProvide credit card detailsSelect gift wrapping optionComplete orderVerify order createdVerify gift wrapping optionVerify credit card debitedVerify email sent

Scenario tests

Wednesday, January 18, 12

Intermittent

Flakey

Flakey

Non-Deterministic

Wednesday, January 18, 12

Flakey

Non-Deterministic

Useless=Wednesday, January 18, 12

Flakey

Non-Deterministic

Useless=worse than

Wednesday, January 18, 12

Wednesday, January 18, 12

Wednesday, January 18, 12

Wednesday, January 18, 12

Wednesday, January 18, 12

Wednesday, January 18, 12

Quarantine

Limit!

Wednesday, January 18, 12

• Interference caused by dependencies• Asynchrony• Remote services• Time• Resource leaks

Causes

http://martinfowler.com/articles/nonDeterminism.html

Wednesday, January 18, 12

Application reference data

Test-specific data

Test reference data

Don’t use production data dumps (except for performance testing and staging)

Ensure tests are independent from each other

Test data

Wednesday, January 18, 12

Don't separate test and dev teams

People with testing capability specify criteria; people with programming capability build test fixture

Use record/playback only for limited cases

Version control tests together with production code

Use mainstream programming environments for your tests

Put time and effort into test data management - don’t use production data

Treat failures seriously

Handle new features iteratively not incrementally

Treat test code as codeUse executable acceptance criteria as a communication mechanism with business

Devs can run a single test or a suite on their dev machine

Build your own abstractions for acceptance criteria

Choosing between UI and subcutaneous tests depends on the architecture of your application

Build tests for parallelizationFix flaky tests

Wednesday, January 18, 12