specification by example and agile acceptance testing

Specification by Example and Agile Acceptance

Testing Bridging the communication gap in

software projectsGojko Adzic

gojko@gojko.net@gojkoadzic

http://gojko.net

Why should you care (PM/BA)?

• Developers will actually read the specifications

• They will understood the stuff correctly

• They will not skip parts of the specs

• You can track the development progress

• Save time on acceptance/smoke testing

Why should you care (dev)?

• Requirements will be unambiguous and without functional gaps

• Business analysts will really understand those special cases you mentioned

• You will have automated tests to guide development

• It will be easier to take-over and hand-over code

Why should you care (QA)

• Finally stop those guys from making the same mistakes over and over

• Avoid doing the same stuff all the time

• Build quality in from the start

• Verify business rules by a click on a button

http://www.flickr.com/photos/lambdachialpha/157986473/

http://www.flickr.com/photos/mulesafpilot/3513588967

http://www.defenseimagery.mil/assetDetails.action?guid=68cf92e35ce13e6c7c9c066f0b48b6daaa9bf8d8

An experiment with four active battalions in US Army

“Commander expectations matched actions in only 34% of the cases”

L.G.Shattuck, 2000http://www.au.af.mil/au/awc/awcgate/milreview/shattuck.pdf

http://www.flickr.com/photos/mataniere/3107073262

The process is very much like a telephone game

How many points are there?

How many points are there?

“Just-in-case code is the biggest source of waste

in software development”

Mary and Tom Poppendieck

Lean Software Development

B2 bomber crashed and $2bn went up in flames

"the aircraft actually performed as it was designed. In other words, all the systems were

functioning normally."

Maj. Gen. Floyd L. Carpenter

http://www.foxnews.com/wires/2008Jun05/0,4670,B2Crash,00.html

http://www.flickr.com/photos/biolog/3457774800

You can't help a lot when the party is already over...

F-16 design team was asked to do the impossible -

a cheap 2.5 Mach airplane! “When asked […] why they need Mach

2 - 2.5, the answer was to be able to escape from combat. Their solution was […] providing acceleration and

maneuverability, not maximum speed.”

http://97-things.near-time.net/wiki/Seek%20the%20value%20in%20requested%20capabilities

Refuse requirements that are a solution to an unknown problem!

http://www.flickr.com/photos/sylvancatharsis/3783608640/

One of the most effective ways of testing requirements is with test cases very much like those for testing the completed system

Donald Gause and Gerald Weinberg

Exploring Requirements - 1989!

As formality increases, tests and requirements become indistinguishable.

Robert C. Martin and Grigori Melnik

Tests and Requirements, Requirements and Tests: a Mobius Strip

IEEE Software January/February Issue 2008

Key practices• Discuss real-world examples to build a

shared understanding of the domain

• Use those examples as an acceptance criteria

• Automate acceptance tests

• Focus the development on those tests

• Use the tests as a live specification to facilitate change

Better names• “Acceptance testing” is

misleading!

• Behaviours

• Executable specifications

• Examples

Jim Shore: “Describe-

Demonstrate-Develop”

A very useful way to think about acceptance tests in practice

http://www.jamesshore.com/Blog/How-I-Use-Fit.html

Iteration flow (just a suggestion)

Step 1:Building a shared

understandingof the domain

Specification workshops

• Business people, developers and QA in the same room

• Transfer the knowledge• Ensure that we all understand

the same thing

Inconsistencies and gaps are easy to spot

when you write the rules down!

Real-world examples help flush out

incorrect assumed rules find real business rules!

People have think at a more detailed level

and can't brush questions off…

People approach the same problem from

different perspectives, so this avoids

groupthink!

Step 2:Select a formal set of acceptance tests and

automate them

The Toyota Way1. Check at the source2. Inexpensive Verifications3. Test to prevent defects,

not to discover them

Save time on manually (acceptance/smoke)

testing

Verify business rules with the click of a button

Automate tests, but still keep them human-readable

And you can add pictures as well….

This is a specification:

It is also a test

This as well

And this

And this

Given a stock of prices 0.5,1.0

When the stock is traded at 2.0

Then the alert status should be OFF

When the stock is traded at 5.0

Then the alert status should be OFF

When the stock is traded at 11.0

Then the alert status should be OFF

A good acceptance test is• Focused on a single thing (rule, step..)

• Specification not a script

• Self-explanatory

• Uses the domain language

• SMART– Specific

– Measurable

– Achievable

– Relevant

– Time-bound

Step 3:Providing focus for

development

No just-in-case code

Developers will have to code exactly what was

specified… not just the rules they see

Automated testreports show where

we are…

When all the tests are green, the job is done

Step 4: Keeping in touch with

changes

Live documentation

As relevant and reliable as executable code, but much easier to read!

Previous examples help you ensure to

discuss all important edge cases.

Automated tests show straight away when

something is obsolete or broken

[tests became a]

“significant and valuable business resource”

Rick Mugridge, Doubling the Value of Automated Tests,Google Tech Talks 09/2006

Recap (PM/BA)

• Developers will have to read the specifications to implement tests

• Discussion makes sure that everyone understood the stuff correctly

• To make all tests green, they cannot skip parts of the specs

• You can track the development progress

• Save time on acceptance testing with automated verifications

Recap (dev)

• Discuss and suggest examples until the gaps and inconsistencies are flushed out

• Make sure that business analysts understood special cases by suggesting them as examples and discussing them

• Acceptance tests are a live specification/documentation for the code

Recap (QA)

• Suggest examples and discuss them to cover mistakes that people make over and over

• Automated tests help you avoid doing the same stuff all the time

• Build quality in from the start by suggesting tests that prevent problems

• Verify business rules by a click on a button

Where next?

http://gojko.net http://agiletesting.org.uk http://acceptancetesting.info

27 Nov – Agile specifications and testing exchange

30 Nov – Agile acceptance testing workshop

4 Dec – Building software that matters