Case: Exploratory Testing in an agile development organization

Johan Åtting, Sectra

@johanatting

IT Quality & Test Management, Copenhagen 12 June 2013

Developer, Tester, Project Manager, Test Manager, Support Manager, Operations Manager, Quality Manager, Line Manager, @johanatting Founded EAST: Östergötlands Association for Software Testers

Conference speaker Currently: Chief Quality Officer at Sectra

Transport, Bank & Finance, Telecom and MedTech.

19 years in the software business

WHY & HOW

we use exploratory testing in our context today

CONTEXT

13%

87% Radiology IT - RIS/PACS

Orthopedic Imaging

Rheumathology

Our mission is to increase effectiveness of healthcare, while maintaining or increasing quality in patient care.

10 Agile development teams

2 Testers + 4 Programmers per team

Development & Testing - One department

Release Test Development ...

...

... . . .

Exploratory Testing - During Sprints - Between Sprints (Cross Team Testing) - During Release Test

What is Exploratory Testing?

Exploration

Wandering

Explore: The jungles of ...

Using: Map, local guides, whip ...

To: Find lost treasures

Explore: export function

To: see how file size effects performance

Using: The image data db, ...

Explore: export function

To: collect info about design consistency

Using: the import function as Oracle

Mission / Plan / Charters

Test...

Adjust Observe

use feedback from the previous test to inform the next

WHY

Benefits of exploratory testing

• It’s agile, flexible and responsive

• Focus & time is spent on testing and not on writing test scripts

• Learning about the system step by step

• We find more bugs

HOW

1 2 3 4 5 6 7 8 9 10

Two week sprint Sp

rin

t P

lan

nin

g

Spri

nt

Dem

o

Testing

Initial test planning

Continuous test re-planning

Initial test planning: 1) Brainstorm test ideas based on:

• Sprint objective

• Quality characteristics (safety, security, performance, reliability, capability, usability…)

• Risks (safety, business, technical)

• ...or any other basis or source for test ideas

=> Documented as e.g. mind maps

Initial test planning: 2) Organize test ideas (mind map) into charters:

• Explore: Area, function, feature...

• To: discover information about...

• Using: resources, oracles, heuristics

Testing

Test...

Adjust Observe

Adjust: next step in the charter

Continuous test re-planning

Test...

Adjust Observe

Adjust: add charters & re-prioritize

When are we done?

We use test levels! (to discuss test coverage)

0 Not tested

1 Sanity Check

2 Common & Critical

3 Reasonable

4 Complex 5 Can’t think of anything more that could give us valuble information

Test Levels

Todays testing tasks progress board

Target area

Test Owner Area Status

Anna Reporting

Ben Import

Chris Export

David Printing

Ellen User roles

Fred Archiving (not started)

Greg Regression (not started)

Weekly test status meeting with all testers

Possible problems* or we suspect there will be

No problems* found or suspicions of any

Known problems*

*Problems = Preventing release or preventing further testing

Output: (from a team sprint)

• Exploratory test records

• Updated (or new) regression tests

• Test cases for the Release Test phase

• Bug reports (only unfixed bugs)

But only if needed

Release Test Development ...

...

... . . .

Exploratory Testing - During Sprints - Between Sprints (Cross Team Testing) - During Release Test

Cross Team Testing Gather all testers to test each others test

objects after every sprint.

We need to get fresh, unbiased, independent eyes on what is being developed.

Sprint Sprint Sprint Sprint

Cro

ss T

eam

Te

stin

g

Cro

ss T

eam

Te

stin

g

Cro

ss T

eam

Te

stin

g

Sprint Sprint Sprint Sprint Sprint Sprint Sprint Sprint Sprint Sprint Sprint Sprint

1 2 3 4 5 6 7 8 9 10

In the beginning of the next two week sprint Sp

rin

t P

lan

nin

g

Spri

nt

Dem

o

Pla

nn

ing

(1-2

ho

urs

)

Test

ing

(6 h

ou

rs)

Cross Team Testing

Planning (1-2 hours) 1. High level presentation of test objects

2. Planning sessions in smaller groups

Testing (6 hours) • Exploratory group testing sessions (*3)

• Debrief to the Test Owner

• Bug reporting

Time Slot Activity

9:20 Test Session #1 70min

10:30 Debriefing Session #1 15min

10:45 FIKA

11:00 Test Session #2 75min

12:15 Debriefing Session #2 15min

12:30 LUNCH

13:30 Test Session #3 75min

14:45 Debriefing Session #3 15min

15:00 -> Bug Reporting / Optional additional testing

Takeaways Cross Team Testing

• Everyone is biased

• There is a need for independent eyes

• Cross Team Testing is one solution

More info: http://vimeo.com/53218576#at=0 (Video from ÖreDev 2012) http://www.slideshare.net/Johantting/cross-team-testing-at-dev-lin2013 (Slides from DevLin 2013)

Release Test Development ...

...

... . . .

Exploratory Testing - During Sprints - Between Sprints (Cross Team Testing) - During Release Test

Release Testing Before we start Relese Testing all areas need to have a ...

Test Owner Area Status

Anna Reporting

Ben Import

Chris Export

David Printing Ellen User roles

Fred Archiving

Greg Regression

... at the end of each row.

Test Owner Area Status

Anna Reporting Ben Import Chris Export David Printing Ellen User roles Fred Archiving

Greg Regression

Start of Release Test Phase

Release Testing • Planned by each test owner

• Coordinated by a test project manager

• Mix of: Retest of new features, workflow based tests, regression tests...

• Mix of exploratory tests & scripted tests

• Test environment is freezed

Test Owner Area Status

Anna Reporting

Ben Import

Chris Export

David Printing Ellen User roles

Fred Archiving

Greg Regression

Continue Release Testing until

Summary

Benefits of exploratory testing

• It’s agile, flexible and responsive

• Focus & time is spent on testing and not on writing test scripts

• Learning about the system step by step

• We find more bugs

We do Exploratory Testing - During Sprints - Between Sprints (Cross Team Testing)

- During Release Test

Twitter: @JohanAtting Linkedin: Johan Åtting E-mail: johan.atting@sectra.com