exploratory testing explained

MJ Half-day Tutorials

5/5/2014 8:30:00 AM

Exploratory Testing

Explained

Presented by:

Paul Holland

Testing Thoughts

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Paul Holland Testing Thoughts

An independent software test consultant and teacher, Paul Holland has more than sixteen years of hands-on testing and test management experience, primarily at Alcatel-Lucent where he led a transformation of the testing approach for two product divisions, making them more efficient and effective. As a test manager and tester, Paul focused on exploratory testing, test automation, and improving testing techniques. For the past five years, he has been consulting and delivering training within Alcatel-Lucent and externally to companies such as Intel, Intuit, Progressive Insurance, HP, RIM, and General Dynamics. Paul teaches the Rapid Software Testing course for Satisfice. For more information visit testingthoughts.com.

Copyright © 1996-2013, Satisfice, Inc.

dddd 1

1

1


Paul Holland, Doran Jones, Inc.

[email protected]

www.doranjones.com

My Background� Managing Director, Testing Practice at Doran Jones

� Independent S/W Testing consultant 4/2012 - 3/2014

� 16+ years testing telecommunications equipment and

reworking test methodologies at Alcatel-Lucent

� 10+ years as a test manager

� Presenter at STAREast, STARWest, Let’s Test,

EuroSTAR and CAST

� Facilitator at 50+ peer conferences and workshops

� Teacher of S/W testing for the past 6 years

� Teacher of Rapid Software Testing

� Military Helicopter pilot – Canadian Sea Kings


dddd 2

2

Let’s Do ET! Ready…?

3

The Roadmap

Want to learn how to test?

OKAY LET’S GO!!!

4

(To do ET well, begin by doing it poorly.)


dddd 3

3

Acquiring the competence, motivation,

and credibility to…

Testing is…

create the conditions necessary to…

…so that you help your clients to make

informed decisions about risk.

evaluate a product by learning

about it through experimentation, which includes to

some degree: questioning, study, modeling,

observation and inference, including…

operating a product

to check specific

facts about it…

Call this “Checking” not Testing

Observe Evaluate Report

Interact with the

product in specific

ways to collect

specific

observations.

Apply algorithmic

decision rules to

those

observations.

Report any

failed checks.

means

operating a product

to check specific

facts about it…


dddd 4

4

Tacit

Test ProceduresConsistency

Oracles

Prospective Testing

Learning and

Teaching

Commitment

Management

(inc. estimation)

Recruiting

Helpers

Managing Testing

Logistics

Test Tooling and

Artifact Development

Test Framing

Bug

Advocacy

& Triage

Project

Post Mortem

Creating Archival

Documentation

Guiding Helpers

Discovery of

Curios, Issues &

Risks

Building the

Test Team

Designing

Checks and Tests

Playing with

the Product

Studying

Results

Galumphing

Configuring

Product & Tools

Schedule

Management

Study

Customer

Feedback

Relationship

Building

Making Failure

Productive

Sympathetic

Testing

Maintaining Personal

Health and Motivation

Team

Leadership

Quasi-Functional

Testing

Playing

Programmer

Testing w/Simulated

Conditions

Testing a Simulation

Creating the Test Lab

Studying Specs

Managing Records

Playing

Business Analyst

Opposition

Research

Testability Advocacy

Cultivate Credibility

Testing vs. Checking

�TESTING (think “what testers

do”): the process of evaluating a product by learning about it

through experimentation, which includes to some degree:

questioning, study, modeling, observation and inference.

�CHECKING (think “fact

checking”): the process of making evaluations by applying algorithmic

decision rules to specific observations of a product.


dddd 5

5

10

Exploratory Testing Is…

� an approach to testing…

� that emphasizes the personal freedom and

responsibility of each tester to continually

optimize the value of his work…

� by treating learning, test design, test

execution and result evaluation as mutually

supportive activities that run in parallel

throughout the project.

(applicable to any test technique)

(optimize how?)


dddd 6

6

Exploration is Not Just Actionarrows and cycles

You can put them together!arrows and cycles

(value seeking)

(task performing)


dddd 7

7




dddd 8

8

QUIZ

� Testing without a specification

� Testing with a specification

� Testing without instructions

� Testing with instructions

� Testing a product for the first time

� Re-testing after a change

� Informal testing

� Formal testing

� Playing… Just playing…

� Fully automated fact checking 15

Which of

these

are

exploratory

testing?

16

The Testing Formality Continuum

Mixing Scripting and Exploration

When I say “exploratory testing” and don’t qualify it,

I mean anything on the informal side of this continuum.

INFORMAL FORMALNot done in any specific way,

nor to verify specific facts.

Done in a specific way,

or to verify specific facts.

Machine

Checking

Human

Checking

Vague/Generic

Test Scripts

“Human

Transceiver”

Matrix/Outline

of Test Conditions

Product

Coverage

OutlinePlaySpecific

Test Data

Survey

Exploratory

Analytical

Exploratory


dddd 9

9

Common Questions in ET

� How do we manage ET?

� How do we record ET?

� How do we measure ET?

� How do we estimate ET?

� How do we learn ET?

� How do we defend ET?

17

“How do we do X with ET that we

could never do with scripted testing

either but nobody noticed?”

18

Tests

Project

Environment

Product

Elements

Quality

Criteria

Perceived

Quality

A Heuristic Test Strategy Model


dddd 10

10

19

Project

Environment

Tests

Product

Elements

Quality

Criteria

Perceived

Quality

A Heuristic Test Strategy Model

This is what people think you do

described actual

“Compare the product to its specification”

20


dddd 11

11

This is more like what you really do

imagined

actualdescribed

“Compare the idea

of the product to

a description of it”

“Compare the actual product to a

description of it”

“Compare the idea

of the product to

the actual product”

21

This is what you find…The designer INTENDS the product

to be Firefox compatible,

but never says so,

and it actually is not.

The designer INTENDS

the product to be Firefox

compatible,

SAYS SO IN THE SPEC,

but it actually is not.

The designer assumes

the product is not

Firefox compatible,

and it actually is not, but the

ONLINE HELP SAYS IT IS.

The designer

INTENDS

the product to be

Firefox compatible,

SAYS SO,

and IT IS.


the product is not

Firefox compatible,

but it ACTUALLY IS, and the

ONLINE HELP SAYS IT IS.

The designer INTENDS the product

to be Firefox compatible,

MAKES IT FIREFOX COMPATIBLE,

but forgets to say so in the spec.


the product is not

Firefox compatible,

and no one claims that it is,

but it ACTUALLY IS.

Imagined

Described Actual


dddd 12

12

23

Oracles

An oracle is a way to recognize

a problem that appears during testing.

“...it appeared at least once to meet some

requirement to some degree.”

“...it works”

24

Consistency (“this agrees with that”)

an important theme in oracles� Familiarity: The system is not consistent with the pattern of any familiar problem.

� Explainability: The system is consistent with our ability to explain it.

� World: The system is consistent with objects and states, in the world, that it represents.

� History: The present version of the system is consistent with past versions of it.

� Image: The system is consistent with an image that the organization wants to project.

� Comparable Products: The system is consistent with comparable systems.

� Claims: The system is consistent with what important people say it’s supposed to be.

� Users’ Expectations: The system is consistent with what users want.

� Product: Each element of the system is consistent with comparable elements in the same system.

� Purpose: The system is consistent with its purposes, both explicit and implicit.

� Statutes & Standards: The system is consistent with applicable laws and standards.


dddd 13

13

Familiar Problems

If a product is consistent with problems we’ve seen before,

we suspect that there might be a problem.

Explainability

If a product is inconsistent with our ability to explain it

(or someone else’s), we suspect that there might be a problem.


dddd 14

14

World

If a product is inconsistent with the way the world works,


History

If a product is inconsistent with previous versions of itself,


Okay,

so how the #&@

do I print now?


dddd 15

15

Image

If a product is inconsistent with an image that

the company wants to project, we suspect a problem.

Comparable Products

WordPad Word

When a product seems inconsistent with a product that is

in some way comparable, we suspect that there might be a problem.


dddd 16

16

Claims

When a product is inconsistent with claims that important

people make about it, we suspect a problem.

User Expectations

When a product is inconsistent with expectations that a

reasonable user might have, we suspect a problem.


dddd 17

17

Purpose

When a product is inconsistent with its designers’ explicit

or implicit purposes, we suspect a problem.

Product

When a product is inconsistent internally—as when it

contradicts itself—we suspect a problem.


dddd 18

18

Statutes and Standards

When a product is inconsistent with laws or widely

accepted standards, we suspect a problem.

Tacit Explicit

Oth

er

Pe

op

leTe

ste

r Your

Feelings &

Mental Models

Shared

Artifacts

(specs, tools,

etc.)

Stakeholders’

Feelings &

Mental Models

Inference

Observable

Consistencies

ReferenceConference

Experience

Oracles From the Inside Out


dddd 19

19

37

ET is a Structured Process� Exploratory testing, as I teach it, is a structured process

conducted by a skilled tester, or by lesser skilled testers or users working under supervision.

� The structure of ET comes from many sources:− Test design heuristics

− Chartering

− Time boxing

− Perceived product risks

− The nature of specific tests

− The structure of the product being tested

− The process of learning the product

− Development activities

− Constraints and resources afforded by the project

− The skills, talents, and interests of the tester

− The overall mission of testing

In other words,

it’s not “random”,

but systematic.

IP Address

38

Heuristics bring useful structure

to problem-solving skill.

� adjective:

“serving to discover.”

� noun:

“a fallible method for solving a problem or

making a decision.”

“The engineering method is the use of heuristics

to cause the best change in a poorly understood situation

within the available resources.”

-- Billy Vaughan Koen, Discussion of The Method


dddd 20

20

39

Test Design and Execution

Guide testers with personal supervision and concise

documentation of test ideas. Meanwhile, train them

so that they can guide themselves and be accountable

for increasingly challenging work.

Test

Ideas

Achieve excellent test design by

exploring different test designs

while actually testing Product

Product

or spec

40

If You are Frustrated

1. Look over your recent tests and find a pattern there.

2. With your next few tests, violate the old pattern.

3. Prefer MFAT (multiple factors at a time).

4. Broaden and vary your observations.

Boundary Testing


dddd 21

21

41

If You are Confused

1. Simplify your tests.

2. Conserve states.

3. Frequently repeat your actions.

4. Frequently return to a known state.

5. Prefer OFAT heuristic (one factor at a time).

6. Make precise observations.

Dice Game

42

Exploratory Branching:

Distractions are good!

New test

idea

New test

idea

New test

idea

New test ideas occur

continually during an

ET session.


dddd 22

22

43

“Long Leash” Heuristic

but periodically take stock

of your status against your mission

Let yourself be distracted…

‘cause you never know

what you’ll find…

44

ET is a Structured Process

In excellent exploratory testing, one structure tends to

dominate all the others:

Exploratory testers construct a compelling story of

their testing. It is this story that

gives ET a backbone.

Pen Test


dddd 23

23

To test is to construct three stories(plus one more)

Level 1: A story about the status of the PRODUCT-…about how it failed, and how it might fail...

…in ways that matter to your various clients.

Level 2: A story about HOW YOU TESTED it-…how you configured, operated and observed it…

…about what you haven’t tested, yet…

…and won’t test, at all…

Level 3: A story about the VALUE of the testing-…what the risks and costs of testing are…

…how testable (or not) the product is…

…things that make testing harder or slower…

…what you need and what you recommend…

(Level 3+: A story about the VALUE of the stories.)…do you know what happened? can you report? does report serve its purpose?

Why should I be pleased

with your work?

The Roadmap

46

� How often do you account for your progress?

� If you have any autonomy at all, you can risk investing some time in − learning

− thinking

− refining approaches

− better tests

Allow some disposable time

Self-management is good!


dddd 24

24

47

Allow some disposable time

� If it turns out that you’ve made a bad

investment…oh well

☺ If it turns out that you’ve made a

good investment, you might have

− learned something about the product

− invented a more powerful test

− found a bug

− done a better job

− avoided going down a dead end for too long

− surprised and impressed your manager

48

“Plunge in and Quit” Heuristic

� This benefits from disposable time– that part of your work not scrutinized in detail.

� Plunge in and quit means you can start something without committing to finish it successfully, and therefore you don’t need a plan.

� Several cycles of this is a great way to discover a plan.

Whenever you are called upon to test

something very complex or frightening, plunge in!

After a little while, if you are very confused

or find yourself stuck, quit!


dddd 25

25

49

The First Law of Documentation

“That should be documented.”

“That should be documented

if and when and how it serves our purposes.”

Who will read it? Will they understand it?

Is there a better way to communicate that information?

What does documentation cost you?

50

Common Problems with

Test Documentation

� Distracts, and even prevents, testers from doing the best testing they know how to do.

� Authors don’t understand testing.

� Authors don’t own format.

� Templates help hide poor thinking.

� Full of fluff.

� Fluff discourages readers and increases cost of maintenance.

� No one knows documentation requirements.

� Too much formatting increases the cost of maintenance.

� Information for different audiences is mixed together.

� Catering to rare audiences instead of probable user.

� Disrupts the social life of information.

� Long term and short term goals conflict.

� Most people don’t read.


dddd 26

26

51

What Does Rapid Testing Look Like?

Concise Documentation Minimizes Waste

Risk ModelCoverage Model Test Strategy

Reference

Risk CatalogTesting Heuristics

General

Project-

Specific Status

Dashboard

Schedule BugsIssues

52

Consider Automatic Logging

� Exploratory testing works better when the product produces an automatic log of everything that was covered in each test.

� You can also use external logging tools such as Spector (www.spectorsoft.com).

� Automatic logging means that you get something like a retrospective script of what was tested.


dddd 27

27

53

Introducing the Test Session

1) Charter

2) Uninterrupted Time Box

3) Reviewable Result

4) Debriefing vs.


dddd 28

28

Charter:

A clear mission for the session

� A charter summarizes the goal or activity of the session.

� Before a session, it is a statement of how the session will be

focused. Afterward, it should be changed, if necessary, to

reasonably characterize the actual focus of the session.

� General charters are the norm, early in the project:

“Analyze the Insert Picture function”

� More specific charters tend to emerge later on:

“Test clip art insertion. Focus on stress and flow techniques, and make

sure to insert into a variety of documents. We’re concerned about

resource leaks or anything else that might degrade performance over

time.”

Charter Patterns:

Evolving test strategy

� Intake Sessions (Goal: negotiate mission)

“Interview the project manager about testing Xmind.”

− Survey Sessions (Goal: learn product)

“Familiarize yourself with Xmind.”

� Setup Sessions (Goal: create testing infrastructure)

“Develop a library of mindmaps for testing Xmind.”

� Analysis Sessions (Goal: get ideas for deep coverage)

“Identify the primary functions of Xmind.”

“Construct a test coverage outline.”

“Brainstorm test ideas.”

“Prepare a state model for state-based testing.”

“Perform a component risk-analysis to guide further testing.”

“Discover all the error messages in Xmind.”


dddd 29

29

Charter Patterns:

Evolving test strategy

� Deep Coverage Sessions (Goal: find the right bugs)

“Perform scenario testing based on the scenario playbook.”

“Perform a tour that achieves double-transition state coverage.”

“Perform steeplechase boundary testing on the major data items.”

“Test each error message in Xmind.”

“Perform a function tour using the 2300 node mindmap.”

� Closure Sessions (Goal: get ready to release)

“Verify the latest fixes.”

“Re-test tutorial with the latest build.”

“Review help files and readme.”

“Go over deferred bugs with Customer Support people.”

“Perform clean-machine install test.”

Time Box:

Focused test effort of fixed duration

A normal session is 90 minutes long.Base timing can be adjusted based on your context

A real session may be somewhat longer or shorter.

A normal session is uninterrupted.

A real session may be somewhat interrupted.

Real sessions are “normed” for the purposes of

reporting metrics. This is so that our clients don’t

get confused by the numbers.


dddd 30

30

Example

Session 1…

Work for 20 minutes

(fifteen minute interruption)

Work for 75 minutes

(ten minute interruption)

Work for 40 minutes

(end the real session)

Session 2…

Work for 35 minutes

(twenty minute interruption)

Work for 85 minutes

(end the real session)

This results in 2 session reports; 5 hours of real time.

It would be reported as 3 normal sessions.

If a session is so interrupted that your

efficiency is destroyed, abort the session

until you get control of your time.


dddd 31

31

Debriefing:

Measurement begins with observation

� The manager reviews session report to

assure that he understands it and that it

follows the protocol.

� The tester answers any questions.

� Session metrics are checked.

� Charter may be adjusted.

� Session may be extended.

� New sessions may be chartered.

� Coaching happens in both directions.

Reviewable Result:

A scannable session sheet

� Charter

− #AREAS

� Start Time

� Tester Name(s)

� Breakdown

− #DURATION

− #TEST DESIGN AND EXECUTION

− #BUG INVESTIGATION AND REPORTING

− #SESSION SETUP

− #CHARTER/OPPORTUNITY

� Data Files

� Test Notes

� Bugs

− #BUG

� Issues

− #ISSUE

CHARTER

-----------------------------------------------

Analyze MapMaker’s View menu functionality and

report on areas of potential risk.

#AREAS

OS | Windows 2000

Menu | View

Strategy | Function Testing

Strategy | Functional Analysis

START

-----------------------------------------------

5/30/00 03:20 pm

TESTER

-----------------------------------------------

Jonathan Bach

TASK BREAKDOWN

-----------------------------------------------

#DURATION

short

#TEST DESIGN AND EXECUTION

65

#BUG INVESTIGATION AND REPORTING

25

#SESSION SETUP

20


dddd 32

32

Reviewable Result:

Various tools are now available.

� Rapid Reporter

� SessionWeb

� SBTExecute

� Atlassian: Bonfire

� Atlassian: Jira

The Breakdown Metrics

Testing is like looking for worms

Test Design and Execution

Bug Investigation and Reporting

Session Setup


dddd 33

33

Challenges of SBTM

� Architecting the system of charters (test planning)

� Making time for debriefings

� Getting the metrics right

� Creating good test notes

� Keeping the technique from dominating the testing

� Managing chronic interruptions

For example session sheets and metrics see

http://www.satisfice.com/sbtm

Whiteboard

� Used for planning and tracking of test execution

� Suitable for use in waterfall or agile (as long as you have control over your own team’s process)

� Use colours to track:− Features, or

− Main Areas, or

− Test styles (performance, robustness, system)


dddd 34

34

Whiteboard

� Divide the board into four areas:− Work to be done

− Work in Progress

− Cancelled or Work not being done

− Completed work

� Red stickies indicate issues (not just bugs)

� Create a sticky note for each half day of work (or mark # of half days expected on the sticky note)

� Prioritize stickies daily (or at least twice/wk)

� Finish “on-time” with low priority work incomplete

Whiteboard Example

End of

week 1

Out of 7

weeks


dddd 35

35

Whiteboard Example

End of

week 6

Out of 7

weeks

Reporting

� An Excel Spreadsheet with:− List of Charters

− Area

− Estimated Effort

− Expended Effort

− Remaining Effort

− Tester(s)

− Start Date

− Completed Date

− Issues

− Comments

� Does NOT include pass/fail percentage or number of test cases


dddd 36

36

Sample Report

Charter AreaEstimated

Effort

Expende

d Effort

Remainin

g Effort Tester

Date

Started

Date

Completed

Issues

Found Comments

Investigation for high QLN spikes on

EVLT

H/W

Performance 0 20 0 acode 12/10/2011 01/14/2012 1617032

Lots of investigation. Problem was on 2-3

out of 48 ports which just happened to be

2 of the 6 ports I tested.

ARQ Verification under different RA

Modes ARQ 2 2 0 ncowan 12/14/2011 12/15/2011

POTS interference ARQ 2 0 0 --- 01/08/2012 01/08/2012

Decided not to test as the H/W team

already tested this functionality and time

was tight.

Expected throughput testing ARQ 5 5 0 acode 01/10/2012 01/14/2012

INP vs. SHINE ARQ 6 6 0 ncowan 12/01/2011 12/04/2011

INP vs. REIN ARQ 6 7 5 jbright 01/06/2012 01/10/2012

To translate the files properly, had to

install Python solution from Antwerp.

Some overhead to begin testing

(installation, config test) but was fairly

quick to execute afterwards

INP vs. REIN + SHINE ARQ 12 12

Traffic delay and jitter from RTX ARQ 2 2 0 ncowan 12/05/2011 12/05/2011

Attainable Throughput ARQ 1 4 0 jbright 01/05/2012 01/08/2012

Took longer because was not behaving as

expected and I had to make sure I was

testing correctly. My expectations were

wrong based on virtual noise not being

exact.

Weekly Report

� A PowerPoint slide indicating the

important issues (not a count but a list)

− “Show stopping” bugs

− New bugs found since last report

− Important issues with testing (blocking bugs,

equipment issues, people issues, etc.)

− Risks (updates and newly discovered)

− Tester concerns (if different from above)

− The slide on the next page indicating progress


dddd 37

37

Sample Report

0

10

20

30

40

50

60

70

80

90

ARQ SRA Vectoring Regression H/W Performance

Effort(person half days)

Feature

"Awesome Product" Test Progress as of 02/01/2012

Original Planned Effort

Expended Effort

Total Expected Effort

Direction of lines indicates

effort trend since last report

Solid centre bar=finished

Green: No concerns

Yellow: Some concerns

Red: Major concerns

74



dddd 38

38

75


Appendix A

Reporting the TBS BreakdownA guess is okay, but follow the protocol

� Test, Bug, and Setup are orthogonal categories.

� Estimate the percentage of charter work that fell into each

category.

� Nearest 5% or 10% is good enough.

� If activities are done simultaneously, report the highest

precedence activity.

� Precedence goes in order: T, B, then S.

� All we really want is to track interruptions to testing.

� Don’t include Opportunity Testing in the estimate.


dddd 39

39

Activity HierarchyAll test work fits here, somewhere

all work

non-session

session

opportunity on charter

test bug setup

inferred

Non-Session

61%

Test

28%

Bug

4%

Opportunity

1%

Setup

6%

Work Breakdown:Diagnosing the productivity

� Do these proportions make sense?

� How do they change over time?

� Is the reporting protocol being followed?

0.0

50.0

100.0

150.0

200.0

250.0

300.0

5/26 6/9 6/23 7/7 7/21 8/4 8/18


dddd 40

40

Coverage:Specifying coverage areas

� These are text labels listed in the Charter

section of the session sheet. (e.g. “insert

picture”)

� Coverage areas can include anything

− areas of the product

− test configuration

− test strategies

− system configuration parameters

� Use the debriefings to check the validity of

the specified coverage areas.

Coverage:

Are we testing the right stuff?

� Is it a lop-sided set of

coverage areas?

� Is it distorted

reporting?

Distribution of On Charter Testing

Across Areas

0

20

40

60

80

100

120

• Is this a risk-based test

strategy?


dddd 41

41

Using the Data

to Estimate a Test Cycle

1. How many perfect sessions (100% on-charter testing) does

it take to do a cycle? (let’s say 40)

2. How many sessions can the team (of 4 testers) do per day?

(let’s say 3 per day, per tester = 12)

3. How productive are the sessions? (let’s say 66% is on-

charter test design and execution)

4. Estimate: 40 / (12 * .66) = 5 days

5. We base the estimate on the data we’ve collected. When

any conditions or assumptions behind this estimate

change, we will update the estimate.

exploratory testing explained

Technology

exploratory testing

checking testing

credibility testing

specification testing

formal testing

testing approach

testing practice

quiz testing