exploratory testing basics, experiences, and future in sstc2016

Exploratory testing – basics, experiences and future

Kari Kakkonen/ [email protected]

1. What is Exploratory Testing

2. How is Exploratory Testing done?

3. Who can do it?

4. What in future?

Kari Kakkonen, Knowit

• Speaks, train, coach and mentor regularly about

• ISTQB Advanced, Foundation and Agile Testing + Knowit Quality Professional

• Quality & Test process and organization development

• Agile testing, Scrum, Kanban, Lean

• Metrics

• Leadership

• Test automation, mobile, cloud, DevOps

• Quality, Cost, Benefits

• Speaking & writing highlights

• EuroSTAR and Iqnite several times

• ASTQB in USA, OOP in Germany, TEST-IT in South-Africa, Nordic Testing Days in Estonia, Testing Days in Czech, Israel Testing Week

• Numerous times in Finland at Testing Assembly, Aalto Testing Days, Tieturi Testing, Talentum Testing Forum, Quality Assurance & Software Testing, ICT Expo, TestIT Summit, Microsoft, HP, IBM, Borland etc. events

• Testing Experience magazine, Quality and Testing magazine, Sytyke-magazine, Tietoviikko

• Education

• ISTQB Expert Level Test Management Full & Advanced Full & Agile Tester certified

• SAFe and Scrum Master certified

• SPICE provisionary assessor certified

• M.Sc, Helsinki University of Technology / Aalto-university

• Marketing studies, University of Wisconsin-Madison

• Professional achievements

• Wide spread of business domain knowledge

• Embedded, Industry, Public,

• Training, Telecom, Commerce,

• Insurance, Banking, Pension

• ISTQB Treasurer, Executive Committee 2015-

• Finnish Software Testing Board FiSTB, chairman

• TestausOSY/FAST founding member

• Knowit, Director, Quality and Competences

• Chairman of research project STX, Lappeenranta University of Technology

• Finnish Software Measurement Association FiSMA ry ex-board member

• Ranked in 100 most influential IT-persons in Finland

© Knowit Oy

Twitter: @kkakkonen

LinkedIn: fi.linkedin.com/in/

karikakkonen/

[email protected]

100+ Mobile apps

20+ Extranet services

50+ Intranet services

25+ Web stores 500+

Web sites

100+ Service design

projects

Knowit – We are known for our work We develop and grow our customers’ business.

© Copyright Knowit Oy 2015 | Confidential | Version 1.0

#1 In Quality Assurance

What it is?

© Copyright Knowit Oy 2015 | Confidential 5

CHARACTERIZATION OF EXPLORATORY TESTING

• ”Planning and execution of testing is done at the same time”

(After James Bach)

• Test cases are not necessarily documented even afterwards (Cem

Kaner)

• Testing is done iteratively piece by piece

• Continuous learning and interpretation of conceptions

• Utilization of knowledge gained from experience


EXPLORATORY TESTING - TERMS

• Adventure may go to sidetrack as

long as you come back to

mainroad again (Kaner)

• Testing area: a bunch of

functionalities

• Testing session

• Duration about ½ - 2 hours

• Time span of concentrated work is

about 20 minutes

• Getting back to work takes about

20 minutes


AN EXAMPLE: WWW.TOPSELLERS.COM, A FICTIONAL E-COMMERCE SHOP

• Test case: ”Log in. Browse the content of your shopping cart. Result:

Shopping cart is empty, because no items have been picked.”

• During testing it is noticed that the shopping cart contains items! Items

have been picked with the same user account and the e-commerce

shop keeps the earlier picked things in shopping cart.

• Aftertaste: ”The test should have considered this. Let’s change the test

data and the test itself, and design a new test case, which takes into

account that the shopping cart can store items.”

• A familiar situation?


LEARNING IN EXPLORATORY TESTING

Testing

Opinion-forming

Reporting

Designing actions

Observations

After reference: Psychology of Usability, Sinkkonen et al.


How is it done?


Step by step

© Copyright Knowit Oy 2015 | Confidential

Plan

• Test charter

Test session

• Notes / Log

• Bugs

Debriefing

• Dashboard

Exploratory testing in Prague – find a park

• Test

charter

• ”look for

green”

19.10.2016 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 17


• Test execution log

• Dashboard



• Defect report


TEST DESIGN

• Define the testing areas of the test object

• Divide each area to one or more test sessions

• Test charter works as a roadmap per test session

• Define test cases to be documented

• heuristic: less than 10% of all tests

• Write down test ideas and/or high level test cases


PURPOSE OF TEST CHARTER(S)

• What will be tested?

• What documents are available?

• What kind of errors are being sought?

• Tasks and what test techniques will be used?

• Targets and outputs (for example reports)

Reference: A practioner’s guide to software test design. Copeland

Ref. Exploratory testing: A multiple case study. Itkonen, Rautiainen


Charter


Area Coverage

and

working

hours

Practice

Documents Result possible

errors

Risks

R1. Customer’s

all selected

items are not

added to order,

Effect: 20

eur/buyer,

probability 5%

R2. Order can

not be

completed after

interruption, 5,

probability: ??

Main page 100%

Path coverage

(direct paths) and

the most common

(80% used) loops

10h

Scripting

with

Functional

Tester-tool

Main page display

description

document, navigation

map (COMING

FROM

DEVELOPMENT)

All pages and

the shopping

cart are

available

Shopping

cart

5h Shopping cart-

UC.doc (use case)

Shopping cart

can be used in

the same way

as a real

shopping cart

The same

product can

not be added

to shopping

cart several

times

Emptying the

shopping cart

causes an

execption

R1

Order ? Order-UC.doc? R2

Charter as an excel


DOCUMENTS SUPPORTING TESTING

• Charter

• List of different testing strategies

• Lists of heuristics

• List of typical errors

• Kaner’s bug taxonomies*)

• Legal notices, standards, de facto-standards

• Requirements and design documentation of the system

• Self made description and models of the system behavior

• User guide *)

*) Reference: Testing Computer Software. Kaner et al.


**) Reference: Exploratory testing: A multiple case study. Itkonen, Rautiainen

Kaner’s bug taxonomies: (Testing computer software, p. 60 – 64)

• User interface errors

• Error handling

• Boundary-related errors

• Calculation errors

• Initial and later states

• Control Flow errors

• Errors in handling or interpreting data

• Race conditions

• Load Conditions

• Hardware

• Source and version control

• Documentation

• Testing errors


Exploratory Testing - Heuristics

• A set of heuristics can be applied when testing.

• A heuristic can guide the tester in how to perform the testing and to

evaluate the results, few examples below


Heuristics Examples

Boundaries:

Approaching the Boundary e.g. almost too big,

almost too small, at the Boundary

CRUD Create, Read, Update, Delete

Configuration variations

Varying the variables related to configuration e.g.

Screen Resolution; Network Speed, Latency, Signal

Strength; etc.

Interruptions e.g., log off, shut down, or reboot

Source: Elisabeth Hendrickson www.testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf

http://www.testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf



PERFORMANCE ATTITUDE - PROFESSIONAL WORKING

• Keep the targets of testing in mind

• You can visit bypaths but only for a moment

• Write down observations and questions about the system

• Report in a disciplined and systematical way

• During the execution, write only the most essential test cases and in

high level

• Test cases can be refined later


TESTS ARE DESIGNED DURING EXECUTION

• Define a test from a question

• Design test on the basis of charter and test

ideas

• A surprising situation may indicate an error:

Utilize the surprise effect!

• Heuristics e.g.

• ”Backwards thinking”: ”This button saves the

definition text. I wonder what other ways are

there for saving the text?”


NOTE TAKING: TEST EXECUTION LOGS

• Keep a test execution log

• Keep track of the tests carried out

• Main thing is that executed tests are noted

• You may create scripted test cases of some of the tests

• Keep the most important test cases that have been executed, which show

how the testing has been done, e.g. what values have been used

• You may record your execution

• Write down also test notes for test session post-mortem and your own

learning


Test charter and test log in mindmap (Xmind)


NOTE TAKING: DEFECT LOGS

• In defect reporting, traceability to the requirements must be

maintained so that coverage can be evaluated

• A well written error log is the best evidence of the existence of a fault

• Report a bug clearly, so that the failure can be repeated

• You may use defect reporting systems


TESTING DASHBOARD AS A TEST REPORT - AN EXAMPLE


Test area Workload Coverage Quality

level/risks

Comment

Main page !Interrupted High,

5h Very high [all

parts + stress

tests etc..]

49: 1435, 36: 1469,

42: 1501

wait for more

pictures of

user interface

Shopping

cart

!Started

High, 2h (reserved

4h)

Low [main parts

to testing] (High)

81: 1425

[probability 9 x

effect 9; error

number. 1425]

Order !Done

6h (reserved 4h) High [all parts]

Feedback !Not done

Low (reserved 1h) Low [main parts

to testing]

MEASURING EXPLORATORY TESTING

• The duration of the session

• The relative change in the number of test cases by the same tester

• Coverage of testing per session

• The number of interruptions (Suspension criteria)

• Number of rejected defects in defect database

• …

• Metrics are the eyeglasses of testing that you need in order to be fully

aware of the situation and potential problems in testing

It is recommended that you choose metrics that are suitable for the

challenges of exploratory testing


Some Tools

• Notetaking: • Screenshots With Annotation

• Video with Annotation

• Intergrated bug reporting

• HP Sprinter

• QA Symphony qTest eXplorer

• Telerik Test Studio Explore

• Bug Magnet

• Notepad++ (pad tools in general)

• Xmind (mindmap tools in general)

• Rapid Reporter

• Pivotal Tracker

• Test charter planning • Task and backlog tools

• Test charter planning

• Trello

• Jira Agile

• HP Agile

• Excel


Who can do it?


WHO TO RECRUIT? THE PROFILE OF AN EXPLORATORY TESTER

• Exploratory testing is particularly well suitable for a person, who…

• likes to take risks

• is not afraid of changes or new things

• is open-minded

• sets challenges and goals to themselves

• is smart and quick in finding test conditions

• ”Pioneer-style”

• Everyone can learn to be an exploratory tester

Reference: Choosing and Managing the Ideal Test Team. Lloyd Roden


EXPLORATORY TESTING REQUIRES LEARNING SKILLS

• Outline the functionality of the system on paper

• Aim at understanding

• Don’t force yourself to remember facts, use documents

• Ask questions about the functionality of the system

• Recognize the items on which you need more information

Reference: Tutkiva oppiminen. Hakkarainen et al.


IMPROVE YOUR TESTING THINKING SKILLS

• Observe your own testing habits

• Recognize your own ways of thinking

• Learn from misunderstandings and mistakes

• Select the testing techniques according to the situation *)

• Improve your skills to deduce the states of the system *)

Adapted from the reference: Tutkiva oppiminen. Hakkarainen et al.,

*) Reference: Rapid Software Testing. James Bach


What in future?


Exploratory future

• Exploratory testing will become the norm in agile testing

• And agile will be the norm for all development

• Coverage will be proven with both last quantities of well-logged

exploratory testing and some formally designed test cases

• With test automation to make sure the basics work, testers will be free

to find important defects with exploratory testing

• People will start exploring also with test automation

• Keyword driven and data-driven approaches

© Copyright Knowit Oy 2015 | Confidential

Thank you

@kkakkonen [email protected]