put risk based testing in place right now!

23/11/2016 1

Eric RIOU du COSQUERMinsk, November 24th 2015

Put Risk Based Testing in place, right now!

23/11/2016 2

You took the risk to attend my presentation

Busines Analyst / Product Owner

Project Manager

Test Manager

Functional or Technical Tester

Software Quality and Testing consultant

Sales person

About you

23/11/2016 3

Eric RIOU du COSQUER, erdc@certilogtest.com

• Business Analysis www.iqbba.org• Member of the executive committee

• Requirements Engineering www.reqb.org• Member of the executive committee

• International Software Testing www.istqb.org • General Secretary from 2011 to 2015, France

Representative afterwards

• French Software Testing Qualification Board www.cftl.fr • Manager since 2013

• Test organizations assessment www.tmmi.org • Lead Assessor since 2015

About me

23/11/2016 4

The goal is to explain how to implement a Risk Based Testing approach based on PRISMA® (Product RIsk MAnagement)

Introduction

Risk Management Basics

RBT approach

What next?

Summary

Agenda

23/11/2016 5

Testing, Risk, and Risk Based Testing

Introduction

23/11/2016 6

Main activities (after ISTQB)

What is testing ?

Planning

Cont

rol

Closure

Acceptance

System

Integration

Component 1

Analysis andDesign

Implementation and Execution Evaluation &

Reporting

Planification

Closure

Cont

rol

23/11/2016 7

Definitions (ISTQB)

Risk• A factor that could result in future negative

consequences; usually expressed as impact and likelihood

Product Risk• A risk directly related to the test object

Project Risk• A risk related to management and control of the (test)

project, e.g. lack of staffing, strict deadlines, changing requirements…

What is a risk ?

23/11/2016 8

Definition

Risk Based Testing• An approach to testing to reduce

the level of product risks and inform stakeholders of their status (…). It involves the identification of product risks and the use of risk levels to guide the process

What is « RBT » ?(Risk Based Testing)

23/11/2016 9

A general risk management approach applied to product risks

Risk Management Basics

23/11/2016 10

A process with 4 main activities

Risk Management

Risk assessment

Identification

Analysis

Risk control

Mitigation

Monitoring

What does the general risk management approach consist in ?

23/11/2016 11

The result is a list of risks

• Advice: 30 risks max !

1/4 Risk Identification

Risks TypeRisk 1 Fonctionnal

Risk 2 Security

Risk 3 Fonctionnal

Risk 4: Reliability

…

23/11/2016 12

Define Likelihood and Impact for each risk, and then a risk level

• Risk Level = Probability * Impact

2/4 Risk Analysis

Risks Type Likelihood Impact LevelRisk 1 Fonctionnal

Risk 2 Security

Risk 3 Fonctionnal

Risk 4: Reliability

… … … … …

23/11/2016 13

The risk level calculation may be supported by a table

2/4 Risk Analysis

23/11/2016 14

Implement actions to reduce the risks

• Four mains options1. Mitigate the risk through preventive measures to reduce likelihood

and/or impact2. Make contingency plans to reduce impact if the risk becomes an

actuality3. Transfer the risk to some other party to handle4. Ignore and accept the risk, which means doing nothing but wait and

see whether the problem occurs or not.

• Mitigation with testing• Associate test cases to the risks

3/4 Risk Mitigation

23/11/2016 15

Periodically review the risk status , identify new risks and communicate

4/4 Risk monitoring

Risks Type Proba. Impact Action Status LevelRisk 1 Fonctionnal

Risk 2 Security

Risk 3 Fonctionnal

Risk 4:

Reliability

… … … … …

New Risk

23/11/2016 16

A practical approach, step by step

RBT approachbased on PRISMA®(Product RISk Management)

23/11/2016 17

The decision to implement an RBTapproach must be made

#1RBT Selection

23/11/2016 18

Possible insights

Exhaustive testing is impossible

The allocated test design and execution time and budget is always reduced

The specifications and requirements may not cover the overall set of expected caracteristics

The quality and success of a product depend on the final users and customers view

How to (be) convice(d) to implement an RBT approach ?

23/11/2016 19

The right people to be involved mustbe identified

#2Stakeholders identification

23/11/2016 20

The Test Manager must select different kind of stakeholders

Who should be involved in the RBT process ?

On the vendor side

On the customer

side

• End user (client of the customer)• Other organizations (regulatory entities,

…)

• Customer representatives (called “Business”)

• Project sponsors• End users (from the customer company)• Installation and Operations personnel• Testers and Quality Assurance staff

• Project managers• Business and System Analysts• Developers and architects• DBA• GUI designers• Technical writers• Testers and Quality Assurance staff

23/11/2016 21

PRISMA provides a checklist for stakeholders identification

Who should be involved in the RBT process ?

- Project manager - Business experts- Designers - Testers- Client / sponsor - End users - Usability experts - Operations- Maintenance team - Security - Safety services - Inspectors- Support / helpdesk - Manufacturing- Marketing - Legal- Professional bodies - Special interest groups- Technology experts - Marketing- Customers - System development- Quality assurance - Regulatory bodies

23/11/2016 22

A first list of risks must be created

#3Risk identification

23/11/2016 23

Different techniques can be combined

How to involve the selected stakeholders in the risk identification ?

• Requirements based

• Interviews

• Workshops and Brainstorming sessions

Risks TypeRisk 1 Fonctionnal

Risk 2 Security

Risk 3 Fonctionnal

Risk 4: Reliability

…

Same result as above

23/11/2016 24

The initial set of product risks mustbe improved

#4#4 Risk triage or extended identification

23/11/2016 25

Review the list and check against requirements

• Remove the less relevant risk from the list

• What to do with• A risk but no requirement• A requirement but no risk

How to keep the most relevant risks in the list ?

Product Risk Requirement

ID Product Risk Risk Type Requirement01 Customer cannot start the

transaction at another bankFunctionality Customer shall be able to

perform a transaction at another bank

02 Customer not issued with receipt at the end of the transaction

Functionality Customer shall receive a receipt at the end of the transaction

03 The system is unavailable to the customer for longer than two hours

Reliability System shall be available to customers 24/7

……

Example of a set of product risks for an after Pinkster]

23/11/2016 26

The impact of each risk needs to be rated

#5Impact Rating

23/11/2016 27

PRISMA® suggested factors

1. Critical areas (damage, cost and consequences of failure)2. Visible areas (external visibility of a failure)3. Most used areas4. Business importance5. Cost of rework

Which factors shall we consider to rate the impact ?

Impact

Factor Criticity Visibility …

Weight 2 1 …

Risk 1 5 3 …

Risk 2 3 5 …

Risk 3 3 2 …

… … … …

23/11/2016 28

The likelihood of each risk needs to be rated

#6Likelihood Rating

23/11/2016 29

PRISMA® suggested factors1. Complexity2. Size3. Number of changes4. New technology and methods5. Inexperience6. New development vs. re-use7. Interfacing8. …

Which factors shall we consider to rate the likelihood ?

Impact LIkelihood

Factor Criticity Visibility … Complexity Size …

Weight 2 1 … 1 2 …

Risk 1 5 3 … 3 5 …

Risk 2 3 5 … 4 1 …

Risk 3 3 2 … 2 4 …

… … … … … … …

23/11/2016 30

Once impact and likelihood are scored, the risks are included in a Matrix

#7Risk Matrix creation

23/11/2016 31

Impact and Likelihood are scored for each risk

• Each risk may be rated by different profiles• Impact: business skills• Likelihood: technical skills

How to visualize the risk distribution ?

Impact Probabilité

Factor Criticity Visibility VALUE Complexity Size VALUE

Weight 2 1 na 1 2 na

Risk 1 5 3 13 3 5 13

Risk 2 3 5 11 4 1 6

… … … … … … …

23/11/2016 32

Each risk will be positioned in a matrix

What is the Product Risk Matrix ?

IIV

II IIII

IIIII

Like

lihoo

d of

Def

ects

(T

echn

ical

Ris

ks)

Impact of Defects(Business Risks)

33

15

15

R1

R2

R3

R4R5

23/11/2016 33

IIV

Consider the following advice1. Avoid the central circle2. Try not to have all the risks in the same areas3. Add a fifth area for safety-critical applications

How to ensure a right distribution of the risks ?

IIV

II IIII

IIIII

Like

lihoo

d

Impact33

15

15

R1

R2

R4R5

R5 R7

R6

23/11/2016 34

The test approach will be basedon the risk distribution

#8Test approach and Test techniques selection

23/11/2016 35

Impact and Likelihood help you focus on the right level(s)

How to allocate the test effort on the different levels ?

IIV

II IIII

IIIII

Like

lihoo

d

Impact33

15

15

component and Integration leveltest (focus on technical risk)

systemand acceptance level test (focus on business risk)

23/11/2016 36

This question should be adressed for each test level

How to select the right techniques and define the associated coverage goals ?

IIV

II IIII

IIIII

Like

lihoo

d

Impact33

15

15

Example for the component level

Decision coverage

(90%)

Code inspection

Instruction coverage

(90%)

Instruction coverage

(70%)

23/11/2016 37

This question should be adressed for each test level

How to select the right techniques and define the associated coverage goals ?

IIV

II IIII

IIIII

Prob

abili

té

Impact33

15

15

Use Case(incl alternative

paths)

Decision table

Use Case(main path)

Equivalence partitioning

Use Case(incl alternative

paths)

Equivalence partitioningUse Case

(main path

Exploratory testing

Example for the acceptance level

23/11/2016 38

The traceability from risks to test casesis implemented

# 9Test Design… and Execution

23/11/2016 39

Use the traceability

How to reach the final Risk Based Test Execution step ?

Product Risk Requirement Test CasesTest

Execution Results

Defects

23/11/2016 40

The risk likelihood and impactmust be reviewed based onthe test execution results

#10Risk Based reporting and Defect correction

23/11/2016 41

Update it !

What to do with the Product Risk Matrix

Product Risk Requirement Test CasesTest

Execution Results

DefectsDefects Likelihood is increased

Passed test cases Likelihood is decreased

New risks ?

23/11/2016 42

Increase your knowledge in RBT and implement it right now!

What next ?

23/11/2016 43

And at any time!

RBT is everywhere in the test process

23/11/2016 44

The Best seller about RBT

• ISBN 9789490986070

Sources

23/11/2016 45

With your own Excel file or the PRISMA® tool

The method can be tooled

23/11/2016 46

ISTQB Advanced Level Test Analyst Syllabus

Additional Sources

• http://www.istqb.org/downloads/send/10-advanced-level-syllabus-2012/53-advanced-level-syllabus-2012-test-analyst.html

23/11/2016 47

TMMi• http://

www.tmmi.org/wp-content/uploads/2016/09/TMMi.Framework.pdf

Additional Sources

23/11/2016 48

Eric RIOU du COSQUERerdc@certilogtest.comwww.certilogtest.com

Thank you !