risk-based testing – an overview

Risk-Based Testing – An Overview

Assurance with IntelligenceSlide 1

Paul GerrardGerrard Consulting1 Old Forge CloseMaidenheadBerkshireSL6 2RD UK

e: [email protected]: http://gerrardconsulting.comt: 01628 639173

I Why Risk-Based Testing? II Introduction to Risk-Management III Risk and Test Objectives IV Designing the Test Process V Project Intelligence, Test Strategy and Reporting V1 Close, Q&A

Here’s the commercial bit:- This material is based on:- Risk-Based E-Business Testing, Gerrard and Thompson,

Artech House, 2002- Visit www.riskbasedtesting.com for more information.

Agenda

Slide 2 Assurance with Intelligence

Why Risk Based Testing?

Requirements

FunctionalSpecification

PhysicalDesign

ProgramSpecification

UserAcceptance

Test

SystemTest

IntegrationTest

UnitTest

V-Model

Is there ever a one-to-one relationship between baseline documents and testing?

Where is the static testing (reviews, inspections, static analysis etc.)?


“Traditional” approach

Test stageTest stageTest stageTest stage

ConsiderSchedule,Environments,Timescales etc.

AcceptanceSystem

TestDevTest

Methodology

Build andExecute tests Not again!

Notfocused

Not Done

StakeholderInvolvement

Are these faultsreally severe?

Too detailedTo understand

We have toTrust them

Slide 5

Sequence of decisions- Stages responsibility capability objectives

Guidance to developers and testers- None, except generic, text book mantras- “demonstrate software meets requirements”

Input of stakeholders- Only when system/acceptance tests reveal problems- Far too late!

Decision making- Timescale driven in early stages- Crisis driven towards the end- Unsatisfactory all round.

Problems with tradition


WriteRequirements

SpecifySystem

DesignSystem

Test theRequirements

Test theSpecification

Test theDesign

UnitTest

AcceptanceTest

SystemTest

IntegrationTest

InstallSystem

BuildSystem

BuildSoftware

WriteCode

W-Model


risk-basedtest reporting

assessproduct risks

Decide

Risk-based testing

Planassess

product risksdefine testobjectives

test techniques,products to test

responsibilityestimation

process

Schedule

focused testdesign andexecution

Implement

If every test aims to address a risk, tests can be prioritised by risk

It’s always going to take too long so…- Some tests are going to be dropped- Some risks are going to be taken

Proposal:- The tester is responsible for making the project

aware of the risks being taken- Only if these risks are VISIBLE, will

management ever reconsider.

Risk-based test planning


Enough testing has been planned when the stakeholders (user/customer, project manager, support, developers) approve:

TESTS IN SCOPE- They address risks of concern and/or give

confidence THE TESTS THAT ARE OUT OF SCOPE

- Risk is low OR these tests would not give confidence

The amount and rigour of testing is determined by CONSENSUS.

How much testing is enough?


Even penguins know how to manage risk!


Do nothing! Pre-emptive risk reduction measures

- information buying- process model- risk influencing- contractual transfer

Reactive risk reduction measures- contingency plans- insurance

The risk that’s left is the residual risk.

Risk response planning

Where testing fits

in


Risk and Test Objectives

If we focus on risks, we know that bugs relating to the selected mode of failure are bound to be important.

If we focus on particular bug types, we will probably be more effective at finding those bugs

If testers provide evidence that certain failure modes do not occur in a range of test scenarios, we will become more confident that the system will work in production.

Why use risks to define test objectives?


Risks and test objectives - examples

There is a risk that… Test Objective

The web site fails to function correctly on the user’s client operating system and browser configuration.

To demonstrate that the application functions correctly on selected combinations of operating systems and browser version combinations.

Bank statement details presented in the client browser do not match records in the back-end legacy banking systems.

To demonstrate that statement details presented in the client browser reconcile with back-end legacy systems.

Vulnerabilities that hackers could exploit exist in the web site networking infrastructure.

To demonstrate through audit, scanning and ethical hacking that there are no security vulnerabilities in the web site networking infrastructure.

Other test objectives relate to broader issues- contractual obligations- acceptability of a system to its users- demonstrating that all or specified functional or non-

functional requirements are met- non-negotiable test objectives might relate to mandatory

rules imposed by an industry regulatory authority and so on

Risk assessment might miss something, or de-scope something important

Generic test objectives- ‘catch all’ measure – e.g. all requirements coverage- complete the definition of your test stages.

Risk-based test objectives are usually not enough


Generic test objectivesTest Objective Typical Test Stage Demonstrate component meets requirements Component Testing Demonstrate component is ready for reuse in larger sub-system

Component Testing

Demonstrate integrated components correctly assembled/combined and collaborate

Integration testing

Demonstrate system meets functional requirements Functional System Testing Demonstrate system meets non-functional requirements

Non-Functional System Testing

Demonstrate system meets industry regulation requirements

System or Acceptance Testing

Demonstrate supplier meets contractual obligations (Contract) Acceptance Testing

Validate system meets business or user requirements (User) Acceptance Testing Demonstrate system, processes and people meet business requirements

(User) Acceptance Testing


Designing the Test Process

Master Test Planning process


Test process worksheet


Failure Mode or Objective

Pro

ba

bility

Co

nseq

ue

nc

e

Te

st Effe

ctivene

ss

RIS

K N

um

ber

Pro

toty

pin

g

Infra

structu

re

Su

b-S

ystem

Ap

plicatio

n S

ystem

No

n-F

un

ction

al T

ests

User A

ccep

tan

ce

Op

eration

al A

ccep

tance (B

TS

)

Live C

on

fide

nce

Cu

stom

er Live T

rial

Test Technique

Client Platform

1 Which browsers, versions and O/S platforms will be supported, includes non-frames, non-graphic browsers etc.)?

SS

2 New platforms: Web TV, Mobile Phones, Palm Pilots etc.

3 Connection through commercial services e.g. MSN, Compuserve, AOL

SS

4 Browser HTML Syntax Checking SS

5 Browser compatibility HTML Checking SS

6 Client configuration e.g. unusable, local character sets being rejected by database etc.

SS

7 Client configuration: Client turns off graphics, rejects cookies, Cookies time out, Client doesn’t have required plug-ins etc.

SS

8 Minimum supported client platform to be determined/validated

SS

Component Functionality

9 Client component functionality SS

10 Client web-page object loading SS

11 Custom-built infrastructure component functionality

SS

12 COTS component functionality SS

13 HTML page content checking - spelling, HTML validation

SS

System/Application functionality

14 End-to-end system functionality CC

15 Loss of context/persistence between transactions

SS CC

Test products through the lifecycle


initial riskassessment

testobjectives

teststages

test processdefinition

master testplanning

test plan/procedures

test specification

test log

test execution

release riskassessment

test results analysis

todayPlannedend

Progress through the test plan

Resi

dual R

isks

start

Project Intelligence, Test Strategy and Reporting


PI and the project lifecycle


ProjectInitiation

Project Planning

PI Management

Acc

ep

tan

ce

Goa

l Ass

ess

me

nt

Goa

l Ass

ess

me

nt

Goa

l Ass

ess

me

nt

Key:

Stakeholder Involvement/Project Assurance/Governance

Project planning and initiation Project Intelligence Activities

Development activities Review and Test activities

TestStrategy

Project Intelligence Planning

ResultsChain

Analysis

PIStrategy

PI Strategy overview


Risks

Coverage goals

Business goals

PI Drivers Ass. Obj.Project Phase

Reqs Design Build Integ Systest UAT Trial Prod.

Objectives for each test phase are easily

identified

PI Process Overview – designed to handle change


Test Process Management

Project RiskManagement

PI Reporting

Test Phase

Coverage Goals

Change

Goals and Risks

3Project or Process

Risks

4Evaluation &

Analysis

13Impact Analysis

1Requirements

Analysis/coverage

2Classification

5Product RisksOut of Scope

7Goals OutstandingRisks Outstanding

Coverage Outstanding

12Goals AchievedRisks Addressed

Coverage Achieved

6Allocation toTest Phase

9RunTest

10Identify

Regression Test

11Run

RegressionTest

8Define/

Design Tests

Risk is not product-related,or is programme-critical

Product risk

Risk will not be tested (no test

objective)

Testobjectivedefined

All new goals/risksare “outstanding”

initially

Tests have revealed a new risk or changes

to a risk

Change affects requirements

(and tests)

Change requires regression tests

Failed tests must be repeated when fixes received

New risk identified

Risk and goal-based reporting


Risks

Coverage goals

Business goals

PI Drivers Ass. Obj.Project Phase

Reqs Design Build Integ Systest UAT Trial Prod.

Progress towards goals is clearly seen.

Outstanding risks are highly visible.

Risk-based reporting

Assurance with IntelligenceSlide 27Progress through the test plan

todayPlanned

end

residual risks of

releasing TODAY

Resi

dual R

isks

start

all risks ‘open’ at the start

Goal based test reporting


Open

Closed

Ris

ks

Open

Open

Closed

Closed

Open

Obje

ctiv

e

Obje

ctiv

e

Obje

ctiv

e

Obje

ctiv

e

Goal

Goal

Goal

Goal

Goal

Benefits available

for releaseO

bje

ctiv

e

Benefit

Closed

RBT approach helps stakeholders:- They get more involved and buy-in to the

approach- They have better visibility of the test process

RBT approach helps testers- Approval to test against risks in scope- Approval to not test against risks out of scope- Clearer test objectives upon which to design

tests RBT approach helps developers

- Specifies their responsibility for testing in detail- “No hiding place”.

Risk-based test approach: planning


RBT approach helps stakeholders:- They have better visibility of the benefits

available and the risks that block benefits RBT approach helps management:

- To see progress in terms of risks addressed and benefits that are available for delivery

- To manage the risks that block acceptance

- To better make the release decision.

Risk-based test approach: execution and reporting


Risk-Based Testing

Any Questions?

riskbasedtesting.comgerrardconsulting.com

uktmf.com

risk-based testing – an overview

Documents

riskmanagementiii risk

residual risk

intelligencewhy risk

riskbased testingassurance

riskbased test planningslide

riskbased ebusiness

intelligenceenough testing

rigour of testing