Can we calculate the profit of testing?Jurian van de LaarPhilips TASS

10th Dutch Testing DayOctober 8th, 2004 - Leiden

Version 2.0

Philips TASS, 20/04/23 2

Agenda

• Introduction• Why talk about ‘profit’ of testing ?• Consequences of insufficient testing• Profit of testing: calculation example• Supporting practices• Profit calculation example refined• Conclusions

Philips TASS, 20/04/23 3

Introduction• Philips TASS International

– Independent subsidiary of Philips Electronics

– Technical and embedded software

• Philips TASS Netherlands– Founded in 1978, Location Eindhoven– 160 software specialists– HW/SW Interfacing, Remote Services,

Integration & Testing (I&T)

• TASS I&T task force:– Increase I&T knowledge (ISEB)– Sharing I&T best practices– Enable customers to

* compose reliable embedded software products

* reduce the risk of product failure.

Philips TASS, 20/04/23 4

Software testing in our line of business

• Large quantities, complex or safety critical• Changing software in operation is difficult• Driven by TTM (market-windows)

• Commitment, metrics, change control• Rigorous testing vs strategy / techniques

Philips TASS, 20/04/23 5

Why talk about ‘profit’ of testing ?

• To quantify the benefits of testing* Emphasis mostly on cost ; benefits hard to explain* Failures after production: taken into consideration?* Early fault detection contributes to total profit

• To make test process more efficient* ‘Calculated risks’ sometimes not really calculated* Finding severe faults in testing prevents high costs

Philips TASS, 20/04/23 6

The ‘profit’ of testing …

Cost of testing Cost of failure

Profit: prevented cost of failure - cost of testing

So, testing is important …but expensive.Rigorous

… but exhaustive is impossible.

‘Bottleneck’ for TTM

… a ‘calculated’risk. How much to invest

in ‘sufficient’ testing?

Philips TASS, 20/04/23 7

Source: Rijkswaterstaat, www.aanlegA50.nl

Source: Eindhovens Dagblad Tuesday July 27, 2004

Source: Eindhovens Dagblad Tuesday August 8, 2004

“Murphy’s Law ..”“Murphy’s Law ..”

“Two failures at the same time ..”“Two failures at the same time ..”

“.. the other failure was in the control software.”“.. the other failure was in the control software.”

“.. expects that the supplier will install faultless software this week.”“.. expects that the supplier will install faultless software this week.”

Philips TASS, 20/04/23 8

Consequences of insufficient testing• Dissatisfied customers (claims)• Bad publicity (image/brand damage)• Service (helpdesk, logistics)• Fixing the faults (repairs)

Philips TASS, 20/04/23 9

Costs of fixing a fault relative to phase

From: E. van Veenendaal, The Testing Practitioner, 2002. Source: B. Boehm, Software Engineering Economics, 1981

• Consequence worse if fault is found late

• Statistics show exponential increase• Operation phase: 1000 x Cost Reqt.

phase

Philips TASS, 20/04/23 10

Can we calculate the profit of testing?• A simplified hypothetical example:

* Costs of testing effort: 180,000 Euro.* Serious problem found in acceptance testing.* Fix in acceptance test phase costs 1,400 Euro.* If not found until in operation: costs 20,000 Euro.* ‘Profit’: n x (20,000 - 1,400) - 180,000 !!

(where n is amount of serious problems found)

Philips TASS, 20/04/23 11

Supporting practices

• Risk based testing• Using intake criteria and acceptance

criteria• Measuring severity• Measuring field call rate

Philips TASS, 20/04/23 12

Risk management in unit testing

• Classify modules using criteria for Impact and Likelihood• E.g. HIGH: give higher priority and more thorough testing• Result: important defects found sooner: lower costs

Philips TASS, 20/04/23 13

Using intake and acceptance criteria• Example 1: Mobile Phone Project

– Intake: Every module tested and reviewed– Acceptance: customer acceptance test

• Example 2: TV Project– Intake procedure: paper check release notes– Acceptance: team required on customer site

Philips TASS, 20/04/23 14

Severity

Severity A: Product can not be produced, block acceptance

Severity B: Serious Problem

Severity C: Cosmetic / Minor issue

Severity D: Acceptable

Philips TASS, 20/04/23 15

After the project: FCR

Indication to what extent the product doesn’t meet the

expectations of the customer, resulting in

complaints.

• Meaning:– Simplified example:

4% FCR: Of every 100 sold products,4 products are returned by the customerto the service organization (warranty period)

– Used for root cause analysis,not a financial driver

Philips TASS, 20/04/23 16

Supporting practices resumed* Risk based testing

– Prioritize & Differentiate => Test Strategy !– More effective testing, Lower cost of failure.

* Using intake criteria and acceptance criteria– More explicit transitions between phases– Reduce costs of fixing faults from earlier phases

* Measuring severity– Monitor seriousness of (potential) failure– Input for calculating cost/benefits of testing

* Measuring field call rate– Analyze failures after product release– Input for calculating costs of product failure

Philips TASS, 20/04/23 17

Can we calculate the profit of testing?

* Found n serious problems (B) in unit testing.

* 2% FCR would cost 20,000 x 50 = 1,000,000 Euro.

* 52 B-problems NOT found would consume all profit!

Taking cost of testing into consideration, e.g. (n x 800) + 180,000: extra profit is 819,200 Euro.

1,000,000 products soldRepair cost: 50 Euro per productSelling price: 500 Euro per product

* 2% FCR reduction = extra profit of 1,000,000 Euro.

Expectations of supplier:10% profit (50,000,000 Euro)4% FCR (2,000,000 Euro)

• Hypothetical example refined:

* Root cause analysis learned: type B yields 2% FCR.

Philips TASS, 20/04/23 18

ConclusionsTesting is not cheap … … insufficient testing is expensive.

Risk based testing and criteria can increase effectiveness.

Measuring during projects is important …… and if we continue to measure after release,costs and profit of testing can be calculated.

Your contribution in refining and improving these calculations is more than welcome !!