advanced defect management

Advanced Defect Management – The IBM Way

Sabarinath Venugopalan

Practice Leader, Nordic – IBM Test Services

© 2009 IBM Corporation

ERUC 2009

Agenda

Defect Reduction Methord

Test Planning and Optimization Workbench (TPOW)

Summary

Client Examples

Introduction to Advanced Defect Management


ERUC 2009

Key Messages

The Next Generation of Testing is

emerging to meet the demands of the

intelligent enterprise

The key to the Next Generation of

Testing is informed actions, enabled by

predictive analytics

Advanced Defect Management is

powered by informed actions. It delivers

significant benefits to the major factors

governing the Next Generation of

Testing


ERUC 2009

Today’s leading Enterprises are Intelligent. They are…

Anticipating

Precise

Aware

Questioning

Empowering

Gathers, senses, and uses

structured and unstructured

information from every

node, person, and sensor

within the environment

Predicts and prepares for the

future and doesn’t only react or

correct actions, but also steers

and evaluates trade-offs

LinkedConnects internal and

external functions front to

back across geographies in

a way that aligns to desired

business outcomes

Enables and extends

employees' memory,

insight and reach, as well

as the authority to decide

and act

Reserves the right to get

smarter by challenging its

status quo while creating

new opportunities

Uses only the most relevant

information to support timely

decisions/actions closer

to the point of impact and

consequence

Source: IBM Study: Business Analytics and Optimization for the Intelligent Enterprise, March 2009


ERUC 2009

Old ways of decision making and management are breaking down. Organizations will need to adopt new ways of working to improve speed to insight and speed to impact

Traditional Approach New Approach

Instinct and intuition Fact driven

Corrective Directive

Efficient Optimized

Years, months, weeks Hours, minutes, seconds

Human insight Applied semantics

Decision support Action support

Source: IBM Study: Business Analytics and Optimization for the Intelligent Enterprise, March 2009


ERUC 2009

The Next Generation of Testing will be driven by four factors. Advanced Defect Management substantially improves these factors.

Quality Improvement

DATA

INSIGHTS

Cost Reduction Risk Management

Schedule Acceleration

3

42

1

Analytical I Insightful I “Smarter” I Linked

Total View of Test

Managing Change will be paramount


ERUC 2009

IBM’s Advanced Defect Management capabilities are based on two related analytical predictive capabilties. These are enabled by significant base of data collected over 25 years.

IBM® Defect Reduction Method™ is a patent pending, statistical,

objective defect classification and analysis process that helps

organizations find and fix defects as early as possible and prevent

future defects from being injected.

IBM® TPOW™ is an advanced, empirically risk-based planning tool

that delivers an optimized test strategy, supporting master test plan

and ongoing project planning. At the root of TPOW is the defect

prediction capabilities that that gives you insights about your

application projects.


ERUC 2009

Advanced Defect Management drives dramatic reductions in cost and schedule while improving overall quality and lowering business risk

Cost Overview

(Total: $13,277,407)

Source: Test Planning and Optimization Workbench example

Cost Overview

(Total: $20,653,285)

Schedule: 89 days

Environment: 14 days

Schedule: 37 days

Environment: 5 hrs

$911,260

$10,240,335

$2,125,812

Defect fix cost

Business cost

Test cost

Defect fix cost $14,439,176

Test cost $1,579,838

Business cost $4,634,271

Cost ReductionDATA

INSIGHTS

35%+

Cost Reduction


ERUC 2009

Car WashCar Wash

Car Wash

Car WashCar Wash

IBM is leveraging on Advanced Defect Management internally and in client engagements to substantially improve the four factors of Next Generation Testing.

TPOW Macro

Planning

(Reference

Project)

TPOW Micro

Planning

Test

Optimizers

Execution

Managed Testing Services

Test

Execution

Implement advanced Test

practices, processes and

optimizers with the full-life cycle.

Execute managed test services

leveraging globally deployed

resources executing test

according to the newly

implemented processes.

1000 – 5000

Function Points

Currenty in

development or

under major

enhancement

1B

2C

Execute detailed bottom-up test planning

leveraging the Macro Planning feature of the

Test Planning and Optimization Workbench.

3

Execute Pilot

TPOW Macro

Planning

(Client

Project Data)

1A2A

2B

2C

Conduct a Defect Analysis Starter which

executes IBM’s Defect Reduction Method to

gather quantitative data needed to customize a

client Reference Project

Execute IBM’s Test Planning and Optimization

Workbenchby starting with Macro Planning. IBM

assesses an existing application and, provides

recommended improvements.

1A

1B

Execute Test Optimizers recommended thru

Macro Planning. 2B

DAS

Execute IBM’s Test Planning and Optimization

Workbench Macro Planning using baseline

data gathered by the Defect Analysis Starter.

2A 3

Select target programs (s)

Applications


ERUC 2009

Agenda



Summary

Client Examples



ERUC 2009

IBM® Defect Reduction Method™ (DRM) provides measurable insight into software quality and development processes

When does it start/end?

Used continually throughout the test life cycle

Defect classification can begin at any point that

defects have been recorded

Comprehensive analysis reports typically run at test

exit for each phase or for the release, but interim

reports should be generated periodically throughout

test execution to gain the most value from DRM

IBM® Defect Reduction Method™ is a patent

pending, statistical, objective defect classification and

analysis process that helps organizations find and fix

defects as early as possible and prevent future defects

from being injected.

DRM Classification DRM Analysis

►Systems engineering life cycle

Test Assess and Planning Test Execution and ReportingTesting project

startupTest Design Test Implementation and Close

Ongoing DRM activity


ERUC 2009

The IBM® Defect Analysis Starter™ quickly identifies risks and helps improve the quality of system and application development. Application of DAS is typically completed in 3 weeks!

X X X

X X X

XX XX XX

XX XX XX

The IBM® Defect Analysis Starter™ provides business value by:

What is it?

A defect based analysis, using IBM®’s Defect Reduction Method™ (DRM), to

identify high priority cost reduction opportunities associated with software

test practices and defect prevention.

The IBM® Defect Analysis Starter™ utilizes:

An IBM approach that encompasses classification and analysis

Benchmarks based on defect data collected for nearly two decades,

mapped to industry, quality level, and test phase/activity

Metrics to objectively quantify and prioritize risk and opportunity, and to

measure quality

The IBM® Defect Analysis Starter™ includes:

Classification of client provided defects

Analysis and measurement of risks (General Test Effectiveness, Test

Design Effectiveness, System Stability and Completeness) and Quality

(Evaluation against Quality Benchmarks, Focus Area Identification, and

Artifact Value Assessment)

A Rough Order of Magnitude business case reflecting cost reduction

opportunity (Invalid Reduction, Production Defect Reduction, Earlier Defect

Removal, Cycle Time Reduction, and Prevention of Defect Injection)

Identifying and recommending specific actions to reduce risk and cost associated with developing and testing

systems or applications

Acknowledging and reinforcing client strengths

Guiding the client to utilize test practices that can increase their ROI while delivering improved quality


ERUC 2009

Agenda



Summary

Client Examples



ERUC 2009

TPOW helps to answer challenging questions …

Am I doing too much/too little testing?

Am I getting my money’s worth out of testing?

Will an increased testing investment drive any further

improvement to quality?

How can I shift more of my investment to earlier project

phases/development?

What is the overall cost, schedule, and quality impact of

moving testing offshore?

How can I determine the quality of my application based

on my testing effort?

I just got a budget cut; what testing should I eliminate?

What impact will it have on application production quality?

How early in the life cycle can I develop an overall test

strategy, including effort, schedule, cost, and quality

predictions?

What if I….?


ERUC 2009

TPOW works at multiple levels

Model initial cost

and benefits

Provides guidance in test

design and execution

Achieve the organization’s

quality, schedule and cost goals

Analytical

Project

Oriented

Micro

Planning

Macro

Planning

Ongoing

Project

Planning

Intelligence

Startup DeploymentSolution

closeSolution

outline

Macro

design

Micro

designBuildSolution

requirements

Industry and client benchmarking

21 3

4


ERUC 2009

Example: Macro Planning - Create a Baseline

Baseline

Derive Function Points for High / Medium and Low

TPOW Baseline

Definition of Test Activities Definition of Triggers Distribution Allocation of Test Effort

– Per Test Activity

– Per Trigger

Definition of Global Resource Model

Benefit of using “Car Wash” Assume 1 Function Point for 10 KLOC

Macro

Planning

Macro

Planning


ERUC 2009

Example: Macro Planning - Assess Process Initiatives

Baseline + DAS Initiative

Complete for medium case

Baseline + DAS

Assume 1 FP for 10 KLOC

Baseline

Outcome

Baseline Das initiative %

Test Cost 4,263,066.00$ 4,430,601.50$ -4%

Defect Fix cost 19,861,196.50$ 10,667,479.75$ 46%

Business cost 5,272,015.00$

Escaped defects 375.00 61 84%

Definition of Test Activities Definition of Triggers Distribution Allocation of Test Effort

– Per Test Activity

– Per Trigger

Definition of Global Resource Model

Benefit of using “Car Wash”

Macro

Planning

Macro

Planning


ERUC 2009

Agenda


Test Planning and Optmization Workbench (TPOW)

Summary

Client Examples



ERUC 2009

Reduction of Client PLM Incidents

PLM Incidents Trend

0

20

40

60

80

100

120

140

160

180

June

July

August

Septem

ber

October

Novem

ber

No.

of I

ncid

ents

Sev2

Sev3

Sev4

Total

Achieved through application of Defect Reduction Methods (DRM)

*Trough in August attributed to vacation


ERUC 2009

Client Example: Selection of Accelerators based on TPOW Analysis

Scenario Escaped

defects 2)

Production

Defect rate

Cost to fix

Production

Defect 1) DKK

Potential

annual saving

Baseline 43.226 23% 562.000.000 0

TPOW 42.031 21% 546.000.000 16.000.000

+Carwash,+ DRM Phase1 +

Test Data Mgt phase 1&2

33.785 17% 439.000.000 123.000.000

+ DRM phase 2+3, + Test

data management phase 3

26.928 14% 350.000.000 212.000.000

TCOE 14.022 7% 182.000.000 380.000.000


ERUC 2009

Team of testers spent 3 1/2 months developing

detailed estimates for a major development

project.

Since the project was in the design phase, they

were trying to develop estimates that would

support a fixed budget for testing.

The client wanted a guarantee that no more than

5% of the defects associated with the application

lifecycle would arise during production.

If weight was a measure of accuracy, the 10,378

person days of effort looked like a solid estimate.

The team asked for an independent estimate to

validate their numbers.

In just 1 1/2 days using advanced defect

methods, the assessment of the teams' estimate

yielded …


ERUC 2009

Increase test effort to meet decreased defect rate ..

PD: Person Day

ST SIT PT UAT Total

XYZ 21.68%

2250

45.37%

4708

10.79%

1120

22.16%

2300

100%

10,378

Improvement

suggestion

26.85%

3812

51%

7242

6.41%

910

15.75%

2236

100%

14,200

DecreaseIncrease

Scenario Output

Projection Defect discovery and escape to production

Defect production rate = 368 / 1921 = 19%

Validation

and ImprovementValidation result

Given the quality objective of production defect rate 5%, the existing estimated test

effort (10,378PD) is not enough

Improvement suggestion1. Add the total test effort to 14,200PD

2. Adjust the test effort distribution on test activities (see table below)


ERUC 2009

Agenda



Summary

Client Examples



ERUC 2009

Summary

Applying predictive analytics to reduce

defects and to move defects earlier in

the lifecycle delivers substantial gains

Taking a total view of testing generates

dramatic benefits

IBM’s capabilities in advanced defect

management are deemed unique by

technology analysts


ERUC 2009

Contact:

Sabarinath Venugopalan

Practise Leader

IBM Test Services, Nordic

+46-70-793 2665

[email protected]

advanced defect management

Documents