Readiness Index – Is your application ready for Production?

Jeff TatelmanSQuAD October 2008

Building a Metrics Program Typical Test Organizations Readiness for Change Metrics and Productivity

Production Readiness Introduction & Objective Four Key Metrics Reporting

Agenda

Typical Test Organizations

No metrics collected No collection of requirements Limited formal reporting on

project status Some central test repositories

but not dash boarding results Minimal effort towards process

improvement

Motivation Investment Skills Education Culture Support Staff Aids/Maturity Process Maturity

Dimensions Of Readiness

Set Measurement Objectives Select Measures Develop Measurement Program How to Use the Results

Getting Started

Determine Approach Which measures will be included as a priority Focus on a few to start with to get ‘quick wins’

Determine Scope One application? One project?

Determine Method Identify best method to track and analyze agreed metrics

Determine Timescale Full project lifecycle? Time boxed approach, eg. 1, 3 or 6 months

Determine Deliverables Report, audience, next steps

Set Measurement Objectives

Metrics and Productivity

Strategic Is it to increase customer satisfaction? Is it to achieve CMM level 2? Is it to achieve industry standards?

Tactical Is it to improve vendor delivery? Is it to increase timeliness of system delivery?

Operational Is it to reduce defects found in testing? Is it to improve requirements definition?

Measuring Unit Examples Strategic

Quality Customer satisfaction Timeliness (delivery) Personnel Industry benchmarking

Tactical Estimating and planning System quality System delivery cost/time Budget Productivity Cost of quality

Operational Schedule tracking Effort tracking Defects Problem resolution System availability

Develop Measurement Program

Establish Specific Measurement Objectives Define a Critical Metrics Set Select Measures to Support the Metrics Set Put Collection Mechanisms into Place Determine Timing of Data Collection Establish Mechanism for Evaluating Results Communicate Results Establish Process for Future Planning Based on

Metrics

How To Use The Results

Conduct root cause analysis of the data Identify areas which would have the biggest

impact and look to how to improve Implement procedures to continually

improve the processes Revisit original list and determine what to

address next

Key Points to Remember

Don’t try to measure too much Understand the goals of your team before

you determine what to measure Determine which metrics support these goals Don’t let your metrics define the behavior of

the team Change or choose a set of relative metrics that can

not be manipulated Monitor and identify trends , define areas for

improvement

Allan Page Measurement that Matter, Better Software October 2005

Production Readiness

Based On Mike Ennis’ Presentation at Star West 2006

How do you know when a product is ready to ship?

Quality Metrics Customer Commitment Release Dates Are Preset Indefinite Testing Adequate Test Coverage Time & Resources Release Criteria

Release Criteria No open critical or high defects Minimal number of medium & low

defects that have been approved by Senior Management

Product is able to run for 72 consecutive hours

No open installation or configuration issues

All pre-defined performance criteria has been met

Determining What to Measure

Test Case Execution Percentage The percentage of tests that have been attempted during

the test cycle Test Case Success Percentage

The percentage of tests that have passed during the test cycle

Number of Unresolved Defects The number of open defects that are currently logged

against the product Defect Arrival Rate

The number of defects found in a given day, week or build

Readiness Metrics Objective

Provide management an overall picture to assess if the project is ready to be placed in production.

Evaluate an application go / no-go production readiness status by Tracking test case execution and defect metrics

over time Calculating the Production Readiness Index based

on the above metrics

Test Case Spreadsheet Example

Defect Spreadsheet Example

Setting Range For Each Metric

Test Case Execution Percentage 10 = 100% 9 = 90-99% 8 = 80-89% 7 = 70-79% … etc

Defect Arrival Rate This Week 10 = 0 9 = 1-2 8 = 3-4 … etc

Creating and Analyzing Readiness Example

RATING RANGESDefects Arrival Rate 9 10 = 0, 9 = 1-2, 8 = 3-4, 7 = 4-5, 6 = 5-6, 5=6-7,

4=…

Total Open Defects 9 10 = 0-5, 9 = 5-10, 8 = 11-15, 7 = 16-20, 6 = 20-25 …

Test Success % 7 10 = 100%, 9 = 99-97%, 8 = 96-94%, 7 = 93-90%...

Test Completion % 6 10 = 100-95%, 9 = 94-90%, 8 = 89-85%, 7 = 84-80%..

TOTAL RATING 31 Green = 35-40, Yellow = 29-34 Red < 28

0

2

4

6

8

10Test Completion

Test Success

Unresolved Defects

Defect Arrival Rate

Spider Chart Example

Test Execution Ratings (green) - scale of 10, goal = 10Defect Ratings (red) - reverse scale of 10, goal = 0

8.9

7.7

3.1

1.1

0.0

2.0

4.0

6.0

8.0

10.0

Test Completion Test Success UnresolvedDefects

Defect ArrivalRate

Management Graph Example

Readiness Index Example

Production Readiness Index - scale of 100

3220

30 34 38 3446 41

73

45 47 51

7281

94100

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Week / Day

Managing the risks Understand the relationships between the metrics Learn to anticipate and minimize the risks before they

happen Always know the information behind the data

Are you ready to release? Redefine your Release Criteria using the individual/overall

rating scale Use colors for presentation & effectiveness Let the data speak for itself

Final Thoughts

Usefulness Does it tell us what’s happening, wrong or need to know?

Trustworthiness Does not provide false indications

Timeliness Does it indicate the health of a system...advanced

warning? Simplicity

Too difficult to read, it will get abandoned Accessibility

Must be visible, easy to get to

Key Attributes Of a Good Measure

Questions

jeffcolorado@juno.com