1QAAC

Metrics: A Path for Success

Kim Mahoney, QA Manager, The Hartfordkim.mahoney@thehartford.com

2QAAC

Session objectives: to leave this room with knowledge of metrics and be able to apply these learning’s to achieve success in your organization.

2

3QAAC

Success

My definition of success is:– Test case pass rates > 70%– Test environment availability/stability > 90%– Requirements are stable with minimal changes– No defects leaked into production– Root causes of defects tell a story

3

What is yours?

4QAAC

AgendaWhat metrics to use in my organization?

Where to get the data to create the metric?

Key metrics for ensuring successes Test case pass rate Defect leakage into production Requirements stability index Test environment availability Root cause of defects Test effort variance Error discovery rate Automation script results

How can metrics pave a path to success?

4

5QAAC

What metrics to use in my organization?

5

6QAAC

What do you want to gauge?•Quality of code deploys•Environment stability

What do you want to determine?•Go/no-go decisions•Quality of requirements

What makes sense?•Not all metrics make sense for every project

Who to distribute to?•Distribute to folks who can do something about it!

6

What metrics to use in my organization?

7QAAC

Where to get the data?

7

8QAAC

Testing Tool Test case execution results Defects & root cause information Production defects Automation results

Requirements Tool Requirements changes

Time Tracking Tool Actual effort

Vendor Partner Environmental information

8

Where to get the data to create the metric?

9QAAC

Key metrics for ensuring successes

9

10QAAC

Test Case Pass Rate

10

11QAAC

Test case pass rateWhat is your target pass rate in your

organization?

Pass rate = # test cases passed / # test cases executed

Sample: 238 test cases passed / 278 test cases executed = 86% pass rate

What can be learned from this 86%?

11

12QAAC

Test Case Pass Rate con’t

86% pretty good, but….• What if the 14% that is failing is the most

critical part of the system?• What if this is the last cycle of testing and

14% of those test cases cannot be fixed before production?

• What was the project goal’s pass rate?

12

13QAAC

Test Case Pass Rate con’t

How to use test pass rate?• Comparing cycle to cycle• Comparing similar test efforts• Review test pass rate during and

after test execution

Who to tell? How to tell them?• Management, Project Team• Lessons learned meeting, post

mortem

13

14QAAC

Defect Leakage into Production

14

15QAAC

Defect Leakage into Production

Capture for each release

Capture root causeConfiguration, data, requirements, training, etc..

Capture severity – impact to bizCritical, high, medium, low

Compare release to release

15

16QAAC

Defect Leakage - Sample

What can we learn from this?

Who would want to know this?

16

17QAAC

Requirement Stability Index

17

18QAAC

Requirements Stability Index

RSI indicates the level of change to the original set of customer approved requirements

Why is this a good metric to measure?• Measuring and controlling RSI within the defined ranges

• Leads to a stabilized & controlled requirement thus reducing rework effort & defect leakage.

• Increases test effectiveness and quality of application implemented in production.

18

19QAAC

RSI cont’d

Calculation:= (Total # Original Requirements + Total # of

Requirements Changed + Total # of Requirements Added + Total # of Requirements deleted) / Total # Original Requirements

GreenGreen 1.00 to 1.1 (requirements are stable)YellowYellow 1.12 to 1.15 (requirements stability is average)RedRed >1.15 (requirements are unstable due to

frequent requirement changes)

19

20QAAC

RSI cont’dWhen will we calculate RSI?

RSI for a project/application will be calculated every time when a change is requested.

Note: RSI can be published at the end of a release during project closure phase.

20

21QAAC

RSI cont’d - Sample• Total number of original

requirements: 28 • Requirement changes: 2• Requirements added: 1• Requirements deleted: 3

So……..

• RSI = (28+2+1+3)/28 = 1.18• RSI > 1.15, Red (requirements are unstable)

21

22QAAC

Test Environment Availability

22

23QAAC

Test environment availability

Test environment availability and stability– Total minutes due to issues / total minutes available

to track• Example: 120 / 480 = .25• 25% of the time the environment was not available for

testing

23

So… what is the impact when the test environment is not so stable?

24QAAC

A picture is worth 1000 words…

What can be said about the above?

24

25QAAC

Test environment availability cont’d

– Track daily and report out by release

– Track for trending

25

26QAAC

Root Cause of Defects

26

27QAAC

Root Cause of DefectsDefects can be caused for a number of reasons:

– Code issues– Ambiguous requirements– Data– Test case– Database– Existing production defect– Configuration– Not an issue (all other)

Track root causes for trending to proactively avoid anticipated defects in future

27

28QAAC

Root Cause of Defect – cont’d

28

29QAAC

Test Effort Variance

29

30QAAC

Test effort variancePlanned vs. actuals for a test effort

– Why track?• Learn from• Refine your estimating skills

– Who cares?• QA management, Project Managers,

Finance

30

– How to mitigate variance• Some reasons for variance: changes in requirements, environmental issues,

offshore network issues, late code deployments, unusually high defects, etc…• Easier to explain a week in variance

– Trending• Are you always over or under estimating?

31QAAC

Error Discovery Rate

31

32QAAC

Error Discovery Rate• EDR = total defects / total test cases executed

32

33QAAC

Automation Script Results

33

34QAAC

Automation script results

What is the value in automation? The reasons are obvious.

What is not so obvious?– The kinds of defects that are found over and over– Script re-work that the automation team put in

due to code changes– Applications that always have a low pass rate

when the automation bed is run

34

35QAAC

Metrics can be a path to success…

35

36QAAC

… because you can…

• Learn from the metrics• Compare similar projects• Make things better• Continuously improve

36

One step at a time to achieve success.

37QAAC

In Summary…• Reliable tools are needed which

house the data• Metrics are objective• Need to know which metrics

make sense for your organization

• Distribute to folks who can make a difference

• Pass rates can be deceiving• Using metrics displays proactive

thought leadership

37

38QAAC 38