Download - QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford [email protected]
![Page 2: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/2.jpg)
2QAAC
Session objectives: to leave this room with knowledge of metrics and be able to apply these learning’s to achieve success in your organization.
2
![Page 3: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/3.jpg)
3QAAC
Success
My definition of success is:– Test case pass rates > 70%– Test environment availability/stability > 90%– Requirements are stable with minimal changes– No defects leaked into production– Root causes of defects tell a story
3
What is yours?
![Page 4: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/4.jpg)
4QAAC
AgendaWhat metrics to use in my organization?
Where to get the data to create the metric?
Key metrics for ensuring successes Test case pass rate Defect leakage into production Requirements stability index Test environment availability Root cause of defects Test effort variance Error discovery rate Automation script results
How can metrics pave a path to success?
4
![Page 5: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/5.jpg)
5QAAC
What metrics to use in my organization?
5
![Page 6: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/6.jpg)
6QAAC
What do you want to gauge?•Quality of code deploys•Environment stability
What do you want to determine?•Go/no-go decisions•Quality of requirements
What makes sense?•Not all metrics make sense for every project
Who to distribute to?•Distribute to folks who can do something about it!
6
What metrics to use in my organization?
![Page 7: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/7.jpg)
7QAAC
Where to get the data?
7
![Page 8: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/8.jpg)
8QAAC
Testing Tool Test case execution results Defects & root cause information Production defects Automation results
Requirements Tool Requirements changes
Time Tracking Tool Actual effort
Vendor Partner Environmental information
8
Where to get the data to create the metric?
![Page 9: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/9.jpg)
9QAAC
Key metrics for ensuring successes
9
![Page 10: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/10.jpg)
10QAAC
Test Case Pass Rate
10
![Page 11: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/11.jpg)
11QAAC
Test case pass rateWhat is your target pass rate in your
organization?
Pass rate = # test cases passed / # test cases executed
Sample: 238 test cases passed / 278 test cases executed = 86% pass rate
What can be learned from this 86%?
11
![Page 12: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/12.jpg)
12QAAC
Test Case Pass Rate con’t
86% pretty good, but….• What if the 14% that is failing is the most
critical part of the system?• What if this is the last cycle of testing and
14% of those test cases cannot be fixed before production?
• What was the project goal’s pass rate?
12
![Page 13: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/13.jpg)
13QAAC
Test Case Pass Rate con’t
How to use test pass rate?• Comparing cycle to cycle• Comparing similar test efforts• Review test pass rate during and
after test execution
Who to tell? How to tell them?• Management, Project Team• Lessons learned meeting, post
mortem
13
![Page 14: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/14.jpg)
14QAAC
Defect Leakage into Production
14
![Page 15: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/15.jpg)
15QAAC
Defect Leakage into Production
Capture for each release
Capture root causeConfiguration, data, requirements, training, etc..
Capture severity – impact to bizCritical, high, medium, low
Compare release to release
15
![Page 16: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/16.jpg)
16QAAC
Defect Leakage - Sample
What can we learn from this?
Who would want to know this?
16
![Page 17: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/17.jpg)
17QAAC
Requirement Stability Index
17
![Page 18: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/18.jpg)
18QAAC
Requirements Stability Index
RSI indicates the level of change to the original set of customer approved requirements
Why is this a good metric to measure?• Measuring and controlling RSI within the defined ranges
• Leads to a stabilized & controlled requirement thus reducing rework effort & defect leakage.
• Increases test effectiveness and quality of application implemented in production.
18
![Page 19: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/19.jpg)
19QAAC
RSI cont’d
Calculation:= (Total # Original Requirements + Total # of
Requirements Changed + Total # of Requirements Added + Total # of Requirements deleted) / Total # Original Requirements
GreenGreen 1.00 to 1.1 (requirements are stable)YellowYellow 1.12 to 1.15 (requirements stability is average)RedRed >1.15 (requirements are unstable due to
frequent requirement changes)
19
![Page 20: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/20.jpg)
20QAAC
RSI cont’dWhen will we calculate RSI?
RSI for a project/application will be calculated every time when a change is requested.
Note: RSI can be published at the end of a release during project closure phase.
20
![Page 21: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/21.jpg)
21QAAC
RSI cont’d - Sample• Total number of original
requirements: 28 • Requirement changes: 2• Requirements added: 1• Requirements deleted: 3
So……..
• RSI = (28+2+1+3)/28 = 1.18• RSI > 1.15, Red (requirements are unstable)
21
![Page 22: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/22.jpg)
22QAAC
Test Environment Availability
22
![Page 23: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/23.jpg)
23QAAC
Test environment availability
Test environment availability and stability– Total minutes due to issues / total minutes available
to track• Example: 120 / 480 = .25• 25% of the time the environment was not available for
testing
23
So… what is the impact when the test environment is not so stable?
![Page 24: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/24.jpg)
24QAAC
A picture is worth 1000 words…
What can be said about the above?
24
![Page 25: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/25.jpg)
25QAAC
Test environment availability cont’d
– Track daily and report out by release
– Track for trending
25
![Page 26: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/26.jpg)
26QAAC
Root Cause of Defects
26
![Page 27: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/27.jpg)
27QAAC
Root Cause of DefectsDefects can be caused for a number of reasons:
– Code issues– Ambiguous requirements– Data– Test case– Database– Existing production defect– Configuration– Not an issue (all other)
Track root causes for trending to proactively avoid anticipated defects in future
27
![Page 28: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/28.jpg)
28QAAC
Root Cause of Defect – cont’d
28
![Page 29: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/29.jpg)
29QAAC
Test Effort Variance
29
![Page 30: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/30.jpg)
30QAAC
Test effort variancePlanned vs. actuals for a test effort
– Why track?• Learn from• Refine your estimating skills
– Who cares?• QA management, Project Managers,
Finance
30
– How to mitigate variance• Some reasons for variance: changes in requirements, environmental issues,
offshore network issues, late code deployments, unusually high defects, etc…• Easier to explain a week in variance
– Trending• Are you always over or under estimating?
![Page 31: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/31.jpg)
31QAAC
Error Discovery Rate
31
![Page 32: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/32.jpg)
32QAAC
Error Discovery Rate• EDR = total defects / total test cases executed
32
![Page 33: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/33.jpg)
33QAAC
Automation Script Results
33
![Page 34: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/34.jpg)
34QAAC
Automation script results
What is the value in automation? The reasons are obvious.
What is not so obvious?– The kinds of defects that are found over and over– Script re-work that the automation team put in
due to code changes– Applications that always have a low pass rate
when the automation bed is run
34
![Page 35: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/35.jpg)
35QAAC
Metrics can be a path to success…
35
![Page 36: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/36.jpg)
36QAAC
… because you can…
• Learn from the metrics• Compare similar projects• Make things better• Continuously improve
36
One step at a time to achieve success.
![Page 37: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/37.jpg)
37QAAC
In Summary…• Reliable tools are needed which
house the data• Metrics are objective• Need to know which metrics
make sense for your organization
• Distribute to folks who can make a difference
• Pass rates can be deceiving• Using metrics displays proactive
thought leadership
37
![Page 38: QAAC 1 Metrics: A Path for Success Kim Mahoney, QA Manager, The Hartford kim.mahoney@thehartford.com](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649cb95503460f9498003a/html5/thumbnails/38.jpg)
38QAAC 38