using metrics to reduce cost of re-work dwight lamppert senior test manager franklin templeton
TRANSCRIPT
![Page 1: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/1.jpg)
Using Metrics to Reduce Cost of Re-work
Dwight LamppertSenior Test Manager
Franklin Templeton
![Page 2: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/2.jpg)
2
Opening Question
“What is the job of testing?”
![Page 3: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/3.jpg)
3
Potential Response
“The job of testing is to find defects.”
![Page 4: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/4.jpg)
4
Myth Exposed
Myth: The Job of Testing Is to Find Defects1
1The Seven Principles of Lean Software Development by Mary Poppendieck
![Page 5: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/5.jpg)
5
“Quality is everyone’s responsibility”
Myth: The Job of Testing Is to Find Defects
“The job of tests, and the people that develop and run tests, is to prevent defects, not find them. A quality assurance organization should champion processes that build quality into the code from the start rather than test quality in later.”1
1The Seven Principles of Lean Software Development by Mary Poppendieck
![Page 6: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/6.jpg)
6
At a recent software seminar
“The common attributes of successful software projects are
Heroes Chaos Re-work Untracked OT”
![Page 7: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/7.jpg)
7
Today’s Discussion
Case Study – 4 projects2 DefinitionsTesting Metrics (what to collect &
analyze)Metrics from first 2 projects Process Improvement StepsSuccess Story – Projects 3 & 4Some trends & conclusions
![Page 8: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/8.jpg)
8
Project Background
Legacy mature system (mainframe) Industry regulatory enhancements Time-boxed: 3 to 4 months Waterfall SDLC
First 2 Projects Project 1 – Dec09 – 1X Project 2 – June10 – 2.3X
![Page 9: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/9.jpg)
9
Definition: Re-work
Re-work – AD time to fix and testing time to report and re-test System Testing and Production defects
![Page 10: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/10.jpg)
10
Definition: Static Testing
Static Testing - Testing of a component or system at specification or implementation level without execution of that software, e.g., reviews or static code analysis.
![Page 11: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/11.jpg)
11
Static vs. Dynamic Testing
From System Testing with an Attitude, p.14, by Nathan Petschenik
![Page 12: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/12.jpg)
12
Testing Defect Analysis
When was the defect introduced (SDLC phase)
When was the defect discovered (testing phase)
![Page 13: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/13.jpg)
13
Points of Origin for Defects2
1. Requirements2. Design3. Code4. User documentsUser documents5. Bad fixes (new defects due to repairs of
older defects)
2Software Engineering Best Practices: Lessons from Successful Projects in the Top Companies by Capers Jones
![Page 14: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/14.jpg)
14
Defect Discovery Phases
1. Static Testing Document walk-throughs & reviews Pre-testing Test Plan review All before Build is complete
2. System Testing
3. Production
![Page 15: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/15.jpg)
15
Tracking the Cost of Re-work
1. Defect Fix Time
2. Re-test Time
COR (Cost of Re-work) – calculated by multiplying time x rate – cost per defect
![Page 16: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/16.jpg)
16
Testing Rework Workflow
Re-work time during Test Execution is tracked as 2 Components
1. Time to Fix a Defect
2. Time to Re-test Defect Fixes
![Page 17: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/17.jpg)
17
Project 1 - Defect Fix Cost
Defect Fix Cost
$0 $21 $49
$406
$1,735
$0
$500
$1,000
$1,500
$2,000
Requirements Design Build System Test Transition
SDLC Phase of Defect Discovery
![Page 18: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/18.jpg)
18
Issues from first 2 projects
1. High % of project = Cost of Re-work
![Page 19: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/19.jpg)
19
COR = 23 to 29% Project
Cost of Rework - % of Total Project
29%
23%
0%5%
10%15%20%25%30%35%
1 2 3 4
Project
% o
f R
ewo
rk
COR - % of Project
![Page 20: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/20.jpg)
20
Issues from first 2 projects
1. High % of project = Cost of Re-work
2. High % of Requirements & Design (R&D) defects found later
![Page 21: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/21.jpg)
21
20-30% R&D Defects found later
% R & D Defects found later
29%
21%
0%
5%
10%
15%
20%
25%
30%
35%
1 2 3 4
Project
%
% R & D Defectsfound later
![Page 22: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/22.jpg)
22
Issues from first 2 projects
1. High % of project = Cost of Re-work
2. High % of Requirements & Design (R&D) defects found later
3. DDP (Defect Discovery %) – slightly better than average
![Page 23: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/23.jpg)
23
Defect Discovery Percentage (DDP)
DDP = % of Total Defects discovered before implementation
Provides a metric to evaluate the overall effectiveness of testing.
90% - Good95% - Very Good
![Page 24: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/24.jpg)
24
DDP = average +
Defect Detection %
89.288.2
80
85
90
95
100
1 2 3 4
Project
% DDP%
![Page 25: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/25.jpg)
25
Process Improvement Steps
1. Developed framework in Quality Center to track COR
2. Presentations to business partners & executives
3. Lively discussion during presentations4. AD appointed a new AD Lead5. AD improved their internal code review6. More emphasis on and participation
during reviews, walk-throughs – Static Testing
![Page 26: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/26.jpg)
26
Process Improvement Steps
7. More collaboration between AD and testing teams
8. Increased emphasis on unit testing
9. Risk-based testing approach
10. Improved regression work
11. Staff increased in domain knowledge
![Page 27: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/27.jpg)
27
Projects 3 & 4 – Process Improved
1. Project 1 – Dec09 – 1x
2. Project 2 – June10 – 2.3x
3. Project 3 – May11 – 3.4x
4. Project 4 – June11 – 2.3x
![Page 28: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/28.jpg)
28
Static Testing emphasized
Defects Found During Static Testing
10
92
339
0
50
100
1 2 3 4
Project
# of
Def
ects
Static Testing Defects
![Page 29: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/29.jpg)
29
System Testing Defects
Defects Found During System Testing
12
23
3
8
05
1015
2025
1 2 3 4
Project
# of
Def
ects
System TestingDefects
![Page 30: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/30.jpg)
30
% R&D Defects found later
% R & D Defects found later
7%
29%
21%
2%
0%
5%
10%
15%
20%
25%
30%
35%
1 2 3 4
Project
%
% R & D Defectsfound later
![Page 31: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/31.jpg)
31
DDP (Defect Discovery %)
Defect Detection %
89.2%88.2%
98.1%
95.5%
80%
85%
90%
95%
100%
1 2 3 4
Project
% DDP%
![Page 32: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/32.jpg)
32
COR as % Project Cost
Cost of Rework - % of Total Project
12%
29%
23%
6%0%5%
10%15%20%25%30%35%
1 2 3 4
Project
% o
f R
ewo
rk
COR - % of Project
![Page 33: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/33.jpg)
33
Defect Discovery Rate
DDR = % of Total Defects discovered during an SDLC or Test Phase
Static Testing System Testing TransitionGoal - Trying to find out if we are
finding defects earlier.
![Page 34: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/34.jpg)
34
DDR Trends
17.6
70.5
11.7
27
62.1
10.8
89.3
7.8
1.9
88.6
6.84.5
0
10
20
30
40
50
60
70
80
90
100
Static System Production
----------------C
od
ing
-----------------
As DDR for Static Testing
increases, DDR for System Testing and Production
Defects decreases
![Page 35: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/35.jpg)
35
When is Unit Testing complete?
“A necessary criterion for completion of developer testing is that the developers believe that the software is ready to be delivered to the users on the day it is delivered to the system-test team.”
From System Testing with an Attitude, p. 33, by Nathan Petschenik
![Page 36: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/36.jpg)
36
Today’s Discussion
Defect metrics – powerful storyStatic Testing worksIt is a collaborative effortThe positive effect ripples into Unit
Testing, System Testing, UAT, and implementation
No shortcuts here – influence your project to do it right
![Page 37: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/37.jpg)
37
Some Final Thoughts
Training needed for our staff (testing, AD, business, PM community) - how to be more effective in document reviews and walk-throughs.
Perspectives Tester – static testing > a paradise (a
new maturity level)
![Page 38: Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton](https://reader035.vdocuments.net/reader035/viewer/2022070305/55140130550346e2488b48ee/html5/thumbnails/38.jpg)
38
Questions & Answers
???