copyright (c) 2003 howard e. dow1 results from inspecting test automation scripts howie dow...
Post on 15-Dec-2015
218 Views
Preview:
TRANSCRIPT
Copyright (c) 2003 Howard E. Dow
1
Results from Inspecting Results from Inspecting Test Automation ScriptsTest Automation Scripts
Howie Dow
howie.dow@rcn.com
hdow@alumni.carnegiemellon.edu
Copyright (c) 2003 Howard E. Dow
2
Agenda
Audience - Project - Terminology
What the team did
Results and data
Benefits, pitfalls and how to
Conclusions
Copyright (c) 2003 Howard E. Dow
3
Car Cah
Parked Pahked
Rs ????Idea
Idear
Copyright (c) 2003 Howard E. Dow
4
Target Audience
EngineersProject leaders
IdeallyTrained on inspectionsSome inspection experience
Copyright (c) 2003 Howard E. Dow
5
Project
Call center system
Functional tests
GoalsDefect free test scripts
On time
Increase skill level
Copyright (c) 2003 Howard E. Dow
6
Skill levelsSkill Level (Red=Available, Green=Needed)
0
0.5
1
1.5
2
2.5
3
3.5
1 2 3 4
Engineer Number
Ski
ll L
eve
l (1
=L
ow
, 2=
Me
diu
m,
3=H
igh
)
Copyright (c) 2003 Howard E. Dow
7
Terminology
Inspection = Software Inspection = Formal Inspection
= “Fagan Style” Inspection
Reviews
Walk-Throughs
Any technique
Copyright (c) 2003 Howard E. Dow
8
Terminology
Inspection characteristicsGoals
Rules, Roles, Responsibilities
Process
Copyright (c) 2003 Howard E. Dow
9
Goals of Inspections
Find defects
Collect data
Archive data
Use data
Learn
Copyright (c) 2003 Howard E. Dow
10
Etiquette, Customs and Rules
Focus on the product
Be prepared
Open mind
Language
Copyright (c) 2003 Howard E. Dow
11
Roles and Responsibilities
Moderator Recorder
Reader Author
Management
Inspector
Copyright (c) 2003 Howard E. Dow
12
Planning
ProductOverview
Preparation
Inspection
Rework
Data analysisand storage
Re-inspect(optional)
Inspection Process
Copyright (c) 2003 Howard E. Dow
13
Audience - Project - TerminologyWhat the team didResults and dataBenefits, pitfalls and how toConclusions
Agenda
Copyright (c) 2003 Howard E. Dow
14
Moderator Recorder
Reader Author Inspector
Manager
Roles and Responsibilities
Copyright (c) 2003 Howard E. Dow
15
ProductOverview
Planning
Preparation
Inspection
Rework
Data analysisand storage
Re-inspect(optional)
Allowedsuggestions
Time not collected
Inspection Process Used
Copyright (c) 2003 Howard E. Dow
16
Summary of key differences
Omit product overview
Involved management
Suggesting fixes allowed
Strong bias to re-examine after rework by another team member
Omission - collecting preparation time
Copyright (c) 2003 Howard E. Dow
17
Ready to go
17 Scripts
3-6 inspectors
Copyright (c) 2003 Howard E. Dow
18
Data CollectedScript identifier
Author
Date completed
Date inspected
Size
Defect: number, location, description
Inspection time
Inspectors
Copyright (c) 2003 Howard E. Dow
19
Audience - Project - TerminologyWhat the team didResults and dataBenefits, pitfalls and how toConclusions
Agenda
Copyright (c) 2003 Howard E. Dow
20
Results
No operational defects found in inspected test scripts
On time delivery
Increased skill level Defects
Copyright (c) 2003 Howard E. Dow
21
Summary Data - 1
TotalsScripts inspected: 17
LOC inspected: 12,949
Defects found: 544
Time spent: 24.2 hours 75 hours
Copyright (c) 2003 Howard E. Dow
22
Summary Data - 2
Number of Scripts Written
0
2
4
6
8
A B C
Engineer
Copyright (c) 2003 Howard E. Dow
23
Summary Data - 3
Calculated Defect density: 42 Defects/KLOC
Inspection rate: 535 LOC/hour 173 LOC/hour
Defect find rate: 22.5 Defects/hour 7 Defects/hr
Copyright (c) 2003 Howard E. Dow
24
Summary Data - 4
Severity type Quantity Percent of total
High 15 2.8%
Medium 209 38.4%
Low 320 58.8%
Total 544 100%
Copyright (c) 2003 Howard E. Dow
25
Percent Severity
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%1 3 5 7 9 11
13
15
17
Inspection Date (earliest to most recent)
% Low PriorityDefects
% Medium PriorityDefects
% High PriorityDefects
Copyright (c) 2003 Howard E. Dow
26
Inspectors per Inspection
0
1
2
3
4
5
6
7
0 2 4 6 8 10 12 14 16 18
Inspection Date (earliest to most recent)
# o
f In
sp
ec
tors
Copyright (c) 2003 Howard E. Dow
27
Defects per KLOC
0.020.040.0
60.080.0100.0120.0
140.0160.0
0 2 4 6 8 10 12 14 16 18
Inspection Date (earliest to most recent)
De
fec
ts/K
LO
C
Copyright (c) 2003 Howard E. Dow
28
Defects Found per Hour
0.0
5.0
10.0
15.0
0 2 4 6 8 10 12 14 16 18
Inspection Date (earliest to most recent)
Copyright (c) 2003 Howard E. Dow
29
Defect Count vs. Size (LOC)
0
1020
3040
50
6070
8090
100
0 500 1000 1500 2000
Size (LOC)
De
fec
t C
ou
nt
Copyright (c) 2003 Howard E. Dow
30
Size vs. Defects/KLOC
0
500
1000
1500
2000
2500
0.0 20.0 40.0 60.0 80.0 100.0 120.0 140.0 160.0
Defects/KLOC
Siz
e
Copyright (c) 2003 Howard E. Dow
31
ObservationDefect Count vs. Size (LOC)
0
1020
3040
50
6070
8090
100
0 500 1000 1500 2000
Size (LOC)
De
fec
t C
ou
nt
Copyright (c) 2003 Howard E. Dow
32
Observation
% Medium Severity Defects
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
0 2 4 6 8 10 12 14 16 18
Inspection Date (earliest to most recent)
Copyright (c) 2003 Howard E. Dow
33
Observation
Size vs. Defects/KLOC
0
500
1000
1500
2000
2500
0.0 20.0 40.0 60.0 80.0 100.0 120.0 140.0 160.0
Defects/KLOC
Siz
e
???
Copyright (c) 2003 Howard E. Dow
34
Size (LOC)
0
500
1000
1500
2000
2500
0 1 2 3 4 5 6 7 8
Size(LOC)Eng A
Size(LOC)Eng B
Size(LOC)Eng C
Observation
Copyright (c) 2003 Howard E. Dow
35
Defects per KLOC
0.0
20.0
40.0
60.0
80.0
100.0
120.0
140.0
160.0
0 1 2 3 4 5 6 7 8
Defects/KLOCEng A
Defects/KLOCEngr B
Defects/KLOCEng C
Excellent!
Observation
Copyright (c) 2003 Howard E. Dow
36
Audience - Project - TerminologyWhat the team didResults and dataBenefits, pitfalls and how toConclusions
Agenda
Copyright (c) 2003 Howard E. Dow
37
Benefits
No operational defects reported
On time delivery
Value of process shown
Some increase in expertise
Copyright (c) 2003 Howard E. Dow
38
Pitfalls
Must be
“Sloppy” data collection
Need for more training
Copyright (c) 2003 Howard E. Dow
39
How to
Find a “critical” project
Suggest doing something different
Keep it simple - be flexible
Propose “Let’s put more eyes on the code.”
“Let’s inspect the code.”
Copyright (c) 2003 Howard E. Dow
40
Audience - Project - TerminologyWhat the team didResults and dataBenefits, pitfalls and how toConclusions
Agenda
Copyright (c) 2003 Howard E. Dow
41
Conclusions
Test scripts can be inspected
Share your resultsOther testers
Developers
Useful as a learning tool
Avoid unrealistic expectations
Copyright (c) 2003 Howard E. Dow
42
ReferencesThe original article on software inspections. “Design and Code Inspections to Reduce Errors in Program Development”, Michael Fagan, IBM Systems Journal, Volume 15, No. 3, 1976. This may be available as reprint at: http://www.research.ibm.com/journal/si/382/fagan.pdf
Handbook of Walkthroughs, Inspections, and Technical Reviews, Daniel P. Freedman and Gerald M. Weinberg, 3rd Edition, Dorset House, 1990
Software Inspection, Tom Gilb and Dorothy Graham, Addison-Wesley, 1990
Copyright (c) 2003 Howard E. Dow
43
Howie Dowhowie.dow@rcn.com
hdow@alumni.carnegiemellon.edu
Thank you for your time.
Questions please
top related