copyright (c) 2003 howard e. dow1 results from inspecting test automation scripts howie dow...

Copyright (c) 2003 Howard E. Dow

1

Results from Inspecting Results from Inspecting Test Automation ScriptsTest Automation Scripts

Howie Dow

[email protected]

[email protected]


2

Agenda

Audience - Project - Terminology

What the team did

Results and data

Benefits, pitfalls and how to

Conclusions


3

Car Cah

Parked Pahked

Rs ????Idea

Idear


4

Target Audience

EngineersProject leaders

IdeallyTrained on inspectionsSome inspection experience


5

Project

Call center system

Functional tests

GoalsDefect free test scripts

On time

Increase skill level


6

Skill levelsSkill Level (Red=Available, Green=Needed)

0

0.5

1

1.5

2

2.5

3

3.5

1 2 3 4

Engineer Number

Ski

ll L

eve

l (1

=L

ow

, 2=

Me

diu

m,

3=H

igh

)


7

Terminology

Inspection = Software Inspection = Formal Inspection

= “Fagan Style” Inspection

Reviews

Walk-Throughs

Any technique


8

Terminology

Inspection characteristicsGoals

Rules, Roles, Responsibilities

Process


9

Goals of Inspections

Find defects

Collect data

Archive data

Use data

Learn


10

Etiquette, Customs and Rules

Focus on the product

Be prepared

Open mind

Language


11

Roles and Responsibilities

Moderator Recorder

Reader Author

Management

Inspector


12

Planning

ProductOverview

Preparation

Inspection

Rework

Data analysisand storage

Re-inspect(optional)

Inspection Process


13

Audience - Project - TerminologyWhat the team didResults and dataBenefits, pitfalls and how toConclusions

Agenda


14

Moderator Recorder

Reader Author Inspector

Manager

Roles and Responsibilities


15

ProductOverview

Planning

Preparation

Inspection

Rework

Data analysisand storage

Re-inspect(optional)

Allowedsuggestions

Time not collected

Inspection Process Used


16

Summary of key differences

Omit product overview

Involved management

Suggesting fixes allowed

Strong bias to re-examine after rework by another team member

Omission - collecting preparation time


17

Ready to go

17 Scripts

3-6 inspectors


18

Data CollectedScript identifier

Author

Date completed

Date inspected

Size

Defect: number, location, description

Inspection time

Inspectors


19


Agenda


20

Results

No operational defects found in inspected test scripts

On time delivery

Increased skill level Defects


21

Summary Data - 1

TotalsScripts inspected: 17

LOC inspected: 12,949

Defects found: 544

Time spent: 24.2 hours 75 hours


22

Summary Data - 2

Number of Scripts Written

0

2

4

6

8

A B C

Engineer


23

Summary Data - 3

Calculated Defect density: 42 Defects/KLOC

Inspection rate: 535 LOC/hour 173 LOC/hour

Defect find rate: 22.5 Defects/hour 7 Defects/hr


24

Summary Data - 4

Severity type Quantity Percent of total

High 15 2.8%

Medium 209 38.4%

Low 320 58.8%

Total 544 100%


25

Percent Severity

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%1 3 5 7 9 11

13

15

17

Inspection Date (earliest to most recent)

% Low PriorityDefects

% Medium PriorityDefects

% High PriorityDefects


26

Inspectors per Inspection

0

1

2

3

4

5

6

7

0 2 4 6 8 10 12 14 16 18


# o

f In

sp

ec

tors


27

Defects per KLOC

0.020.040.0

60.080.0100.0120.0

140.0160.0

0 2 4 6 8 10 12 14 16 18


De

fec

ts/K

LO

C


28

Defects Found per Hour

0.0

5.0

10.0

15.0

0 2 4 6 8 10 12 14 16 18



29

Defect Count vs. Size (LOC)

0

1020

3040

50

6070

8090

100

0 500 1000 1500 2000

Size (LOC)

De

fec

t C

ou

nt


30

Size vs. Defects/KLOC

0

500

1000

1500

2000

2500

0.0 20.0 40.0 60.0 80.0 100.0 120.0 140.0 160.0

Defects/KLOC

Siz

e


31

ObservationDefect Count vs. Size (LOC)

0

1020

3040

50

6070

8090

100

0 500 1000 1500 2000

Size (LOC)

De

fec

t C

ou

nt


32

Observation

% Medium Severity Defects

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

0 2 4 6 8 10 12 14 16 18



33

Observation

Size vs. Defects/KLOC

0

500

1000

1500

2000

2500

0.0 20.0 40.0 60.0 80.0 100.0 120.0 140.0 160.0

Defects/KLOC

Siz

e

???


34

Size (LOC)

0

500

1000

1500

2000

2500

0 1 2 3 4 5 6 7 8

Size(LOC)Eng A

Size(LOC)Eng B

Size(LOC)Eng C

Observation


35

Defects per KLOC

0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

160.0

0 1 2 3 4 5 6 7 8

Defects/KLOCEng A

Defects/KLOCEngr B

Defects/KLOCEng C

Excellent!

Observation


36


Agenda


37

Benefits

No operational defects reported

On time delivery

Value of process shown

Some increase in expertise


38

Pitfalls

Must be

“Sloppy” data collection

Need for more training


39

How to

Find a “critical” project

Suggest doing something different

Keep it simple - be flexible

Propose “Let’s put more eyes on the code.”

“Let’s inspect the code.”


40


Agenda


41

Conclusions

Test scripts can be inspected

Share your resultsOther testers

Developers

Useful as a learning tool

Avoid unrealistic expectations


42

ReferencesThe original article on software inspections. “Design and Code Inspections to Reduce Errors in Program Development”, Michael Fagan, IBM Systems Journal, Volume 15, No. 3, 1976. This may be available as reprint at: http://www.research.ibm.com/journal/si/382/fagan.pdf

Handbook of Walkthroughs, Inspections, and Technical Reviews, Daniel P. Freedman and Gerald M. Weinberg, 3rd Edition, Dorset House, 1990

Software Inspection, Tom Gilb and Dorothy Graham, Addison-Wesley, 1990


43

Howie [email protected]

[email protected]

Thank you for your time.

Questions please

copyright (c) 2003 howard e. dow1 results from inspecting test automation scripts howie dow...

Documents

copyright c

inspection experience

preparation time slide

responsibilities process

conclusions agenda slide

idea idear slide

data archive data use

data benefits