www.ischool.drexel.edu info 631 prof. glenn booker week 1 – defect analysis and removal 1info631...
TRANSCRIPT
www.ischool.drexel.edu
INFO 631 Prof. Glenn Booker
Week 1 – Defect Analysis and Removal
1INFO631 Week 1
www.ischool.drexel.eduINFO631 Week 1 2
Motivation
• Much of software measurement is devoted to improving the quality of the product
• To do so, it helps to understand how, when, and why mistakes are made during the software life cycle, resulting in defects
• It is assumed that defects are queued for being fixed (removed) after they are detected
www.ischool.drexel.eduINFO631 Week 1 3
Defect Analysis Overview
• Defect analysis looks at:– When a defect was created (“injected”)– When a defect was found (“detection”)– What caused a defect (type of defect, and/or
orthogonal defect classification)– How a defect was found (“triggers”)
• Yes, there are often many terms for the same activity!
www.ischool.drexel.eduINFO631 Week 1 4
Defect Creation (Injection)
• Defects may be created, detected, and removed during every phase of the software life cycle, including during maintenance– Most are created during requirements
analysis, design, and coding– Testing and maintenance create relatively
few defects
www.ischool.drexel.eduINFO631 Week 1 5
Waterfall Life CycleRequirements
Analysis
High LevelDesign
Low Level(Detailed)
Design
Coding
Unit TestingComponent(Integration)
Testing
SystemTesting
Maintenance
www.ischool.drexel.eduINFO631 Week 1 6
Defect Detection
• Defects may be found:– During quality assurance activities between
phases of the life cycle (such as during major reviews and inspections); and often
– Within each phase (during preliminary reviews, peer reviews and testing activities)
• Here we mostly consider the former (between-phase activities)
www.ischool.drexel.eduINFO631 Week 1 7
Defect Detection
• Defects are found (discovered) by– Reviews and formal inspections
• For requirements analysis, high level design, low level design, and coding
– Testing• Unit testing• Integration testing• System testing• Acceptance testing
www.ischool.drexel.eduINFO631 Week 1 8
Origin of Defects & Type
• What kind of defects are made in each life cycle phase?– Requirements - incorrect specification;
missing requirements– High level design - design does not cover all
requirements, or is inflexible– Low level design - mismatch between HLD
and LLD; design does not cover all requirements
www.ischool.drexel.eduINFO631 Week 1 9
Origin of Defects & Type
– Coding - code errors – Integration – interface, compatibility problems– Unit testing - bad fixes – Component testing - bad fixes– System testing - bad fixes– Acceptance testing - bad fixes– Maintenance (enhancement) - bad
requirements, design, coding, testing
www.ischool.drexel.eduINFO631 Week 1 10
Defect Causal Analysis
• Regardless of when defects are created or discovered, they may be analyzed to determine their cause– Causes may range from the mundane to the
esoteric– Once identified, causes should be recorded
to support defect prevention activities
www.ischool.drexel.eduINFO631 Week 1 11
Types of Defect Causes
• Some causes may include:– Unclear requirements– Incorrect architecture – Incomplete design– Unfamiliarity with programming language– Typos– Inadequate understanding of interfaces– Inadequate understanding of standards (e.g.
TCP/IP, ODBC, SQL, etc.)
www.ischool.drexel.eduINFO631 Week 1 12
Types of Defect Causes
• and:– Poor documentation for legacy system– Design based on outdated requirements– Code written to outdated design– Poor design for future expansion (e.g.
hardwired constants, buried assumptions, etc.)
– Conflicting requirements– Bad fix of a previous problem– …and many more
www.ischool.drexel.eduINFO631 Week 1 13
Orthogonal Defect Classification
• ODC is a formal type of defect causal analysis
• Classify defects by the type of defect, and the life cycle phase when it was created
• This particular scheme is still somewhat experimental, but the basic concept is widely accepted
www.ischool.drexel.eduINFO631 Week 1 14
ODC - Defect Type
• A possible set of definitions:– Function error–affects capability, user, product or
hardware interfaces, global data structures– Assignment error – errors with initialization of
control blocks or data structure– Interface error – errors in interacting with other
components, modules, or or device drivers– Checking – errors in validating data and values
www.ischool.drexel.eduINFO631 Week 1 15
ODC - Defect Type
– Timing/serialization – errors in management of shared and real-time resources
– Build/package/merge – errors due to mistakes in library systems, change management, or version control
– Documentation – errors in publications and maintenance notes
– Algorithm – errors regarding efficiency or correctness that affect the task
www.ischool.drexel.eduINFO631 Week 1 16
Orthogonal Defect Classification
Type Phase CausedFunctional DesignInterface Low Level Design
(LLD) Checking LLD or Coding Assignment CodingTiming LLDBuild/package Library toolsDocumentation PublicationsAlgorithm LLD
www.ischool.drexel.eduINFO631 Week 1 17
Defect Trigger
• Is a condition that allows a defect to surface (become visible); examples include:
• Design conformance• Logic or data flow incomplete• Workload or stress (performance)• Boundary conditions (extremes)• Bug fix• Recovery timing• User code
www.ischool.drexel.eduINFO631 Week 1 18
Defect Removal Modeling
• Defect removal is critical to reducing (development or maintenance) cycle time and cost, and improving quality
• Michael Fagan and Capers Jones are noteworthy authors
• See, for example, “Fagan Inspections”
www.ischool.drexel.eduINFO631 Week 1 19
Defect Removal Model (DRM)
• Software development phase-based DRM covers– Defect injection into a phase – Defect removal during a phase– Defect carryover across phases– Effectiveness of defect removal within a
phase
www.ischool.drexel.eduINFO631 Week 1 20
Defect Removal Model
• DRM is a quality management tool• Provides insights into the defect removal process
and where this process might be improved• Used after a project is completed; provides post
mortem information about that project• If defect removal process is similar for a new
project, then the DRM can be used to improve the defect removal process of that new project
www.ischool.drexel.eduINFO631 Week 1 21
Defect Removal Model
• Defects are created (injected) by incorrect requirements gathering, analysis, design, coding, or by bad fixes– Any given defect can be removed in the
phase in which it was injected or in a later phase
• Remove defect means fix bug, redo specs and/or design as needed
www.ischool.drexel.eduINFO631 Week 1 22
Defect Removal Effectiveness
• Phase based defect detection activities– Requirements analysis & inspection– Design inspection– Code inspection, Build verification testing– Unit, string, integration, regression, system
testing– Fix verification inspection & testing
www.ischool.drexel.eduINFO631 Week 1 23
Defect Removal Model
Applies to each life cycle phase
Life Cycle Phase
Defects Injected (new mistakes)
NetDefectsfrom previous Phase
Defect Detection
(inspection)
Known Defects
Undetected Defects
Fixed Defects
Unfixed Defects + Bad Fix Defects
Net Defects
Bad Fix Defects
www.ischool.drexel.eduINFO631 Week 1 24
Defect Removal Effectiveness
• For any given phase of defect removal
• Defects present at removal operation– Defects found during removal operation +
defects found later– May be determined ex post facto (after the
fact), or using a statistical prediction model
Removal Effectiveness = Defects found by removal operation
Defects present at removal operation________________________________ X 100
www.ischool.drexel.eduINFO631 Week 1 25
Defect Removal Effectiveness
• Defect removal effectiveness = (# of defects found by inspection) /(# of defects originally present) *100
• Early detection percentage = (# of major inspection errors) /(# of major and minor errors) * 100
www.ischool.drexel.eduINFO631 Week 1 26
Defect Matrix Assumptions
• Defects are removed in the same life cycle phase when they are found
• No defects are knowingly left unfixed• No bad fixes
– Or at least they are blended into the number of defects created in that life cycle phase
www.ischool.drexel.eduINFO631 Week 1 27
Sample Defect Matrix—When Originated vs. When Found
When Originated (injected, or created)When Found/Fixed
Rqmts Top Lev. Design
Low Lev Design
Coding Unit Testing
Compon Testing
System Testing
Field Total
Rqmts Top Lev Design 49 681 730 Low Lev Design 6 42 681 729 Coding 12 28 114 941 1095 Unit Testing 21 43 43 223 2 332 ComponTesting 20 41 61 261 4 387 SystemTesting 6 8 24 72 1 111 Field 8 16 16 40 1 81 Total 122 859 939 1537 2 4 1 3465
1
0 0
www.ischool.drexel.eduINFO631 Week 1 28
Defect Removal Effectiveness
Life cycle Phase(s)
Defects passed from Previous life cycle phases
Defects Created this Phase
Defects Found & removed this Phase
Defects passed to Next life cycle phase
Next = (Previous + Created) – FoundDRE = Found / (Previous + Created) * 100
www.ischool.drexel.eduINFO631 Week 1 29
High Level Design Effectiveness
• There are no requirements defects removed; 122 defects are passed to HLD
• High (Top) Level Design (I0) Inspection Effectiveness– Defects found and removed at I0: 730– Defects existing on step entry (escapes from
requirements phase: 122– Defects injected in current phase: 859– E(I0) = 730/(122+859) x 100 = 74%
www.ischool.drexel.eduINFO631 Week 1 30
Low Level Design Effectiveness
• Low Level Design (I1) Inspection Effectiveness– Defects found and removed at I1: 729– Defects existing on step entry (escapes from
requirements phase and I0): 122+859-730 = 251
– Defects injected in current phase: 939– E(I1) = 729/(251+939) x 100 = 61%
www.ischool.drexel.eduINFO631 Week 1 31
Code Inspection Effectiveness
• Code Inspection (I2) Effectiveness– Defects found and removed at I2: 1095– Defects present on step entry (escapes from
requirements phase, I0, and I1): 251+939-729 = 461
– Defects injected in current phase: 1537– E(I2)= 1095/(461+1537) x 100 = 55%
www.ischool.drexel.eduINFO631 Week 1 32
Unit Testing Effectiveness
• Unit Testing (UT) Effectiveness– Defects found and removed at UT: 332– Defects existing on step entry (escapes from
previous phases): 461+1537-1095 = 903– Defects injected in current phase (bad fixes):
2– E(UT) = 332/(903+2) x 100 = 37%
• Can follow the same pattern for the other testing phases and post-release
www.ischool.drexel.eduINFO631 Week 1 33
Summary Effectiveness Measures
• Overall Design & Coding Inspection Effectiveness– IE = (730+729+1095)/(122+859+939+1537) x 100 =
74%• Overall Effectiveness of all Testing activities
– TE = (332+387+111)/(903+2+4+1)x100 = 91%• Overall Defect Removal Effectiveness of the
development process (not including release)– DRE = (0+730+729+1095+332+387+111)/(122+859+
939+1537+2+4+1) x 100 = 3384/3464*100 = 97.7%
www.ischool.drexel.eduINFO631 Week 1 34
Rayleigh Model
• The Rayleigh Model describes the number of defects which will be discovered, by development phase
• It’s a special case of the Weibull family of distributions, which we’ll cover later
www.ischool.drexel.eduINFO631 Week 1 35
Rayleigh Model Assumptions
• Defect rate observed during development process is positively correlated with defect rate in field
• Given the same defect injection rate, if more defects are discovered and removed earlier, fewer will remain in later stages, leading to fewer defects in the field
www.ischool.drexel.eduINFO631 Week 1 36
Rayleigh Model Inspections
• I0 = High Level Design Inspection• I1 = Low Level Design Inspection• I2 = Code Inspection• UT = Unit Testing • CT = Component (Integration) Testing• ST = System Testing• GA = after General Availability (release or
Fielding of the system)
www.ischool.drexel.eduINFO631 Week 1 37
Rayleigh Model
Number of Defects
Development PhaseI0 I1 I2 UT CT ST GA
Each bar represents the number of defects found during that life cycle phase’s inspection activity
www.ischool.drexel.eduINFO631 Week 1 38
Desired Rayleigh Curve TrendsEffect of Early Defect Removal and Reducing Error Injection
www.ischool.drexel.eduINFO631 Week 1 39
Cost of Defect Removal by Phase
• Defect removal and rework is less costly the closer that the defects are found relative to the phase in which they are injected– Rework in the I0, I1, and I2 inspection levels can be
10 to 100 times less expensive than if it is done during formal testing
– Reviews can reduce number of defects reaching testing phases by factor of 10• These reductions cut testing costs by 50-80%, even
including the review costs
www.ischool.drexel.eduINFO631 Week 1 40
Cost Effectiveness of DRM
• Relative cost of fixing a problem found in design/coding, testing, or after release are: 1:20:82 (Remus, 1983) 1:13:92 (Kan, 1989)
• Cost of defect removal can be analyzed by inspection type, testing phase, defect severity, defect origin, etc.