1 experience-driven process improvement boosts software quality © software quality week 1996...
TRANSCRIPT
1Experience-Driven Process Improvement Boosts Software Quality©
Software Quality Week 1996
Experience-Driven Process Improvement
Boosts Software Quality
-
Otto Vinter
Manager Software Technology and Process Improvement
email: [email protected]
2Experience-Driven Process Improvement Boosts Software Quality©
Experience-Driven Process Improvement Boosts Software Quality
Brüel & KjaerSkodsborgvej 307, DK-2850 Naerum, Denmark
Tel: +45 4280 0500, Fax: +45 4280 1405
High-Precision Electronic Instrumentation for
• Sound
• Vibration
• Condition Monitoring
• Gas Measurements
3Experience-Driven Process Improvement Boosts Software Quality©
European System and Software Initiative (ESSI)
An Accompanying Measure to ESPRIT• The European Strategic Programme for Research and
Development in Information Technologies
ESSI Objectives• Promote Improvements in the Software Development
Process in Industry
• Improve Current Practice by Applying State-of-the-art in Software Engineering
• Evaluate State-of-the-art Supports
• Disseminate Experience across Borders and Industrial Sectors
ESSI Lines of Actions• Assessments
• Process Improvement Experiments
• Dissemination
4Experience-Driven Process Improvement Boosts Software Quality©
The PET Process Improvement Experiment
The Prevention of Defects through
Experience-Driven Test Efforts
(PET)
PET Objectives• Extract knowledge on frequently occurring problems in
the development process for embedded software
• Change the development process by defining the optimum set of methods and tools available to prevent these problems reappearing
• Measure the impact of the changes in a real-life development project
Partner in the Consortium: DANFOSS
• a leading manufacturer of mechatronic products
• performing a similar experiment
5Experience-Driven Process Improvement Boosts Software Quality©
Defect Analysis from Error Logs
Error Logs Analysed• Embedded software development projects
• Project sizes app. 7 manyears
• 1100 bugs analyzed from the error logs
• Bugs are anything between serious defects and suggestions for improvements
• Bug reporting starts in the integration phase
• Bug reports covered a period until 18 months after first release
6Experience-Driven Process Improvement Boosts Software Quality©
Defect Analysis from Error Logs
Bug Categorisation• Based on a bug classification scheme by Boris Beizer:
- Boris Beizer: Software Testing Techniques, Second Edition, Van Nostrand Reinhold
• comprehensive set of bug categories
• contains statistics from many projects
• categorization performed in teams
- 1-2 developers and 1-2 process consultants
- app. 5 minutes / bug
7Experience-Driven Process Improvement Boosts Software Quality©
The Beizer Bug Classification Scheme
1. Requirements and Features
2. Functionality as Implemented
3. Structural Bugs
4. Data
5. Implementation (standards violation, and documentation)
6. Integration
7. System and Software Architecture
8. Test Definition or Execution Bugs
9. Other Bugs, Unspecified
Each category detailed to a depth of up to 4 levels
8Experience-Driven Process Improvement Boosts Software Quality©
Defect Analysis from Error Logs
Category Our Analysis Beizer Statistics
1.Requirements 23,5 % 8,1 %
2. Functionality 24,3 % 16,2 %
3. Structural 20,9 % 25,2 %
4. Data 9,6 % 22,4 %
5. Implementation 4,3 % 9,9 % (5,9 %)
6. Integration 5,2 % 9,0 %
7.Architecture 0,9 % 1,7 %
8. Test 6,9 % 2,8 %
9.Unspecified 4,3 % 4,7 %
TOTAL 100,0 % 100,0 %
9Experience-Driven Process Improvement Boosts Software Quality©
Defect Analysis from Error Logs
Other Questions to Capture Subjective Information on the Bugs • when was the bug found in the development life-cycle
• frequency of bugs found over time
• in which part (module) of the product
• who found the bug
• what could prevent the bug
10Experience-Driven Process Improvement Boosts Software Quality©
Defect Analysis from Error Logs
Example of an Error-prone Product
Weeks
Bu
gs R
epor
ted
Trial-releaseProduction-release
11Experience-Driven Process Improvement Boosts Software Quality©
Defect Analysis from Error Logs
Results of the Analysis• no special bug class dominates embedded software
development
• requirements problems, and requirements related problems, are the prime bug cause (36%)
• problems due to lack of systematic unit testing is the second largest bug cause (22%)
• management attention to software process issues
Actions to Improve Unit Testing • introduction of static and dynamic analysis
• host/target tools
• basic set of metrics
12Experience-Driven Process Improvement Boosts Software Quality©
The PET Experiment
Original Objective:• to implement changes to the testing process in the
development of the next version of a product and measure the results
Revised Objective:• to assess a trial-release of a product
• to increase test coverage to industry best practice (branch coverage > 85%)
• to measure the effect after production release
• and determine the effectiveness of static/dynamic analysis
13Experience-Driven Process Improvement Boosts Software Quality©
Static / Dynamic Analysis Results
108 Bugs Found before Trial Release• 73 Bugs found by regression testsuite
- 66% branch coverage achieved
• 105 Person days used
60 Bugs Found by Static / Dynamic Analysis• 33 Bugs found by static analysis
• 27 Bugs found by dynamic analysis
- 93% branch coverage achieved
• 40 Person days used
46% Improvement in testing efficiency
14Experience-Driven Process Improvement Boosts Software Quality©
Results of Static Analysis
Type of Bug Distribution
• Use of uninitialised variable: 15 %
• Variable defined but not used in scope: 24 %
• Variable redefined with no use in between: 36 %
• Parameter mismatch: 6 %
• Unreferenced procedure: 18 %
• Declared but not used variable: 0 %
• Other types of static bugs: 0 %
Complexity Metrics
• 88 % correlation between procedures with McCabe > 10 and XLOC > 100
• Neither McCabe’s Metric nor Code Size correlated with bugs per line of code
15Experience-Driven Process Improvement Boosts Software Quality©
McCabe’s Cyclomatic Complexity Metric
Trial-release Complexity Distribution
34,1
20,9
13,5
7,34,9
3,2 2,7 1,3 1,1 1,6 0,7 0,7 1,0 1,0 0,7 0,7 0,4 0,4 0,3 0,13,3
0
5
10
15
20
25
30
35
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 >20
% Procedures
McCabe's complexity
McCabe > 10 : 9,3 %
16Experience-Driven Process Improvement Boosts Software Quality©
Static Flowgraph (McCabe = 10)
17Experience-Driven Process Improvement Boosts Software Quality©
Static Flowgraph (McCabe = 20)
18Experience-Driven Process Improvement Boosts Software Quality©
Trial-release Error Density
Error Density
0,1
0,5
1,0
00,20,40,60,8
1
1-10 11-20 >20
McCabe's complexity
Errors / Procedure
Error Density
0,1
0,8 0,8
0
0,2
0,4
0,6
0,8
1-100 101-200 >200
XLOC
Errors / Procedure
19Experience-Driven Process Improvement Boosts Software Quality©
Trial-release Error Average
Error Average
3,4
5,6
1,6
0
2
4
6
1-100 101-200 >200
XLOC
Errors / KXLOCError Average
1,11,4
0,4
0
0,5
1
1,5
1-10 11-20 >20
McCabe's complexity
Errors / KXLOC
20Experience-Driven Process Improvement Boosts Software Quality©
Dynamic Analysis Results
Test System for Dynamic Analysis
PC withStimGateInterface StimGate
PC withIEEEInterface
Testcases
IO
21Experience-Driven Process Improvement Boosts Software Quality©
Dynamic Analysis Results
Trial-release Branch Coverage Distribution
18%
0% 0% 0% 0% 1% 1% 1% 1% 0%4%
1% 3% 3% 2%5% 6%
10%
3%
41%
0%5%
10%15%20%25%30%35%40%45%
0-4
5-9
10-1
4
15-1
9
20-2
4
25-2
9
30-3
4
35-3
9
40-4
4
45-4
9
50-5
4
55-5
9
60-6
4
65-6
9
70-7
4
75-7
9
80-8
4
85-8
9
90-9
4
95-1
00
% branch coverage
Procedures
22Experience-Driven Process Improvement Boosts Software Quality©
Dynamic Analysis Results
# Tested Branches + # Inspected Branches ------------------------------------------------------------------- >= 85%
# Total Branches - # Dead Branches
• Final Coverage 93%
• Tested Branches 75%
• Dead Branches 9.5%
• Inspected Branches 9.5%
• Instrumented Code Expansion approximately 40%
• Massive Data Output During Execution (1 GB)
23Experience-Driven Process Improvement Boosts Software Quality©
Dynamic Analysis Results
Branch Coverage0%100%
% branch coverage%
Branch Coverage Distribution Achieved
1% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 1%5% 4%
10%3%
74%
0%
10%
20%
30%
40%
50%
60%
70%
80%
0-4
5-9
10-1
4
15-1
9
20-2
4
25-2
9
30-3
4
35-3
9
40-4
4
45-4
9
50-5
4
55-5
9
60-6
4
65-6
9
70-7
4
75-7
9
80-8
4
85-8
9
90-9
4
95-1
00
% branch coverage
Procedures
24Experience-Driven Process Improvement Boosts Software Quality©
Comparison of Test Efficiency
Hours per bug
• Static Analysis 1,6
• Current Development 7,2
• Dynamic Analysis 9,0
• Current Maintenance 14,0
25Experience-Driven Process Improvement Boosts Software Quality©
Measurements on Production Release
75% Reduction in Production-Release Bugs• Compared to Trial-Release
70% Requirements Bugs in Production-Release=> Increased focus on improving the requirements process
26Experience-Driven Process Improvement Boosts Software Quality©
Conclusions on Static/Dynamic Analysis
Performance Improvement • an efficient way to remove bugs
• marginal delay on trial release date
• marginal increase in the testing resources required
• immediate payback on tools, training & implementation
• remarkably improved test coverage
• increased quality
• reduced maintenance costs
• increased motivation
• applicable to the whole software development industry, incl. embedded software
27Experience-Driven Process Improvement Boosts Software Quality©
In Conclusion
Defect Analysis from Error Logs• is a simple and effective way to assess the software
development process
The Analysis of Bugs• has had a significant impact on the way we now look at
our software development process
• has established a basic set of metrics for test activities
• starting point for process improvement programmes in companies