software defect modeling at jpl john n. spagnuolo jr. and john d. powell 19th international forum on...
Post on 21-Dec-2015
214 views
TRANSCRIPT
Software Defect Modeling at JPL
John N. Spagnuolo Jr. and John D. Powell19th International Forum on
COCOMO and Software Cost Modeling
10/27/2004
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
2
Measurement & BenchmarkingProcess & Product Definition
Capture, define, and refine repeatable processes
and a set of engineering practices for project use
Provide measurement infrastructure for projects,
conduct empirical analysis, and package experiences for future use
Software Engineering Technology Infusion
Deployment
Identify, evaluate, and support software tools and techniques
to facilitate process and product improvement
Infuse practices into project use;provide training, products, mentoring
and consulting for projects
SQI Project EngineeringProvide overall technical infrastructure
and work element integration
SQI Project EngineeringProvide overall technical infrastructure
and work element integration
The JPL SQI Project
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
3
SQI Measurement & Benchmarking• The objective of the SQI Measurement Program is to provide
the basis for a quantitatively based software management approach– Define models and measures– Create an infrastructure– Provide consulting and support– Produce Handbooks & Training
• Cost estimation and planning– Help develop total cost, schedule and plan for project activities and phases
• Quality planning and assessment– Help predict and assess the quality of products
• Management tracking – Help managers plan / monitor activities and assess risks during project execution
• Guiding improvement– Help JPL assess the overall effectiveness of software processes
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
4
Quality Purpose and Goals
• The purpose of this work is to predict software quality via the development of Defect Prediction Models
– Predict defects and their effects on software projects early in project’s life cycle
• The current goal of this work is to make use of analysis of JPL defect data to support decision making.
– Determine and make use of existing software defect trends– Determine the driving forces behind exceptions to trends
• Critical Discriminators (CDs) are the distinguishing characteristics of a project(s) that capture/quantify the driving force behind a deviation from over all JPL defect trends
• Combine Critical discriminators along with trends to assess threats to software project goals
– Provide a means for managers to make use of Trends and CDs for • Planning• Prediction• Corrective Actions• Process Improvement
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
5
Project-Defect-SLOC-ChartJ PL PROJECT Partial Defects All Defects Analyzed Defects Code Data Correlated Influence
1 Project F12 Project F23 Project F34 Project F45 Project F56 Project F67 Project F78 Project F89 Project F910 Project F1011 Project F1112 Project F1213 Project F1314 Project F1415 Project F1516 Project F1617 Project F1718 Project F18
19 Project G120 Project G221 Project G322 Project G423 Project G524 Project G625 Project G726 Project G827 Project G928 Project G1029 Project G1130 Project G1231 Project G1332 Project G1433 Project G1534 Project G1635 Project G1736 Project G1837 Project G1938 Project G2039 Project G2140 Project G2241 Project G2342 Project G2443 Project G2544 Project G26
Flight Software
Ground Software
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
6
Basic Defect Prediction Approach
0
500
1000
1500
2000
2500
0 100 200 300 400 500 600 700 800 LKSLOC
Nu
mb
er
of
defe
cts
FSW
FSW & GSW
GSW
Early Defect Measurement
• Average Defect Density for– Flight Software (FSW)– Ground Software (GSW)– FSW & GSW
• Plot Lines with Slopes = Densities
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
7
One Perspective of Data:
Average Work Hours per SDS vs SLOC • Formed Two Linear Trends• Philosophical Connection with Subsequent Charts
Average WH's/PFR vs LKSLOC
6.3
0
4.94
4.6042
11.4
8.85
2
4.293
5.539
14.4055
0
2
4
6
8
10
12
14
16
0 20 40 60 80 100 120 140 160 180 200
LKSLOC
Avera
ge W
H's
per
PFR
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
8
Preview of Major Findings
• New Trends: Given the basic Defect prediction Approach (Previous slide), the actual JPL defect data shows defects trends that are:– Are Counter intuitive on first inspection– More complex on further inspection
• New CDs: Newly developed software size CDs simplify the complex and counterintuitive Defect versus Size trend.– Managers may quickly make use of simple valuable sub-trends that apply
to their projects:• Without dealing with overall trend complexities that may be based on factors
that do not apply to their project• Based on characteristics such as size that may be estimated early in the
software lifecycle and easily revised for accuracy as the project progresses.– The new CD provides a means for managers to make use of the trends for
planning, prediction, corrective actions and process improvement.
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
9
The High Level Story
•Difference between Fault and Failure–Fault : error which may or may not have been discovered–Failure : Outward sign of an error (Discovered Fault)–In This presentation Failure = Defect
•Related Literature suggests that faults vs. code size may fit to an exponential curve
•Relation of exponential curve to Project G4 Data Plots–Linear fit is reasonable for Project G4 defects vs. size–Not necessarily a contradiction
•All faults may not have to be fixed•More testing resources and/or time
•Project G4 defect curve ~ exponential fault curve
•Expand and correlate Project G4 analysis to all FSW and GSW Data collected thus far
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
10
Project G4 and JPL Projects
• Project G4 DATA : Number of Defects vs. SDS SLOC– Project G4 TEST (SDS’s) - Quadratic Fit the best – Project G4 OPS (SDS’s) - Quadratic Fit is the best – Project G4 TEST & OPS (SDS’s) - Quadratic Fit is the best– Project G4 TEST + OPS (SDS’s) - Quadratic Fit is the best
• JPL Projects: Number of Defects vs. Project Size– All FSW - Quadratic fit is the best– All GSW - Quadratic Fit is the best– All FSW & GSW - Quadratic Fit is the best
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
11
All Collected FSW-TEST-June 30-2004
y = -0.0749x 2 + 28.781x - 363.52
R2 = 0.9981
0100200300400500600700800900
10001100120013001400150016001700180019002000210022002300240025002600
0 50 100 150 200 250 300 350
LKSLOC
Num
ber
of
Defe
cts
FSW
Poly. (FSW)
Interesting Fit for ALL JPL FSW
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
12
JPL GSW -TEST June 30 2004
y = -0.0113x 2 + 8.7463x - 8.1403
R2 = 0.8374
0
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1400
1500
1600
1700
1800
0 100 200 300 400 500 600 700 800
LKSLOC
Nu
mber
of
Defe
cts
All JPL FSW & GSW:TEST June 30 2004
y = -0.0104x 2 + 7.8956x + 135.87
R2 = 0.689
0
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1400
1500
1600
1700
1800
0 100 200 300 400 500 600 700 800
LKSLOC
Num
ber o
f Def
ects
JASON1 GSW TEST
y = -0.0023x 2 + 0.7257x + 4.3511
R2 = 0.5132
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
0 50 100 150 200 250 300
LKSLOC
Num
ber
of D
efe
cts
JASON1 GSW : OPS
y = -0.0029x 2 + 0.5949x + 2.3269
R2 = 0.5848
0
2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
32
34
36
38
40
0 20 40 60 80 100 120 140 160 180 200
LKSLOC
Num
ber
of D
efe
cts
Curve Extends to GSW, GSW&FSWProject G4 TEST, OPS, TEST & OPS,TEST+OPS
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
13
Trends and Potential Explanation
• Upper bound on amount of testing resources & time– Testing resources / KSLOC is higher for smaller modules than for larger modules.– One possible explanation of defects / size ratios– Smaller Modules
• As code gets bigger more people tend to test more• Defect curve increases to a certain point
– Medium Modules• With further size increases, testing resource approach their upper limit• Defect curve begins to “level off”
– Larger Modules• Upper bound on testing resources for Project G4• Testing resources/KSLOC not big enough to maintain defect / size ratio
observed in smaller modules• Eventually defects / KSLOC trend curve begins to decrease
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
14
Al FSW & GSW < 248 LKSLOC
y = 65.224e0.0317x
R2 = 0.7548
0100200300400500600700800900
10001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500
0 20 40 60 80 100 120 140
FSW&GSW< 248 LKSLOCKExpon. (FSW&GSW< 248 LKSLOCK)
Where Does the Exponential come in? All FSW and GSW: Exponential Fit for LKSLOC < 247
All FSW & GSW
y = 220.32e 0.0005x
R2 = 0.0138
y = -0.0107x 2 + 8.0852x + 156.95
R2 = 0.7076
y = 0.0004x + 497.46
R2 = 6E-08
050
100150200250300350400450500550600650700750800850900950
10001050110011501200125013001350140014501500155016001650170017501800
0 100 200 300 400 500 600 700 800
All FSW & GSWVertical1Vertical2Expon. (All FSW & GSW)Poly. (All FSW & GSW)Linear (All FSW & GSW)Linear (Vertical1)Linear (Vertical2)
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
15
Exponential Fits for JPL FSW, GSW Project G4 GSW TEST and Project G4
GSW OPSFSW < 160
LKSLOC
y = 19.295e 0.0661x
R2 = 0.9207
0
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1400
1500
1600
1700
1800
1900
2000
0 10 20 30 40 50 60 70 80
GSW < 247 LKSLOC
y = 125.53e 0.0189x
R2 = 0.9157
0
50
100
150
200
250
300
350
400
450
500
550
600
650
700
750
800
850
0 10 20 30 40 50 60 70 80 90 100
JASON1 GSW: Exponential Fit for LKSLOC < 103
y = 1.6196e0.0992x
R2 = 0.874
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
0 5 10 15 20 25 30 35 40
LKSLOC
# o
f D
efe
cts
JASON1 OPS # of pfr's vs size <57K
y = 2.1255e 0.0777x
R2 = 0.8899
0
3
6
9
12
15
18
21
24
27
30
33
36
39
42
0 5 10 15 20 25 30 35 40
LKSLOC
# o
f pf
r's
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
16
• Examination of Lower SLOC range: Project G4– Computed (max LKSLOC - min LKSLOC)/3– Project G4 TEST- Excellent fit to Exponential Curve < 103 LKSLOC– Project G4 OPS - Excellent fit to Exponential Curve < 57 LKSLOC– Project G4 TEST & OPS - Excellent fit to Exponential Curve < 103
LKSLOC– Project G4 TEST + OPS - Excellent fit to Exponential Curve < 103
LKSLOC
• Examination of Lower SLOC range: FSW, GSW, FSW & GSW
– Computed (max LKSLOC - min LKSLOC)/3– All FSW - Excellent fit to Exponential Curve < 118 LKSLOC– All GSW - Excellent fit to Exponential Curve < 247 LKSLOC– All FSW and GSW - Excellent fit to Exponential Curve < 247 LKSLOC
Relevance to Exponential Curve
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
17
Observations at this Point
• The Empirical Evidence thus far seems to suggest:
– Number of defects seems to increase as code size increases, peaks out, then begins a decreasing trend.
– The exponential relationship between number of faults vs. code size, as indicated by the “Exponential curve”, appears to be most closely approximated by the relationship between the number of defects vs. code size for smaller sized Projects/SDS’s here at JPL
– Code does not require exponential discovery of faults to work appropriately for JPL purposes
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
18
Summary• Defect Differences from Test to Operations by SDS
• The Size Critical Discriminator (CD) for Defect Rates– Size CD is a code size threshold
• Represents successful development of a new and valuable CD• Distinguishes between very different expected defects rate behaviors• Allows defect prediction superior to simple “defects per lines of code” models
– Significance of the Size CD• Intuitive results below Size CD threshold• Counterintuitive results above the Size CD threshold
– Plausible and Reasonable explanations – Valuable rules for projects
– Calculation of the Size CD and Differing Trends • Analysis of Size CD and Defect Trends gives reasonable confidence level• Very High Statistical Confidence Levels (R2, etc..) but more data is need to make
claims of high confidence overall• Actual Data driven results• Trends persist
– Across multiple analyses of the data– Across various subsets of the data
10/27/2004 John N. Spagnuolo Jr. &John D. Powell
19
• Relations between criticality 1 and criticality 2 defects• For each “Number of PFR’s vs.. size” chart, construct a chart
corresponding to defect density vs.. size – determine corresponding hump in trend curves
• Identify and Classify More Critical Discriminators– New individual CDs– Interaction of multiple CDs– For each CD associate modules, characteristics and data– Establish Trends for all modules corresponding to each CD
• Investigate interaction of multiple CDs at work in a given project / SDS
Future Work