a comparative analysis of the efficiency of change metrics and static code attributes for defect...
TRANSCRIPT
A Comparative Analysis of the Efficiency of Change Metrics and Static Code Attributes for Defect Prediction
Raimund Moser, Witold Pedrycz, Giancarlo Succi
Free University of Bolzano-Bozen
University of Alberta
Defect Prediction
quality standards customer satisfaction
Questions:
1. Which metrics are good defect predictors?
2. Which models should be used?
3. How accurate those models?
4. How much does it cost? Benefits?
Approaches for Defect Prediction
1. Product-centric
2. Process-centric
3. Combination of both
Change history of source files (number or size of modifications, age of a file)
Changes in the team structure Testing effort Technology Other human factors to software defects
measures extracted from the: • static/dynamic structure of source code • design documents • design requirements
Previous Work
The relationship between software defects and code metrics
No agreed answerNo cost-insensitive prediction
Impact of the software process on the defectiveness of software
Two aspects of defect prediction:
Questions to Answer by This Work
Questions:
1. Which metrics are good defect predictors?
2. Which models should be used?
3. How accurate those models?
4. How much does it cost? Benefits?
Are change metrics more useful?
Which change metrics are good?
How can cost-sensitive analysis be used?
Not how many defects are present in a subsystem butis source file defective?
Outline
Experimental Set-Up Assessing Classification Accuracy
Accuracy Classification Results Cost-Sensitive Classification
Cost-Sensitive Defect Prediction
Experiment Using a Cost Factor of 5
Data & Experimental Set-Up
Public data set from the Eclipse CVS repository (releases 2.0, 2.1, 3.0) by Zimmermann et al.
18 change metrics concerning change history of files 31 static code attributes metrics that Zimmerman has
used at a file level (correlation analysis, logistic regression, and ranking analysis)
One Possible Proposal of Change Metrics
renaming or moving software elements
the number of files that have been committed together with file x
in weeks, starting from release date to its first appearance
Experiments
1. Change Model uses proposed change metrics
2. Code Model uses static code metrics
3. Combined Model uses both types of metrics
Build Three Models for Predicting the Presence or Absence of
Defects in Files
Outline
Experimental Set-Up Assessing Classification Accuracy
Accuracy Classification Results Cost-Sensitive Classification
Cost-Sensitive Defect Prediction
Experiment Using a Cost Factor of 5
ResultsAssessing Classification Accuracy
Accuracy Classification Results
By analyzing the decision trees:
Defect Free: Large MAX_CHANGESET or
Low REVISIONS Smaller MAX_CHANGESET and
Low REVISIONS and
REFACTORINGS
Defect Prone: High number of BUGFIXES
Outline
Experimental Set-Up Assessing Classification Accuracy
Accuracy Classification Results Cost-Sensitive Classification
Cost-Sensitive Defect Prediction
Experiment Using a Cost Factor of 5
Cost-Sensitive Classification
Cost-sensitive classification - costs associated with different errors made by a model
min
>1 FN implicate higher costs than FP
Costly to fix an undetected defect in post release cycle than to inspect defect-free file
Cost-Sensitive Defect PredictionResults for J48 Learner, Release 2.0
Use some heuristics to stop increasing the recall
FP<30%
=5
Experiment Using a Cost Factor of 5
Defect predictors based on change data outperform those based on static code attributes.
Reject
€
H0
Limitations
Dependability on a specific environment Conclusions are on only three data miners Choice for code and change metrics Reliability of the data
mapping between defects and locations in source code
extraction of code or change metrics from repositories
Conclusions
18 change metrics, J48 learner, =5 give accurate results for 3 releases of the Eclipse project:
>75% of correctly classified files>80% recall< 30% FP rate
Hence, the change metrics contain more discriminatory and meaningful information about the defect distribution that the source code itself.
Important change metrics: Defect prone
files with high revision numberslarge bug fixing activities
Defect-freefiles that are large CVS commitsrefactored several times files
Future Research
Which information in change data is relevant for defect prediction?
How to extract this data automatically?