it’s not a bug, it’s a feature:does misclassification affect bug localization?

It’s Not a Bug, It’s a Feature:Does Misclassification Affect Bug Localization?

Pavneet Singh Kochhar, Tien-Duy B. Le, David LoSingapore Management University

2/24

Misclassification of Issue Reports

BUG

Herzig et al. *• 1/3 issue reports are wrongly classified as bugs.• 40% of issue reports are misclassified.

* It’s not a Bug, it’s a Feature: How Misclassification Impacts Bug Prediction, K. Herzig, S. Just, A. Zeller, ICSE 2013

DOCUMENTATIONIMPROVEMENT

REFACTORING

BACKPORTCLEANUP

DESIGN DEFECT

TASK

TEST

3/24

Bug Localization

Thousands of Source Code Files

GOAL: Find the buggy files ------>

4/24

Our Study

• Dataset (3 Software Systems): HTTPlient, Jackrabbit, Lucene-Java

• Over 7000 issue reports• 3 Research Questions• Suggest mitigation strategy

How misclassification impacts bug localization?

7/24

RQ1– Effect of Misclassification on Bug Localization

Projects Reported Actual DifferenceHTTPClient 0.429 0.419 -2.33%

Jackrabbit 0.302 0.339 12.25%

Lucene-Java 0.301 0.322 6.98%

Difference of -2.33% to 12.25% between MAP scoresSignificant differences (Mann-Whitney Wilcoxon test)

Mean Average Precision (MAP) Scores

7/24

RQ2– Effect of Different Misclassification Types

Actual to Reported HC JB LJ OverallNone 0.429 0.302 0.301 0.312

IMPROVEMENT to BUG 0.416 0.299 0.295 0.307

TEST to BUG 0.429 0.328 0.313 0.334

13 different categories: BUG, RFE, IMPROVEMENT, DOCUMENTATION, REFACTORING, BACKPORT, CLEANUP, SPEC, TASK, TEST, BUILD_SYSTEM, DESIGN_DEFECT, and OTHERS

TEST to BUG & IMPROVEMENT to BUG have the most impact.

Mean Average Precision (MAP) Scores

Note: HC- HTTPClient, JB- Jackrabbit, LJ- Lucene-Java

7/24

RQ3– Mitigation Strategy

Remove issue reports where no source code files are modified.Remove issue reports which explicitly mention the buggy files in summary or description.

7/24

Conclusion

Difference of -2.33%, 12.25% & 6.98% between MAP scores for 3 projects.TEST to BUG and IMPROVEMENT to BUG have significant impactMitigation:

Remove issue reports which do not change source code files.

Remove issue reports which specify buggy files in summary or description section.

Appendix (Statistical Analysis)

• Mann-Whitney-Wilcoxon (MWW) test: Given a significance level = 0.001,if p-value <, then the test rejects the null hypothesis.

9

Appendix (RQ2 Results)

10

Actual to Reported HC JB LJ Overall

None 0.429 0.302 0.301 0.312

RFE to BUG 0.427 0.303 0.304 0.313

DOCUMENTATION to BUG 0.430 0.304 0.305 0.315

IMPROVEMENT to BUG 0.416 0.299 0.295 0.307

REFACTORING to BUG 0.428 0.301 0.301 0.311

BACKPORT to BUG 0.430 0.303 0.300 0.313

CLEANUP to BUG 0.429 0.303 0.303 0.314

SPEC to BUG 0.435 0.302 0.301 0.312

TASK to BUG 0.432 0.302 0.301 0.312

TEST to BUG 0.429 0.328 0.313 0.334

BUILD_SYSTEM to BUG 0.429 0.306 0.303 0.315

DESIGN_DEFECT to BUG 0.424 0.301 0.301 0.311

OTHERS to BUG 0.439 0.303 0.301 0.313