icse 2011 ds_1

33
Pragmatic Prioritization of Software Quality Assurance Efforts Emad Shihab Queen’s University 1

Upload: sailqu

Post on 14-Apr-2017

125 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Icse 2011 ds_1

1

Pragmatic Prioritization of Software Quality Assurance Efforts

Emad ShihabQueen’s University

Page 2: Icse 2011 ds_1

2

Motivation

Software ^ has bugsalways

always

How to allocate quality assurance efforts for legacy and new code?

Q:

managers ^ have limited resources

Page 3: Icse 2011 ds_1

3

One Answer: Defect Prediction

Many recent studies focused on defect prediction

….However adoption in practice remains low

Page 4: Icse 2011 ds_1

4

An Industrial Example

How to prioritize the creation of unit tests for a large legacy system

Page 5: Icse 2011 ds_1

5

Criteria for Pragmatic QA Approaches

Focused Granularity

Unit of prioritization must be focused (e.g., function or method)

Current Approach Desired Approach

File, Folder or subsystem level (Thousands of LOC)

Page 6: Icse 2011 ds_1

6

Criteria for Pragmatic QA Approaches

Responsive Feedback

Continuous reflection of system changes.

Current Approach Desired Approach

Predict for next release

Page 7: Icse 2011 ds_1

7

Criteria for Pragmatic QA Approaches

Consider Effort

Consider effort in recommendations (e.g., use size)

Current Approach Desired Approach

No consideration for effort

In addition, need to evaluate the generality of the approaches to understand implications of findings

Page 8: Icse 2011 ds_1

8

Proposed Pragmatic SQA Approaches

Unit test creation

How to allocate quality assurance efforts for legacy code?Q:

A:

Change risk

How to allocate quality assurance efforts for new code?

Q:

A:

Complete

In Progress

Page 9: Icse 2011 ds_1

9

Unit Test CreationApproach

1. Search modification record comments for keywords, bug identifiers and modified files

2. Extract source code of modified file(s) and compare to previous version to identify changed functions

3. Combine data from 1 and 2 to calculate function-level heuristics

Page 10: Icse 2011 ds_1

10

Unit Test CreationHeuristics

Most Frequently Modified (MFM)Most Recently Modified (MRM)

Most Frequently Fixed (MFF)Most Recently Fixed (MRF)

Largest Fixed (LF)Largest Modified (LM)

Change Risk (CR)Size Risk (SR)

Random

Modification

Fix

Size

RiskRandom

Page 11: Icse 2011 ds_1

11

Unit Test CreationHeuristic Performance

Based on a heuristic, generate list of 10 functions to write unit tests for

Use size of function to measure effort required to write unit test

Page 12: Icse 2011 ds_1

12

Unit Test Creation Usefulness: Was writing the unit test useful?

Time to write unit testA

B

C

6 bug fixes

2 bug fixes

0 bug fixes

Usefulness = 2/3 = 66.67%

Page 13: Icse 2011 ds_1

13

Unit Test CreationUsefulness of Heuristics

Random

Most Recently Modified (MRM)

Size Risk (SR)

Change Risk (CR)

Most Recently Fixed (MRF)

Most Frequently Modified (MFM)

Most Frequently Fixed (MFF)

Largest Modified (LM)

Largest Fixed (LF)

0 10 20 30 40 50 60 70 80 90 100

27.7

43.1

48.8

55

56.9

80

83.8

84.7

87

8.5

9.9

12.6

17.4

16

28.1

32.3

32.9

44.7

Open SourceCommercial

Page 14: Icse 2011 ds_1

Revisiting the Pragmatic QA CriteriaUnit Test Creation

Unit Test CreationFunction level recommendations

Heuristic lists are continuously updated

Size of function used to consider effort required

Focused Granularity

Consider Effort

Responsive Feedback

Evaluate Generality: Studied Open Source and Commercial systems

Page 15: Icse 2011 ds_1

15

Change RiskProposed Approach

1. Mine changes and extract change metrics

2. Identify bug-inducing changes

3. Predict risky changes

Page 16: Icse 2011 ds_1

Revisiting the Pragmatic QA CriteriaChange Risk

Change RiskMap changes to modified functions

Flag risky changes while they are fresh in developer’s minds

Size of change used to consider effort

Focused Granularity

Consider Effort

Responsive Feedback

Evaluate Generality: Study on Open Source and Commercial systems

Page 17: Icse 2011 ds_1

Thank youI am grateful for this opportunity!

Page 18: Icse 2011 ds_1

Backups

Page 19: Icse 2011 ds_1

Revisiting the Pragmatic SQA Criteria

Unit Test Creation Change Risk

Focused Granularity

Estimate Effort

Evaluate Generality

Timely Feedback

Page 20: Icse 2011 ds_1

Pragmatic SQA Criteria

What we have What we wantFile level Unit of prioritization

must be focused (e.g., function)

Act on results in future release

Must be able to act on results of approach quickly

No effort provided Provide an estimate of the effort required for the task

Focused Granularity

Estimate Effort

Timely Feedback

Evaluate Generality

To help understand implications of findings

Page 21: Icse 2011 ds_1

21

Changes RiskNew code

Identify risky changes for closer examination before they are integrated into the code base

Page 22: Icse 2011 ds_1

22

Unit Test CreationImplemented code

Prioritize the creation of unit tests for implemented code

Page 23: Icse 2011 ds_1

23

Revisiting the Pragmatic-aware Criteria

Focused Granularity Timely feedback Estimate Effort Evaluate Generality

Page 24: Icse 2011 ds_1

24

Avoid the most bugs effectively!

Write unit tests for functions with best Return on Investment (ROI)

How can we avoid the most bugs given limited resources?

Page 25: Icse 2011 ds_1

25

Defect Prediction... An Example

Uses code metrics and pre-release defects to predict files with post-release defects in the Eclipse project.

Page 26: Icse 2011 ds_1

26

Motivation

Software has^ bugs and

managers have ^ limited resources

always

always

How to allocate quality assurance efforts for implemented and new code?

Q:

A: Re-active and pro-active SQA

Page 27: Icse 2011 ds_1

27

Proposed ApproachExtracting Historical Data

1. Extract change data to calculate metrics (e.g., churn)2. Use SZZ to identify and predict bug-inducing changes3.

Page 28: Icse 2011 ds_1

28

Answer … Defect Prediction!

Use code metrics and process metrics to predict which files are most likely to contain defects in future releases

Page 29: Icse 2011 ds_1

29

Defect Prediction... The Response

“The results are interesting, but of limited use to us ...”

... Why?

Page 30: Icse 2011 ds_1

30

Defect Prediction... Lessons Learned

1. Identifying files is too coarse of a granularity

2. Having to wait till the next release is too late

3. Need an estimate of effort required

4. Why should we believe this will work for us?

Page 31: Icse 2011 ds_1

31

Criteria for Pragmatic SQA Approaches

Timely Feedback

Must be able to act on results of approach quickly

Page 32: Icse 2011 ds_1

32

Criteria for Pragmatic SQA Approaches

Estimate Effort

Provide an estimate of the effort required for the task

Page 33: Icse 2011 ds_1

33

Criteria for Pragmatic SQA Approaches

Evaluate Generality

Evaluate on multiple projects/domains to clearly understand implications of the findings