writing acceptable patches: an empirical study of open source project patches

Writing Acceptable Patches: An Empirical Study of Open Source Project Patches Yida Tao*, DongGyun Han§ and Sunghun Kim**The Hong Kong University of Science and Technology§KAIST Institute for IT Convergence, Korea Advanced Institute of Science and Technology

Developers submit many patches…but not all of them can be accepted

Code Reviewer

2

Developers submit many patches…but not all of them can be accepted

Patch Reviewer

“IT’S A NO FROM ME!”

3

0%

10%

20%

30%

40%

50%

60%

70%

Firefox(Baysal et al. WCRE’12)

Apache(Rigby and German, 2006)

FLAC,OpenAFS(Weißgerber et al. MSR’08)

Patch rejection & resubmission

56%

40%

60%

4

Patch rejection & resubmission = ? increased interruption and workload

5

Patch rejection & resubmission = increased interruption and workload

6

We hope our patches to be accepted right after the first or early rounds of code review

7

How to write acceptable patches?

8

How to write acceptable patches?

• Learn from “past failures”• Q1: Why are patches rejected?

•Understand the priority• Q2: Which reasons are more decisive to reject a patch?• Q3: Which reasons are more difficult to judge?

•Analyze primary stakeholders• Q4: Do patch reviewers and writers have unanimous

patch evaluation criteria?

9

How to write acceptable patches?

• Learn from “past failures”• Q1: Why are patches rejected?

•Understand the priority• Q2: Which reasons are more decisive to reject a patch?• Q3: Which reasons are more difficult to judge?

•Analyze primary stakeholders• Q4: Do patch reviewers and writers have unanimous

patch evaluation criteria?

10

How to write acceptable patches?

• Learn from “past failures”• Q1: Why are patches rejected?

•Understand the priority• Q2: Which reasons are more decisive to reject a patch?• Q3: Which reasons are more difficult to judge?

•Analyze primary stakeholders• Q4: Do patch reviewers and writers have unanimous

patch evaluation criteria?

11

• Learn from “past failures”• Q1: Why are patches rejected?

•Understand the priority• Q2: Which reasons are more decisive to reject a patch?• Q3: Which reasons are more difficult to judge?

•Analyze primary stakeholders• Q4: Do patch reviewers and writers have unanimous

patch evaluation criteria?

Patch extraction & inspection

Developer survey

Literature12

Patch Extraction & Inspection

13

Extracting Rejected Patches

• Projects• Eclipse and Mozilla

• Bug tracking system• Bugzilla

• Criterion for rejected patch• Patches with explicit review- flag

• Number of patches• 300 randomly inspected

• 162 Eclipse + 138 Mozilla

Patch attachments for Eclipse bug#233156

spell checking performance test (patch)2009-11-23

no flags

performance test (5.69 KB, patch)2009-11-30

no flags

performance test + fix (12.95 KB, patch)2009-12-01

review-

fix (7.63 KB, patch)2010-01-04

review-

performance test (6.53KB, patch)2010-01-04

review+

fix (10.08 KB, patch)2010-01-14

review+

14

Patch Inspection

•Open coding• Review comments of 300 rejected patches

• # open codes: 637

“The patch fixes CCE but introduces a new bug: … Also the parameter should be renamed to …“

Open code:• new bug introduced• bad parameter name

15

Patch Inspection

Open code:“malformed patch”

Open code: “wrong indentation”

Theme:Violating coding styles

16

•Open coding• Review comments of

300 rejected patches

• # open codes: 637

•Axial coding

Patch Inspection

Open code:“malformed patch”

Open code: “wrong indentation”

Theme:Violating coding styles

17

•Open coding• Review comments of

300 rejected patches

• # open codes: 637

•Axial coding• 12 themes why patches

are rejected

Theme Frequency

24.6%

18

Suboptimal solution

Theme Frequency

24.6%

19

Incomplete fix22.3%

Theme Frequency

24.6%

20

22.3%

Including unnecessary changes11.9%

Violating coding styles9.7%

Theme Frequency

24.6%

21

22.3%

11.9%

Bad Naming

Inconsistent, misleadingdocumentationMissing documentation

Test Failure

Introducing New Bugs

Compilation errors

Duplicate code

6.3%

6.1%

5.7%

5.5%

Theme Frequency

22

24.6%

22.3%

11.9%

9.7%

Suboptimal solution, 24.6%

Incomplete fix, 22.3%

Including unnecessary

change, 11.9%Violating coding style guidelines,

9.7%

Bad naming, 6.3%

Inconsistent, misleading

documentation, 6.1%

Missing documentation,

5.7%

Test failures, 5.5%

Introducing new bugs, 4.1%

Compilation errors, 1.9%Duplication, 1.6%

Patch size too large, 0.3%

Developer survey12 patch rejection reasons from manual inspection

• Decisive score of each reason• Difficulty to judge each reason246 / 1785 (14%)

Eclipse and Mozilla developers

23

Q2: How decisive each rejection reason is?

24

Patch reviewers have very low tolerance for patches that fail compilation or tests, or introduce new bugs

25

Inconsistent or misleading documentation is highly unacceptable and way worse than missing documentation

26

Inconsistent or misleading documentation is highly unacceptable and way worse than missing documentation

27

Reviewers reject a large patch not solely because of its size, but mainly because the underlying reasons that induce its large size

28

Reviewers reject a large patch not solely because of its size, but mainly because the underlying reasons that induce its large size

29

Q3: Which patch-rejection reasons are more difficult to judge?

30

Ver

y d

iffi

cult

to

jud

geV

ery

easy

Not decisive Very decisive to reject a patch 31

Test failures

Compile errors

Inconsistent, misleading documentation

Ver

y d

iffi

cult

to

jud

geV

ery

easy

Not decisive Very decisive to reject a patch 32

Ver

y d

iffi

cult

to

jud

geV

ery

easy

Not decisive Very decisive to reject a patch

Unnecessary changes

Duplication

33

Ver

y d

iffi

cult

to

jud

geV

ery

easy

Not decisive Very decisive to reject a patch

Incomplete fixSuboptimal solution

Introduce new bugs

34

Q4: Do patch reviewers and writers have unanimous patch evaluation criteria?

35

Distinguish patch reviewers and patch writers

•Method• Patch writer: write ≥ 1 patch• Patch reviewer: review ≥ 1 patch or both

•Responses• 246 respondents• 131 patch writers, 115 patch reviewers

36

37

2 3 4 5

Compilation errors

Test Failures

Introducing new bugs

Inconsistent Documentation

Suboptimal solution

Incomplete fix

Duplication

Unnecessary changes

Violating coding styles

Bad naming

Missing documentation

Patch size too large

Patch reviewer Patch writer

Avg. Decisive Score

38

2 3 4 5

Compilation errors

Test Failures

Introducing new bugs

Inconsistent Documentation

Suboptimal solution

Incomplete fix

Duplication

Unnecessary changes

Violating coding styles

Bad naming

Missing documentation

Patch size too large

Patch reviewer Patch writer

Avg. Decisive Score

39

2 3 4 5

Compilation errors

Test Failures

Introducing new bugs

Inconsistent Documentation

Suboptimal solution

Incomplete fix

Duplication

Unnecessary changes

Violating coding styles

Bad naming

Missing documentation

Patch size too large

Patch reviewer Patch writer

Avg. Decisive Score

Patch reviewers tend to have stricter criteria on patch quality compared to patch writers

Why are patches rejected?

40

Why are patches rejected?

Patch inspection

41

Why are patches rejected?

Patch inspection Developer survey

+

42

Why are patches rejected?

Patch inspection Developer survey Literature

+ +

43

30 patch rejection reasons in 5 categories

Why are patches rejected?

Patch inspection Developer survey Literature

+ +

44

30 patch rejection reasons in 5 categories

1. Problematic implementation or solution

2. Difficult to read or maintain

3. Lack of communication or trust

4. Deviating from the project focus or scope

5. Affecting the development schedule

45

• Compilation & test failures

• Incomplete

• Suboptimal

• Solution too aggressive or hostile for end users

46

1. Problematic implementation or solution

47

2. Difficult to read or maintain

• Coding style violation

• Inconsistent documentation

• Using deprecated API

• No accompanied test cases

3. Lack of communication or trust• No discussion before patch submission• Patch writers are not responsive

4. Deviating from the project scope or focus• Irrelevant or obsolete

5. Affecting the development schedule• Freeze the progress of other features

48

Please check our paper for more details on patch rejection reasons

49

Summary

Writing acceptable patches

50

Checklist

* Compile error* Test failure* Documentation* Duplication* Unnecessary changes

51

Checklist

* Compile error* Test failure* Documentation* Duplication* Unnecessary changes

52

Communication

* Discussion prior to patch submission

* Timely response to review comments

Checklist

* Compile error* Test failure* Documentation* Duplication* Unnecessary changes

53

Communication

* Discussion prior to patch submission

* Timely response to review comments

Call for industrial &research efforts

* Introduce new bugs

* Suboptimal solutions

* Incomplete fix