automatically generated patches as debugging aids
TRANSCRIPT
![Page 1: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/1.jpg)
Automatically Generated Patches as Debugging Aids: A
Human Study
Yida Tao, Jindae Kim, Sunghun Kim
Dept. of CSE, The Hong Kong University of Science and Technology
Chang Xu
State Key Lab for Novel Software Technology, Nanjing University
![Page 2: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/2.jpg)
• Promising research progress• ClearView1: Prevent all 10 Firefox
exploits• GenProg2: Fix 55/105 real bugs
[1] Automatically Patching Errors in Deployed Software. Perkins et al. SOSP’09[2] A systematic study of automated program repair: fixing 55 out of 105 bugs for $8 each. Le Goues et al. ICSE’122
Automatic Program Repair
![Page 3: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/3.jpg)
3
Automatic Program Repair
![Page 4: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/4.jpg)
4
- Slashdot discussion: http://science.slashdot.org/story/09/10/29/2248246/Fixing-Bugs-But-Bypassing-the-Source-Code
“It won't get your bug patched any quicker. You’ll just have shifted the coders' attention away from their own app's bugs, and onto the repair tool’s bugs.”
Automatic Program Repair
![Page 5: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/5.jpg)
#what-could-possibly-go-wrong• Blackbox repair• Increasing maintenance
cost• Vulnerable to attack
- Slashdot discussion: http://science.slashdot.org/story/09/10/29/2248246/Fixing-Bugs-But-Bypassing-the-Source-Code- A human study of patch maintainability. ISSTA’12- Automatic patch generation learned from human-written patches. ICSE’13
5
![Page 6: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/6.jpg)
- Slashdot discussion: http://science.slashdot.org/story/09/10/29/2248246/Fixing-Bugs-But-Bypassing-the-Source-Code- A human study of patch maintainability. ISSTA’12- Automatic patch generation learned from human-written patches. ICSE’13
#program-out-of-control
6
#what-could-possibly-go-wrong• Blackbox repair• Increasing maintenance
cost• Vulnerable to attack
![Page 7: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/7.jpg)
7
Use automatically generated patches as debugging aids
![Page 8: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/8.jpg)
8
Use automatically generated patches as debugging aids
Our Human Study• Investigate the usefulness of
generated patches as debugging aids
• Discuss the impact of patch quality on debugging performance
• Explore practitioners’ feedback on adopting automatic program repair
![Page 9: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/9.jpg)
9
Methodology
![Page 10: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/10.jpg)
10
Bugs
Participants
Debugging aid
Debugis given to
![Page 11: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/11.jpg)
BugsParticipantsDebugging aid 11
![Page 12: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/12.jpg)
BugsParticipantsDebugging aid 12
Low-quality generated patch
![Page 13: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/13.jpg)
BugsParticipantsDebugging aid 13
Low-quality generated patch
High-quality generated patch
![Page 14: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/14.jpg)
BugsParticipantsDebugging aid 14
Low-quality generated patch
High-quality generated patch
Buggy method location
![Page 15: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/15.jpg)
BugsParticipantsDebugging aid 15
Grad: 44
Engr: 28
MTurk: 23
95 Participants
CS graduate students
Industrial software engineers
Amazon Mechanical Turk workers
![Page 16: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/16.jpg)
BugsParticipantsDebugging aid 16
![Page 17: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/17.jpg)
BugsParticipantsDebugging aid 17
44 Graduate students• Between-group
design
14 students
15 students
15 students
![Page 18: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/18.jpg)
BugsParticipantsDebugging aid 18
44 Graduate students• Between-group
design
Low-quality generated patch
High-quality generated patch
Buggy method location
14 students
15 students
15 students
![Page 19: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/19.jpg)
BugsParticipantsDebugging aid 19
44 Graduate students• Between-group
design• Onsite setting• Eclipse IDE• Supervised session
Low-quality generated patch
High-quality generated patch
Buggy method location
14 students
15 students
15 students
![Page 20: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/20.jpg)
BugsParticipantsDebugging aid 20
Low-quality generated patch
High-quality generated patch
Buggy method location
Remote participants(28 Engr + 23 MTurk)• Within-group design
![Page 21: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/21.jpg)
BugsParticipantsDebugging aid 21
Remote participants(28 Engr + 23 MTurk)• Within-group design• Online debugging system
Low-quality generated patch
High-quality generated patch
Buggy method location
![Page 22: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/22.jpg)
BugsParticipantsDebugging aid 22
![Page 23: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/23.jpg)
BugsParticipantsDebugging aid 23
Bug Selection Criteria
• Real bugs• The bug has accepted patches written by developers• Proper number of bugs• The bug has generated patches with different quality
![Page 24: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/24.jpg)
BugsParticipantsDebugging aid 24
Automatic patch generation learned from human-written patches. Kim et al. ICSE’13
![Page 25: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/25.jpg)
BugsParticipantsDebugging aid 25
Automatic patch generation learned from human-written patches. Kim et al. ICSE’13
Auto-generated patch A Auto-generated patch B
for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) if(sub!=null){ args[i+1] = sub.toString(); }}
for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) args[parenCount+1] = new Integer(reImpl.leftContext.length);}
![Page 26: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/26.jpg)
BugsParticipantsDebugging aid 26
Automatic patch generation learned from human-written patches. Kim et al. ICSE’13
Auto-generated patch A Auto-generated patch B
avg. ranking from 85 devs and students
for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) if(sub!=null){ args[i+1] = sub.toString(); }}
for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) args[parenCount+1] = new Integer(reImpl.leftContext.length);}
1.6
2.8
![Page 27: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/27.jpg)
BugsParticipantsDebugging aid 27
Automatic patch generation learned from human-written patches. Kim et al. ICSE’13
Auto-generated patch A Auto-generated patch B
avg. ranking from 85 devs and students
High-Quality Patch Low-Quality patch
for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) if(sub!=null){ args[i+1] = sub.toString(); }}
for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) args[parenCount+1] = new Integer(reImpl.leftContext.length);}
1.6
2.8
![Page 28: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/28.jpg)
BugsParticipantsDebugging aid 28
![Page 29: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/29.jpg)
BugsParticipantsDebugging aid 29
Participants submit 337 patches as their debugging outcome
![Page 30: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/30.jpg)
BugsParticipantsDebugging aid 30
Participants submit 337 patches as their debugging outcome
Location109
LowQ112
HighQ116# submitted
patches w.r.t debugging aid
![Page 31: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/31.jpg)
BugsParticipantsDebugging aid 31
Participants submit 337 patches as their debugging outcome
Location109
LowQ112
HighQ116# submitted
patches w.r.t debugging aid
Bug166
Bug274
Bug359
Bug476
Bug562
# submitted patches w.r.t bugs
![Page 32: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/32.jpg)
32
Evaluation of debugging performance
![Page 33: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/33.jpg)
Patch CorrectnessCorrectness
33
![Page 34: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/34.jpg)
Patch Correctness
• Passing test casesCorrectness
34
![Page 35: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/35.jpg)
Patch Correctness
• Passing test cases• Matching the semantics of original
accepted patches
Correctness
35
![Page 36: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/36.jpg)
Patch Correctness
• Passing test cases• Matching the semantics of original
accepted patches• 3 evaluators
Correctness
36
![Page 37: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/37.jpg)
Debugging Time
• Eclipse Plug-in•Website Timer
Correctness
Debugging time
37
![Page 38: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/38.jpg)
Correctness
Debugging time
• Independent variables• Debugging aids• Bugs• Participant types• Programming
experience
38
![Page 39: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/39.jpg)
Multiple Regression Analysis
Correctness
Debugging time
• Independent variables• Debugging aids• Bugs• Participant types• Programming
experience
correctness=α0+α 1 ∙ x1+α 2 ∙ x2+α 3 ∙ x 3+α 4 ∙ x4d ebugging time=β0+β1 ∙ x1+β2 ∙ x2+β3 ∙ x3+β 4 ∙ x4
39
![Page 40: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/40.jpg)
Post-study Survey
• Helpfulness of debugging aids• Difficulty of bugs• Opinions on using generated patches as
debugging aids
Correctness
Debugging time
Survey feedback
40
![Page 41: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/41.jpg)
41
Results
![Page 42: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/42.jpg)
42
High-quality patches significantly improve debugging correctness
1
48%
33%
71%
![Page 43: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/43.jpg)
43
High-quality patches significantly improve debugging correctness
1
48%
33%
71%
Lo-catio
n
LowQ HighQ
% of correct patches
48%
71%
![Page 44: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/44.jpg)
44
Lo-catio
n
LowQ HighQ
% of correct patches
High-quality patches significantly improve debugging correctness
1
Positive Coefficient =
1.25p-value= 0.00 < 0.05
48%
71%
![Page 45: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/45.jpg)
45
Lo-catio
n
LowQ HighQ
% of correct patches
Low-quality patches slightly undermine debugging correctness
2
48%
33%
71%
![Page 46: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/46.jpg)
46
Lo-catio
n
LowQ HighQ
% of correct patches
Low-quality patches slightly undermine debugging correctness
2
Negative Coefficient = -0.55p-value= 0.09 48%
33%
71%
![Page 47: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/47.jpg)
47
Lo-catio
n
LowQ HighQ
% of correct patches
Low-quality patches can undermine debugging correctness
2
Negative Coefficient = -0.55p-value= 0.09 48%
33%
71%
![Page 48: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/48.jpg)
48
High-quality patches are more useful for difficult bugs3
![Page 49: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/49.jpg)
49
High-quality patches are more useful for difficult bugs3
2
3
4
5
Bug Difficulty
Bug1Math-280
Bug2Rhino-114493
Bug3Rhino-192226
Bug4Rhino-217379
Bug5Rhino-76683
![Page 50: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/50.jpg)
50
High-quality patches are more useful for difficult bugs3
Bug1 Bug2 Bug3 Bug4 Bug50%
10%
20%
30%
40%
50%
60%
70%
80%
90%
% of correct patches
Location LowQ HighQ
2
3
4
5
Bug Difficulty
Bug1Math-280
Bug2Rhino-114493
Bug3Rhino-192226
Bug4Rhino-217379
Bug5Rhino-76683
![Page 51: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/51.jpg)
51
4The type of debugging aid does not affect debugging time
![Page 52: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/52.jpg)
52
4The type of debugging aid does not affect debugging time
Location LowQ HighQ0
20
40
Debugging time (min)
![Page 53: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/53.jpg)
53
5Other factors’ impact on debugging performance
Difficult bugs significantly slow down debuggingEngr and MTurk are more likely to debug correctly Novices tend to benefit more from HighQ patches
![Page 54: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/54.jpg)
Helpfulness of de-bugging aidsVery
helpful
Helpful
Medium
Slightly Helpful
Not Helpful
54
Participants consider high-quality generated patches much more helpful than low-quality patches
Low-quality generated patch
High-quality generated patch
Mann-Whitney U testp-value =
0.001
6
![Page 55: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/55.jpg)
55
Feedback
![Page 56: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/56.jpg)
56
![Page 57: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/57.jpg)
57
Quick starting point• Point to the buggy
area• Brainstorm
“They would seem to be useful in helping find various ideas around fixing the issue, even if the patch isn’t always correct on its own.”
![Page 58: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/58.jpg)
58
Quick starting point• Point to the buggy
area• Brainstorm
Confusing, incomplete, misleading• Wrong lead, especially for
novices• Require further human
perfection“They would seem to be useful in helping find various ideas around fixing the issue, even if the patch isn’t always correct on its own.”
![Page 59: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/59.jpg)
59
“Generated patches would be good at recognizing obvious problems”
“…but may not recognize more involved defects.”
![Page 60: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/60.jpg)
60
“Generated patches would be good at recognizing obvious problems”
“…but may not recognize more involved defects.”
“Generated patches simplify the problem”
“…but they may over-simplify it by not addressing the root cause.”
![Page 61: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/61.jpg)
61
“I would use generated patches as debugging aids, as they provide extra diagnostic information”
![Page 62: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/62.jpg)
62
“I would use generated patches as debugging aids, as they provide extra diagnostic information”
“…along with access to standard debugging tools.”
![Page 63: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/63.jpg)
63
Threats to Validity
![Page 64: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/64.jpg)
64
Threats to Validity
• Bugs and generated patches may not be representative• Quality measure of generated patches may not
generalize• May not generalize to domain experts• Possibility of blindly reusing generated patches• Remove patches that are submitted less than 1 minute
![Page 65: Automatically generated patches as debugging aids](https://reader036.vdocuments.net/reader036/viewer/2022062313/55b8b807bb61eb8f368b470f/html5/thumbnails/65.jpg)
65
Takeaway
• Auto-generated patches can be useful as debugging aids• Participants fix bugs more correctly
with auto-generated patches
• Quality control is required• Participants’ debugging correctness is
compromised with low-quality generated patches
• Maximize the benefits• Difficult bugs• Novice developers