because you can’t fix what you don’t know is broken
TRANSCRIPT
Because you can’t fix what you don’t know is broken…
About me
2
• Initiator of the Automated Error Reporting Initiative • Project Lead of Eclipse Code Recommenders • Eclipse Committer since 2010 • Plug-in Developer for 10+ years (Eclipse 2.1) • Member of the Eclipse Architecture Council • Co-Lead of Java User Group Darmstadt • CEO of Codetrails
• Passion to improve developers day-to-day work with intelligent and practical tools - mostly using Data Mining on Big (Software Engineering) Data.
@marcelbruchmarcelbruch
Adding a new character to the plot
3
SoftwareUser
{ — — –}ApplicationDeveloper
Product Manager
Imagine the following situation…
4
SoftwareUser
{ — — –}ApplicationDeveloper
Product Manager
But when the user starts the app…
5
6Can you feel their pain…?
This image is not available in your country
In the logs you’d find…
7
✘ ✘ ✘
The tragedy of manual error reports
8
✘ ✘
!
maintenance
✘ ✘ ✘
Release SR 2
production
SR 1
✘
!
The tragedy of manual error reports
9
maintenance
Release SR 2
production
SR 1
✘ ✘ ✘
10Not too happy but feeling okay…
This image is not available in your country
What can we do?
11
Application Developer
?{ — — –}
The Automated Error Reporting Initiative
12
Live Demo
13
About server responses,Bugzilla, and immediate feedback
„Keep users informed about your progress…“
14
Systems Overview
15
AERI Web UI Eclipse BugzillaEclipse IDEs
„Direct Reporter Feedback“
Not yet tracked problems…
16
Committers needinfo…
17
Status: – – –
Resolution: – – –
Bug 442500 – ….
Fixed…
18
Status: – – –
Resolution: – – –
Bug 442500 – ….
Won’t Fix…
19
Status: – – –
Resolution: – – –
Bug 442500 – ….
Invalid (aka log messages)
20
Status: – – –
Resolution: – – –
Bug 442500 – ….
About caching, filtering,anonymization, and other shortcuts…
„What goes to eclipse.org?“
21
Do we send all errors to Eclipse…?
22
really
The error reporting funnel
23
✘
structural filter
#1: Structural Filtering
24
java.net.UnknownHostException: marketplace.eclipse.org at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) at java.net.Socket.connect(Socket.java:538) at com.example.HttpClient.openServer(HttpClient.java:527) at com.example.http.HttpClient.<init>(HttpClient.java:211) at com.example.HttpURLConnection.connect(HttpURLConnection.java:932) at org.eclipse.recommenders.internal.news.rcp.PollFeedJob.run(PollFeedJob.java:69) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Error connecting to ‘http://marketplace.eclipse.org/rss.xml’.
#2: Did I already sent this?
25
✘
structural filter
personal filter
#3: Did anyone else sent this?
26
✘
structural filter
personal filter
global hist.
structural filter
personal filter
If you hit a known issue…
27
✘
global hist.
#4: Review
28
review
✘
structural filter
personal filter
global hist.
#4: Review
29
The server sideWalkthrough through the committer front-end
30
The core concepts
31
The Core Concepts
32
✘✘
✘ ✘✘✘✘✘✘ Incidents
Problems
Projects
The committer front-end
33
Creating bug reports
34
Creating bug reports
35
Notifying committers
36
Project Alerts
37
Weekly or daily email digests
38
Some fun factse = mc2 —> errors = (more code)2
39
DisclaimerDon’t draw preliminary conclusions!
Please consult your personal statistican…
How many milestones testers?
450 simrel committers 1200 eclipse committers Testers?~7000 error reporters
Milestones adoption
42
RC3 RC2 RC1 M7 M6
M5 M4 M3 M2 M1
1000
220
160
650
450400
560
600Do new milestones introduce new problems?
70
10
# reporters one week high
SR2 SR1 R
LunaMars
0
350
700
1050
1400
4.4.0 4.4.1 4.4.2 M1 M2 M3 M4 M5 M6 M7 RC1 RC2 RC3
Milestones adoption
43
Christmas
Addition to all EPP packages
RCs
Addition to first EPP packages
How hard do they suffer?
44
300 - 3000 reports 25 testers
50-300 reports 330 testers
3-50 reports 3600 testers
1-2 reports 3000 testers
What do they report?
45
0
750
1500
2250
3000
‚Normal‘ ErrorsOut of MemoryFreezesStack Overflows3rd Party Involved
Top 100 of 7000
Me
What kind of errors are send?
46
3rd Party Involved 16 %
Stack Overflows 0 %
Freezes 9 %
Out Of Memory 1 %
'Normal' Errors 74 %
What are the most popular exceptions?
47
32 %
12 %
25 %
3 %
8 %2 %
18 %
What are the most popular exceptions?
48
Other 32 %
No {Class | Method | Field} DefFoundError 12 %
NullPointerException 25 %
IndexOutOfBoundsException 3 %
IllegalArgumentException 8 %
ClassCastException 2 %
{File} IOException 18 %
NullPointerExceptions by Projects
49
0
1000
2000
3000
NullPointerException
Code Recommenders
65
Statisticians, note: Projects automatically marked as „being responsible“ fixing these NullPointerExceptions but does not mean the NPE happened in their code.
Tip: Enable null analysis
50
@Overrideprotected void run(@Nullable List<Incident> slice) {
for (Incident incident : slice) {
JDT offers a lot…
51
Tip: Never return null
52
@Overridepublic Optional<String> getReason() { return Optional.ofNullable(cause);}
Use java.util.Optional or com.google.common.base.Optional instead
Oh Class, Were Art Thou?
53
Use proper import ranges and define uses on your exports to get rid of {Class | Method | Field} DefNotFoundErrors, LinkageErrors, and some ClassCastExceptions. OSGI needs that information to work properly…
Export-Package: org.eclipse.recommenders.utils; uses:="org.osgi.framework, org.eclipse.core.runtime, com.google.common.base, com.google.common.collect"
Import-Package: com.google.common.annotations;version="[15.0.0,16.0.0)",
Index out of bounds?
54
a b c d e
-1 and array.length do not work. No matter how often you try…
There is no tool support for this. Just unit tests and code that tries to fail gracefully.
Tip: If error messages are not helpful…
55
try { // …} catch (Exception e) {log(„An error occurred“, e);
}
21% of all logged errors do not contain stack traces.
try { // …} catch (Exception e) {log(„Invalid argument {}“, value, e);
}
Are your users ready for Java 8?
56
both2,5%
Java 1.733%
Java 1,8 64%
Java 1.90,3%
Which VM do users run the Eclipse IDE with?
Do people anonymize their messages?
57
Sometimes 3 %
No 81 %
Yes 16 %
Are you concerned that your exception message may reveal sensitive data?
Bugfixing RatesSpeak plain: Does it work?
58
Bugzilla Summary
59
1170 bugs
275 fixed (24%)
100 invalid
70 not eclipse
467 in progress (40%)
172 duplicates (15%)69 works 4 me
28 wont fix
Bugzilla
Fixed bugs by projects
60
More (35) 84
--- 11
EGit 15
Papyrus 19
Recommenders 41
Xtext 42
Oomph 58
In progress & fixed bugs
61
0
30
60
90
120
Oomph TMF Recommenders Papyrus EGit ---
97
1914
1729
20
111519
4142
58
FixedIn Progress
Future Work?Feature requests are tracked in bugzilla. Make sure your request is in there and vote for it.
Client:• Separate view to „review, analyze and send later“ • Making client reusable in other (Eclipse- or SWT-based) applications • Additional client-side automated analysis
Server:• Ensure scalability of the system • Improve duplicate detections • Go Neon if feedback is good.
Hosting an error reporting service for Java software?• Need one? TTML.
62
How to reach EPP Logging
63
• Code: Check out the EPP Logging Oomph profile • Wiki : https://wiki.eclipse.org/EPP/Logging • Manual: https://dev.eclipse.org/recommenders/community/confess/#/about • Forum: https://www.eclipse.org/forums/index.php/f/69/ • Bugzilla: https://bugs.eclipse.org/bugs/buglist.cgi?component=logging&product=EPP • Mail: [email protected]
Credits
• Daniel Haftstein, Codetrails – Eclipse client conception and implementation • Marcel Bruch, Codetrails – Server-side conception and implementation
• Webmasters, Eclipse Foundation – Eclipse server infrastructure & integration • Thanks to many early adopters and committers providing feedback
64
Feedback
65
Contact Us
66
Codetrails GmbH Robert-Bosch-Str. 7 64293 Darmstadt, Germany
www: codetrails.com phone: 06151 / 2767 092 email: [email protected]
Dr. Marcel Bruch CEO
mobile: 0179 / 1317 721 email: [email protected]
Feedback
67