evaluating how developers use general-purpose web-search...
TRANSCRIPT
Evaluating How Developers Use General-Purpose Web-Search for Code Retrieval
Date: May 29, 2018
�1
Md Masudur Rahman, Jed Barson, Sydney Paul, Joshua Kayan, Federico Andres Lois, Sebastian Fernandez Quezada, Christopher Parnin, Kathryn T. Stolee, Baishakhi Ray
Coding Task
�2
Convert a date string to a time object
�3
string to time
�4
string to time
string to time
Search Log
�5
string to time
Java
string to time
Search Log
�6
string to time using java
string to time
Search Log
�7
string to time using java
Search Log
string to time using javastring to time
�8
string to time using java
Search Log
string to time using javastring to time
�9
string to time using java
DateTime
Search Log
string to time using javastring to time
�10
date string to DateTime using java
string to time using javastring to time
Search Log
�11
date string to DateTime using java
Search Log
string to time using javadate string to DateTime using java
string to time
�12
date string to DateTime using java
Joda Time library
Search Log
string to time using javadate string to DateTime using java
string to time
�13
date string to DateTime using Joda Time library
string to time using javadate string to DateTime using java
string to time
Search Log
�14
date string to DateTime using Joda Time library
Search Log
string to time using javadate string to DateTime using javadate string to DateTime using Joda…
string to time
�15
date string to DateTime using Joda Time library
�16
date string to DateTime using Joda Time library
X
X
X
�17
world cup fixtures
Search Log
string to time using javadate string to DateTime using javadate string to DateTime using Joda …
string to time
�18
world cup fixtures
string to time using javadate string to DateTime using javadate string to DateTime using Joda …world cup fixtures
string to time
�19
place to visit in gothenburg
Search Log
string to time using javadate string to DateTime using javadate string to DateTime using Joda …world cup fixtures
string to time
�20
Search Log
string to time using javadate string to DateTime using javadate string to DateTime using Joda …world cup fixturesplace to visit in gothenburg
string to time
place to visit in gothenburg
Code Query
Code
Query string to timestring to time using javadate string to DateTime using javadate string to DateTime using Joda Time library
�21
Search Task
Code
Query string to timestring to time using javadate string to DateTime using javadate string to DateTime using Joda Time library
�22
Convert a date string to a DateTime object using Joda Time library
Search Task
Code vs Non-code
Code Non-Code
Query world cup fixturesplace to visit in gothenburghotel in gothenburg
�23
Query string to timestring to time using javadate string to DateTime using javadate string to DateTime using Joda Time library
General Purpose Search Engine for Code Retrieval
Code Non-Code
Query world cup fixturesplace to visit in gothenburghotel in gothenburg
�24
Query string to timestring to time using javadate string to DateTime using javadate string to DateTime using Joda Time library
Research Goal
Code Non-Code
Query world cup fixturesplace to visit in gothenburghotel in gothenburg
�25
๏ Query characteristics๏ User behaviorQuery
string to timestring to time using javadate string to DateTime using javadate string to DateTime using Joda Time library
Dataset
�26
Query Search Log
string to time using javadate string to DateTime using javadate string to DateTime using Joda Time libraryworld cup fixturesplace to visit in gothenburg
string to time
Users: 310 (mostly developer)
Consist of code and non-code queries
Total query: 150K
Chrome plugin
hotel in gothenburg
Dataset
�27
Query Search Log
?No label
Code or Non-codestring to time using javadate string to DateTime using javadate string to DateTime using Joda Time libraryworld cup fixturesplace to visit in gothenburg
string to time
hotel in gothenburg
Dataset
�28
Query Search Log
?No label
Code or Non-code
Query Classifier
string to time using javadate string to DateTime using javadate string to DateTime using Joda Time libraryworld cup fixturesplace to visit in gothenburg
string to time
hotel in gothenburg
�29
Intent-based Query Classification
Code Intent Analysis
�30
Query: javascript function to get mp3 play length
Code Intent Analysis
�31
Query: javascript function to get mp3 play length CodeScore?
Code Intent Analysis
�32
Token Code Intent
S = set of code related tags n = popularity of a tag
Query: javascript function to get mp3 play length CodeScore17 7 0 6 5 8 3 ?
Code Intent Analysis
�33
Query: javascript function to get mp3 play length CodeScore17 7 0 6 5 8 3 46
Token Code Intent Query Code Intent
Query Code Score
�34
Query Code Scorestring to time 12string to time using java 20date string to DateTime using java 22.5world cup fixtures 0messi curly goal 2.6place to visit in gothenburg 0
Query Code Score
�35
Query Code Score Labelstring to time 12 ?string to time using java 20 ?date string to DateTime using java 22.5 ?world cup fixtures 0 ?messi curly goal 2.6 ?place to visit in gothenburg 0 ?
Query Code Score
�36
Query Code Score Labelstring to time 12 ?string to time using java 20 ?date string to DateTime using java 22.5 ?world cup fixtures 0 ?messi curly goal 2.6 ?place to visit in gothenburg 0 ?
Classifier Evaluation
Precision: 87%Recall: 86%F1-score: 87%
Threshold = 10
Manually annotated 380 queries
Query Code Score
�37
Query Code Score Labelstring to time 12 Codestring to time using java 20 Codedate string to DateTime using java 22.5 Codeworld cup fixtures 0 Non-codemessi curly goal 2.6 Non-codeplace to visit in gothenburg 0 Non-code
Classifier Evaluation
Precision: 87%Recall: 86%F1-score: 87%
Threshold = 10
Manually annotated 380 queries
Query Code Score
�38
Query Code Score Labelstring to time 12 Codestring to time using java 20 Codedate string to DateTime using java 22.5 Codeworld cup fixtures 0 Non-codemessi curly goal 2.6 Non-codeplace to visit in gothenburg 0 Non-code
Code : 89K (59%)Non-code : 61K (41%)
Annotated Data
Classifier Evaluation
Precision: 87%Recall: 86%F1-score: 87%
Threshold = 10
Manually annotated 380 queries
Research Questions
�39
Query Characteristics
User Behavior
RQ1. How do query characteristics differ for code and non-code queries?
RQ2. How do search behaviors vary for code and non-code related queries?
RQ3. How do task sessions vary for code and non-code related search tasks?
Results
�40
RQ1: Query Characteristics
�41
Code queries often longer (more tokens) than non-code
date string to DateTime using java
date string to DateTime using Joda Time library
world cup fixtures
messi curly goal
hotel in gothenburgjavascript function to get mp3 play length
Code Non-code
RQ1: Query Characteristics
�42
date string to DateTime using javadate string to DateTime using Joda Time library
world cup fixturesmessi curly goal
hotel in gothenburgjavascript function to get mp3 play length
Code Non-code
RQ1: Query Characteristics
�43
date string to DateTime using javadate string to DateTime using Joda Time library
world cup fixturesmessi curly goal
hotel in gothenburgjavascript function to get mp3 play length
Code Non-code
RQ1: Query Characteristics
�44
Code Non-code
16K 12K 33K
Code queries contain less vocabulary (unique tokens) than non-code
�45
RQ2: Query Search BehaviorQuery # term added # term deleted
Code
string to time - -string to time using java 2 -date string to DateTime using Joda Time library 4 2
Non-codehotel in gothenburg - -best hotel in gothenburg 1 -
�46
RQ2: Query Search BehaviorQuery # term added # term deleted
Code
string to time - -string to time using java 2 -date string to DateTime using Joda Time library 4 2
Non-codehotel in gothenburg - -best hotel in gothenburg 1 -
Edited query
�47
User often add/delete more terms (avg. 2) to a code compared to non-code (avg. 1)
RQ2: Query Search BehaviorQuery # term added # term deleted
Code
string to time - -string to time using java 2 -date string to DateTime using Joda Time library 4 2
Non-codehotel in gothenburg - -best hotel in gothenburg 1 -
�48
RQ2: Query Search BehaviorQuery # term added # term deleted Code Score
Code
string to time - - 12string to time using java 2 - 20date string to DateTime using Joda Time library 4 2 30.5
�49
RQ2: Query Search BehaviorQuery # term added # term deleted Code Score
Code
string to time - - 12string to time using java 2 - 20date string to DateTime using Joda Time library 4 2 30.5
Edit query to increase code intent
�50
RQ3: Task Search BehaviorQuery # query Task intent
Code Task
string to time
4Converting a date string to a Time
objectstring to time using javadate string to DateTime using Joda Time library
Non-code Task
hotel in gothenburg2 Hotel booking in
Gothenburgbest hotel in gothenburg
More queries required to complete a code task
�51
RQ3: Task Search BehaviorQuery Task intent Search duration
(minute) # web visit
Code Task
string to timeConverting a date string to a Time
object6 15string to time using java
date string to DateTime using Joda Time library
Non-code Task
hotel in gothenburg Hotel booking in Sweden 2 5
hotel in stockholm
More time and website visit required to complete code related tasks
Summary
Code Non-Code
�52
Code queries are linguistically different Users modify code queries more often Users give significantly more effort for code task
General Search Engine
Summary
Code Non-Code
General Search Engine
�53
Code queries are linguistically different Users modify code queries more often Users spend significantly more effort for code task
Code search is less effective
Summary
Code Non-Code
General Search Engine
�54
Code queries are linguistically different Users modify code queries more often Users spend significantly more effort for code task
Code search is less effective
Special treatment required to improve code retrieval
Question?
Code Non-Code
General Search Engine
�55
Code queries are linguistically different Users modify code queries more often Users spend significantly more effort for code task
Code search is less effective
Special treatment required to improve code retrieval