code hunt: experience with coding contests at scale judith bishop, r nigel horspool, tao xie,...
TRANSCRIPT
Code Hunt: Experience with Coding Contests
at ScaleJudith Bishop, R Nigel Horspool, Tao Xie,
Nikolai Tillmann, Jonathan de HalleuxMicrosoft Research, University of Victoria and UIUC
Working and learning for fun
Enjoyment adds to long term retention on a taskDiscovery is a powerful driver, contrasting with direct instructionsGaming joins these two, and is hugely popularCan we add these elements to coding?
Code Hunt can!
www.codehunt.com
Websites
GameProjectCommunityData ReleaseBlogs
www.codehunt.comresearch.microsoft.com/codehuntresearch.microsoft.com/codehuntcommunitygithub.com/microsoft/code-huntLinked on the Project page
The Code Hunt Game
Code Hunt
• Is a serious programming game• Works in C# and Java (Python coming)• Appeals to coders wishing to hone their programming skills• And also to students learning to code• Code Hunt has had over 300,000 users since launching in March 2014
with around 1,000 users a day• Stickiness (loyalty) is very high
The Story
Gameplay
1. User writes code in browser2. Cloud analyzes code – test cases show differences
As long as there are differences: User must adapt code, repeatWhen they are no more differences: User wins level!
secret
code
test cases
It’s a game!
1. iterative gameplay2. adaptive3. personalized4. no cheating5. clear winning criterionScore is based on • how many puzzles solved, • how well solved, and • when solved
secret
code
test cases
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
a.Length>0
a[0]==123…
TF
T
F
Fa==null
T
Constraints to solve
a!=null a!=null &&a.Length>0
a!=null &&a.Length>0 &&a[0]==123456890
Input
null{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890
Done: There is no path left.
Dynamic Symbolic Execution
Available in Visual Studio since 2010(as Pex and Smart Unit Tests)
Powered by a IntelliTest in Visual Studio 2015
Nikolai Tillmann, Jonathan de Halleux, Tao Xie:Transferring an automated test generation tool to practice: from pex to fakes and code digger. ASE 2014: 385-396
Code Hunt Contests
Code Hunt Contests
Code Hunt Properties Code Hunt Contests
Browser-based Have world-wide reach on all platforms
Automatic grading Are cost effective
Cloud based hosting Scale to 100,000s players
Clear scoring criteria Determine winners precisely
Results available Can identify top coders
The game is fun, fresh and different.
The ah-ha moment
Code Hunt can identify top coders
30 Contests in the past year
What the player sees
Dashboard and Leaderboard
Publically visible when the contest ends
Visible only to the organizer
Puzzles and Analytics
Puzzles
• About 400 puzzles• Classified and rated by difficulty• Puzzle Bank is highly protected
Puzzle Group Description SourceDifficult
yCalculat
ed Players
P249-1 numbers Compute 23*X-42 NH 2 1 335
P277 string Convert N to base 4 string NH 2 3 332
P115 array Convert array of scores to letter grades APCS 3 2 329
P002 array Count occurrences of a string in an array APCS 1 2 325
P260-1 bools Compute x && (y || !z) NH 1 2 325
Number: P067Group: ArraysDescription: Remove duplicates
from an arraySource: NHDifficulty: 2
Puzzle Bank Statistics
Modify the Calculated rating based on user experience.
D = a + tries * b + tries * c * distancewhere a, b and c are constants with values
a = 1 b = 0.02 c = 0.05
tries is the average attempts for all players who solved the puzzledistance is the number of levels solved so far.
Tries does depend on• Mix of students who enter• The internet speed• Whether students use an IDE in addition
Laser, Sept 2014
Original Difficulty
Average Calculated Difficulty s
Total Try Count
Average Try Count
Max Try Count
Total Solved Users
24 2.21 4.601.1 2 1.79 106 8.83 27 121.2 2 1.42 56 4.67 13 121.3 3 2.42 158 15.80 45 101.4 2 2.43 130 13.00 31 101.5 2 4.05 211 23.44 73 91.6 1 3.29 122 15.25 52 82.1 3 4.29 116 19.33 43 62.2 3 1.87 23 4.60 7 52.3 2 1.84 20 4.00 8 52.4 2 9.05 70 35.00 68 22.5 1 3.10 42 8.40 27 52.6 2 18.10 190 63.33 138 3
How players perceive difficulty
Beauty of Programming Subjective difficulty
Perceived difficulty
Players who started
Qualification 1.59 2.72 1511Preliminary A 2.17 1.84 251Preliminary B 2.50 1.84 247Semi-Final 2.60 2.22 140
CSTA and TEALS Subjective difficulty
Calculated difficulty
Players who scored
Students 1.96 5.22 61Teachers 1.96 4.38 14
Code Hunt - the APCS (default) Zone
• Opened in March 2014• 129 problems covering the Advanced Placement Computer Science course• By August 2014, over 45,000 users started.
1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 1.14 1.15 2.1 2.2 2.3 2.4 2.5 2.6 2.7 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.80
5000
10000
15000
20000
25000
30000
35000
40000
45000
50000
APCS Zone, First three sectors, 45K to 1K
Sector and Level
Play
ers
Effect of difficulty on drop off in sectors 1-3
Yellow – DivisionBlue – OperatorsGreen - Sectors
Aug 2014 and Feb 2015
1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 1.14 1.15 2.1 2.2 2.3 2.4 2.5 2.6 2.7 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8
-10
0
10
20
30
40
50
60
Effect of Puzzle Difficulty on Drop off
Aug Feb-A
Perc
enta
ge D
rop
Off
Puzzle Level Aug Feb-ACompute -X 1.1 17 22Compute 4 / X 1.6 18 21Compute X-Y 1.7 18 22Compute X/Y 1.11 32 38Compute X%3+1 1.13 15 18Compute 10%X 1.14 12 16Construct a list of numbers 0..N-1 2.1 37 48Construct a list of multiples of N 2.2 19 23Compute x^y 3.1 11 18Compute X! the factorial of X 3.2 16 19Compute sum of i*(i+1)/2 3.5 17 22
One week 3-10 October, 2014
One year May 2014 – April 2015
Survey results – before and after contest season How much did the puzzle aspect of Code Hunt keep you interested in reaching a solution?
Till Sept 2014
Sept 2014 – May 2015
Survey Results cont.In your opinion, were your final solutions well-structured code?
Till Sept 2014
Sept 2014 – May 2015
Gender and Code Hunt Female Male
Respondents 12% 80%
Played as practice for a course in C#
11% 18%
Played as practice for a course in Java
40% 23%
Played for own enjoyment
47% 57%
Open survey with 850 respondents
Beyond contests
Towards a Course Experience
Total Try Count
Average Try Count
Max Try Count
Total Solved Users
13374 363 1306 1581
Public Data release in open source
For ImCupSept257 users x 24 puzzles x approx. 10 tries = about 13,000 programs
For experimentation on how people program and reach solutions
Github.com/microsoft/code-hunt
Upcoming events
PLOOC 2015 at PLDI 2015, June 14 2015, Portland, OR, USA
CHESE 2015 at ISSTA 2015, July 14, 2015, Baltimore, MD, USA
Worldwide intern and summer school contests
Public Code Hunt Contests are over for the summer
Special ICSE attendees Contest. Register at
aka.ms/ICSE2015
Code Hunt Workshop February 2015
Summary:Code Hunt: A Game for Coding
1. Powerful and versatile platform for coding as a game2. Unique in working from unit tests not specifications3. Contest experience fun and robust4. Large contest numbers with public data sets from cloud data
• Enables testing of hypotheses and making conclusions about how players are mastering coding, and what holds them up
5. Has potential to be a teaching platform• collaborators needed
Websites
GameProjectCommunityData ReleaseBlogsOffice MixICSE Contest
www.codehunt.comresearch.microsoft.com/codehuntresearch.microsoft.com/codehuntcommunitygithub.com/microsoft/code-huntLinked on the Project pagemix.office.comaka.ms/icse2015 to registerAka.ms/icse2015codehunt to play
Grazie!
Appendix
Results and rewards
Detailed results are presentedRanking is always based on score, not on time or attemptsScore is based on • how many puzzles solved, • how well solved, and • when solved
Special rewards for contests
Comparing Code Hunt
Code Hunt
A serious online coding game
Built on symbolic execution
Addressing various audiences
Data available in the cloud
IPCC
Annual programming contest
Scored by unit testing
Addressing university students
Questions available on the web
BoP China 2014