turkit: a toolkit for human computation algorithms
TRANSCRIPT
1
TurKit: A Toolkit for Human Computation Algorithms
Rob Miller & Greg LittleUser Interface Design Group
MIT CSAIL
Joint work with Lydia Chilton, Max Goldman, Jeff Bigham, Aubrey Tatarowicz, Rajeev Najak, Michael Bernstein
June 10, 2010
Outline
• Iterative human computation– TurKit: a toolkit for human
computation algorithms
• Systems with human computation inside– VizWiz: vision for blind users with camera phones– Soylent: putting a crowd inside Microsoft Word
June 10, 2010 2
What is TurKit?
2 Ways
What is TurKit?
Way 1
Way 1
ADD 1, 2
Way 1
ADD 1, 2
JMP somewhere
Way 1
ADD 1, 2
JMP somewhere…
Way 1
ADD 1, 2
JMP somewhere…
TURK “How do you feel?”
Way 1
ADD 1, 2
JMP somewhere…
TURK “How do you feel?”
Way 2
Way 2
Way 2
Way 2
Way 2
Way 2
Way 2
Way 2
Way 2
Way 2
Demo
Demo
TurKit
What can you do with TurKit?
x50
31
19
Improve
Improve
Improve
Vote
Improve
Vote
Handwriting Recognition
Image Description
Progression of Other Offer Description:• $10,000 more• better package• far more competitive offer• more competitive offer• competing offer
Outline to Prose
Picture Sorting
x
Subjective Ratings
Subjective Ratings
Subjective Ratings
Workflow Comparisons
Workflow Comparisons
Workflow Comparisons
Workflow Comparisons
Workflow Comparisons
Iterative
Iterative
Parallel
Iterative
Parallel
Iterative
Parallel
Which is better?
Which is better?
• 30 images
Which is better?
• 30 images• 6 iterations
Which is better?
• 30 images• 6 iterations
Which is better?
Which is better?
Which is better?
Larger Algorithms
Larger Algorithms
Larger Algorithms
Outline
• Iterative human computation– TurKit: a toolkit for human
computation algorithms
• Systems with human computation inside– VizWiz: vision for blind users with camera phones– Soylent: putting a crowd inside Microsoft Word
June 10, 2010 58
VizWiz: Helping the Blind See
59
Which door is the women’s restroom?
joint work with Jeff Bigham (University of Rochester)
June 10, 2010
Helping the Blind See
60
MechanicalTurk
1. the left one2. LEFT3. on the left
June 10, 2010
Field Study
• Field deployment with 11 blind iPhone users– Answers received within a minute for ~5 cents a question– Latency can be reduced to less than 30 seconds by keeping
workers warmed up (at $4 per hour)
June 10, 2010 61
What color is this pillow? What denomination is this bill?
Do you see picnic tables across the parking lot?
What temperature is my oven set to?
Can you please tell me what this can is?
What kind of drink does this can hold?
(89s) I can’t tell.(105s) multiple shades of soft green, blue and gold
(24s) 20(29s) 20
(13s) no(46s) no
(69s) it looks like 425 degrees but the image is difficult to see.(84s) 400(122s) 450
(183s) chickpeas.(514s) beans(552s) Goya Beans
(91s) Energy(99s) no can in the picture(247s) energy drink
All Mobile Users Are Situationally Disabled
62
When is Barack Obama speaking at MIT?
MechanicalTurk
joint work with Rajeev Najak
June 10, 2010
Soylent: Putting a Crowd inside MS Word
June 10, 2010
joint work with Michael Bernstein
63
Crowd-Driven Proofreading
June 10, 2010 64
Crowd-Driven Shortening
June 10, 2010 65
Bigger Idea: REAL Wizard of Oz
• Wizard of Oz is a tried-and-true prototyping technique in AI and HCI– Putting a human behind the curtain until we figure out how to
put software there
• Crowd computing enables Wizard of Oz systems that are useful and deployable– So we can start collecting data about how the system is
really used in practice– Adding the AI backend becomes a performance or cost
optimization
66June 10, 2010
Outline
• Iterative human computation– TurKit: a toolkit for human
computation algorithms
• Systems with human computation inside– VizWiz: vision for blind users with camera phones– Soylent: putting a crowd inside Microsoft Word
• Funded in part by
June 10, 2010 67