massolution nyc 2013: crowdsourcing best practices | amazon mechanical turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 1/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++

amazonweb services

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 2/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++22

Crowdsourcing myths

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 3/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++

 _  It’s cheaper _ 

 It’s actually more efficient

 _  It’s faster _   It’s actually more scalable

 _  It’s not accurate _   It’s actually more accurate

3

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 4/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++44

Crowdsourcing BestPractices

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 5/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++

 _  Consider the question

carefully _   Workers answer what you ask 

 _  Select your workers _ 

 Perspective and skills vary

 _  Monitor and improve _   Adjust for optimal results

5

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 6/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++6

The question[  ]6

You will get an answer to thequestion that you ask. Focus onasking the right question

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 7/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++7

Choosing your workers[  ]7

Workers are different– from

language and culturaldifferences – to varying skills.Test and monitor.

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 8/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++8

Monitor and Improve[  ]8

Monitor key metrics, adjust and 

measure key attributes impacton those metrics.

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 9/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++

 _   Accuracy

 _   Know your current

 _   Throughput

 _   Understand both turnaround and scale requirements

 _   Cost _   Measure against a budget – as cost can impact the

other two

9

Key Metrics

[ ]9

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 10/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++10

Accuracy[  ]10

error has two sources: human

error and systematic error.Isolating human error and solving for systematic error gives

a better chance for success.

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 11/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++11

Accuracy[  ]11

After solving for systematic error

choosing the best workers and monitoring those workersprovides the next step towards

high accuracy.

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 12/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++12

Throughput[  ]12

Many factors impact throughput;

Reputationergonomics

clarity of task 

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 13/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++13

Cost[  ]13

Many factors impact cost;

Accuracy and throughput tradeoff

ergonomics

sociology

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 14/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++14

http://www.mturk.com

14

JohnHoskins,AmazonMechanicalTurk

hoskins@amazon.com

amazonweb services