crowdsourcing and all-pay auctions milan vojnović microsoft research joint work with dominic...

Crowdsourcing and All-Pay Auctions

Milan VojnovićMicrosoft Research

Joint work with Dominic DiPalantino

UC Berkeley, July 13, 2009

Examples of Crowdsourcing• Crowdsourcing = soliciting solutions via open calls to

large-scale communities– Coined in a Wired article (’06)

• Taskcn– 530,000 solutions posted for 3,100 tasks

• Innocentive– Over $3 million awarded

• Odesk– Over $43 million brokered

• Amazon’s Mechanical Turk– Over 23,000 tasks

2

Examples of Crowdsourcing (cont’d)

• Yahoo! Answers– Lunched Dec ’05– 60M users / 65M answers (as of Dec ’06)

• Live QnA– Lunched Aug ’06 / closed May ’09– 3M questions / 750M answers

• Wikipedia

3

Incentives for Contribution• Incentives

– Monetary

$$$

– Non-momentary

Social gratification and publicityReputation pointsCertificates and “levels”

• Incentives for both participation and quality

4

Incentives for Contribution (cont’d)• Ex. Taskcn

5

Reward range (RMB)

Cont

est d

urati

onN

umbe

r of s

ubm

issi

ons

Num

ber o

f reg

istr

ants

Num

ber o

f vie

ws

100 RMB $15 (July 09)

Incentives for Contribution (cont’d)• Ex. Yahoo! Answers

6

Points Levels

Source: http://en.wikipedia.org/wiki/Yahoo!_Answers

Questions of Interest

• Understanding of the incentive schemes– How do contributions relate to offered rewards?

• Design of contests– How do we best design contests?– How do we set rewards?– How do we best suggest contests to players and

rewards to contest providers?

7

Strategic User Behavior

• From empirical analysis of Taskcn by Yang et al (ACM EC ’08) – (i) users respond to incentives, (ii) users learn better strategies– Suggests a game-theoretic analysis

8

User Strategies on Taskcn.com User Strategies on Taskcn.com

Outline• Model of Competing Contests

• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills

• Design of Contests

• Experimental Validation

• Conclusion9

Single Contest Competition

10

c1

c2

c3

c4

R

ci = cost per unit effort or quality produced

contest offeringreward Rplayers

Single Contest Competition (cont’d)

11

Outcome

-c1b1

R - c2b2

-c3b3

-c4b4

c1

c2

c3

c4

b1

b2

b3

b4

R

All-Pay Auction

12

Outcome

-b1

v2 - b2

-b3

-b4

v1

v2

v3

v4

b1

b2

b3

b4

Everyone pays their bid

Competing Contests

13

R1

R2

RJ

...

Rj...

contestsusers

1

2

u

N

),,( ,1, Juuu vvv

juv ,

......

Incomplete Information Assumption

Each user u knows

= total number of usersN

= his own skilluv

= skills are randomly drawn from FF

14

We assume F is an atomless distribution with finite support [0,m]

Assumptions on User Skill1) Player-specific skill

random i.i.d. across u (ex. contests require similar skills or skill determined by player’s opportunity cost)

),,( uu vvv

2) Contest-specific skill

random i.i.d. across u and j (ex. contests require diverse skills)

),,( ,1, Juu vvv

juv ,

uv

15

Bayes-Nash Equilibrium

• Mixed strategy

• EquilibriumSelect contest of highest expected profit

where expectation with respect to “beliefs” about other user skills

)(, vju = prob. of selecting a contest of class j

jub , = bid

16Contest class = set of contests that offer same reward

User Expected Profit

• Expected profit for a contest of class j

v

Ncjjjj dxxFpRvg

0

1)(1)(

= prob. of selecting a contest of class j

jp

= distribution of user skill conditional on having selected contest class j

()jF

17

vn

jn

jjujj dxxFvFvRnvg0

, )()(),(

)),((E)( Mvgvg jj

),1(Bin~ jpNM

Outline• Model of Competing Contests

• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills

• Design of Contests

• Experimental Validation

• Conclusion18

Equilibrium Contest Selection

m

0

1

2

3

4

5

1v2

v3

v4

2

3

4

skilllevels

contestclasses

19

Threshold Reward

• Only K highest-reward contest classes selected with strictly positive probability

)(

11:max

~],1[

],1[

1

1

RHJ

RiK ii

Ni

1

11

)(

AkkJ

JA

N

A

k RRH

Ak

kA JJ

20kJ = number of contests of class k

Partitioning over Skill Levels

• User of skill v is of skill level l if

KlRH

RJvF

l

lll

N ~,,1 for ,

)(11)(

],1[],1[

11

),[ 1 ll vvv

where

KKlv l ,,~

for ,0

21

Contest Selection

• User of skill l, i.e. with skill selects a contest of class j with probability

Klj

ljR

R

vl

kk

j

j N

N

,,10

,,1)(

1

11

11

),[ 1 ll vvv

22

Participation Rates

• A contest of class j selected with probability

KKj

Kj

R

RH

Jp Nj

K

Kj

,,1~

0

~,,1

)(111

1

1

]~

,1[

]~

,1[

23

• Prior-free – independent of the distribution F

Large-System Limit

• For positive constants

where K is a finite number of contest classes

J

NNlim

kk

N J

J lim

kkN Np lim

Kkkk ,,1 , , ,

KRRR 21

24

Skill Levels for Large System

• User of skill v is of skill level l if

KlR

RvF

l

l

kk

ll

lk

~,,1 for ,log1)( 1

/

],1[

],1[

),[ 1 ll vvv

where

KKlvl ,,1~ for ,0

25

Participation Rates for Large System

• Expected number of participants for a contest of class j

,K,Kj

Kj

R

RK

kk

j

Kj

Kk

1~

0

~,,1log ~

1

/]~

,1[ ]~

,1[

],1[],1[

1

/:max~ iik eRRiKi

kki

26

• Prior-free – independent of the distribution F

Contest Selection in Large System• User of skill l, i.e. with skill selects a

contest of class j with probability

Klj

ljJv lj

,,10

,,11

)( ],1[

),[ 1 ll vvv

m

0

1

2

34

5

123

4

1/3

1/3

1/3

27

• For large systems, what matters is which contests are selected for given skill

Proof Hint for Player-Specific Skills

28

• Key property – equilibrium expected payoffs as showed

vm0 v1v2v3

g1(v)

g2(v)

g3(v)

g4(v)

4321 RRRR

Outline

• Model of Competing Contests

• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills

• Design of Contests

• Experimental Validation

• Conclusion29

Contest-specific Skills

• Results established only for large-system limit

• Same equilibrium relationship between participation and rewards as for player-specific skills

30

Proof Hints

• Limit expected payoff – For each ],0[ mv

veRvg jjjN

)(lim

• Balancing – Whenever 0j

keReR kjkj all for ,

• Asserted relations for follow from above

),,( 1 K 31

Outline• Model of Competing Contests

• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills

• Design of Contests

• Experimental Validation

• Conclusion32

System Optimum Rewards

33

K

kkk

K

kkkk RCRU

11

)())((

RR

K

kkk

1

maximise

over

subject to

SYSTEM

• Set the rewards so as to optimize system welfare

Example 1: zero costs(non monetary rewards)

34

Assume are increasing strictly concave functions. Under player-specific skills, system optimum rewards:

()kU

KjN

UcR

N

jj ,,1 ,

)(1

)1(1'

for any c > 0 where is unique solution of

K

kkkU

1

1' )(

• Rewards unique up to a multiplicative constant – only relative setting of rewards matters

Example 1 (cont’d)

35

• For large systems

Assume are increasing strictly concave functions. Under player-specific skills, system optimum rewards:

()kU

KjceR jUj ,,1 ,)(1'

for any c > 0 where is unique solution of

K

kkkU

1

1' )(

Example 2: optimum effort

36

• Consider SYSTEM with

)))(1(1())(( )(Rjjjj

jeRmRR

)))((())(( RVRU jjjjj

)()1()( )(jj

Rj RDeRC j

exerted effort

{cost of

giving Rj (budget constraint)

{

prob. contest attended

{

Utility:

Cost:

Outline• Model of Competing Contests

• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills

• Design of Contests

• Experimental Validation

• Conclusion37

Taskcn• Analysis of rewards and participation across

tasks as observed on Taskcn– Tasks of diverse categories: graphics, characters,

miscellaneous, super challenge– We considered tasks posted in 2008

38

Taskcn (cont’d)

39

reward

number of views

number of registrants

number of submissions

Submissions vs. Reward

• Diminishing increase of submissions with reward

40

Graphics Characters Miscellaneous

linear regression

Submissions vs. Rewardfor Subcategory Logos

• Conditioning on the more experienced users, the better the prediction by the model

41

any rate once a month every fourth day every second day

• Conditional on the rate at which users submit solutions

model

Same for the Subcategory 2-D

42

any rate once a month every fourth day every second day

model

Conclusion• Crowdsourcing as a system of competing contests

• Equilibrium analysis of competing contests– Explicit relationship between rewards and participations

• Prior-free– Diminishing increase of participation with reward

• Suggested by the model and data

• Framework for design of crowdsourcing / contests

• Base results for strategic modelling– Ex. strategic contest providers

43

More Information

• Paper: ACM EC ’09

• Version with proofs: MSR-TR-2009-09– http://research.microsoft.com/apps/pubs/default.

aspx?id=79370

44