crowd centrality
DESCRIPTION
Crowd Centrality. David Karger Sewoong Oh Devavrat Shah MIT and UIUC. Crowd Sourcing. Crowd Sourcing. $30 million to land on moon. $0.05 for Image Labeling Data Entry Transcription. Micro-task Crowdsourcing. Micro-task Crowdsourcing. Left. Left. - PowerPoint PPT PresentationTRANSCRIPT
CROWD CENTRALITY
David Karger Sewoong Oh Devavrat Shah
MIT and UIUC
CROWD SOURCING
CROWD SOURCING $30 million to land on moon
$0.05 forImage LabelingData EntryTranscription
MICRO-TASK CROWDSOURCING
Which door is the women’s restroom?
Right
Left
Left
MICRO-TASK CROWDSOURCING
MICRO-TASK CROWDSOURCING
Undergrad Intern:Mturk (single label):
200 image/hr, cost: $15/hr4000 image/hr, cost: $15/hr
Reliability
90% 65%
Mturk (mult. labels): 500 image/hr, cost: $15/hr 90%
Find cancerous tumor cells
THE PROBLEM
Goal: Reliable estimate the tasks with minimal cost
Operational questions: Task assignment Inferring the “answers”
TASK ASSIGNMENT
Random ( , )-regular bipartite graphs Locally Tree-like
Sharp analysis
Good expander
High Signal to Noise Ratio
Tasks
Batches
MODELING THE CROWD
Binary tasks: Worker reliability:
Necessary assumption: we know
Aij
+ - + - +
INFERENCE PROBLEM Majority:
Oracle:
ti
++
-
p1 p2 p3 p4 p5
INFERENCE PROBLEM Majority:
Oracle:
Our Approach:p1 p2 p3 p4 p5
PREVIEW OF RESULTS
Distribution of {pj}: observed to be Beta distribution by Holmes ‘10 + Ryker et al ‘10 EM algorithm : Dawid, Skene ‘79 + Sheng, Provost, Ipeirotis ‘10
PREVIEW OF RESULTS
ITERATIVE INFERENCE Iteratively learn
Message-passing O(# edges) operations
Approximate MAPp1 p2 p3 p4 p5
EXPERIMENTS: AMAZON MTURK
Learning similarities Recommendations Searching, …
EXPERIMENTS: AMAZON MTURK
Learning similarities Recommendations Searching, …
EXPERIMENTS: AMAZON MTURK
TASK ASSIGNMENT: WHY RANDOM GRAPH
KEY METRIC: QUALITY OF CROWDCrowd Quality Parameter
p1 p2 p3 p4 p5
Theorem (Karger-Oh-Shah). Let n tasks assigned to n workers as per
an (l,l) random regular graph Let ql > √2 Then, for all n large enough (i.e. n =Ω(lO(log(1/q)) elq))) after O(log (1/q)) iterations of the algorithm
If pj = 1 for all j q = 1
If pj = 0.5 for all j q = 0
q different from μ2 = (E[2p-1])2
q≤μ≤√q
HOW GOOD IS THIS ? To achieve target Perror ≤ε, we need
Per task budget l = Θ(1/q log (1/ε))
And this is minimax optimal
Under majority voting (with any graph choice) Per task budget required is l = Ω(1/q2 log (1/ε))
no significant gain by knowing side-information(golden question, reputation, …!)
ADAPTIVE TASK ASSIGNMENT: DOES IT HELP ?
Theorem (Karger-Oh-Shah). Given any adaptive algorithm,
let Δbe the average number of workers required per task
to achieve desired Perror ≤ε Then there exists {pj} with quality q so that
gain through adaptivity is limited
WHICH CROWD TO EMPLOY
BEYOND BINARY TASKS Tasks:
Workers:
Assume pj ≥ 0.5 for all j Let q be quality of {pj}
Results for binary task extend to this setting
Per task, number of workers required scale as O(1/q log (1/ε) + 1/q log K) To achieve Perror ≤ ε
BEYOND BINARY TASKS Converting to K-1 binary problems
each with quality ≥ q
For each x, 1 < x ≤ K: Aij(x) = +1 if Aij ≥ x, and -1 otherwise ti(x) = +1 if ti ≥ x, and -1 otherwise Then
Corresponding quality q(x) ≥ q
Using result for binary problem, we have Perror(x) ≤ exp(-lq/16) Therefore
Perror ≤ Perror(2) + … + Perror(K) ≤ K exp(-lq/16)
WHY ALGORITHM WORKS? MAP estimation
Prior on probability {pj} Let f(p) be density over [0,1]
Answers A=[Aij] Then,
Belief propagation (max-product) algorithm for MAP With Haldane prior: pj is 0 or 1 with equal probability Iteration k+1: for all task-worker pairs (i,j)
Xi/Yjrepresent log likelihood ratio for ti/pj= +1 vs -1
This is exactly the same as our algorithm! And our random task assignment graph is tree-like
That is, our algorithm is effectively MAP for Haldane prior
A minor variation of this algorithm Ti
next = Tijnext
= Σ Wij’ Aij’ = Σ Wj’ Aij’ Wj
next = Wijnext
= Σ Ti’j Ai’j = Σ Ti’ Ai’j
Then, Tnext = AAT T
(subject to this modification) our algorithm is computing Left signular vector of A (corresponding to largest s.v.)
So why compute rank-1 approximation of A ?
WHY ALGORITHM WORKS?
WHY ALGORITHM WORKS?
Random graph + probabilistic model E[Aij] = (ti pj - (1-pj)ti) l/n = ti (2pj-1)l/n E[A] = t (2p-1)T l/n That is,
E[A] is rank-1 matrix And, t is the left singular vector of E[A]
If A ≈ E[A] Then computing left singular vector of A makes sense
Building upon Friedman-Kahn-Szemeredi ‘89 Singular vector of A provides reasonable approximation
Perror = O(1/lq) Ghosh, Kale, Mcafee ’12
For sharper result we use belief propagation
CONCLUDING REMARKS Budget optimal micro-task crowd sourcing via
Random regular task allocation graph Belief propagation
Key messages All that matters is quality of crowd Worker reputation is not useful for non-adaptive tasks Adaptation does not help due to fleeting nature of
workers Reputation + worker id needed for adaptation to be effective Inference algorithm can be useful for assigning reputation
Model of binary task is equivalent to K-ary tasks
ON THAT NOTE…