pattern-based specification of crowdsourcing applications
Post on 23-Aug-2014
633 Views
Preview:
DESCRIPTION
TRANSCRIPT
Pattern-Based Specification of Crowdsourcing Applications
Alessandro Bozzon (TU Delft) Marco Brambilla (Politecnico di Milano)Stefano Ceri (Politecnico di Milano)Andrea Mauri (Politecnico di Milano)Riccardo Volonterio (Politecnico di Milano)
Crowdsourcing and Human Computation
It works like magic!
Endless Applications
Endless Success Stories
2008 Olympics Opening Ceremony
Actually… .
Often a Try&HopeError process
Task Design Matters
Crowd can be unreliable
($) Incentives Matter
Quality Control Matters
Platform of Execution Matters
Ok, so what? .
#Workers #Useful Workers #Executions Cost $ Time
(hours) Precision
Setting Nº1 732 44
(6%) 488 72 ~40 ~67%
Setting Nº2 498 25
(5%) 547 48 ~169 ~67%
Setting Nº3 1420 464
(32%) 3387 83 ~184 ~75%
Total 2152 508 4422 203 ~16 days
GOAL !
Simplify and systematize the design, deploy, and monitoring of applications (including experiments)
Contributions
An Abstract Model of
Crowdsourcing Activities
A Composition model for
Crowdsourcing Activities
A Library of crowdsourcing
Patterns
• A conceptual framework • A specification paradigm • A reactive execution control
environment
Crowd Task
[T operation types](intra-task patterns)
Object Type
block sizemin #obj(cons)
input buffer
batch flow (on closed task)stream flow (on closed object)
MicroTask[MT operation types]
r
data manipulator
Case Study: Movie Scene Analysis
Scenario 1: Scene Positioning
Spoiler Alert!
Order Scenes
Scene in Beg/Mid/End
Scenario 2: Actor Identification
Find Actor
Validate Actor
Position Scenes
[Classify](Static Agreement@3)
MicroTask [Classify]
Scene
block 1min 1
Spoiler Scenes
[Like] (Static Agreement@3)
MicroTask [Like]
Scene
block 1min 1
5
Order Scenes
[Order] (SortByLiking)
MicroTask [Like]
Scene
block 2min 2
[Class=E]
[Class=B OR M]
Example of Scenario 1 Model
Intra-Task
Pre-Processing Post-Processing
Task
Consensus
Splitting
Assignment
AggregationmicroTaskmicroTaskmicroTask
microTaskmicroTaskmicroTaskmicroTaskmicroTaskmicroTask
• Consensus • Join • Sort
• Grouping • Performer Control
• Planning • Assignment • Aggregation • Quality & Performer
Auxiliary Intra-Task
Pre-Processing Post-Processing
Task
Consensus
Splitting
Assignment
AggregationmicroTaskmicroTaskmicroTask
microTaskmicroTaskmicroTaskmicroTaskmicroTaskmicroTask
• Pruning • Tie Breaking
• Operations before or after the execution
Workflow• Set of heterogeneous
tasks
Create Decide
Improve Compare / Verify
Find Fix
(a)
(b)
(c)
Auxiliary Task
Create/Decide
Improve/Compare
Find/Fix/Verify
Streaming Vs. Batch Execution
Position Scenes
[Classify] (Static Agreement@3)
MicroTask [Classify]
Scene
block 1min 1
Spoiler Scenes
[Like](Static Agreement@3)
MicroTask [Like]
Scene
block 1min 1
7
Order Scenes
[Order](SortByLiking)
MicroTask [Like]
Scene
block 2min 2
[Class=E]
[Class=B OR M]
(P1)5
3
Position Scenes
[Classify](Static Agreement@3)
MicroTask [Classify]
Scene
block 1min 1
Spoiler Scenes
[Like](Static Agreement@3)
MicroTask [Like]
Scene
block 1min 1
7
Order Scenes
[Order](SortByLiking)
MicroTask [Like]
Scene
block 2min 2Cons.
[Class=E]
[Class=B OR M]
(P2)5
3
P.Beg P.Mid P.EndP1 0.5 1 0.11P2 0.5 0.8 0.33
Spear.Beg Spear. Mid.MidP1 0.5 0.54
P2 0.9 0.51
PositionOrder
P1PositionOrder
P2
b) Elapsed Time (Mins)
#Clo
sed
Obj
ects
11020304050607080
5 60 120 180 240 300 360 5 60 120 180 240 300 360
PositionOrder
P1PositionOrder
P2
b) Elapsed Time (Mins)
#Clo
sed
Obj
ects
11020304050607080
5 60 120 180 240 300 360 5 60 120 180 240 300 360
PositionOrder
P1PositionOrder
P2
b) Elapsed Time (Mins)
#Clo
sed
Obj
ects
11020304050607080
5 60 120 180 240 300 360 5 60 120 180 240 300 360
Intra-Task Consensus Vs. Workflow Decision
A4 A5 A6Precision 0.99 0.95 0.89
Recall 0.90 1 0.96F-Score 0.93 0.97 0.90
Find Actors
[Tag] (Static Agreement@3)
MicroTask [Tag]
Scene
block 1min 1
Validate Actors
[Like]
MicroTask [Like]
Scene+Actor
block Allmin 1
5
(A4)
Find Actors
[Tag]
MicroTask [Tag]
Scene
block 1min 1
Validate Actors
[Like] (Majority Voting@2)
MicroTask [Like]
Scene+Actor
block Allmin 1
5 3
(A5)
(A6)
Find Actors
[Tag](Static Agreement@3)
MicroTask [Tag]
Scene
block 1min 1
Validate Actors
[Like] (Majority Voting@2)
MicroTask [Like]
Scene+Actor
block Allmin 1
5 3
count(Actor.Like)<=1
ActorValidate
A3ActorValidate
A4ActorValidate
A5
a) Elapsed Time (Mins)
ActorValidate
A6
#Clo
sed
Obj
ects
0102030405060
5 30 60 90 120 160
5 30 60 90 120 160 5 60 120 180 240 60 300 540 780
ActorValidate
A3ActorValidate
A4ActorValidate
A5
a) Elapsed Time (Mins)
ActorValidate
A6
#Clo
sed
Obj
ects
0102030405060
5 30 60 90 120 160
5 30 60 90 120 160 5 60 120 180 240 60 300 540 780
ActorValidate
A3ActorValidate
A4ActorValidate
A5
a) Elapsed Time (Mins)
ActorValidate
A6
#Clo
sed
Obj
ects
0102030405060
5 30 60 90 120 160
5 30 60 90 120 160 5 60 120 180 240 60 300 540 780
ActorValidate
A3ActorValidate
A4ActorValidate
A5
a) Elapsed Time (Mins)
ActorValidate
A6
#Clo
sed
Obj
ects
0102030405060
5 30 60 90 120 160
5 30 60 90 120 160 5 60 120 180 240 60 300 540 780
ActorValidate
A3ActorValidate
A4ActorValidate
A5
a) Elapsed Time (Mins)
ActorValidate
A6
#Clo
sed
Obj
ects
0102030405060
5 30 60 90 120 160
5 30 60 90 120 160 5 60 120 180 240 60 300 540 780
ActorValidate
A3ActorValidate
A4ActorValidate
A5
a) Elapsed Time (Mins)
ActorValidate
A6
#Clo
sed
Obj
ects
0102030405060
5 30 60 90 120 160
5 30 60 90 120 160 5 60 120 180 240 60 300 540 780
Take-Home Message
Engineering approaches should be applied to people management too
In crowdsourcing, control and monitoring are key for successful outcomes
top related