crowdsourcing - brown universitycs.brown.edu/courses/cs195w/slides/crowdsourcing.pdfan sql extension...
TRANSCRIPT
Definition Mechanical Turk Quality Control Techniques CrowdDB
Crowdsourcing
Nickolai Riabov, Kenneth Tiong
Brown University
Fall 2013
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Structure of the Talk
DefinitionMechanical TurkQuality ControlCrowdDB
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
What is Crowdsourcing?
The practice of obtaining needed services, ideas, orcontent by soliciting contributions from a large group ofpeople, and especially from an online community, ratherthan from traditional employees or suppliersAllows for large-scale and on-demand invocation ofhuman input for data-gathering and analysisDistinct from outsourcing in that the work comes from anundefined public rather from a specific group
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Crowdsourcing Overview
Requester:People who submit tasks and collect answers
Platform:Performs task management
Worker:People who work on tasks
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Machine Translation
Problem:Manual Evaluation of quality is slow and expensive
Crowdsourcing:Low costs of non-experts, $0.10 to translate a sentenceHigh agreement between experts and non-expertsGood framework for complex tasks like human-assistedtranslation edit rate (i.e. how much editing a humanwould have to perform to change a system output so thatit exactly matches a reference translation)
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Li, Guoliang, Crowdsourcing @ HotDB2012
Definition Mechanical Turk Quality Control Techniques CrowdDB
Painting Similarity
How similar is the artistic style in the paintings above?Very SimilarSimilarSomewhat DissimilarVery Dissimilar
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Lease, M and Kovashka, A., Human and Machine Detection of Stylistic Similarity in Art.CrowdConf 2010
Definition Mechanical Turk Quality Control Techniques CrowdDB
Image Search
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Tingxin Yan, Vikas Kumar, Deepak Ganesan: CrowdSearch: exploiting crowds for accuratereal-time image search on mobile phones. MobiSys 2010:77-90
Definition Mechanical Turk Quality Control Techniques CrowdDB
Examples of Crowdsourcing Platforms
Most Famous: WikipediaMechanical Turk: Marketplace for (usually small) tasksCrowdDB: Uses crowd to answer DB queries
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
When to Crowdsource
Computers cannot do the task (e.g. translation)A single person cannot do the taskThe work can be split into many small tasks
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Different Slide Deck
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Different Slide Deck
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
CrowdDB
Relational Database FailSELECT market_capitalization FROM company
WHERE name = "I.B.M.";
Query returns an empty answer if the company tableinstance in the database does not contain a record for"I.B.M."
Why?Could have been deleted by accidentCould be under I.B.N.Could be under International Business Machines
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Issues with Relational Databases
Closed World AssumptionInformation not in database is either false or nonexistent
Relational databases are extremely literalExpect data to have been properly cleaned andvalidated before entry; no native tolerance ofinconsistency in data or queries
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Issues with Relational Databases
Let’s say you were to run a query like the one below:SELECT image FROM picture
WHERE topic = "Business Success"
ORDER BY relevance LIMIT 1;
Unless someone had previously sorted the pictures byspecific topic, there is no good way to run a query likethis
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
CrowdDB
Use the crowd to answer DB queriesFind missing dataMake a subjective comparison
Recognize patternsMain operations
JoinSort
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
CrowdSQL
An SQL extension that supports crowdsourcing (and istherefore the language for crowdDB)Involve missing data and subjective comparisonsFor traditional databases, equivalent to SQLDevelopers don’t have to be aware that their codeinvolves crowdsourcing
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
CrowdSQL
SQL DDL ExtensionsSpecific attributes of tuples can be crowdsourcedEntire tuples can be crowdsourcedKeyword: CROWD
CrowdDB does not impose any limitations with regard toSQL types and integrity constraintsCROWD tables must have a primary key
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
CrowdDB
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Sample Code
Column "url" marked as crowdsourced
CREATE TABLE Department (university STRING,
name STRING,
url CROWD STRING,
phone STRING,
PRIMARY KEY (university, name));
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Sample Code
"Professor" table to be crowdsourced
CREATE CROWD TABLE Professor (
name STRING PRIMARY KEY,
email STRING UNIQUE,
university STRING,
department STRING,
FOREIGN KEY (university, department)
REF Department(university, name) );
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Comparisons
CROWDEQUAL – ask the crowd if two objects are equal
SELECT profile FROM department
WHERE name ∼= "CS";
CROWDORDER – ask the crowd to arrange the objectsin order of importance
CREATE TABLE picture (
p IMAGE,
subject STRING
);
SELECT p FROM picture
WHERE subject = "Golden Gate Bridge"
ORDER BY CROWDORDER(p, "Which picture visualizes
better %subject");
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
User Interface Generation
Automatically generates user interfacesTwo-step process in CrowdDBUser interfaces are in HTML and JavaScript
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
What the worker sees
The title of the HTML is the name of the tableFields ask the worker to input the missing informationCopies the known field values into the HTML formGenerates JavaScript code to check for correct types ofinput
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Multi-Relation Interfaces
Foreign key references a non-crowdsourced tableGenerated user interface shows a drop-down boxCrowdDB supports two types of user interfaces
NormalizedDenormalized
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Crowd Operators
Implements all operators of the relational algebra, justlike any traditional database systemInitialized with a user interface template and thestandard HIT parametersQuality control carried out by majority vote
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
CrowdDB has three crowd operators
CrowdProbe: Crowdsources missing information ofcrowd columnsCrowdJoin: Implements an index nested-loop join overtwo tablesCrowdCompare: Implements the CROWDEQUAL andCROWDORDER functions
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
CrowdSQL in practice
Minimal extension to SQLCrowdSQL changes the closed-world to an open-worldassumptionCost and response time of queries can be unboundedProvide a way to define a budget for a query – using theLIMIT operator
Constrains the number of tuples returned as a result ofthe queryImplicit constraint on cost and result time of query
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Shortcomings of CrowdSQL
No explicit constraint on budgetLIMIT only constrains number of responses to query
No accounting for lineageTurker #5 is a spammer. Currently no way to identify andremove all data from him
No entity resolution of crowdsourced data.Not a problem if all the workers use exactly the sameliteralsIn general, makes data from different sources difficult toclean
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Benchmarks
Workers were asked to fill in missing data for a tablewith two crowdsourced columnsFor 3607 business names in 40 cities, the turkers had tofind the phone # and address of the business
CREATE TABLE businesses (
name VARCHAR PRIMARY KEY,
phone_number CROWD VARCHAR(32),
address CROWD VARCHAR(256)
);
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Experiment: Vary hit groups, track responsetime
Response times decrease dramatically as size of HITgroups increases
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Experiment: Vary hit groups, track responsetime
But, there is a tradeoff between size of HIT group andhow much of that HIT group is actually completed
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Experiment: Responsiveness, vary reward
For the particular task the experimenters assigned,paying the turkers more resulted in increasedperformance
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Experiment: Worker affinity and quality
Analysis of the distribution of work among workers andanswer qualitySome workers begin to specialize in a particularrequester’s requestsThis does not decrease error frequencyReward and group size also has no effect on errorfrequency
Nickolai Riabov, Kenneth Tiong Crowdsourcing
Definition Mechanical Turk Quality Control Techniques CrowdDB
Observations
Crowd resources have long-term memory that impactperformance
If the requester rejects too many HITs, workers stopworking for requesterBugs leading to error messages can alarm the turkers
User interface design and precise instructions cangreatly increase reliability of results
Nickolai Riabov, Kenneth Tiong Crowdsourcing