crowdsourcing€¦ · crowdsourcing • crowdsourcing is a process where work is outsourced to an...
TRANSCRIPT
![Page 1: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/1.jpg)
Crowdsourcing
Sasha Mile Rudan
![Page 2: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/2.jpg)
Crowdsourcing • Crowdsourcing is a process where work is
outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine
• often been used in the past as a competition in order to discover a solution
• Micro-tasks, repetition, learning, fatigue • Collaborative human interpreter
• first proposed as a programming language on the blog Google Blogoscoped (Philipp Lenssen)
![Page 3: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/3.jpg)
Mechanical Turk • Automaton Chess Player • a fake chess-playing
machine constructed in the late 18th century
• Human “supported”/ “augmented” machine
![Page 4: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/4.jpg)
Amazon Mechanical Turk (1/2) • Collaborative human interpreter
• first proposed as a programming language on the blog Google Blogoscoped (Philipp Lenssen)
• HIT: A Human Intelligent Task • Assignment
• Every HIT can be replicated into multiple assignments • enable majority voting for quality assurance
• HIT Group: • AMT automatically groups similar HITs together
![Page 5: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/5.jpg)
Amazon Mechanical Turk (2/2) • AMT Workflow: requester, workers (turks) • 2 kinds of user interface for workers • Relationship & Reputation • Workers has online-forums about requesters • AMT has REST APIs • Applications: Computational difficult tasks, Missing persons
searches, Social science experiments, Artistic & educational research
![Page 6: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/6.jpg)
Social Turing Tests: Crowdsourcing Sybil Detection
• Feasibility of a crowdsourced Sybil detection system for OSNs
• Sybil, psychiatric case of Shirley Mason who reportedly had 16 personalities
• Detection accuracy (experts, turkers) • Scalability, efficiency
![Page 7: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/7.jpg)
Automated techniques of detection • SybilRank, … • difficulty friending legitimate users
• Own communities • community detection techniques, social graph
• New malicious users generations • infiltrate communities of legitimate users • copying profile data, recruiting real users to
customize them, crowdsourced, paid, anti-CAPTCHA
![Page 8: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/8.jpg)
Ground-truth data collection • Renren
• Sybils directly from Renren Inc • Zhubajie, crowdsourcing platform, instead of AMT
• Facebook USA, Facebook India • Sybil accounts by crawling
• Legitimate profiles: seeding from 4 trusted friends => friends-of-friends => unknown
![Page 9: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/9.jpg)
Social-Turing tests • Social-Turing tests are resilient to
changing attacker strategies • Crowdsourcing is much cheaper than hiring
full-time content moderators • How accurate are users at distinguishing between real
and fake profiles? Are there demographic factors? Does survey fatigue impact detection accuracy? Is crowdsourced Sybil detection cost effective?
![Page 10: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/10.jpg)
Datasets
![Page 11: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/11.jpg)
User Study • User questionnaire • 2 languages • no malicious testers • Wall and photos are
under • Chance to change
the previous answer
![Page 12: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/12.jpg)
Demographic
![Page 13: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/13.jpg)
Testers’ accuracy • Individual accuracy
• Chinese and Indian turkers perform the worst, with half achieving ≤65% accuracy
![Page 14: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/14.jpg)
False positive and False negative • False positives are uniformly lower than
false negatives
![Page 15: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/15.jpg)
Improvements, majority voting • FN error increase with adding more turkers
![Page 16: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/16.jpg)
Reasons for Suspicion
![Page 17: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/17.jpg)
Turker Accuracy Analysis • Demographic factors
• Previous slide with demography
• Temporal Factors and Survey Fatigue
![Page 18: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/18.jpg)
Turker Selection • can we continue to improve the overall
accuracy of turkers by simply adding more of them
• Filtering Inaccurate Turkers • pre-screening test
![Page 19: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/19.jpg)
Profile Difficulty • Are there extremely difficult “stealth” • Turker luck?
![Page 20: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/20.jpg)
System design
![Page 21: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/21.jpg)
• One-layer schema • Two-layer schema • Simulation • Privacy
![Page 22: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/22.jpg)
CrowdDB: Answering Queries with Crowdsourcing
• human input for providing information that is missing from the database
• computationally difficult functions, matching, ranking, or aggregating results based on fuzzy criteria
• Closed-world assumption • Amazon Mechanical Turk
![Page 23: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/23.jpg)
RDBMS • Limitations of the technology are becoming
more apparent • Assumptions about the correctness,
completeness and unambiguity of the data they store • When these assumptions fail to hold, relational
systems will return incorrect or incomplete answers
![Page 24: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/24.jpg)
• SELECT market_capitalization FROM company WHERE name = "I.B.M.";
• Typing error • inadvertently deleted, entered incorrectly • Multiple ways to refer to the same real-
world entity
![Page 25: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/25.jpg)
• SELECT image FROM picture WHERE topic = "Business Success” ORDER BY relevance LIMIT 1;
• could easily be answered by people
![Page 26: Crowdsourcing€¦ · Crowdsourcing • Crowdsourcing is a process where work is outsourced to an undefined group of people • Jeff Howe and Mark Robinson, Wired Magazine • often](https://reader034.vdocuments.net/reader034/viewer/2022042811/5fa5d209b7889627de7081c1/html5/thumbnails/26.jpg)
AMT • Relatively Small Worker Pool