Crowdsourcing ontology engineeringElena SimperlWeb and Internet Science, University of Southampton 11 April 2013

2

Overview• "online, distributed problem-

solving and production model“ [Brabham, 2008]

• Varieties: wisdom of the crowds/collective intelligence, open innovation, human computation...

• Why is it a good idea?– Cost and efficiency savings – Wider acceptance, closer to

user needs, diversity

• Approaches– Collaborative

ontology engineering– Challenges/

competitions– Games with a purpose – Microtask/paid

crowdsourcing• In combination with

automatic techniques

3

Crowdsourcing ontology alignment• Experiments using MTurk, CrowdFlower and established benchmarks• Enhancing the results of automatic techniques• Fast, accurate, cost-effective

[Sarasua, Simperl, Noy, ISWC2012]

CartP301-304

100R50PEdas-Iasted

100R50PEkaw-Iasted

100R50PCmt-Ekaw

100R50PConfOf-Ekaw

Imp301-304

PRECISION 0.53 0.8 1.0 1.0 0.93 0.73

RECALL 1.0 0.42 0.7 0.75 0.65 1.0

Open questions• Quality assurance and evaluation• Incentives and motivators

• Choice of crowdsourcing approach and combinations of different approaches• Reusable collection of algorithms for quality assurance, task assignment,

workflow management, results consolidation etc• Schemas recording provenance of crowdsourced data

• Descriptive framework for classification of human computation systems

– Types of tasks and their mode of execution– Participants and their roles – Interaction with system and among participants– Validation of results– Consolidation and aggregation of inputs into complete solution

Theory and practice of social machines

www.sociam.org