The Wisdom of the Crowd through Gamification

Lora Aroyo, Merel van Empel & Carlos Martinez Ortiz

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

massive  digital  content  online  …  

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

needs  massive  resources  to  process  …  

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

crowdsourcing  –  interpreta5on  at  scale  …  

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

paid  crowdsourcing  solves  only  part  …  

there  are  massive  niches  of  experts  online  …  

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

but  they  need  to  be  engaged  …  

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

gamifica5on  is  the  answer  …  

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

•  entertaining  •  compe--ve  •  quick  •  s-cky  •  targeted  campaigns  •  effec-ve  engagement  

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

games  with  a  purpose  (GWAP)  

Winner EuroITV CompetitionBest Archives on the Web Award

h7p://waisda.nl  

Video Tagging – time-stamped tags

Image Tagging – understanding & training crowds

h7p://annotate.accurator.nl  

Nichesourcing domain expertise

diversity  of  opinions  from  the  crowd  mul-tude  of  contexts  

collected  in  a  decentralized  way  independent  interpreta5ons  

aggregated  view  –  the  big  picture  

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

understanding  perspec5ves  

a  new  approach  to  understanding  seman5cs  harnessing  the  power  &  diversity  of  the  crowd  

 harnessing  disagreement  

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

CrowdTruth.org  

informs  a  vector  space  model  of  truth    instead  of  a  boolean,  fuzzy  or  sta-s-cal  model  

 disagreement  is  signal  

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

CrowdTruth.org  

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

CrowdTruth.org

http://game.CrowdTruth.org

h7p://game.crowdtruth.org    

h7p://game.crowdtruth.org    

h7p://game.crowdtruth.org    

h7p://game.crowdtruth.org    

h7p://game.crowdtruth.org    

h7p://game.crowdtruth.org    

h7p://game.crowdtruth.org    

h7p://game.crowdtruth.org    

h7p://game.crowdtruth.org    

h7p://game.crowdtruth.org    

•  Games engage niche crowds - effective in filling the gap of expert annotations in traditional crowdsourcing

•  Annotator disagreement is signal, not noise:

•  indicative of the variation in human semantic interpretation of signs

•  can indicate ambiguity, vagueness, similarity, over-generality, etc, as well as quality

•  Data analysis challenges

Take Home Message

h7p://game.crowdtruth.org    

CrowdTruth  Team:  Lora  Aroyo,  Chris  Welty    Robert-­‐Jan  Sips,  Carlos  Mar5nez  Or5z,  Anca  Dumitrache,  Oana  Inel,  Benjamin  Timmermans,  Susanna  van  de  Ven,  Merel  van  Empel,  Jelle  v.d.  Ploeg,  Ta-ana  Cristea,  Khalid  Khamkham,  Harrië7e  Smook,  Rens  van  Honschooten,  Arne  Rutjes  

CrowdTruth.org github.com/CrowdTruth

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

CrowdTruth.org

http://game.CrowdTruth.org

massive  crowdsourcing  in  cultural  heritage  …  

http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo