crowdtruth - eswc phd symposium
TRANSCRIPT
![Page 1: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/1.jpg)
Crowdsourcing Disagreement for Collecting Semantic Annotation
Anca Dumitrache@anouk_anca
Extended Semantic Web ConferencePhD Symposium
June 1st 2015
![Page 2: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/2.jpg)
annotatorannotation
example
![Page 3: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/3.jpg)
annotatorannotation
exampleproblem:
![Page 4: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/4.jpg)
annotatorannotation
exampleproblem:
●
![Page 5: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/5.jpg)
annotatorannotation
exampleproblem:
●
●
![Page 6: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/6.jpg)
annotatorannotation
exampleproblem:
●
●
●
![Page 7: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/7.jpg)
annotatorannotation
exampleproblem:
●
●
●
quality = inter-annotator agreement
![Page 8: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/8.jpg)
crowdannotation
examplesolution:
![Page 9: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/9.jpg)
annotation
examplesolution:
●
●
●
crowd
![Page 10: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/10.jpg)
annotation
examplesolution:
●
●
●
crowd
![Page 11: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/11.jpg)
Hypothesis:
![Page 12: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/12.jpg)
annotation
exampleapproach:
crowd
![Page 13: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/13.jpg)
annotation
example ambiguityapproach:
crowd
![Page 14: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/14.jpg)
annotation clarity
exampleapproach:
crowd
![Page 15: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/15.jpg)
worker qualityannotation
exampleapproach:
![Page 16: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/16.jpg)
annotation
exampleapproach:
crowd
![Page 17: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/17.jpg)
annotation
exampleapproach:
crowd
![Page 18: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/18.jpg)
annotation
exampleapproach:
●
●
crowd
![Page 19: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/19.jpg)
annotation
example
crowd
approach:
![Page 20: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/20.jpg)
annotation
example
●
●
crowd
approach:
![Page 21: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/21.jpg)
Crowdsourcing Methodology
![Page 22: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/22.jpg)
annotation
example
experimental plan:
crowd
![Page 23: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/23.jpg)
annotation
example
experimental plan:
crowd
![Page 24: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/24.jpg)
annotation
example
experimental plan:
crowd
![Page 25: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/25.jpg)
annotation
example
experimental plan:
crowd
![Page 26: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/26.jpg)
annotation
example
experimental plan:
crowd
![Page 27: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/27.jpg)
annotation
example
crowd
current work:
![Page 28: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/28.jpg)
annotation
example
approach:
crowd
![Page 29: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/29.jpg)
Crowdsourcing task workflow
![Page 30: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/30.jpg)
annotation
example
evaluation:
crowd
![Page 31: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/31.jpg)
Classifier evaluation results
![Page 32: CrowdTruth - ESWC PhD Symposium](https://reader030.vdocuments.net/reader030/viewer/2022032619/55bea9c5bb61eb9b038b4677/html5/thumbnails/32.jpg)
Try it out!
CrowdTruth.org
github.com/CrowdTruth.org
@anouk_anca
#CrowdTruth