Upload File

lessons learned building machine learning models ...€¦ · lessons learned building machine...

  • Home
  • Documents
  • Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani
27
Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani User:Ladsgroup @AmirSarabadani

Upload: others

Post on 25-Sep-2020

14 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

Lessons learned building machine learning models

for WikidataWikimania 2016, Esino Lario, 24.06.20016

Amir Sarabadani User:Ladsgroup@AmirSarabadani

Page 2: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

Wikidata?

● free knowledge base by the Wikimedia movement

● structured data● user created content● linked data● people, places, events, …● powered by Wikibase (open source)● data: CC-0

Page 3: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

Quality control in Wikidata

● Quality of Wikidata is important● Vandalism is a threat!● But the backlog is too big

Page 4: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani
Page 5: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

Wikidata gets 80,000 human edits per day, 320 will be reverted and 80 of them will have been vandalism.

0.4% of edits are reverted, 0.1% of reverted edits are vandalism

80,000 edits per day(133 hours per day)

320 reverted

80 vandal

Page 6: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2015-12-09/Op-ed

“For Wikidata to truly give more people more access to more knowledge, the data in Wikidata needs to be of high quality.”

“And probably the most important part is machine-learning tools like ORES that help us find bad edits and other issues.”

https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2015-12-09/Op-ed
https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2015-12-09/Op-ed
https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2015-12-09/Op-ed
https://www.wikidata.org/wiki/Wikidata:Third_Birthday/Presents/ORES
Page 7: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

Machine learning classifiers

● Machine learning in Wikipedia● But there is no infrastructure ● Here it comes the ORES

Page 8: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

BAD!

is_anonchrs_addedchrs_removedcust_commentrepeated_chrslongest_tokenbadwords_added

?

Good.

Page 9: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

80kHumanEditsPer Day

Need review

Probably OKhttps://commons.wikimedia.org/wiki/File:Flickr_-_Official_U.S._Navy_Imagery_-_Sailor%27s_daughter_operates_a_fire_hose_with_crew_member_assistance..jpg

Page 10: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

ORES

Page 11: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
Page 12: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560

“Wikidata”

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
Page 13: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560

Is this edit damaging?

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
Page 14: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
Page 15: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
Page 16: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560

{ "267233560": { "prediction": true, "probability": { "false": 0.111, "true": 0.889 } }}

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
Page 17: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

https://ores.wikimedia.org/scores/wikidatawiki/damaging/286961313

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560

{ "267233560": { "prediction": true, "probability": { "false": 0.111, "true": 0.889 } }}

{ "286961313": { "prediction": false, "probability": { "false": 0.946, "true": 0.054 } }}

https://ores.wikimedia.org/scores/wikidatawiki/damaging/286961313
https://ores.wikimedia.org/scores/wikidatawiki/damaging/286961313
https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
Page 18: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

https://ores.wikimedia.org/scores/wikidatawiki/damaging/286961313

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560

{ "267233560": { "prediction": true, "probability": { "false": 0.111, "true": 0.889 } }}

{ "286961313": { "prediction": false, "probability": { "false": 0.946, "true": 0.054 } }}

https://ores.wikimedia.org/scores/wikidatawiki/damaging/286961313
https://ores.wikimedia.org/scores/wikidatawiki/damaging/286961313
https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560
Page 19: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

ORES in Wikipedia vs. Wikidata

● Wikitext-based vandalism detection● Wikibase-based vandalism detection● Percentage of vandalism

Wikipedia: 1-10% (Mostly ~7%)Wikidata: ~0.1%

Page 20: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

ORES extension

● Database cached = speed and filtering● Hit record

Page 21: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani
Page 22: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

Kian

Page 23: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

Problem of harvesting data

● Different languages● Type of statements● Why NLP and categories are not options

Page 24: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani
Page 25: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

How Kian works

● Extracting features● Training

Page 26: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

high_density_settlement_cats = 1

medium_density_settlement_cats = 0

low_density_settlement_cats = 0

?

Settlement

Not a settlement

Kian

Page 27: Lessons learned building machine learning models ...€¦ · Lessons learned building machine learning models for Wikidata Wikimania 2016, Esino Lario, 24.06.20016 Amir Sarabadani

Questions?

Revision Scoring team:Aaron Halfaker ([email protected])Amir Sarabadani

Me?User:[email protected]@AmirSarabadani (twitter)

mailto:[email protected]
mailto:[email protected]

Lessons Learned and Lessons to Be Learned: Investment Law & … · 2014. 4. 30. · 1 Lessons Learned and Lessons to Be Learned: Investment Law & Development for Developed Countries

Wikimania esino lario 2016 presentazione luglio 2014 esino lario

Lessons Learned Remote PM+ Cohort Integrated K+S · Lessons Learned Below are key lessons learned on providing Remote PM+ Training of Helpers. These lessons learned are based off

ASSESSMENT & LESSONS LEARNED FROM AFRICAN DIASPORA MARKETPLACEpdf.usaid.gov/pdf_docs/PDACT256.pdf · ASSESSMENT & LESSONS LEARNED FROM AFRICAN ... ASSESSMENT & LESSONS LEARNED FROM

Lessons Learned Handbook · 2011-10-12 · • Lessons Learned, an adjective, describes anything related to a Lessons Learned procedure. E.g. Lessons Learned process, Lessons Learned

Lessons learned from accident investigation Seminar/IHS - Day 2... · LESSONS LEARNED FROM OTHERS LESSONS LEARNED FROM UAE ... Lessons Learned . ... Dealing with survivors and bereaved

LEARNED LESSONS:

Lessons learned & not learned at MSU

Lessons Learned

Lessons&Learned&in&Deploying&Akka& Persistencedownloads.typesafe.com/website/presentations/lessons-learned-akk… · Lessons&Learned&in&Deploying&Akka& Persistence Duncan&K.&DeVore

LESSONS LEARNED AND GOOD PRACTICES...Lessons Learned Lessons are learned for sharing and applying to similar projects. Project lessons are learned from experiences, both inspiring

Lessons Learned from “Lessons Learned” · 2 LESSONS LEARNED FROM “LESSONS LEARNED” 2. The NRC set safety goals in its Safety Goal Policy Statement, initiated not long after

Lean lessons learned - lean startup methodology and lessons learned from Blossom

Lessons Learned Events at SLAC · 2012-09-14 · Lessons Learned Events: Summary Comments . Important to identify and communicate lessons learned! Lessons learned and near misses

Lessons learneD

Lessons Learned:

Esino una volta, Esino futura

Lessons Learned from Failures Dec 2014 - MTS Houston · 212 on board, 123 perished, 89 survivors ... Lessons Learned Lessons Learned. ... Lessons Learned from Failures Dec 2014.pptx

Lessons Learned Globally - Campaign for Tobacco-Free Kids...Process lessons learned 2. Content lessons learned See next page for lists of these lessons learned. 7 | Lessons Learned