leveraging the power of social media
DESCRIPTION
A light intro to natural language processing on social media, presented as an invited talk at the University of Sheffield Engineering Symposium 2014 in the AI session. As well as an introduction to the area, this presentation covers powerful real-world applications of social media, and touches on the work we do in the Sheffield NLP group. Video cast: https://www.youtube.com/watch?v=QUbRmUinhHw&feature=youtu.beTRANSCRIPT
![Page 1: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/1.jpg)
Leveraging the Power of Social Media
Leon Derczynski
Natural Language Processing GroupDepartment of Computer Science
Faculty of EngineeringUniversity of Sheffield
![Page 2: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/2.jpg)
work in the field of “computational linguistics”
focus on turning textinto “understanding”
and “decision support”
![Page 3: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/3.jpg)
the “AI effect”
Pamela McCorduck
artificial intelligence is less impressive when we know how it works
![Page 4: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/4.jpg)
the “AI effect”
Pamela McCorduck
artificial intelligence is less impressive when we know how it works
..so this talk won't have deep technical detail
![Page 5: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/5.jpg)
language
![Page 6: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/6.jpg)
![Page 7: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/7.jpg)
(note huge evolutionary advancement)
![Page 8: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/8.jpg)
![Page 9: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/9.jpg)
![Page 10: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/10.jpg)
![Page 11: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/11.jpg)
![Page 12: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/12.jpg)
![Page 13: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/13.jpg)
??
![Page 14: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/14.jpg)
social media
![Page 15: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/15.jpg)
social media – a poster child for big data
![Page 16: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/16.jpg)
![Page 17: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/17.jpg)
big data:
promises new insights
is (was) a cool buzzword
causes headaches
what is it?
![Page 18: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/18.jpg)
V: velocity
twitter: 255 000 000 users / month
Facebook: 1 280 000 000 users / month
![Page 19: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/19.jpg)
VV: volume
reddit:34 000 000 posts / month
twitter:650 000 000 messages / month
![Page 20: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/20.jpg)
VVV: variety
![Page 21: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/21.jpg)
![Page 22: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/22.jpg)
![Page 23: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/23.jpg)
![Page 24: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/24.jpg)
![Page 25: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/25.jpg)
![Page 26: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/26.jpg)
there are many online social networks
we need one of these
![Page 27: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/27.jpg)
there are many online social networks
we need one of these
![Page 28: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/28.jpg)
artificial intelligence
![Page 29: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/29.jpg)
“Human knowledge is expressed in language. So computational linguistics is very important.”
- Mark Steedman
![Page 30: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/30.jpg)
Start: sequence of bytes
[naturallanguage
processinggoeshere]
End: actionable knowledge
![Page 31: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/31.jpg)
why bother programming at all?
![Page 32: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/32.jpg)
why bother programming at all?
… let the computer program itself!
![Page 33: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/33.jpg)
machine learning:
make decisions about tasks based on things you've seen before
a little bit like human learning
![Page 34: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/34.jpg)
give text and examples of what we want done
machine learns to from these examples
![Page 35: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/35.jpg)
![Page 36: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/36.jpg)
![Page 37: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/37.jpg)
understanding language
![Page 38: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/38.jpg)
social media text is surprisingly formal
![Page 39: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/39.jpg)
they see me rollin
- a typo?
![Page 40: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/40.jpg)
they see me rollinthey hatin
- perhaps not. G-dropping mapped from speech
![Page 41: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/41.jpg)
they see me rollinthey hatinpatrollin
- incidentally,this linguistic phenomenon is a good predictor of
education level
![Page 42: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/42.jpg)
they see me rollinthey hatinpatrollin
tryna catch me ridin dirty
- a new style! flawless; not a single mistake
![Page 43: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/43.jpg)
omb x
- surely they mean “omg”?
![Page 44: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/44.jpg)
omb ✔
- the keys are like, right next to each other
Xreally? this guy?
![Page 45: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/45.jpg)
Shall we go out for dinner this evening?
Ey yo wen u gon let me tap dat
![Page 46: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/46.jpg)
spelling ability distribution in net slang users
![Page 47: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/47.jpg)
with spelling ability distribution in non-slang users
![Page 48: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/48.jpg)
Do you feel luccy, punk?
![Page 49: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/49.jpg)
Do you feel luccy, punk?
![Page 50: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/50.jpg)
challenge 1: what language is this anyway
je bent Jacques cousteau niet die een nieuwe soort heeft ontdekt, het is duidelijk, ze bedekken hun gezicht. Get over it
RT @TomPIngram: VIVA LAS VEGAS 16 - NEWS #constantcontact http://t.co/VrFzZaa7
![Page 51: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/51.jpg)
challenge 2: pls type better
I wonde rif Tsubasa is okay..
- misplaced space = two new words
no homwork tonight.. suprising??
- maybe there should be!
![Page 52: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/52.jpg)
challenge 3: finding names
derekx is a person
milesx might be a person
Marie Clairex should not be a person
Exodus Porter x probably an OK person, but actually a beer
![Page 53: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/53.jpg)
challenge 3: finding names
Spicy Pickle Jr. x apparently actually a person
![Page 54: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/54.jpg)
challenge 3: finding names
Spicy Pickle Jr. x apparently actually a person
???
![Page 55: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/55.jpg)
old news
![Page 56: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/56.jpg)
social media defends against earthquakes2010
![Page 57: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/57.jpg)
Japanese and US quake response times:
down from ~20s to ~17.5s
![Page 58: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/58.jpg)
social media predicts epidemics2012
exhibit a: one dead crow
![Page 59: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/59.jpg)
social media mentions of dead crows predict WNV in humans
''There's a dead crow in my garden''
![Page 60: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/60.jpg)
social media predicts you getting flu2012
![Page 61: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/61.jpg)
@mari: i think im sick ugh..
great potential for misuse :)
![Page 62: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/62.jpg)
this november:
social media dispatches fire engines2014
![Page 63: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/63.jpg)
![Page 64: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/64.jpg)
trust
![Page 65: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/65.jpg)
if hospitals and fire stations act based on tweets,
wrong information is extra-harmful
![Page 66: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/66.jpg)
rumoursspeculation
misinformationdisinformation
who can you trust online?Imagine a lie detector for politicians / Fox News
![Page 67: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/67.jpg)
responsibility
![Page 68: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/68.jpg)
1. Collect tweets2. ????3. Profit!
![Page 69: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/69.jpg)
how long do we keep them for?
- “15 years is OK, right?” - NSA
what do we store and process?
- “just metadata, it's harmless” - GCHQ
![Page 70: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/70.jpg)
(from Kurt Opshal's slides at the Chaos Communication Congress, photo by Marion Marschalek)
![Page 71: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/71.jpg)
bias
![Page 72: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/72.jpg)
newsstyle
socialmedia
![Page 73: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/73.jpg)
most of our language AI was trained on news text
the bias is:
- middle class- white
-working age- educated
- male- 1980s/1990s- from the US
- journalist- following AP guidelines
![Page 74: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/74.jpg)
your phone rewards you if you talk and write like
(ok.. sort of)
![Page 75: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/75.jpg)
your phone rewards you if you talk and write like
(ok.. sort of)
.. and punishes you when you don't.
(not cool!)
![Page 76: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/76.jpg)
twitter bias is different
- not German or Nordic- are young(ish)
lower requirements
- you can publish even if you're not a journalist- still operates beyond the 1990s
some new requirements
- you do need access to the internet...- ...and twitter (对不起,中国人 )
![Page 77: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/77.jpg)
the big picture
![Page 78: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/78.jpg)
we're racing ahead and improving life quality
there is immense value in “trivia”
understanding social media lets ushelp people better
![Page 79: Leveraging the Power of Social Media](https://reader034.vdocuments.net/reader034/viewer/2022042623/53ed84fe8d7f7289708b5d11/html5/thumbnails/79.jpg)
understanding social media lets ushelp people better
Thank you!
Leon Derczynski