sentiment is shite (2014)
DESCRIPTION
Slide deck from a talk given at the BreakingNews.ie Measurement Conference on 10th September 2014. It's an updated version of a talk I gave (only once) in 2011. My understanding (and the technology) have developed since then.TRANSCRIPT
![Page 1: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/1.jpg)
NEGATIVE SENTIMENT another grumbly presentation from @mediaczar
![Page 2: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/2.jpg)
TWITTER VOLUME
0
5
10
15
1 Jun 2014 1 Jul 2014
Thou
sand
s
“facebook” AND (“newsfeed” OR “news feed”) June 1 – July 31 2014
Data: Netbase
![Page 3: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/3.jpg)
WHAT HAPPENED HERE?
aND HOW DOES THAT MAKE YOU FEEL?
Topic Analysis Emotions
Data: Brandwatch, Netbase
![Page 4: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/4.jpg)
DELOITTE: BALANCE OF SENTIMENT AFFECTS SALES
The balance of sentiment in Tweets is a more powerful driver of sales than reach (or volume) alone, with positive Tweets having generally a higher impact than negative Tweets. Therefore, to gain the most out of the online word- of-mouth embodied by Tweets, companies would be best served by
addressing the balance of sentiment
about their games through increasing the
number of positive Tweets.
6.10%
3.30%
1.60%
30% more positive Tweets
30% fewer negative Tweets
30% more non-Twitter advertising
Deloitte, “Tweets for Sales, Gaming” (2013)
![Page 5: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/5.jpg)
We want to increase the positive buzz around Your
brand
“SENTIMENT” HAS BECOME A MARKETING OBJECTIVE
![Page 6: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/6.jpg)
“SENTIMENT” HAS BECOME A MARKETING OBJECTIVE
Real case study. ���Name obscured to protect the innocent.
![Page 7: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/7.jpg)
THIS IS HONEST ABE
Public sentiment is everything.
With public sentiment, nothing
can fail.
Without it, nothing can succeed.
![Page 8: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/8.jpg)
SENTIMENTANALYSIS
I’M GOING TO BE EVEN MORE HONEST
DON'T HOLD BACK...TELL US WHAT YOU REALLY THINK!
SENTIMENTANALYSIS
IS SHITE!
![Page 9: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/9.jpg)
THE PROMISE OF SOCIAL INTELLIGENCE
AN ALMOST INFINITE SOURCE OF QUAL & QUANT
DATA!
FINALLY THEY WILL REVEAL WHAT THEY REALLY THINK!
![Page 10: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/10.jpg)
VOLUME
RELEVANCE
AUTHORITY
TOPICS
SENTIMENT SENTIMENT
![Page 11: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/11.jpg)
WEIGHTED SENTIMENT, MRS BROWN’S BOYS, JULY 2014
14.8%
12.2%
11.4%
7.3%
-4.1%
-5.2%
-6.3%
-8.5%
POSITIVE
4 DIFFERENT TOOLS GIVE 4 DIFFERENT MEASURES OF SAME DATA
NEGATIVE
![Page 12: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/12.jpg)
METHODOLOGICAL PROBLEMS
LEXICAL ANALYSIS CLASSIFICATION MANUAL INPUT
![Page 13: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/13.jpg)
LEXICAL ANALYSIS
visit: http://www.wjh.harvard.edu/~inquirer/spreadsheet_guide.htm
e.g. Harvard General Enquirer: 11.8K categorised, tagged words
![Page 14: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/14.jpg)
QUICK & DIRTY DIY SENTIMENT ANALYSIS TOOL
![Page 15: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/15.jpg)
SOME OBVIOUS PROBLEMS…
![Page 16: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/16.jpg)
CAN’T HANDLE IDIOMS, SYNONYMS (OR IRONY)
WELL, THAT'S JUST GREAT
![Page 17: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/17.jpg)
“Posts were determined to be positive or negative if they contained at least one positive or negative word”
Visit: http://www.pnas.org/content/111/24/8788.full LIWC: http://www.liwc.net/
FACEBOOK USED LEXICAL APPROACH
0
5
10
15
Thou
sand
s
![Page 18: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/18.jpg)
CLASSIFIERS & SUPERVISED LEARNING…
TRAIN MODEL
TEST MODEL TAGGED DATA
TRAINING DATA
TEST DATA
POSITIVE
NEUTRAL
NEGATIVE
![Page 19: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/19.jpg)
THIS PRESENTATION IS GOOD
visit: text-processing.com/demo/sentiment
CLASSIFIER SAYS “POSITIVE”
![Page 20: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/20.jpg)
THIS PRESENTATION IS BAD
CLASSIFIER SAYS “NEGATIVE”
visit: text-processing.com/demo/sentiment
![Page 21: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/21.jpg)
THIS PRESENTATION IS NOT GOOD
CLASSIFIER SAYS “NEGATIVE”
visit: text-processing.com/demo/sentiment
![Page 22: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/22.jpg)
THESE ARE BOTH POSITIVE
analysed
analysed
![Page 23: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/23.jpg)
THIS IS NEUTRAL
analysed
![Page 24: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/24.jpg)
WHAT ABOUT THIS?
![Page 25: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/25.jpg)
DOES IT UNDERSTAND WHAT IT’S READING?
2. Randomise the word order 1. Take the original text 3. Re-test
recipe: http://stackoverflow.com/questions/17825945/generating-a-list-of-random-words-in-excel-but-no-duplicates
![Page 26: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/26.jpg)
WORD ORDER MAKES NO DIFFERENCE
![Page 27: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/27.jpg)
WORD ORDER MAKES NO DIFFERENCE
![Page 28: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/28.jpg)
WORD ORDER MAKES NO DIFFERENCE
![Page 29: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/29.jpg)
DOMAIN SPECIFIC
Models trained on one set of data ���may not work well on other sets
![Page 30: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/30.jpg)
WHAT ABOUT HUMAN MARKERS?
![Page 31: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/31.jpg)
FAIRLY EASY TO ASSESS ENTERTAINMENT CATEGORY
![Page 32: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/32.jpg)
EXPERIMENT: IS GUINNESS GOOD FOR YOU?
Selected 50 positive and 50 negative tweets as scored by classifier. Passed these tweets to human markers. Each tweet scored 3 times (5 point scale) Average score compared to classifier.
![Page 33: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/33.jpg)
MECHANICAL TURKS!
Visit: http://www.crowdflower.com/
![Page 34: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/34.jpg)
RESULTS: IS GUINNESS GOOD FOR YOU?
NEG NEUT POS
CLASSIFIER 50 0 50
MANUAL 23 30 47
AGREEMENT 40% 0% 72% (Agreement based on #tweets with matched judgments)
![Page 35: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/35.jpg)
Advert used to say #Guinness is good for you
but I think it is not acceptable to say that these days, but in moderation I thrive on it at
70
It's called a rotten apple
#twobeersonecup #Guinness #angryorchard
#delicious http://t.co/GUFxrYoF6i
HUMANS DON’T ALWAYS AGREE…
![Page 36: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/36.jpg)
CLASSIFY THIS…
OR
THERE ARE HUGE LINES AT
THE APPLE STORE TODAY
![Page 37: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/37.jpg)
BIG ENOUGH NUMBERS If the sample is large enough, won’t these problems get ironed out?
![Page 38: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/38.jpg)
SAMPLE BIAS (1936 US PRESIDENTIAL ELECTION)
See: Tim Harford, “Big Data: are we making a big mistake?” (FT Magazine, 28 March 2008)
SURVEY SIZE ROOSEVELT
LITERARY DIGEST 2,400,000 43%
GALLUP 50,000 54%
ACTUAL 61%
ALF LANDON
I THINK YOU'll find iT'S 48 times bigger
![Page 39: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/39.jpg)
WIN 30 MINS OF FREE CONSULTANCY (VALUE £750)
I'm loving #breakingnewsconf,
@mediaczar.
IT’S FAR TOO EASY TO GAME “POSITIVE SENTIMENT” METRICS
![Page 40: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/40.jpg)
RECOMMENDATIONS
PUT GREAT TECH TO MORE MEANINGFUL USE
DON’T LET SENTIMENT BECOME A KPI
MAKE SENTIMENT A TOOL FOR MORE COMPLEX RESEARCH
LIFE ISN’T A POPULARITY CONTEST
![Page 41: Sentiment is Shite (2014)](https://reader033.vdocuments.net/reader033/viewer/2022061123/54705134af79590e1b8b4938/html5/thumbnails/41.jpg)
THANK YOU! PLEASE DON’T ASK ME ANY TRICKY QUESTIONS THAT WILL MAKE ME LOOK STUPID I’M @MEDIACZAR ON TWITTER FEEL FREE TO COME AND TALK TO ME AFTERWARDS