detecting sarcasm in multimodal social platforms
TRANSCRIPT
![Page 1: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/1.jpg)
Eureka Presentation
ROSSANO SCHIFANELLA, Paloma de Juan, Joel Tetreault, Liangliang Cao
@ACM Multimedia 2016, Amsterdam
DETECTING SARCASM IN MULTIMODAL SOCIAL PLATFORMS
![Page 2: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/2.jpg)
Eureka Presentation
SARCASM
![Page 3: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/3.jpg)
Eureka Presentation
Because who doesn’t love finishing the slides late at night the day before the talk… #acmmm2016 #hangover
![Page 4: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/4.jpg)
Eureka Presentation
WHAT IS SARCASM?
LITERAL INTENDED≠
![Page 5: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/5.jpg)
Eureka Presentation
Great day today
![Page 6: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/6.jpg)
Eureka Presentation
LEXICAL and LINGUISTIC MARKERS
![Page 7: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/7.jpg)
Eureka Presentation
INTERJECTIONS, INTENSIFIERS, HYPERBOLES
Well, really great day today
![Page 8: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/8.jpg)
Eureka Presentation
Great day today !?!?!?!?
PUNCTUATION
![Page 9: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/9.jpg)
Eureka Presentation
CONTEXT
![Page 10: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/10.jpg)
Eureka Presentation
Great day today! #epicfail
HASHTAGS
![Page 11: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/11.jpg)
Eureka Presentation
Great day today! #winning
HASHTAGS
![Page 12: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/12.jpg)
Eureka Presentation
Great day today! 😭 ⛈ ☔
EMOJIS
![Page 13: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/13.jpg)
Eureka Presentation
Great day today! 😍 🏖☀
EMOJIS
![Page 14: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/14.jpg)
Eureka Presentation
Great day today
Third car accident in a mile!
1
2
PREVIOUS POSTS
@RSCHIFAN
@RSCHIFAN
![Page 15: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/15.jpg)
Eureka Presentation
AUTHOR PROFILE, PROPENSITY TO SARCASTIC UTTERANCES
Great day today
Well this is not stressful at all #sarcasm
1
3
@RSCHIFAN
@RSCHIFAN
2 I looooove Trump’s hair! #sarcasm@RSCHIFAN
![Page 16: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/16.jpg)
Eureka Presentation
SOCIAL MEDIA IS MULTIMODAL
![Page 17: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/17.jpg)
Eureka Presentation
METADATA VISUALS
TEXT
![Page 18: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/18.jpg)
Eureka Presentation
Great day today
![Page 19: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/19.jpg)
Eureka Presentation
Great day today
![Page 20: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/20.jpg)
Eureka Presentation
Text+Image
Image as a contextual clue
![Page 21: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/21.jpg)
Eureka Presentation
POSTS CONTAINING #SARCASM OR #SARCASTIC
DATA
517K 63K 20K99% 40% 7.56%
TEXT+IMAGE TEXT+IMAGE TEXT+IMAGE
![Page 22: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/22.jpg)
Eureka Presentation
CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components
1
![Page 23: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/23.jpg)
Eureka Presentation
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION IS THE TEXT ENOUGH?
YES NO
DO
ES T
HE IM
AG
E H
ELP
?
YE
SN
O
![Page 24: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/24.jpg)
Eureka Presentation
-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.
TAKEAWAYS
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION IS THE TEXT ENOUGH?
YES NO
DO
ES T
HE IM
AG
E H
ELP
?
YES
NO
TEXT-ONLY
![Page 25: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/25.jpg)
Eureka Presentation
-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.
TAKEAWAYS
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION
It was a beautiful spring day today! I almost went out outside in shorts it was so nice! 😭 😣 😭 #spring #sarnia #winter #allgonein24hours
![Page 26: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/26.jpg)
Eureka Presentation
-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.
-Text+Image: multimodality is key
TAKEAWAYS
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION IS THE TEXT ENOUGH?
YES NO
DO
ES T
HE IM
AG
E H
ELP
?
YES
NO
TEXT+IMAGE
![Page 27: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/27.jpg)
Eureka Presentation
-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.
-Text+Image: multimodality is key
TAKEAWAYS
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION
Seriously cute cat just wandered into my garden, sweet little thing 😍 #cat #photogenic #cute #garden
![Page 28: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/28.jpg)
Eureka Presentation
-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.
-Text+Image: multimodality is key
TAKEAWAYS
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION
So happy I brought the nice weather back with me...
![Page 29: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/29.jpg)
Eureka Presentation
-Text-Only: sometime even if the textual component is enough to detect the sarcastic tone, the image has an important role in terms of explainability, interpretability and engagement.
-Text+Image: multimodality is key -Not Sarcastic: #sarcasm is not always
sufficient to mark the content as sarcastic, users have often their own definition of sarcasm that is close to humour, fun, silly content.
TAKEAWAYS
-100 posts per platform -Two questions:
A. Is the text enough? B. Does the image help?
MANUAL ANNOTATION IS THE TEXT ENOUGH?
YES NO
DO
ES T
HE IM
AG
E H
ELP
?
YE
SNO NOT SARCASTIC
![Page 30: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/30.jpg)
Eureka Presentation
COLLECT A GROUND TRUTH FOR SARCASMA. Evaluate the impact of visuals as a source for context B. Identify sarcastic posts with a high level of agreement
CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components
1
2
![Page 31: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/31.jpg)
Eureka Presentation
ASK THE CROWD!
1K POSTS
5 JUDGEMENTS
![Page 32: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/32.jpg)
Eureka Presentation
SECOND EXPERIMENT
For all the posts that are judged not sarcastic in the previous step, show the text and the image
FIRST EXPERIMENT
Show only the textual component of a post
![Page 33: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/33.jpg)
Eureka Presentation
Text+Image 37,4%
Text Only 37,8%
Not Sarcastic 24,8%
Text+Image 44,5%
Text Only 23,6%
Not Sarcastic 31,9%
\
![Page 34: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/34.jpg)
Eureka Presentation
COLLECT A GROUND TRUTH FOR SARCASMA. Evaluate the impact of visuals as a source for context B. Identify sarcastic posts with a high level of agreement
DETECT SARCASMSVM Fusion+Deep learning fusion approaches
CHARACTERISE THE ROLE OF IMAGESStudy of the interplay between textual and visual components
1
2
3
![Page 35: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/35.jpg)
HOW CAN WE DETECT SARCASM IN MULTIMODAL POSTS?
1 SVM
![Page 36: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/36.jpg)
Eureka Presentation
-LEXICAL -SUBJECTIVITY -1,2-GRAMS -WORD2VEC -COMBINATION
NLP FEATURES VISUAL SEMANTIC FEATURES
-YFCC100M DATASET -1,570 CONCEPTS VIA CONVOLUTIONAL NEURAL
NETWORK -EACH CONCEPT AS A ONE-HOT FEATURE
![Page 37: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/37.jpg)
Eureka Presentation
-LEXICAL -SUBJECTIVITY -1,2-GRAMS -WORD2VEC -COMBINATION
NLP FEATURES VISUAL SEMANTIC FEATURES
-YFCC100M DATASET -1,570 CONCEPTS VIA CONVOLUTIONAL NEURAL
NETWORK -EACH CONCEPT AS A ONE-HOT FEATURE
+
LINEAR SVM
+FEATURES VECTOR
FUSION
![Page 38: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/38.jpg)
HOW CAN WE DETECT SARCASM IN MULTIMODAL POSTS?
2 DEEP
LEARNING
![Page 39: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/39.jpg)
Eureka Presentation
1 K. Chateld, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: delving deep into convolutional nets. In BMVC, 2014.
Adapted Visual Representation1 (trained on ImageNet)
NLP Multilayer Perceptron (based on unigrams)
CO
NC
ATE
NA
TIO
N
LAY
ER
NO
N-L
INE
AR
LA
YE
RS
SARCASM DETECTION
![Page 40: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/40.jpg)
Eureka Presentation
EVALUATIONGOLD SET
2K EXAMPLES
![Page 41: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/41.jpg)
Eureka Presentation
D-50 D-80 D-100
baseline (1,2-grams) 81.7 82.5 80.2
baseline + VSF +6% +6.3% +4.3%
D-50 D-80 D-100
baseline (1,2-grams) 88.8 86.0 84.4
baseline + VSF -0,04% +2.1% +6.2%
SVM
1
![Page 42: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/42.jpg)
Eureka Presentation
D-50 D-80 D-100
baseline 77 74.6 74.8
baseline + AVR +1% +5.1% +3.7%
D-50 D-80 D-100
baseline 75.8 74.6 75.5
baseline + AVR +2.4% +1.4% -1%
DEEP LEARNING
2
![Page 43: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/43.jpg)
Eureka Presentation
Future Directions
SVM: IMPROVE THE FUSION METHOD, ADD SEMANTICS
DEEP LEARNING APPROACH: A LOT TO DO!
VISUAL SENTIMENT CONCEPTS
AUTOMATIC GENERATION OF SARCASTIC IMAGE CAPTION
SARCASTIC CONVERSATIONAL BOTS
![Page 44: Detecting Sarcasm in Multimodal Social Platforms](https://reader030.vdocuments.net/reader030/viewer/2022020301/58a5b2671a28ab1a628b66a1/html5/thumbnails/44.jpg)
Eureka Presentation
Questions?
@rschifan
http://www.di.unito.it/~schifane
THANKS FOR THE VERY INTERESTING TALK!