enquiring minds: early detection of rumors in social media from enquiry posts zhe zhaopaul...
TRANSCRIPT
Enquiring Minds: Early Detection of Rumors in Social Media from Enquiry Posts
Zhe Zhao Paul Resnick Qiaozhu Mei
PRESENTATION GROUP 2
Outline
Introduction Background Study Approach For Detection Experimental Setup Evaluation Conclusion
Rumor is a controversial, fact-checkable statement
Malaysia airline MH370 is missing
Malaysia airline MH370 crashed
Rumor is a controversial, fact-checkable statement
Malaysia airline MH370 is missing
Malaysia airline MH370 crashed
Recreational Marijuana should be made legal
Recreational Marijuana becomes legal in Michigan
Introduction
It is very difficult to claim that every post on social media is a factual claim
The broad success of online social media has created fertile soil for the emergence and fast spread of rumors.
This paper proposes an automated tool to identify potential Rumors
Spread of Rumor
Oh my god is this real? RT @AP: Breaking: Two Explosions in the White House and Barack Obama is injured
Is this true? Or hacked account? RT @AP Breaking: Two Explosions in the White House and Barack Obama is injured
Is this real or hacked? RT @AP: Breaking: Two Explosions in the White House and Barack Obama is injured
Is this legit? RT @AP Breaking: Two Explosions in the White House and Barack Obama is injured
Detecting Rumor
Rumors are basically judge on the key phrases it has – “Is this true?”
“Really?”
“What?
The paper proposes algorithm for identifying newly emerging, controversial topics that is scalable to massive stream of tweets i.e. signal tweets
Then it identifies a set of regular expressions that define the set of signal tweets
Related Work
Detection Problems in Social Media!
The work on detecting rumor has started in recent years.
Sharing/ Retweeting / Trending determines it’s a rumor or not.
Question Asking in Social Media
Another detection feature used in related work is question asking. Mendoza et al. found on a small set of cases that false tweets were questioned much more than confirmed truths.
Detection using question mark!
Previous work has shown that only one third of tweets with question marks are real questions, and not all questions are related to rumors.
Problem Statement
Rumor Cluster We define a rumor cluster R as a group of social media posts that are either declaring, questioning, or denying the same fact claim, s, which may be true or false. Let S be the set of posts declaring s, E be the set of posts questioning s, and C be the set of tweets denying s, then R = S ∪ E ∪ C. We say s is a candidate rumor if S ≠ ∅ and E ∪ C ≠ ∅.
The paper’s objective is to minimize the delay from the time when the first tweet about the rumor is posted to the detection time.
RUMOR
Fact Checka
ble
Controversial
Detection of rumors
1) Identify Signal Tweets
2) Identify Signal Clusters
3) Detect Statements
4) Capture Non-signal Tweets
5) Rank Candidate Rumor Cluster
Identify Signal Rumor
If we want to detect rumors, the first thing we should know is what rumors look like.
Author defines rumors as a verification of a piece of factual knowledge, i.e. “According to the Mayan Calendar, does the world end on Dec 16th, 2013?”.
Or as corrections (debunks) of a question. i.e. “This new is true!”
What we need is more than theory
Using Porter Stemmer and Chi-Squared algorithm on 10417 tweets, with 3423 tweets labeled as verification or correction, and we draw the pattern of good signals.
Identify Signal Clusters
What is Signal Cluster?
After a rumor tweet emerges, people might follow, i.e. retweet it or come up with a new one containing similar information, thus forming a group or cluster.
What? An eight year girl died at Boston marathon explosion.
Is it true? Two explosions in the White house and Barack Obama is injured!
The shocking new is tested be to wrong!
How do we do it?
Use connected component clustering algorithm, Jaccard Similarity algorithm and Minhash algorithm to achieve it.
What??!! Two Explosions in the White House and Barack Obama is Injured in head.Is it true?? Two Explosions in the White House and Barack Obama is Injured on arm.Really?? @AP: Two Explosions in the White House and Barack Obama is not Injured.
Detect Statement
Right now what we get is a few clusters of potential rumors, not sure about the content.
Our goal is the rumor content, not the pattern.
Which one to draw out?
A way out
Just pick out the statement that appears more often than 80% of other statements.
Why 80? Have higher probability to be a rumor!
What??!! Two Explosions in the White House and Barack Obama is InjuredIs it true?? Two Explosions in the White House and Barack Obama is InjuredReally?? @AP: Two Explosions in the White House and Barack Obama is Injured
Compare Non-Signal Tweets
Remember when we detect rumor clusters, using signals.
Tweets not belong to verification or correction, but also can bear rumor information.
Match those statements with non-signal tweets.
Also using Jaccard similarity. If the score > 0.6, we can say they matched.
Rank candidate rumor clusters
Till now, in network, we have got several rumor clusters.
Each cluster stands for one rumor statement.
But output should be one, the most potential rumor.
Popularity? NO! i.e. funny picture or touching
video.
Ranking rumor clusterPercentage of signal tweets
Entropy ratio
Tweet lengths
Retweets
URLs
Hashtags
@ Mentions
Data Sets
BOSTON MARATHON BOMBING (high-profile event) Two bombs exploded at the finish line of the annual Boston Marathon competition on April 15th, 2013 which contains 30,340,218 unique tweets.
GARDENHOSE (random sample) Collected a tweet stream in a random month of the year 2013 (November 1 to November 30, 2013) which contains 1,242,186,946 tweets.
Baselines and Variants of Methods
1. Trending Topics
2. Hash tag Tracking
3. Corrections Only
4. Enquiries and Corrections
Rank candidate rumors purely by popularity, the number of tweets in the cluster.(identify signal tweets)
5. SVM ranking
6. Decision tree ranking
Use both enquiry and correction tweets as signals.(rank the candidate rumor clusters)
Effectiveness of Enquiry Signals
Precision of Candidate Rumor Clusters
Precision of rumor detection using different signals. Candidate rumors ranked by popularity only. Maximum number of output rumor clusters: 10 per hour for BOSTON and 50 per day for GARDENHOSE.
Effectiveness of Enquiry Signals
Earliness of Detection
Earliness of detection comparing to Enquiries+ Corrections: enquiry signals help to detect rumors hours
Ranking Candidate Rumor Clusters
@N is the percentage of real rumors among the top N candidate rumor clusters output by the a method.
Precision@N of different ranking methods
Effectiveness of Enquiry Signals
In order to verify that the ranking algorithm is not overfitting only one data set, We also applied the decision tree trained using 7 days of labeled results in GARDENHOSE data set to rank rumor clusters detected hourly from BOSTON data set.
Precision@N if rumor clusters are ranked by the Decision Tree. One third of top 50clusters are real rumors.
Efficiency of Framework
Filtering of tweets
Clustering
Potential rumor statements
The cost is significantly reduced as compared to approach which first generates trending topics and then identify rumors.
Time Comparison
Trending Topics:
Clustering
Hashtag Tracking:
Filtering & Clustering
This Method:
Filtering, Clustering then
retrieving back Same clustering and ranking implementation was used except filtering tweets with enquiry and tweets were not retrieved back after clustering.
Conclusion
Method which capitalizes on verification questions which also appear sooner facilitating early detection.
Cluster only those tweets that contain enquiry patterns, extract the statements and use them to pull back in the rest of the non-signal tweets.
Robust even with tweets exceeding 100 million.
Future work-
• Signal labelled by humans to have iterative improvements
• Improving the filtering of enquiry and correction signal by training a classifier