losing my revolution long paper tpdl2012
TRANSCRIPT
![Page 1: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/1.jpg)
Losing My Revolution:
Old Dominion UniversityDepartment of Computer Science
Hany SalahEldeen & Michael Nelson Losing My Revolution
How Many Resources Shared on Social Media Have Been Lost?
Hany M. SalahEldeen & Michael L. Nelson
![Page 2: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/2.jpg)
All tweets are equal…
Hany SalahEldeen & Michael Nelson Losing My Revolution 2
…but some are more equal than the others
![Page 3: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/3.jpg)
Research Questions:
Hany SalahEldeen & Michael Nelson Losing My Revolution 3
• How long would this last?• And if lost, is there a backup somewhere?• Finally, can we model this existence?
![Page 4: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/4.jpg)
Phase 1:Data Gathering
Hany SalahEldeen & Michael Nelson Losing My Revolution 4
![Page 5: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/5.jpg)
We decided to collect as many posts on social media as possible satisfying these conditions:• Has embedded resources.• Has a time stamp.• From different sources.• Related to socially significant events.
Data Gathering
Hany SalahEldeen & Michael Nelson Losing My Revolution 5
![Page 6: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/6.jpg)
• From Twitter, Websites, Books:• The Egyptian revolution.
• From Twitter Only:• Stanford’s SNAP dataset:• Iranian elections.• H1N1 virus outbreak.• Michael Jackson’s death.• Obama’s Nobel Peace Prize.
• Twitter API:• The Syrian uprising.
Six Socially Significant Events
Hany SalahEldeen & Michael Nelson Losing My Revolution 6
![Page 7: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/7.jpg)
Preparation:Stanford’s SNAP Dataset
Hany SalahEldeen & Michael Nelson Losing My Revolution 7
Extracted tweets in English only.
Contain hash tagsContain embedded resources
![Page 8: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/8.jpg)
• With start with initial tags manually assigned related to the event and extract co-occurring ones
Twitter Tag Expansion
Hany SalahEldeen & Michael Nelson Losing My Revolution 8
Event Initial Hashtags Top Co-occurring Hashtags
H1N1 Outbreak #h1n1 = 61,351
#swine = 61,829#swineflu = 56,419
#flu = 8,436#pandemic = 6,839#influenza = 1,725
#grippe = 1,559#tamiflu = 331
#cnn = …….#health = …….
![Page 9: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/9.jpg)
• We repeat this with all the other 3 events from SNAP
Twitter Tag Expansion
Hany SalahEldeen & Michael Nelson Losing My Revolution 9
![Page 10: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/10.jpg)
• Using the expanded tags we sort them according to number of tweets and filter them by co-occurrence.
Tweet Filtration
Hany SalahEldeen & Michael Nelson Losing My Revolution 10
Event Hashtags selected for filteration Tweets Extracted
H1N1 Outbreak
#h1n1 = 61,351
#h1n1 & #swine = 44,972
#h1n1 & #swine & #swineflu = 42,574
#h1n1 & #swine & #swineflu & #pandemic = 5,517
Final Dataset Size = 5,517
![Page 11: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/11.jpg)
• We repeat this for all the other 3 events.• We might need further random sampling to reduce the size of
the dataset
Tweet Filtration
Hany SalahEldeen & Michael Nelson Losing My Revolution 11
![Page 12: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/12.jpg)
• The social media played a key role in documenting and driving the revolution.
• Millions of tweets, Facebook posts, videos, and images have been shared during the 18 days of the 25th January 2011 revolution.
• We manually extracted all the resources we can from the period of 20th January till March 1st.
• Hard to extract.
Egyptian Revolution Dataset
Hany SalahEldeen & Michael Nelson Losing My Revolution 12
![Page 13: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/13.jpg)
Sources Utilized
Hany SalahEldeen & Michael Nelson Losing My Revolution 13
Tweets From Tahrir
IAmJan25.com
Storify.com
![Page 14: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/14.jpg)
• Since this event was a current event we utilized the Twitter search API in the extraction process.
• Similar to the SNAP dataset, we applied hashtag expansion and filtration.
Syrian Uprising Dataset
Hany SalahEldeen & Michael Nelson Losing My Revolution 14
Initial Hashtags Top Co-occurring Hashtags
#syria
#bashar#risedamascus
#genocideinsyria#stopassad2012
#assadcrimes#assad
![Page 15: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/15.jpg)
What are people sharing?
Hany SalahEldeen & Michael Nelson Losing My Revolution 15
![Page 16: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/16.jpg)
For all the collected data, how many URIs are:1. unique and how many are repeated? 2. still active on the live web and how
many died?3. archived in one of the public web
archives?
Data Analysis
Hany SalahEldeen & Michael Nelson Losing My Revolution 16
![Page 17: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/17.jpg)
Phase 2:Uniqueness and Existence
Hany SalahEldeen & Michael Nelson Losing My Revolution 17
![Page 18: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/18.jpg)
UniquenessA URL can take many different forms utilizing numerous URL shortners
http://www.cnn.com
Could be:
http://bit.ly/2EEjBlhttp://goo.gl/2ViC
Hany SalahEldeen & Michael Nelson Losing My Revolution 18
% curl -I http://goo.gl/2ViCHTTP/1.1 301 Moved PermanentlyContent-Type: text/html; charset=UTF-8Cache-Control: no-cache, no-store, max-age=0, must-revalidatePragma: no-cacheExpires: Fri, 01 Jan 1990 00:00:00 GMTDate: Tue, 18 Sep 2012 01:08:44 GMTLocation: http://www.cnn.com/Server: GSETransfer-Encoding: chunked
![Page 19: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/19.jpg)
Uniqueness
• Thus, we resolve all the URLs extracted and keep the final destination URL after redirects (30X redirects).
• Then we extract all the unique URLs and remove redundancies.
Hany SalahEldeen & Michael Nelson Losing My Revolution 19
![Page 20: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/20.jpg)
Uniqueness
Hany SalahEldeen & Michael Nelson Losing My Revolution 20
Collection All Resources Unique Resources
5,517H1N1 Outbreak 1,645
2,293Michael Jackson 1,187
3,429Iran 1,340
1,118Obama 370
7,313Egypt 6,154
1,955Syria 355
![Page 21: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/21.jpg)
Existence on the live-web
• For each unique URL we resolved the final HTTP response and considered 2 classes:• Success: 200 OK• Failure: 4XX, 50X families and the 30X loop
redirects or soft 404s.
Hany SalahEldeen & Michael Nelson Losing My Revolution 21
![Page 22: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/22.jpg)
Existence on the live-web
Hany SalahEldeen & Michael Nelson Losing My Revolution 22
Collection Resources Missing Percentage Missing
394H1N1 Outbreak 23.95%
397Michael Jackson 33.45%
339Iran 25.30%
92Obama 24.86%
645Egypt 10.48%
25Syria 7.04%
![Page 23: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/23.jpg)
Existence in Public Web-Archives
• For each unique URL we downloaded its timemap utilizing Memento.
• The aggregator checks 10+ public web archives for the existence of snapshots.
• The resource is declared to be archived if it has at least one Memento.
Hany SalahEldeen & Michael Nelson Losing My Revolution 23
![Page 24: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/24.jpg)
Existence in Public Web-Archives
Hany SalahEldeen & Michael Nelson Losing My Revolution 24
Collection Resources Archived Percentage Archived
693H1N1 Outbreak 42.12%
406Michael Jackson 34.20%
516Iran 38.51%
176Obama 47.57%
1242Egypt 20.18%
19Syria 5.35%
![Page 25: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/25.jpg)
Phase 3:Existence as a Function of Time
Hany SalahEldeen & Michael Nelson Losing My Revolution 25
![Page 26: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/26.jpg)
Timeline of Events
Hany SalahEldeen & Michael Nelson Losing My Revolution 26
List of events
Social Events Having a Bimodal Time Distribution
![Page 27: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/27.jpg)
Resources Missing & Archived
Hany SalahEldeen & Michael Nelson Losing My Revolution 27
Collection Percentage Missing Percentage Archived
23.49%H1N1 Outbreak 41.65%
36.24%Michael Jackson 39.45%
26.98%Iran 43.08%
24.59%Obama 47.87%
10.48%Egypt 20.18%
7.04%Syria 5.35%
31.62% 30.78%
24.47% 36.26%
25.64% 43.87%
26.15% 46.15%
![Page 28: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/28.jpg)
Resources Missing & Archived
Hany SalahEldeen & Michael Nelson Losing My Revolution 28
![Page 29: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/29.jpg)
Curve Fitting The Data
Hany SalahEldeen & Michael Nelson Losing My Revolution 29
![Page 30: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/30.jpg)
Conclusions
• Measured 21,625 resources from 6 data sets in archives & live web.
• After a year from publishing about 11% of content shared on social media will be gone.
• After this we are losing roughly 0.02% daily.
Hany SalahEldeen & Michael Nelson Losing My Revolution 30
![Page 31: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/31.jpg)
Appendix A:Extra Slides
Hany SalahEldeen & Michael Nelson Losing My Revolution
![Page 32: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/32.jpg)
Stanford’s SNAP Dataset:• Collection of about 50 large network datasets.• Twitter posts dataset comprises nearly ½ Billion
Tweet.• Posted from June 1st 2009 till December 31st
2009.• Nearly 17 million users.• Nearly 20-30% of the total posts published by
Twitter during this period.
Data Gathering
Hany SalahEldeen & Michael Nelson Losing My Revolution
![Page 33: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/33.jpg)
Existence as a function of timeDual-Peaked Events:• Iranian Elections:• 13th Jun. 2009: Protests and elections• 1st Aug. 2009: Trials
• Michael Jackson’s Death:• 25th Jun. 2009: Death announcement• 10th Jul. 2009: Death unnatural causes
• H1N1 Outbreak:• 11th Sept. 2009: Worldwide outbreak• 5th Oct. 2009: Vaccine release
• Obama’s Nobel Peace Prize:• 9th Oct. 2009: Prize announcement.• 10th Dec. 2009: Nobel Ceremony
Hany SalahEldeen & Michael Nelson Losing My Revolution
Back
![Page 34: Losing My Revolution Long Paper TPDL2012](https://reader035.vdocuments.net/reader035/viewer/2022062405/554e833ab4c90545698b542b/html5/thumbnails/34.jpg)
Future WorkIn the next steps we will:• expand the datasets.• cover the uncovered temporal areas in 2010 and
before 2009.• examine closely the extended points and tune the
function with time.• analyze the other factors like: publishing venue,
rate of sharing, popularity of authors, and the nature of the event.
Hany SalahEldeen & Michael Nelson Losing My Revolution