alphago vs lee se-dol: tweeter analysis using hadoop and spark
TRANSCRIPT
Jongwook Woo
HiPIC
CSULA
Alphago vs Lee Se-DolTweeter Analysis using Hadoop and
Spark
March 18 2016Jongwook Woo, PhD
High-Performance Information Computing Center (HiPIC)California State University Los Angeles
High Performance Information Computing CenterJongwook Woo
CSULA
Content
Hadoop and SparkIBM DashDBConclusion
High Performance Information Computing CenterJongwook Woo
CSULA
Hadoop and Spark Environment
SystemsAzure HDInsights Spark8 Nodes
–40 cores: 2.4GHz Intel Xeon–Memory - Each Node: 28 GB
Data SourceKeyword ‘alphago’ from Tweeter via Apache NiFi
Data Size 63,193 tweets
Real Time Data Collection period03/12 – 03/17/2016
–No data collected on 03/13
High Performance Information Computing CenterJongwook Woo
CSULA
Top 10 Countries that Tweets “Alphago”
High Performance Information Computing CenterJongwook Woo
CSULA
Top 10 Countries
# of Tweets per CountryUSA: > 11,000Japan: > 9,000Korea: > 1,900Russia, UK: > 1,600Thai Land, France : > 1,000 Netherland, Spain, Ukraine: > 600
High Performance Information Computing CenterJongwook Woo
CSULA
Top 10 Countries Sentiment
Positive Negative
High Performance Information Computing CenterJongwook Woo
CSULA
Top 10 Countries
Most Tweeted Countries All countries show more positive tweets
–Korea, Japan, USA
Country Positive Negative
USA 5070 3567
Japan 8118 217
…
Korea 1053 407
…
High Performance Information Computing CenterJongwook Woo
CSULA
Daily Tweets in 03/12 – 03/17/2016
3/12/2016 3/13/2016 3/14/2016 3/15/2016 3/16/2016 3/17/20160
5000
10000
15000
20000
25000
30000
35000
40000
45000
50000
Alphago vs Lee Sedol
Game 4: Mar 13 Lee Se-Dol win
Game 5: Mar 15
Game 3: Mar 12
High Performance Information Computing CenterJongwook Woo
CSULA
Ngram words
3 word in row right after Go-Champion “sedol” and “se-dol”
sedol
se-dol3-grams FrequencyAgain-to-win 1,187
Is-something-I’ll 369
Is-something-i 199
In-go-tournament 168
High Performance Information Computing CenterJongwook Woo
CSULA
Sentiment Map of Alphago
PositiveNegative
High Performance Information Computing CenterJongwook Woo
CSULA
Sentiment Map of Lee Se-Dol vs Alphago
YouTube video: the sentiment of the World https://youtu.be/vAzdnj4fkOg?list=PLaEg1tCLuW0BYLqVS5RTbToiB8wQ2w14a
High Performance Information Computing CenterJongwook Woo
CSULA
Tweeter Analysis using IBM DashDB
Environment:DashDB and Tweets Services of IBM Bluemix
–Load existing data Period: by March 16 2016
Authors and Followers of the Tweets
High Performance Information Computing CenterJongwook Woo
CSULA
Top 10 Tweet Countries
With Hashtag “#Alphago”
United States: >10,000Japan: >8,000Korea: >1,800
High Performance Information Computing CenterJongwook Woo
CSULA
Hashtags Frequency
High Performance Information Computing CenterJongwook Woo
CSULA
Sentiment at #Alphago
High Performance Information Computing CenterJongwook Woo
CSULA
Gender Counts Who Tweets
female male unknown
Unknown
High Performance Information Computing CenterJongwook Woo
CSULA
Tweets counts per months
Aug-2014 Feb-2015 Feb-2016 Jan-2015 Jan-2016 Mar-20160
2000
4000
6000
8000
10000
12000
Tweets counts per months
High Performance Information Computing CenterJongwook Woo
CSULA
Daily Tweets During Games
3/9/2016 3/10/2016 3/11/2016 3/12/2016 3/13/2016 3/14/2016 3/15/2016 3/16/20160
500
1000
1500
2000
2500
3000
3500
Daily Tweets during GamesGame 4: Mar 13 Lee Se-Dol win
Game 5: Mar 15 Game 3: Mar 12
Game 1: Mar 9
Game 2: Mar 10
High Performance Information Computing CenterJongwook Woo
CSULA
Conclusion
Analyze Tweeters with “Alphago” USA and Japan dominates the tweets
More than KoreaEuropean countries as well
More Positive tweetsAlphago and Lee Sedol both become popular