big data and nosql in real time

Big Data and NoSQL in REAL TIMEFacebook and Twitter Examples

Ron Zavner

2

Agenda

Our real time world… Flavors of Big Data Facebook messaging and real time analytics system Twitter analytics system Winning architecture?

® Copyright 2011 Gigaspaces Ltd. All Rights Reserved

What is Real Time?

3® Copyright 2011 Gigaspaces Ltd. All Rights Reserved

We’re Living in a Real Time World…Homeland Security

Real Time Search

Social

eCommerce

User Tracking & Engagement

Financial Services

® Copyright 2011 Gigaspaces Ltd. All Rights Reserved4

Big Data Predictions

“Over the next few years we'll see the adoption of scalable frameworks and platforms for handling streaming, or near real-time, analysis and processing. In the same way that Hadoop has been borne out of large-scale web applications, these platforms will be driven by the needs of large-scale location-aware mobile, social and sensor use.”

Edd Dumbill, O’REILLY

5® Copyright 2011 Gigaspaces Ltd. All Rights Reserved


The Two Vs of Big Data

Velocity Volume

The Flavors of Big Data Analytics

Counting Correlating Research


Analytics – Counting

How many signups, tweets, retweets for a topic?

What’s the average latency?

Demographics Countries and cities Gender Age groups Device types …


Analytics – Correlating

What devices fail at the same time?

What features get user hooked?

What places on the globe are “happening”?


Analytics – Research

Sentiment analysis “Obama is popular”

Trends “People like to tweet

after watching American Idol”

Spam patterns How can you tell when

a user spams?


It’s All about Timing

• Event driven / stream processing • High resolution – every tweet gets counted

• Ad-hoc querying • Medium resolution

• Long running batch jobs (ETL, map/reduce) • Low resolution (trends & patterns)


This is what we’re here to discuss

FACEBOOK REAL-TIMEANALYTICS SYSTEM

12

13

Store 135+ Billion Messages A Month


14

The actual analytics.. Like button analytics

Comments box analytics


15

Goals

Show why plugins are valuable Make the data more actionable Make the data more timely Remove point of failures Handle massive load - 200K events per second


16

Technology Evaluation

MySQL DB Counters In-Memory Counters MapReduce Cassandra HBase


PTail

Scribe

Puma

HbaseFACEBOOK

Log

FACEBOOK

Log

FACEBOOK

Log

HDFS

Real Time Long Term

Batch1.5 Sec

The solution..10,000 write/sec per server

Keep Things In Memory

Facebook keeps 80% of its data in Memory (Stanford research)

RAM is 100-1000x faster than Disk (Random seek)• Disk: 5 -10ms • RAM: ~0.001msec

TWITTER REAL-TIMEANALYTICS SYSTEM

19


Twitter Reach – Here’s One Use Case


Let’s start with some statistics ….

Twitter in Numbers (March 2011)

Source: http://blog.twitter.com/2011/03/numbers.html


It takes a week for users to

send 1 billion Tweets.




On average,

140 million tweets get sent every day.




The highest throughput to date is

6,939 tweets/sec.




460,000 new accounts

are created daily.



26

5% of the users generate

75% of the content.


Twitter in Numbers

Source: http://www.sysomos.com/insidetwitter/

Challenge – Word Count

Word:Count

Tweets

Count?® Copyright 2011 Gigaspaces Ltd. All Rights Reserved27

• Hottest topics• URL mentions• etc.

(Tens of) thousands of tweets per second to process Assumption: Need to process in near real time

Aggregate counters for each word A few 10s of thousands of words (or hundreds of

thousands if we include URLs) System needs to linearly scale System needs to be fault tolerant

Word Count - Analyze the Problem



Use EDA (Event Driven Architecture)

TokenizerRaw FiltererTokenized CounterFiltered

Sharding (Partitioning)

Tokenizer1 Filterer 1

Tokenizer2 Filterer 2

Tokenizer 3 Filterer 3

Tokenizer n Filterer n

Counter Updater 1

Counter Updater 2

Counter Updater 3

Counter Updater n


Computing Reach with Event Streams

Twitter Storm



Storm Overview


Storm Cluster


Streaming word count with Storm


Storage Data Persistency Querying

Storm LimitationSpouts

Bolt

Topologies


Event driven / flow Reliable Storage Data Persistency Querying

Winner is… storm & in memory data grids


Facebook messages http://

highscalability.com/blog/2010/11/16/facebooks-new-real-time-messaging-system-hbase-to-store-135.html

Facebook Real time analytics http://

highscalability.com/blog/2011/3/22/facebooks-new-realtime-analytics-system-hbase-to-process-20.html

Learn and fork the code on github: https://github.com/Gigaspaces/rt-analytics

Detailed blog posthttp://bit.ly/gs-bigdata-analytics

Twitter in numbers: http://blog.twitter.com/2011/03/numbers.html

Twitter Storm: http://bit.ly/twitter-storm

References

http://highscalability.com/blog/2010/11/16/facebooks-new-real-time-messaging-system-hbase-to-store-135.html



http://highscalability.com/blog/2011/3/22/facebooks-new-realtime-analytics-system-hbase-to-process-20.html



https://github.com/Gigaspaces/rt-analytics

https://github.com/Gigaspaces/rt-analytics

http://bit.ly/gs-bigdata-analytics

http://bit.ly/gs-bigdata-analytics

http://blog.twitter.com/2011/03/numbers.html

http://blog.twitter.com/2011/03/numbers.html

http://bit.ly/twitter-storm

[email protected]


Q&A