big data use cases in the cloud

Big Data Use Cases in the cloudPeter Sirota, GM Elastic MapReduce @petersirota

What is Big Data?

Computer generated data Application server logs (web sites, games) Sensor data (weather, water, smart grids) Images/videos (traffic, security cameras)

Human generated data Twitter “Firehose” (50 mil tweets/day 1,400% growth

per year) Blogs/Reviews/Emails/Pictures

Social graphs Facebook, linked-in, contacts

Big Data is full of valuable, unanswered questions!

Why is Big Data Hard (and Getting Harder)?

Data Volume Unconstrained growth Current systems don’t scale



Data Structure Need to consolidate data from multiple data sources

in multiple formats across multiple businesses


Changing Data Requirements Faster response time of fresher data Sampling is not good enough and history is important Increasing complexity of analytics Users demand inexpensive experimentation

We need tools built specifically for Big Data!

Innovation #1:

Apache HadoopThe MapReduce computational paradigm Open source, scalable, fault‐tolerant, distributed system

Hadoop lowers the cost of developing a distributed system for data processing

Innovation #2:

Amazon Elastic Compute Cloud (EC2) “provides resizable compute capacity in the cloud.”

Amazon EC2 lowers the cost of operating a distributed system for data processing

Amazon Elastic MapReduce =

Amazon EC2 + Hadoop

Elastic MapReduce applications

Targeted advertising / Clickstream analysisSecurity: anti-virus, fraud detection, image recognitionPattern matching / Recommendations Data warehousing / BIBio-informatics (Genome analysis) Financial simulation (Monte Carlo simulation)File processing (resize jpegs, video encoding)Web indexing

Clickstream Analysis –

Big Box Retailer came to Razorfish 3.5 billion records 71 million unique cookies 1.7 million targeted ads required per day

Problem: Improve Return on Ad Spend (ROAS)


Targeted Ad

User recently purchased a sports movie and is searching for video games (1.7 Million per day)


Lots of experimentation but final design: 100 node on-demand Elastic MapReduce cluster running Hadoop


Processing time dropped from 2+ days to 8 hours (with lots more data)


Increased Return On Ad Spend by 500%

World’s largest handmade marketplace 8.9 million items 1 billion page view per month $320MM 2010 GMS

• Easy to ‘backfill’ and run experiments just boot up a cluster with 100, 500, or 1000 nodes

Production DB snapshots

Production DB snapshots

Web event logs

Web event logs ETL – Step

1ETL – Step

1ETL – Step

2ETL – Step

2

JobJob

JobJob

JobJob

Recommendations

The Taste Test http://www.etsy.com/tastetest

http://www.etsy.com/tastetest

Recommendations

etsy.com/gifts

Gift Ideas for Facebook Friends

•

• Yelp generates close to 400GB of logs per day

Yelp

• Yelp does not have a physical MapReduce cluster

• Running 250 production clusters per week

• All of those run on Elastic MapReduce

MapReduce at Yelp

Features driven by MapReduce

• Analyze ad stats (reporting, billing, algorithm inputs)

• Analyze A/B test results

• Detect duplicate business listings

• Email bounce processing

• Identify bots based on traffic patterns

More MapReduce uses

9/23/2011 Amazon EMR Strata Justin Moore - @injust

Big Data @ foursquare


How do we use EMR?

• Map-Reduce– Run algorithms on our entire dataset– Streaming jobs, complex analyses

• Hive– Business intelligence– Exploratory analyses– Infographics!


How big is our data?

• Global reach (North Pole, Space)• Native app for almost every smartphone, SMS,

web, mobile-web• 10M+ users, 15M+ venues, ~1B check-ins• Terabytes of log data


Our Stack


Computing venue-to-venue similarity

• Spin up 40 node cluster• Submit Ruby streaming job

– Invert User x Venue matrix– Grab Co-occurrences– Compute similarity

• Spin down cluster• Load data to app server


Who is checking in?


What are people doing?


Where are our users?


When do people go to a place?

Thursday Friday Saturday Sunday


Why are people checking in?

• Explore their city, discover new places• Find friends, meet up• Save with local deals• Get insider tips on venues• Personal analytics, diary• Follow brands and celebrities• Earn points, badges, gamification of life• The list grows…


How can we leverage these insights?


Join us!

foursquare is hiringwww.foursquare.com/jobs

Justin Moore@injust

[email protected]

http://www.foursquare.com/jobs

http://aws.amazon.com/elasticmapreduce/

big data use cases in the cloud

Documents