20100806 cloudera 10 hadoopable problems webinar

10 Common Hadoop-able Problems

August 5, 2010

Topics

• Introduction

• 10 Common Hadoop-able Problems

• Summary

• Questions

Copyright 2010 Cloudera Inc. All rights reserved 2

Today’s speaker - Jeff Hammerbacher

• [email protected]

• Studied Mathematics at Harvard

• Worked as a Quant on Wall Street

• Conceived, built, and led Data team at Facebook

• Nearly 30 amazing engineers and data scientists

• Several open source projects and research papers

• Founder of Cloudera

• Chief Scientist

• Also, check out the book “Beautiful Data”


What is Hadoop?

• A scalable fault-tolerant distributed system for data storage and processing (open source under the Apache license)

• Scalable data processing engine• Hadoop Distributed File System (HDFS): self-healing high-bandwidth

clustered storage• MapReduce: fault-tolerant distributed processing

• Key value• Flexible -> store data without a schema and add it later as needed• Affordable -> cost / TB at a fraction of traditional options• Broadly adopted -> a large and active ecosystem• Proven at scale -> dozens of petabyte + implementations in

production today

Copyright 2010 Cloudera Inc. All Rights Reserved. 4

Cloudera’s Distribution for Hadoop, version 3

• Open source – 100% Apache licensed

• Simplified – Component versions & dependencies managed for you

• Integrated – All components & functions interoperate through standard API’s

• Reliable – Patched with fixes from future releases to improve stability

• Supported – Employs project founders and committers for >70% of components


Hue Hue SDK

OozieOozie

HBaseFlume, Sqoop

Zookeeper

Hive

Pig/Hive

The industry’s leading Hadoop distribution

How does Cloudera know which problems are Hadoop-able?

• Talking to 1000s of users

• Supporting 100s of implementations

• Experience putting Hadoop into production with customers across a range of industries


Summary – 10 Common Hadoop-able Problems

1. Modeling true risk

2. Customer churn analysis

3. Recommendation engine

4. Ad targeting

5. PoS transaction analysis

6. Analyzing network data to predict failure

7. Threat analysis

8. Trade surveillance

9. Search quality

10. Data “sandbox”


What is common across Hadoop-able problems?

Nature of the data

• Complex data

• Multiple data sources

• Lots of it

Nature of the analysis


• Batch processing

• Parallel execution

• Spread data over a cluster of servers and take the computation to the data

What Analysis is Possible With Hadoop?

• Text mining

• Index building

• Graph creation and analysis

• Pattern recognition

• Collaborative filtering

• Prediction models

• Sentiment analysis

• Risk assessment


Benefits of Analyzing With Hadoop


• Previously impossible/impractical to do this analysis

• Analysis conducted at lower cost

• Analysis conducted in less time

• Greater flexibility

Topics

• Introduction


• Summary

• Questions


1. Modeling True Risk


1. Modeling True Risk

Solution with Hadoop


• Source, parse and aggregate disparate data sources to build comprehensive data picture

• e.g. credit card records, call recordings, chat sessions, emails, banking activity

• Structure and analyze

• Sentiment analysis, graph creation, pattern recognition

Typical Industry

• Financial Services (Banks, Insurance)

2. Customer Churn Analysis


2. Customer Churn Analysis



• Rapidly test and build behavioral model of customer from disparate data sources

• Structure and analyze with Hadoop

• Traversing

• Graph creation


Typical Industry

• Telecommunications, Financial Services

3. Recommendation Engine


3. Recommendation Engine



• Batch processing framework

• Allow execution in in parallel over large datasets

• Collaborative filtering

• Collecting ‘taste’ information from many users

• Utilizing information to predict what similar users like

Typical Industry

• Ecommerce, Manufacturing, Retail

4. Ad Targeting


4. Ad Targeting



• Data analysis can be conducted in parallel, reducing processing times from days to hours

• With Hadoop, as data volumes grow the only expansion cost is hardware

• Add more nodes without a degradation in performance

Typical Industry

• Advertising

5. Point of Sale Transaction Analysis


5. Point of Sale Transaction Analysis






• Optimizing over multiple data sources

• Utilizing information to predict demand

Typical Industry

• Retail

6. Analyzing Network Data to Predict Failure


6. Analyzing Network Data to Predict Failure



• Take the computation to the data

• Expand the range of indexing techniques from simple scans to more complex data mining

• Better understand how the network reacts to fluctuations

• How previously thought discrete anomalies may, in fact, be interconnected

• Identify leading indicators of component failure

Typical Industry

• Utilities, Telecommunications, Data Centers

7. Threat Analysis


7. Threat Analysis



• Parallel processing over huge datasets

• Pattern recognition to identify anomalies i.e. threats

Typical Industry

• Security

• Financial Services

• General: spam fighting, click fraud

8. Trade Surveillance


8. Trade Surveillance






• Detect trading anomalies and harmful behavior

Typical Industry

• Financial services

• Regulatory bodies

9. Search Quality


9. Search Quality



• Analyzing search attempts in conjunction with structured data


• Browsing pattern of users performing searches in different categories

Typical Industry

• Web

• Ecommerce

10. Data “Sandbox”


10. Data “Sandbox”



• With Hadoop an organization can “dump” all this data into a HDFS cluster

• Then use Hadoop to start trying out different analysis on the data

• See patterns or relationships that allow the organization to derive additional value from data

Typical Industry

• Common across all industries

Topics

• Introduction


• Summary

• Questions


Summary – 10 Common Hadoop-able Problems

1. Modeling true risk

2. Customer churn analysis

3. Recommendation engine

4. Ad targeting

5. PoS transaction analysis

6. Threat analysis

7. Analyzing network data to predict failure

8. Trade surveillance

9. Search quality

10. Data “sandbox”


Who is Cloudera?

• Enterprise software & services company providing the industry’s leading Hadoop-based data management platform• Founding team came from large Web companies

• Products: Cloudera Enterprise & Cloudera’s Distribution for Hadoop• All necessary packages, matched, tested and supported

• Tools to support production use of Hadoop

• The leading distribution for the enterprise

• Contributors and committers• Fixing, patching and adding features

34

Hear More Examples @ Hadoop World 2010

• 2nd annual event focused on practical applications of Hadoop

• Date: October 12th 2010

• Location: Hilton New York

• Keynote from Tim O’Reilly – founder O’Reilly Media

• Pre and post conference training available for Hadoop and related projects

• 36 business and technical focused sessions


Confirmed speakers from

http://www.cloudera.com/company/press-center/hadoop-world-nyc/

Questions?


20100806 cloudera 10 hadoopable problems webinar

Documents

data storage

analysis analysis

data scientists

data team

data sandboxcopyright

network data

common hadoopable problems1

datawhat analysis