2011 - tdwi big data forum - the new analytics
DESCRIPTION
Presentation by Casey Kiernan at the 2011 TDWI Big Data Forum - Orlando, FL - Oct 30 / Nov 1 - 2011TRANSCRIPT
Hadoop&
The New Analytics
Casey Kiernan
Sr. Director / Data Architecture - Shopzilla.com
November 1, 2011
2
Agenda The New “Data”
The New Business Model
The New Analytic Scenarios
The New Analytic Architectures
The New Analytic Technologies
And, Yes… The New Data-Center
The World as I See it
SERVICES • SOA • JSON • AVRO • APPLICATIONS • HTML • JAVA • C# • THE CLOUD • HADOOP • OLAP
SQL SERVER • ORACLE • UNIX • SUBVERSION • COMPLIANCE • SECURITY • SALESFORCE • MYSQL
FIN
AN
CE
• ID
• C
UST
OM
ER •
EX
PER
IAN
PYTH
ON
• SITES • TENA
NT • O
RG
• SSN
My Mountain Bike
Guidance
PerformanceRate of ClimbCalories BurnedMiles ObtainedTotal ClimbedElapsed Time
Current, Average, Max Values
Data CollectionSpeed / Trip Miles
Data CollectionCadence / RPM
Data Collection Heart Rate
Data Collection AltitudeTemperatureTime
Data Architecture - on a Local Wireless Network (ANT+ Protocol)
as a Data Platform
BUSINESS INTELLIGENCE.DATA WAREHOUSE/OLAP.
OLTP DATA.
“Business” Analytics
What are our most profitable Movie titles?
6
What will Happen?What did Happen?
StrategicTactical AnalyticsOperational Reporting
Months WeeksWeeks Months Years
“Business” Analytics
“Personal” Analytics
SELF-SERVICE.GUIDANCE.
BEHAVIOURS.
What Movie should I watch tonight?
8
What will Happen?What did Happen?
StrategicTactical AnalyticsHistorical Behaviors
Months WeeksWeeks Months Years
What is Happening NOW?
“Personal” Analytics
9
GU
IDA
NC
E
DATA COLLECTION
Meaningful
Massive
10
11
12
OLTP App
Data Warehouse
OLTP to OLAP Mapping
OLAP / Reports
Orders App
FIN App
BusinessAnalyst
What are our most profitable Movie titles?
Stag
ing
“Business” Analytics
13
Application
AnalyticsData
End User
What Movie should I watch tonight?
“Personal” Analytics
14
Analytics“Read” Performance
App Persistence“State” PersistencePersistence/Analytics
Big DataBehaviors / “Write” Performance
PersonalizedRecommendations
Personalization,Preferences, State
End-User ExperienceBrowser, Tablet,
Mobile,…Self-Service Application
“Personal Analytics” Data Architecture
Late-Binding Structures
Non-Formal Intake (“Copy”)
Minimal Transaction Semantics
Target Scenario - Writers
Procedural Query Syntax - MapReduce
Hadoop
15
RDBMSHighly Structured Environment
Formalized intake process
ACID Transactional Semantics
Target Scenario – OLTP R/W
High Level Query Syntax - SQL
The New Technology Stack
Technology Data Warehousing New Analytics
Analytics OLAP OLAP + Open-Source
Data Movement ETL Tool MapReduce
SQL RDBMS Hive
Schema Metadata RDBMS JSON / AVRO
Indexing (Readers) RDBMS HBase
RI RDBMS Application Logic
App Store (Objects) RDBMS Key/Value - Cassandra,…
Schema / Columns RDBMS Column Families / Dynamic
Logs (Writers) RDBMS Scalable - Hadoop
Infrastructure Data-Center Cloud
16
Specialization / Individual Scalability / Late-Binding - for each component
17
AnalyticsHbase (Column-Families)
App PersistenceCassandra (JSON)Persistence/Analytics
Data-Center or Cloud
Big DataHadoop (AVRO)
PersonalizedRecommendations
Personalization,Preferences, State
End-User ExperienceBrowser, Tablet,
Mobile,…Self-Service Application
Specialization of Data Technologies
SQLHive
MapReduce
Personal Analytics + Business Intelligence
18
Data Warehouse
OLTP to OLAP Mapping
OLAP / Reports
OLTP App
OLTP App
BusinessAnalyst
Stag
ing
App
19
Contact InformationIf you have further questions or comments:
Casey Kiernan
Sr. Director / Data Architecture
Shopzilla.com
BLOG: www.the-data-platform.com
A recent ride in AZ