hadoop, oracle and the industrial revolution of data
DESCRIPTION
Presentation given at Oracle Open World 2012TRANSCRIPT
![Page 1: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/1.jpg)
© 2012 Quest Software Inc. All rights reserved.
Hadoop, Oracle and the industrial revolution of data
Guy HarrisonVP R&D, Database Management
![Page 2: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/2.jpg)
Hadoop, Oracle and the industrial revolution of data
Guy HarrisonExecutive Director, R&D Business Intelligence Software
![Page 3: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/3.jpg)
Pg. 3© 2012 Quest Software Inc. All rights reserved.
Introductions
www.guyharrison.net [email protected]
http://twitter.com/guyharrison
![Page 4: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/4.jpg)
Pg. 4© 2012 Quest Software Inc. All rights reserved.
Quest
![Page 5: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/5.jpg)
Pg. 5© 2012 Quest Software Inc. All rights reserved.
![Page 6: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/6.jpg)
Pg. 6© 2012 Quest Software Inc. All rights reserved.
![Page 7: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/7.jpg)
Pg. 7© 2012 Quest Software Inc. All rights reserved.
![Page 8: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/8.jpg)
![Page 9: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/9.jpg)
Pg. 9© 2012 Quest Software Inc. All rights reserved.
![Page 10: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/10.jpg)
Pg. 10© 2012 Quest Software Inc. All rights reserved.
![Page 11: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/11.jpg)
Pg. 11© 2012 Quest Software Inc. All rights reserved.
Blue
Yellow
Red
0 10 20 30 40 50 60 70 80
Star trek shirt fatality analysis
Pct
![Page 12: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/12.jpg)
Pg. 12© 2012 Quest Software Inc. All rights reserved.
![Page 13: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/13.jpg)
Pg. 13© 2012 Quest Software Inc. All rights reserved.
![Page 14: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/14.jpg)
Pg. 14© 2012 Quest Software Inc. All rights reserved.
What is Big Data?
![Page 15: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/15.jpg)
Pg. 15© 2012 Quest Software Inc. All rights reserved.
The 3-4 V’s
VolumeTerabytesPetabytesExabytesZetabytes
VarietyStructuredUnstructuredHuman GeneratedMachine Generated
VelocityUser populations xTransaction rates xMachine data
Value Competitive or Community advantage
![Page 16: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/16.jpg)
Pg. 16© 2012 Quest Software Inc. All rights reserved.
Volume Data volumes have always been increasing
2006 Perspective
![Page 17: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/17.jpg)
Pg. 17© 2012 Quest Software Inc. All rights reserved.
But the vastness is becoming mind boggling
Human Brain
Living Human Genomes
Digital information 2008
Total Digital capacity
Digital information created 2011
1.00E+09 1.00E+11 1.00E+13 1.00E+15 1.00E+17 1.00E+19 1.00E+21 1.00E+23
2.81E+15
1.10E+17
5.48E+18
4.87E+18
1.18E+21
2.13E+21
Gigabyte Terabyte Petabyte Exabyte zettabyte
![Page 18: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/18.jpg)
Pg. 18© 2012 Quest Software Inc. All rights reserved.
Velocity
![Page 19: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/19.jpg)
Pg. 19© 2012 Quest Software Inc. All rights reserved.
Fail whales
![Page 20: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/20.jpg)
Pg. 20© 2012 Quest Software Inc. All rights reserved.
The Industrial Revolution of Data
Variety
![Page 21: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/21.jpg)
Pg. 21© 2012 Quest Software Inc. All rights reserved.
![Page 22: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/22.jpg)
Pg. 22© 2012 Quest Software Inc. All rights reserved.
![Page 23: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/23.jpg)
Pg. 23© 2012 Quest Software Inc. All rights reserved.
Big Data is driven by the smallest devices
![Page 24: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/24.jpg)
Pg. 24© 2012 Quest Software Inc. All rights reserved.
Samsung Galaxy S IIII specifications
Quad-core 1.4 GHz CPU
1GB RAM
64GB Storage
1080p display
GSM/Bluetooth/WiFi Network
8MP Camera
GPS & Compass
![Page 25: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/25.jpg)
Pg. 25© 2012 Quest Software Inc. All rights reserved.
![Page 26: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/26.jpg)
Pg. 26© 2012 Quest Software Inc. All rights reserved.
![Page 27: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/27.jpg)
Pg. 27© 2012 Quest Software Inc. All rights reserved.
![Page 28: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/28.jpg)
Pg. 28© 2012 Quest Software Inc. All rights reserved.
![Page 29: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/29.jpg)
Pg. 29© 2012 Quest Software Inc. All rights reserved.
![Page 30: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/30.jpg)
Pg. 30© 2012 Quest Software Inc. All rights reserved.
![Page 31: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/31.jpg)
Pg. 31© 2012 Quest Software Inc. All rights reserved.
![Page 32: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/32.jpg)
Pg. 32© 2012 Quest Software Inc. All rights reserved.
![Page 33: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/33.jpg)
Pg. 33© 2012 Quest Software Inc. All rights reserved.
![Page 34: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/34.jpg)
Pg. 34© 2012 Quest Software Inc. All rights reserved.
![Page 35: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/35.jpg)
35
Name: Willy Bowman
Nationality: German
DON’T MENTION THE WAR
![Page 36: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/36.jpg)
Pg. 36© 2012 Quest Software Inc. All rights reserved.
Data Input
![Page 37: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/37.jpg)
Pg. 37© 2012 Quest Software Inc. All rights reserved.
![Page 38: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/38.jpg)
From now on, I’ll call you ‘An Ambulance’. OK?
“Siri call me an ambulance”
I found 14 bridges nearby:
“I want to jump off a bridge”
Siri
![Page 39: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/39.jpg)
Pg. 39© 2012 Quest Software Inc. All rights reserved.
![Page 40: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/40.jpg)
Pg. 40© 2012 Quest Software Inc. All rights reserved.
![Page 41: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/41.jpg)
Pg. 41© 2012 Quest Software Inc. All rights reserved.
Brain Control
![Page 42: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/42.jpg)
Pg. 42© 2012 Quest Software Inc. All rights reserved.
![Page 43: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/43.jpg)
Pg. 43© 2012 Quest Software Inc. All rights reserved.
![Page 44: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/44.jpg)
Pg. 44© 2012 Quest Software Inc. All rights reserved.
![Page 45: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/45.jpg)
Pg. 45© 2012 Quest Software Inc. All rights reserved.
![Page 46: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/46.jpg)
Pg. 46© 2012 Quest Software Inc. All rights reserved.
![Page 47: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/47.jpg)
Pg. 47© 2012 Quest Software Inc. All rights reserved.
All of this requires and Generates Big Datasets
But what are they good for?
![Page 48: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/48.jpg)
Pg. 48© 2012 Quest Software Inc. All rights reserved.
Value?
Achieve competitive advantage
From Big Data using
Collective Intelligence,
Machine Learning
and Predictive Analytics
![Page 49: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/49.jpg)
Machine LearningPrograms that evolve with “experience”
Collective IntelligencePrograms that use inputs from “crowds’ to seem intelligent
Predictive AnalyticsPrograms that extrapolate from existing data into the future
Big Data AnalyticsHow do we derive value from the data?
![Page 50: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/50.jpg)
Pg. 50© 2012 Quest Software Inc. All rights reserved.
![Page 51: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/51.jpg)
Pg. 51© 2012 Quest Software Inc. All rights reserved.
![Page 52: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/52.jpg)
Pg. 52© 2012 Quest Software Inc. All rights reserved.
![Page 53: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/53.jpg)
Pg. 53© 2012 Quest Software Inc. All rights reserved.
![Page 54: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/54.jpg)
Pg. 54© 2012 Quest Software Inc. All rights reserved.
![Page 55: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/55.jpg)
Pg. 55© 2012 Quest Software Inc. All rights reserved.
![Page 56: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/56.jpg)
Pg. 56© 2012 Quest Software Inc. All rights reserved.
![Page 57: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/57.jpg)
Pg. 57© 2012 Quest Software Inc. All rights reserved.
![Page 58: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/58.jpg)
Pg. 58© 2012 Quest Software Inc. All rights reserved.
![Page 59: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/59.jpg)
Pg. 59© 2012 Quest Software Inc. All rights reserved.
![Page 60: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/60.jpg)
Pg. 60© 2012 Quest Software Inc. All rights reserved.
![Page 61: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/61.jpg)
Pg. 61© 2012 Quest Software Inc. All rights reserved.
Applications
Collective Intelligence
Search Optimization
Recommendation Systems
Security•Vulnerability•Penetration Detection
Fraud Detection
Predictive Analytics•Churn •Defaults
Medical•Risk analysis•Diagnosis•Prognosis
Game optimization
Advertising•Targeting•Tailoring
![Page 62: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/62.jpg)
Pg. 62© 2012 Quest Software Inc. All rights reserved.
Collective Intelligence beats Artificial Intelligence
?
![Page 63: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/63.jpg)
Pg. 63© 2012 Quest Software Inc. All rights reserved.
![Page 64: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/64.jpg)
Pg. 64© 2012 Quest Software Inc. All rights reserved.
![Page 65: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/65.jpg)
Pg. 65© 2012 Quest Software Inc. All rights reserved.
![Page 66: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/66.jpg)
Pg. 66© 2012 Quest Software Inc. All rights reserved.
![Page 67: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/67.jpg)
Pg. 67© 2012 Quest Software Inc. All rights reserved.
![Page 68: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/68.jpg)
Pg. 68© 2012 Quest Software Inc. All rights reserved.
For the past 40 years, AI has been consistently disappointing
![Page 69: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/69.jpg)
Pg. 69© 2012 Quest Software Inc. All rights reserved.
![Page 70: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/70.jpg)
Pg. 70© 2012 Quest Software Inc. All rights reserved.
![Page 71: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/71.jpg)
Pg. 71© 2012 Quest Software Inc. All rights reserved.
![Page 72: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/72.jpg)
Pg. 72© 2012 Quest Software Inc. All rights reserved.
![Page 73: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/73.jpg)
Pg. 73© 2012 Quest Software Inc. All rights reserved.
![Page 74: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/74.jpg)
Pg. 74© 2012 Quest Software Inc. All rights reserved.
![Page 75: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/75.jpg)
Pg. 75© 2012 Quest Software Inc. All rights reserved.
![Page 76: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/76.jpg)
Pg. 76© 2012 Quest Software Inc. All rights reserved.
![Page 77: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/77.jpg)
Pg. 77© 2012 Quest Software Inc. All rights reserved.
![Page 78: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/78.jpg)
Pg. 78© 2012 Quest Software Inc. All rights reserved.
Google: pioneers of big data
![Page 79: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/79.jpg)
Pg. 79© 2012 Quest Software Inc. All rights reserved.
![Page 80: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/80.jpg)
Pg. 80© 2012 Quest Software Inc. All rights reserved.
![Page 81: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/81.jpg)
Pg. 81© 2012 Quest Software Inc. All rights reserved.
![Page 82: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/82.jpg)
Pg. 82© 2012 Quest Software Inc. All rights reserved.
![Page 83: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/83.jpg)
Pg. 83© 2012 Quest Software Inc. All rights reserved.
Google File System (GFS)
Map Reduce BigTableChubby
Google Applications
Google Software Architecture
![Page 84: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/84.jpg)
Pg. 84© 2012 Quest Software Inc. All rights reserved.
START REDUCEMAPMAP
MAPMAP
MAPMAP
MAPMAP
MAPMAP
MAPMAP
MAP
MAPMAP
MAPMAP
MAPMAP
MAPMAP
MAPMAP
MAPMAP
MAPMAP
MAPMAP
MAPMAP
MAPMAP
MAPMAP
Map Reduce
![Page 85: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/85.jpg)
Pg. 85© 2012 Quest Software Inc. All rights reserved.
HDFS
MAPPER
MAPPER
MAPPER
MAPPER
MAPPER
MAPPER
MAPPER
MAPPER
SCANSORT
MAPPER
MAPPER
MAPPER
MAPPER
AGGREGATE
REDUCECLIENT
Multi-stage Map-Reduce
![Page 86: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/86.jpg)
Pg. 86© 2012 Quest Software Inc. All rights reserved.
Hadoop: Open Source Map-Reduce Stack
![Page 87: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/87.jpg)
Pg. 87© 2012 Quest Software Inc. All rights reserved.
Hadoop at Yahoo!
Yahoo! Hadoop cluster:− 4000 nodes− 16PB disk− 64 TB of RAM− 32,000 Cores
![Page 88: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/88.jpg)
Pg. 88© 2012 Quest Software Inc. All rights reserved.
![Page 89: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/89.jpg)
Pg. 89© 2012 Quest Software Inc. All rights reserved.
MAP REDUCE (DISTRIBUTED PROCESSING)
HADOOP CLIENT (JAVA, PIG, HIVE)
HDFS (DISTRIBUTED
STORAGE)
JOB TRACKER
DATA NODE TASK TRACKER
DATA NODE TASK TRACKER
DATA NODE TASK TRACKER
DATA NODE TASK TRACKER
NAME NODE
DATA NODE TASK TRACKER
DATA NODE TASK TRACKER
DATA NODE TASK TRACKER
DATA NODE TASK TRACKER
SECONDARY NAME NODE
DATA NODE TASK TRACKER
DATA NODE TASK TRACKER
DATA NODE TASK TRACKER
DATA NODE TASK TRACKER
DATA NODE TASK TRACKER
DATA NODE TASK TRACKER
DATA NODE TASK TRACKER
Hadoop Architecture(1.0)
![Page 90: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/90.jpg)
Pg. 90© 2012 Quest Software Inc. All rights reserved.
Schema on Read vs Schema on Write
![Page 91: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/91.jpg)
Pg. 91© 2012 Quest Software Inc. All rights reserved.
Data
Analyse
Aggregate
Normalize
Cleanse
Code
Extract Load TransformData Warehouse
Utilize
Data LoadHadoop
Analyse
Cleanse
Code
Utilize
Schema on Write
Schema on Read
![Page 92: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/92.jpg)
Pg. 92© 2012 Quest Software Inc. All rights reserved.
Hadoop Ecosystem
Hadoop File System (HDFS)
Hadoop Map ReduceHbase
(Database)ZooKeeper(Locking)
SQOOP(RDBMS loader)
Hive(Query)
Pig(Scripting)
Flume(Log Loader)
Oozie (Workflow manager)
![Page 93: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/93.jpg)
Pg. 93© 2012 Quest Software Inc. All rights reserved.
HBase
![Page 94: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/94.jpg)
Pg. 94© 2012 Quest Software Inc. All rights reserved.
HBase is a real-time database built on Hadoop
HBase
ASM
Datafiles
Buffer Cache
Table Table
Redo
Disks
LogBuffer
HDFS
HFile
MemStore
Table Table
WA Log
Disks
HFile
![Page 95: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/95.jpg)
Name Site Counter
Dick Ebay 507,018
Dick Google 690,414
Jane Google 716,426
Dick Facebook 723,649
Jane Facebook 643,261
Jane ILoveLarry.com 856,767
Dick MadBillFans.com 675,230
NameId Name
1 Dick
2 Jane
SiteId SiteName
1 Ebay
2 Google
3 Facebook
4 ILoveLarry.com
5 MadBillFans.com
NameId SiteId Counter
1 1 507,018
1 3 690,414
2 3 716,426
1 3 723,649
2 3 643,261
2 4 856,767
1 5 675,230
Id Name Ebay Google Facebook (other columns) MadBillFans.com
1 Dick 507,018 690,414 723,649 . . . . . . . . . . . . . . 675,230
Id Name Google Facebook (other columns) ILoveLarry.com
2 Jane 716,426 643,261 . . . . . . . . . . . . . . 856,767
Hbase Data Model
![Page 96: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/96.jpg)
Pg. 96© 2012 Quest Software Inc. All rights reserved.
Hive
![Page 97: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/97.jpg)
Pg. 97© 2012 Quest Software Inc. All rights reserved.
![Page 98: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/98.jpg)
Pg. 98© 2012 Quest Software Inc. All rights reserved.
SQL
JAVA
Resu
lts
![Page 99: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/99.jpg)
Pg. 99© 2012 Quest Software Inc. All rights reserved.
Pig
![Page 100: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/100.jpg)
Pg. 100© 2012 Quest Software Inc. All rights reserved.
Pig Latin
SQL or Hive QL
![Page 101: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/101.jpg)
Pg. 101© 2012 Quest Software Inc. All rights reserved.
Meanwhile, back at the Death Star….
![Page 102: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/102.jpg)
![Page 103: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/103.jpg)
Pg. 103© 2012 Quest Software Inc. All rights reserved.
![Page 104: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/104.jpg)
Pg. 104© 2012 Quest Software Inc. All rights reserved.
Oracle Exadata
Database servers64 cores, 576 GB
RAM
Storage Servers112 cores, 100 TB SAS or336 TB SATA plus5 TB SSD
![Page 105: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/105.jpg)
Pg. 105© 2012 Quest Software Inc. All rights reserved.
Exadata
Hadoop
$0 $1,000 $2,000 $3,000 $4,000 $5,000 $6,000
$4,911
$750
Exadata vs Hadoop $$/TB (Hardware only)
Economies
![Page 106: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/106.jpg)
Pg. 106© 2012 Quest Software Inc. All rights reserved.
![Page 107: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/107.jpg)
Pg. 107© 2012 Quest Software Inc. All rights reserved.
![Page 108: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/108.jpg)
Pg. 108© 2012 Quest Software Inc. All rights reserved.
18 Sun X4270 M2 servers− 48GB RAM per node (864GB total)− 2x6 Core CPU per node (216 total)− 12x2TB HDD per node (216 spindles,
864 TB)− 40Gb/s Infiniband between nodes− 10Gb/s Ethernet to datacentre
Competitive Pricing
www.oracle.com/us/bigdata/index.html
Oracle Big Data Appliance
![Page 109: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/109.jpg)
Pg. 109© 2012 Quest Software Inc. All rights reserved.
Big Data Appliance Software
Cloudera Enterprise
Oracle Enterprise R
Oracle NoSQL
Oracle Big Data Connectors
![Page 110: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/110.jpg)
Pg. 110© 2012 Quest Software Inc. All rights reserved.
Oracle’s Storage Hierarchy
ORACLEEXADATA
ORACLEEXALOGIC
ORACLEBIG DATA
APPLIANCE
ORACLE NOSQL
ORACLE LOADER FOR HADOOP
APACHEHADOOP ORACLE
RDBMS
ORACLE WEBLOGIC
ORACLE EXALYTICS
ORACLE ESSBASE
ORACLE TIMES TEN
Latency
Storage Costs
![Page 111: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/111.jpg)
Pg. 111© 2012 Quest Software Inc. All rights reserved.
111
![Page 112: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/112.jpg)
Pg. 112© 2012 Quest Software Inc. All rights reserved.
![Page 113: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/113.jpg)
Pg. 113© 2012 Quest Software Inc. All rights reserved.
Hadoop and RDBMS integration
![Page 114: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/114.jpg)
Pg. 114© 2012 Quest Software Inc. All rights reserved.
Scenario #1: Reference data in RDBMS
CUSTOMERS
WEBlOGS
PRODUCTS
HDFS
RDBMS
![Page 115: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/115.jpg)
Pg. 115© 2012 Quest Software Inc. All rights reserved.
Scenario #2: Hadoop for off-line analytics
CUSTOMERS
PRODUCTS
RDBMS
SALESHISTORY
HDFS
![Page 116: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/116.jpg)
Pg. 116© 2012 Quest Software Inc. All rights reserved.
Scenario #3: MapReduce output to RDBMS
WEBLOGSSUMMARY
RDBMS
DB QUERYTOOL
WEBLOGS
HDFS
![Page 117: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/117.jpg)
Pg. 117© 2012 Quest Software Inc. All rights reserved.
Scenario #4: Hadoop as RDBMS “active archive”
SALES 2011
HDFS
RDBMS
QUERYTOOL
SALES 2010
SALES 2009
SALES 2008
SALES 2009
SALES 2008
![Page 118: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/118.jpg)
Pg. 118© 2012 Quest Software Inc. All rights reserved.
The Big Data Stack
![Page 119: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/119.jpg)
The Big Data Stack
HDFS
MAP-REDUCE HBASE
PIG
CASCADING
MAHOUT
JAVA APIHIVE
R (ET AL)
JAVA API
DATA SCIENTIST
![Page 120: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/120.jpg)
![Page 121: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/121.jpg)
The Big Data Stack
HDFS
MAP-REDUCE HBASE
PIG
CASCADING
MAHOUT
JAVA API HIVE
R (ET AL)
JAVA API
DATA SCIENTISTBIG DATA ANALAYTIC PLATFORM
![Page 122: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/122.jpg)
Big Data Analytics Platform
BIG DATA ANALYTICS
INDEXING AND SEARCH
VISUALIZATION
RECOMMENDERS
CLUSTERING
CLASSIFICATION
EXPERT SYSTEMS (LIKE WATSON)
OPTIMIZATION
ADVERTISING
BASKET ANALYSIS
SENTIMENT ANALYSIS
![Page 123: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/123.jpg)
Pg. 123© 2012 Quest Software Inc. All rights reserved.
In Summary
![Page 124: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/124.jpg)
Pg. 124© 2012 Quest Software Inc. All rights reserved.
Hadoop is….
![Page 125: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/125.jpg)
Pg. 125© 2012 Quest Software Inc. All rights reserved.
Exadata
Hadoop
$0 $1,000 $2,000 $3,000 $4,000 $5,000 $6,000
$4,911
$750
Exadata vs Hadoop $$/TB (Hardware only)
Economical
![Page 126: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/126.jpg)
Pg. 126© 2012 Quest Software Inc. All rights reserved.
Scalable
• 4000 nodes at Yahoo!• >100 PB at Facebook• 10,000 node design
goal for Hadoop 2.0
![Page 127: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/127.jpg)
Pg. 127© 2012 Quest Software Inc. All rights reserved.
A platform for AI, CI & analytics
![Page 128: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/128.jpg)
Pg. 128© 2012 Quest Software Inc. All rights reserved.
ETL “Free”
Data
Analyse
Aggregate
Normalize
Cleanse
Code
Extract Load TransformData Warehouse
Utilize
Data LoadHadoop
Analyse
Cleanse
Code
Utilize
Schema on Write
Schema on Read
![Page 129: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/129.jpg)
Pg. 129© 2012 Quest Software Inc. All rights reserved.
The most concrete technology enabling the Big Data revolution
![Page 130: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/130.jpg)
Pg. 130© 2012 Quest Software Inc. All rights reserved.
Hadoop is not….
![Page 131: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/131.jpg)
Pg. 131© 2012 Quest Software Inc. All rights reserved.
But future Enterprise Data Architectures will likely incorporate Hadoop side by side with RDBMS
A replacement for RDBMS
![Page 132: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/132.jpg)
Pg. 132© 2012 Quest Software Inc. All rights reserved.
Though OLTP systems can be built with Hadoop-compatible NoSQL systems such as HBase and Cassandra
Suitable for OLTP
![Page 133: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/133.jpg)
Pg. 133© 2012 Quest Software Inc. All rights reserved.
Hadoop alone only solves the storage challenge of Big Data
A complete solution
![Page 134: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/134.jpg)
Pg. 134© 2012 Quest Software Inc. All rights reserved.
Shameless plugs
![Page 135: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/135.jpg)
![Page 136: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/136.jpg)
Pg. 136© 2012 Quest Software Inc. All rights reserved.
Toad for Cloud Databases
Work with Hive, Hbase, Oracle, SQL Server, Cassandra, MySQL, MongoDB, BI servers and other NoSQL and SQL datastores
![Page 137: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/137.jpg)
Pg. 137© 2012 Quest Software Inc. All rights reserved.
Toad for Cloud Databases• Federated SQL queries across Hive, Hbase, NoSQL, RDBMS
Toad for Cloud Databases
![Page 138: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/138.jpg)
Pg. 138© 2012 Quest Software Inc. All rights reserved.
0 5 10 15 20 25 30 350
1,000
2,000
3,000
4,000
5,000
6,000
7,000
50M row, 50GB Oracle table to 16-node Hadoop clusterSQOOP
SQOOP with Quest Connector
Number of mappers
Ela
pse
d T
ime
(ms)
Quest Connector for Oracle and Hadoop
Hi-speed, bi-directional data transfer between Hadoop, Hive and Oracle
![Page 139: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/139.jpg)
Pg. 139© 2012 Quest Software Inc. All rights reserved.
Business Intelligence solutions with first class support for Hadoop, Oracle and many other platforms
Toad BI Suite
![Page 140: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/140.jpg)
Pg. 140© 2012 Quest Software Inc. All rights reserved.
Redo-logs
Change Data Capture
JMS Queue Hadoop Poster
BatchedHDFS File Copy
Audit / Change Data
HBase RealTime replication
SharePlex® for Hadoop
![Page 141: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/141.jpg)
Pg. 141© 2012 Quest Software Inc. All rights reserved.
• Hive Query IDE
• Oracle <-> Hadoop data management
• Basic Hadoop administration
• ETA beta H1 2013
Toad for Hadoop
![Page 142: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/142.jpg)
![Page 143: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/143.jpg)
Pg. 143© 2012 Quest Software Inc. All rights reserved.
![Page 144: Hadoop, oracle and the industrial revolution of data](https://reader036.vdocuments.net/reader036/viewer/2022081412/54564254af79594d148b9260/html5/thumbnails/144.jpg)
Pg. 144© 2012 Quest Software Inc. All rights reserved.
Summary:
The future belongs to those of us prepared to wear funny hats and glasses
The connected and mobile internet requires and produces “big data” that is qualitatively different from the data we’ve had before− Requiring different types of datastores
Enterprise can leverage big data for competitive advantage− Requiring different types of analytical engines