tackling big data with the elephant in the room
TRANSCRIPT
TACKLING BIG DATA WITH THE ELEPHANT IN THE ROOM
WHAT’S THE PROBLEM WITH BIG DATA?
Volume Variety Velocity
WHAT’S THE SOLUTION TO BIG DATA?
“In pioneer days they used oxen for heavy pulling, and when one oxen couldn’t budge
a log, they didn’t try to grow a larger ox. We shouldn’t be trying for bigger
computers, but for more systems of computers.” – Grace Hopper
HADOOP’S SOLUTION
Sqoop
Pig Hive
HBase Mahout Flume
Oozie …
Hadoop Distributed File System
MapReduce
Hadoop Core
Components
Hadoop Ecosystem
WHAT IS
HDFS?
HOW DOES HDFS WORK?
Large Data File
Block #1
Block #2
HOW DOES HDFS WORK?
Large Data File
Block #1
Block #2
Block #1
Block #1
Block #1
HOW DOES HDFS WORK?
Large Data File
Block #1
Block #2
Block #1
Block #1
Block #1
Block #2
Block #2
Block #2
HOW DOES HDFS WORK?
Large Data File
Block #1
Block #2
Block #1
Block #1
Block #1
Block #2
Block #2
Block #2
WHAT IS MAP-REDUCE? Core Ideas
– Data Locality – Parallelism – Block Independence
Three Stages 1. Map 2. Swap & Sort 3. Reduce
WORD COUNT MAP
the cat sat on the mat the aardvark sat on the …
Node 1
the mahout drove the ….
Node 2
the cat sat on the mat The aardvark sat on the … The mahout drove the …
Mapper
WORD COUNT MAP
the cat sat on the mat the aardvark sat on the …
Node 1
the mahout drove the ….
Node 2
Mapper
map()
map()
Mapper
WORD COUNT MAP
the cat sat on the mat the aardvark sat on the …
Node 1
the mahout drove the ….
Node 2
Mapper
map()
map()
the 1
cat 1
sat 1
on 1
the 1
mat 1
the 1
mahout 1
drove 1
the 1
Mapper
WORD COUNT MAP
the cat sat on the mat the aardvark sat on the …
Node 1
the mahout drove the ….
Node 2
Mapper
map()
map()
the 1
cat 1
sat 1
on 1
the 1
mat 1
the 1
mahout 1
drove 1
the 1
map() the 1
aardvark 1
sat 1
on 1
the 1
WORD COUNT SWAP & SORT the 1
cat 1
sat 1
on 1
the 1
mat 1
the 1
mahout 1
drove 1
the 1
the 1
aardvark 1
sat 1
on 1
the 1
WORD COUNT SWAP & SORT the 1
cat 1
sat 1
on 1
the 1
mat 1
the 1
mahout 1
drove 1
the 1
the 1
aardvark 1
sat 1
on 1
the 1
aardvark 1
cat 1
mat 1
on 1,1
sat 1
the 1,1,1,1
drove 1
mahout 1
the 1,1
WORD COUNT SWAP & SORT the 1
cat 1
sat 1
on 1
the 1
mat 1
the 1
mahout 1
drove 1
the 1
the 1
aardvark 1
sat 1
on 1
the 1
aardvark 1
cat 1
mat 1
on 1,1
sat 1
the 1,1,1,1
drove 1
mahout 1
the 1,1
aardvark 1
cat 1
mat 1
mahout 1
sat 1
drove 1
on 1,1
the 1,1,1,1,1,1
Node 3
Node 4
Node 5
WORD COUNT REDUCER aardvark 1
cat 1
mat 1
mahout 1
sat 1
drove 1
on 1,1
the 1,1,1,1,1,1
Node 3
Node 4
Node 5
Reducer 0
Reducer 1
Reducer 2
aardvark 1
cat 1
mat 1
mahout 1
sat 1
drove 1
on 2
the 6
TAKE-AWAYS
Sqoop
Pig Hive
HBase Mahout Flume
Oozie …
Hadoop Distributed File System
MapReduce
Hadoop Core
Components
Hadoop Ecosystem
QUESTIONS?