february 2014 hug : introduction to tez
DESCRIPTION
February 2014 HUG : Introduction to TezTRANSCRIPT
Tez, An Introduction
Page 1
Alan F. Gates
Founder & Architect
@alanfgates
© 2014 Hortonworks
In The Beginning of Hadoop…
Page 2
...there was MapReduce–It could handle data sizes way beyond those of its competitors–It was resilient in the face of failure–It made it easy for users to bring their code and algorithms to the data (i.e. free to program in Java instead of just SQL)
© 2014 Hortonworks
But, It Was Too Low Level
Page 3
© 2014 Hortonworks
But it was too rigid
Page 4
© 2014 Hortonworks
But, It Was Batch
Page 5
© 2014 Hortonworks
YARN to the Rescue
Page 6
© 2014 Hortonworks
Why Tez? Enable Data Processing In Many Tools
Page 7
•An execution engine that can be used by Hive, Pig, Cascading, and others
•Right now SQL on hadoop is hot, and we want to enable that
•But we also want to keep in mind that there’s a lot else to be done in Hadoop (machine learning, ETL, graph processing, etc.) and we want to open up the work we’re doing to those groups as well.
© 2014 Hortonworks
Why Tez? Span Batch and Interactive
Page 8
•It’s hard for customers to use different tools depending on their data size
•It’s hard for applications like Hive to use different back end engines depending on the inputs and outputs
© 2014 Hortonworks
Why Tez? Preserve MapReduce Experience
Page 9
•MapReduce represents engineering centuries of work
•Much has been learned (mostly the hard way) about scale and resiliency
•We are not excited to reinvent those wheels, we would rather rebuild the vehicle on top of them