1

Headline Goes HereSpeaker Name or Subhead Goes Here

DO NOT USE PUBLICLY PRIOR TO 10/23/12Beyond Batch

Doug Cutting October 2012

2

Hadoop Started As Batch

MapReduce• Simple, powerful• Kills a lot of birds

• Efficient, scalable• Compute at storage

• Shared platform• Used by Pig, Hive, etc.

• Incredibly useful!• But not sufficient

3

Big Data Is Not (Just) Batch

Its true themes are:• Scalability

• Affordability• Commodity hardware• Open-source software

• Distributed & reliable• Schema on read• Data beats algorithms

4

HBase: First Non-Batch Component

Online key/value store• Complement to batch

• Online put/get• Batch load & analyze• Best of both• Popular combination

• A step towards the future…

5

Holy Grail Of Big Data

• Open source, commodity HW, etc.• Linear scaling

• To scale, just buy more hardware• On many axes

• Storage capacity• Throughput & latency

• of batch & query• Transactions, Joins, Indexes

• and batch!

6

Google Gives Us A Map

Google publication Apache project

2004 GFS & MapReduce 2006 Hadoop batch programs

2005 Sawzall 2008 Pig & Hive batch queries

2006 BigTable 2008 HBase online key/value

... ... ... ... ...

2012 Spanner ? ? holy grail?

Google publication Open source project

2004 GFS & MapReduce 2006 Hadoop batch programs

2005 Sawzall 2008 Pig & Hive batch queries

2006 BigTable 2008 HBase online key/value

... ... ... ... ...

2012 Spanner ? ? transactions, etc.

5 years – 26 authors!

7

Impala Is Latest Step

Google publication Apache project

2004 GFS & MapReduce 2006 Hadoop batch programs

2005 Sawzall 2008 Pig & Hive batch queries

2006 BigTable 2008 HBase online key/value

... ... ... ... ...

2012 Spanner ? ? holy grail?

Google publication Open source project

2004 GFS & MapReduce 2006 Hadoop batch programs

2005 Sawzall 2008 Pig & Hive batch queries

2006 BigTable 2008 HBase online key/value

2010 Dremel/F1 2012 Impala online queries

2012 Spanner ? ? transactions, etc.

8

@cutting #bigquestions