FASTER, FASTER, FASTER: THE TRUE STORY OF A MOBILE ANALYTICS DATA MART ON HIVE

Mithun RadhakrishnanJosh Walters

3

• Mithun Radhakrishnan• Hive Engineer at Yahoo• Hive Committer • Has an irrational fear of

spider monkeys• mithun@apache.org• @mithunrk

About myself

4

RECAP

55 2015 Hadoop Summit, San Jose, California

6

From: The [REDACTED] ETL teamTo: The Yahoo Hive TeamSubject: A small matter of size...

Dear YHive team,

We have partitioned our table using the following 6 partition keys: {hourly-timestamp, name, property, geo-location, shoe-size, and so on…}. For a given timestamp, the combined cardinality of the remaining partition-keys is about 10000/hr.

If queries on partitioned tables are supposed to be faster, how come queries on our table take forever just to get off the ground?

Yours gigantically, Project [REDACTED]

7

ABOUT ME• Josh Walters• Data Engineer at Yahoo• I build lots of data pipelines• Can eat a whole plate of deep fried cookie

dough• http://joshwalters.com• @joshwalters

8

WHAT IS THE CUSTOMER NEED?• Faster ETL

• Faster queries

• Faster ramp up

9

CASE STUDY: MOBILE DATA MART• Mobile app usage data

• Optimize performance

• Interactive analytics

10

LOW HANGING FRUIT• Tez Tez Tez!

• Vectorized query execution

• Map-side aggregations

• Auto-convert map join

11

DATA PARTITIONING• Want thousands of partitions

• Deep data partitioning

• Difficult to do at scale

12

DEEP PARTITIONING• Greatly helps with compression

• 2015 XLDB talk on methods used

• https://youtu.be/P-vrzYYdfL8

• http://www-conf.slac.stanford.edu/xldb2015/lightning_abstracts.asp

13

SOLID STATE DRIVES• Didn’t really help

• Ended up CPU bound

• Regular drives are fine

14

ORC!• Used in largest data systems

• 90% boost on sorted columns

• 30x compression versus raw text

• Fits well with our tech stack

15

SKETCH ALL THE THINGS• Very accurate

• Can store sketches in Hive

• Union, intersection, difference

• 75% boost on relevant queries

16

SKETCH ALL THE THINGSSELECT COUNT(DISTINCT id)FROM DB.TABLEWHERE ...; -- ~100 seconds

SELECT estimate(sketch(id))FROM DB.TABLEWHERE ...; -- ~25 seconds

17

SKETCH ALL THE THINGSStandard Deviation 1 2 3

Confidence Interval 68% 95% 99%

K = 16 25% 51% 77%

K = 512 4% 8% 13%

K = 4096 1% 3% 4%

K = 16384 < 1% 1% 2%

18

MORE SKETCH INFO• Summarization, Approx. and Sampling: Tradeoffs for Improving Query, Hadoop Summit, 2015

• http://datasketches.github.io

19

ADVANCED QUERIES• Desire for complex queries

• Retention, funnels, etc

• A lot can be done with UDFs

20

FUNNEL ANALYSIS• Complex to write, difficult to reuse

• Slow, requires multiple joins

• Using UDFs, now runs in seconds, not hours

• https://github.com/yahoo/hive-funnel-udf

21

REALLY FAST OLAP• OLAP type queries are the most common

• Aggregate only queries: group, count, sum, …

• Can we optimize for such queries?

22

OLAP WITH DRUID• Interactive, sub-second latency

• Ingest raw records, then aggregate

• Open source, actively developed

• http://druid.io

23

BI TOOL• Many options

• Don’t cover all needs

• Need graphs and dashboards

24

CARAVEL• Hive, Druid, Redshift, MySQL, …

• Simple query construction

• Open source, actively developed

• https://github.com/airbnb/caravel

25

WHAT WE LEARNED• Product teams need custom data marts

• Complex to build and run

• Just want to focus on business logic

26

DATA MART IN A BOX!• Generalized ETL pipeline

• Easy to spin-up

• Automatic continuous delivery

• Just give us a query!

27

DATA MART ARCHITECTURE

28

INFRASTRUCTURE WORK• We didn’t do this alone

• Partners in grid team fixed many pain points

Y!HIVE

30

Dedicated Queue Metrics:

Shared Cluster Metrics:

Hive on Tez - Interactive Queries in Shared Clusters

31

32

Hive 0.2

Hive 0.3

Hive 0.4

Hive 0.5

Hive 0.6

Hive 0.7

Hive 0.8

Hive 0.9

Hive 0.10

Hive 0.11

Hive 0.12

Hive 0.13

Hive 0.14

Hive 1.0

Hive 1.1

Hive 1.2

Hive 1 Hive 2.0

Hive Master

0

500

1000

1500

2000

2500

3000

3500

4000

4500

Increased Configurability or Increased complexity?

LOC

33

• Out of the box:• Tez container reuse

• set tez.am.container.reuse.enabled=true;• Tez speculative execution

• set tez.am.speculation.enabled=true;• Reduce-side vectorization

• set hive.vectorized.execution.reduce.enabled=true;• set hive.vectorized.execution.reduce.groupby.enabled=true;

Performance Tuning

34

• Understand your data:• Use ORC’s index-based filtering:

• set hive.optimize.index.filter=true;• Bloom filters

• ALTER TABLE my_orc SET TBLPROPERTIES(“orc.bloom.filter.columns”=“foo,bar”);• Cardinality?

• Sort on filter-column• Trade-offs: Parallelism vs. filtering

Performance Tuning

35

• Understand your queries:• Prefer LIKE and INSTR over REGEXP*• Compile-time date/time functions:

• current_date()• current_timestamp()

• Queries generated from UI tools

Performance Tuning

36

• Index-based filtering available to Pig / MR users• HCatLoader, HCatInputFormat

• Split-calculation improvements• Block-based BI• Parallel ETL

• Disabled dictionaries for Complex data types• OOMs

Performance Improvements - ORC

37

• Skew Joins• Already solved for Pig• Hive for ETL• Current Hive solution: Explicit values. (Wishful thinking)• Poisson sampling

• Faster sorted-merge joins• Wide-tables• SpillableRowContainers

Performance Improvements - Joins

38

• Improvements for data-discovery• HCatClient-based users• Oozie, GDM• 10x improvement

• Fetch Operator improvements:• SELECT * FROM partitioned_table LIMIT 100;• Lazy-load partitions

Performance Improvements – Various Sundries

39

• Avro Format is popular• Self describing• Flexible• Generic• Quirky

• Intermediate stages in pipelines• Development

Performance Improvements: Hive’s AvroSerDe

40

“There is no mature, no stable. The only constant is change… ... [Our] work on feeds often involves new columns, several times a day.”

41

42

• AvroSerDe needs read-schema at job-runtime (i.e. map-side)• Stored on HDFS

• ETL Jobs need 10-20K maps• Replication factor• Data-node outage

• It gets steadily worse• Block-replication on node-loss• Task attempt retry• More nodes lost• Rinse and repeat

The Problem

43

44

• Reconcile metastore-schema against read-schema?• toAvroSchema( fromAvroSchema( avroSchema )) != avroSchema

• Store schema in TBLPROPERTIES?• Cache read-schema during SerDe::initialize()

• Once per map-task• Prefetch read-schema at query-planning phase

• Once per job• Separate optimizer

The Solution

4545

• Row-oriented format• Skew-join• Stats storage

We’re not done yet

46

• Team effort• Chris Drome• Selina Zhang• Michael Natkovich• Olga Natkovich• Sameer Raheja• Ravi Sankurati

Thanks

Q&A