compaction and splitting in apache accumulo

© Hortonworks Inc. 2012

Compaction and Splitting in Apache AccumuloBillie [email protected] 24, 2012


What are compaction and splitting?

•Accumulo tables are divided into non-overlapping key ranges called tablets

•Compaction selects a set of sorted files for a single tablet and rewrites them into one file

•Splitting divides a tablet into two tablets


Tablet Overview

•When memory fills, new sorted files are created by flushing

•Sorted files are combined together into fewer sorted files


How much data are you writing?

•If you never compact – O(N)

•If you always compact – O(N2)

…

…


Accumulo Compaction Algorithm

•Compact a set of files when:

size of the largest file

compaction ratio

sum of the sizes of files× ≤

table.compaction.major.ratio


In Action (r = 3, N = 1, W = 1)


In Action (r = 3, N = 2, W = 2)


In Action (r = 3, N = 3, W = 3)


In Action (r = 3, N = 3, W = 6)


In Action (r = 3, N = 4, W = 7)


In Action (r = 3, N = 5, W = 8)


In Action (r = 3, N = 6, W = 9)


In Action (r = 3, N = 6, W = 12)


In Action (r = 3, N = 7, W = 13)


In Action (r = 3, N = 8, W = 14)


In Action (r = 3, N = 9, W = 15)


In Action (r = 3, N = 9, W = 24)


In Action (r = 3, N = 27, W = 90*)


Amount of data written

•W(rk) = (k+1)rk – (k-1)rk-1

•Thus, W(N) ≈ O(N log N)


HBase Compaction Algorithm



sum of the sizes of

smaller files≤ compaction

ratio×

hbase.hstore.compaction.ratio


HBase Compaction Algorithm



sum of the sizes of

smaller files≤ compaction

ratio×

HBase ratio = Accumulo ratio – 1

1


Other Compaction-related Properties

•Accumulo

•Hbase

table.file.maxtserver.compaction.major.thread.files.open.maxtserver.compaction.major.delaytable.compaction.major.everything.idle

hbase.hstore.compactionThresholdhbase.hstore.blockingStoreFileshbase.hstore.blockingWaitTimehbase.hstore.compaction.minhbase.hstore.compaction.maxhbase.hstore.compaction.min.sizehbase.hstore.compaction.max.size


Accumulo Splitting

•Always check to see if a split is needed before compacting

•If it is needed, split first•File names stored in metadata table

split threshold


Accumulo Splitting Process

•Tablet closed, no new writes•Three writes to the metadata table–tablet made smaller & marked as splitting–new tablet added–original tablet's splitting marks removed

•Tablet server swaps new tablets for old tablet in its online tablet list

•Master informed


Accumulo Splitting Recovery

•Whenever a tablet is brought online, the tablet server checks to see if it has split marks.

•If so, it assumes the splitting process was interrupted and finishes making changes to the metadata table.


1

• Simplify deployment to get started quickly and easily

• Monitor, manage any size cluster with familiar console and tools

• Only platform to include data integration services to interact with any data

• Metadata services opens the platform for integration with existing applications

• Dependable high availability architecture

• Tested at scale to future proof your cluster growth

Hortonworks Data Platform

Reduce risks and cost of adoption Lower the total cost to administer and provision Integrate with your existing ecosystem


Hortonworks Training

The expert source for Apache Hadoop training & certification

Role-based Developer and Administration training

– Coursework built and maintained by the core Apache Hadoop development team.– The “right” course, with the most extensive and realistic hands-on materials– Provide an immersive experience into real-world Hadoop scenarios– Public and Private courses available

Comprehensive Apache Hadoop Certification

– Become a trusted and valuable Apache Hadoop expert


Next Steps?

• Expert role based training• Course for admins, developers

and operators• Certification program• Custom onsite options

Download Hortonworks Data Platformhortonworks.com/download

1

2 Use the getting started guidehortonworks.com/get-started

3 Learn more… get support

• Full lifecycle technical support across four service levels

• Delivered by Apache Hadoop Experts/Committers

• Forward-compatible

Hortonworks Support

hortonworks.com/training hortonworks.com/support


[email protected]

compaction and splitting in apache accumulo

Technology