tuning elasticsearch for multi-terabyte analytics
Post on 26-Jan-2015
Embed Size (px)
DESCRIPTIONA talk by Andrew Clegg at the ElasticSearch London meetup in November 2013 on how Pearson does large-scale analytical queries on ElasticSearch.
- 1. ElasticSearch London Tuning ElasticSearch for multi-terabyte analytics or Counting stuff is hard Andrew Clegg Data Analytics & Visualization Team Pearson @andrew_clegg
2. Introduction 3. Our data Over 11 billion docs in production cluster. Each doc is around 1-2KB of JSON. ~60 million docs/day == ~700 docs/sec. Higher than this during peak times. Much higher when backfilling historical data. Conversely: not many end users yet: 5-20 on a typical day. 4. Our architecturePalomino 5. Getting data in Hardware (Yes, actual hardware!) Cisco UCS servers, 24 cores, 96GB memory. 8 x 1TB disks. 7 for data, 1 for log files, temp files, etc. Reads/writes parallelized across segments. Currently 5 of these in production cluster. 10GB switch. 6. Getting data in Index configuration We dont store any data in ElasticSearch. All we need is facet counts. Disable _source, _all, and individual field storage. Disable term vectors and norms. No analysis on text fields (just unbroken strings). No date autodetection. 7. Getting data in Weekly rolling indices mean shard level can increase as traffic does Time (new index each week) Shard countNB currently we have steady state so its set to 5 shards each week. 3 replicas per shard (including primary). Real-time implies: cant disable replication during indexing! 8. Getting data in Client configuration Multiple writer threads on multiple machines: currently 6 x 3. Bulk API: currently up to 1000 docs per batch. Incoming docs queued until batch limit, or time or size limits, reached. (e.g. 1000 docs or 100000 bytes or 2 secs since last batch) 9. Getting data in Other things we could do -- but currently dont Tune indexer thread pool size? Tune segment merge policy? Reduce flush interval? Even without these, our current record is over 20,000 docs indexed/sec. (And think the bottleneck was the client machines) 10. Getting data out Typical queries Date histogram and terms facet are the most common by far. So we wrote our own versions with some optimizations :-) https://github.com/pearson-enabling-technologies/elasticsearch-approx-plugin Field data cache size important for speed: currently 30% of 80GB heap. (In fact it actually uses much more than this, with ES 0.90.2. Upgrade planned!) We always use search_type=count. 11. Getting data out Facet workflow Client requestData nodes: Find matching records Perform groupings and counts (and any other calculations) Return to masterArbitrary master node: Parses query Distributes subqueries to data nodes (including itself) Combines results (reduce function) Returns to client 12. Getting data out Facet plugin optimizations Approximate data structures and sampling mode: Trade between speed/memory and accuracy. Uses Lucenes BytesRef & BytesRefHash instead of String & HashSet. Micro-caching of local calculations, e.g. timestamp rounding. Explicit render phase after reduce phase: Defer as much as possible until then. 13. Getting data out General advice for plugin writers Minimize object creation/destruction and type conversions. Use arrays of primitives, or Trove collections, where possible. Reuse buffers. Release objects as soon as possible when no longer needed. Lucene has some neat tricks: bit fields, fast hashing algorithms. So does ElasticSearch: CacheRecycler lets you reuse collections. 14. Getting data out Hints for query performance tuning Tools like jmap, jstat, Visual VM and MAT are very helpful. Use ES hot threads API to see where its spending its time. Set up unit/integration tests with time and RAM instrumentation. 15. Getting data out Other things we could do -- but currently dont Non-data nodes to parse queries, and handle reduce & render phases. Garbage collector tuning. (Note to self: see if Trove still crashes Java 7 JVM under G1 GC) Use SSDs :-) 16. Thanks! Any questions? https://github.com/pearson-enabling-technologies/ https://twitter.com/andrew_clegg