scaling massive elastic search clusters - rafał kuć - sematext
DESCRIPTION
Rafał Kuć presentation on "Scaling Massive ElasticSearch Clusters" given during Berlin Buzzwords 2012TRANSCRIPT
![Page 1: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/1.jpg)
Scaling Massive ElasticSearch
Clusters
Rafał Kuć – Sematext International
@kucrafal @sematext sematext.com
![Page 2: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/2.jpg)
Who Am I
• „Solr 3.1 Cookbook” author
• Sematext software engineer
• Solr.pl co-founder
• Father and husband :-)
Copyright 2012 Sematext Int’l. All rights reserved
![Page 3: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/3.jpg)
What Will I Talk About ?
• ElasticSearch scaling
• Indexing thousands of documents per second
• Performing queries in tens of milliseconds
• Controling shard and replica placement
• Handling multilingual content
• Performance testing
• Cluster monitoring
Copyright 2012 Sematext Int’l. All rights reserved
![Page 4: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/4.jpg)
The Challenge
• More than 50 millions of documents a day
• Real time search
• Less than 200ms average query latency
• Throughput of at least 1000 QPS
• Multilingual indexing
• Multilingual querying
Copyright 2012 Sematext Int’l. All rights reserved
![Page 5: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/5.jpg)
Why ElasticSearch ?
• Written with NRT and cloud support in mind
• Uses Lucene and all its goodness
• Distributed indexing with document
distribution control out of the box
• Easy index, shard and replicas creation on live
cluster
Copyright 2012 Sematext Int’l. All rights reserved
![Page 6: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/6.jpg)
Index Design
• Several indices (at least one index for each day
of data)
• Indices divided into multiple shards
• Multiple replicas of a single shard
• Real-time, synchronous replication
• Near-real-time index refresh (1 to 30 seconds)
Copyright 2012 Sematext Int’l. All rights reserved
![Page 7: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/7.jpg)
Shard Deployment Problems
• Multiple shards per node
• Replicas on the same nodes as shards
• Not evenly distributed shards and replicas
• Some nodes being hot, while others are cold
Copyright 2012 Sematext Int’l. All rights reserved
![Page 8: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/8.jpg)
Default Shard Deployment
ElasticSearch Cluster
Node 1 Node 2
Node 3
Shard 1 Shard 2 Shard 3 Replica 1
Replica 2
Replica 3
Copyright 2012 Sematext Int’l. All rights reserved
![Page 9: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/9.jpg)
What Can We Do With Shards Then ?
• Contol shard placement with node tags:
– index.routing.allocation.include.tag
– index.routing.allocation.exclude.tag
• Control shard placement with nodes IP addresses:
– cluster.routing.allocation.include._ip
– cluster.routing.allocation.exclude._ip
• Specified on index or cluster level
• Can be changed on live cluster !
Copyright 2012 Sematext Int’l. All rights reserved
![Page 10: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/10.jpg)
Shard Allocation Examples
• Cluster level:
curl -XPUT localhost:9200/_cluster/settings -d '{
"persistent" : {
"cluster.routing.allocation.exclude._ip" : "192.168.2.1"
}
}'
• Index level:
curl -XPUT localhost:9200/sematext/ -d '{
"index.routing.allocation.include.tag" : "nodeOne,nodeTwo"
}'
Copyright 2012 Sematext Int’l. All rights reserved
![Page 11: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/11.jpg)
Number of Shards Per Node
• Allows one to specify number of shards per
node
• Specified on index level
• Can be changed on live indices
• Example:
curl -XPUT localhost:9200/sematext -d '{
"index.routing.allocation.total_shards_per_node" : 2
}'
Copyright 2012 Sematext Int’l. All rights reserved
![Page 12: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/12.jpg)
Controlled Shard Deployment
ElasticSearch Cluster
Node 1 Node 2
Node 3
Shard 1
Shard 2
Shard 3 Replica 1Replica 2
Replica 3
Copyright 2012 Sematext Int’l. All rights reserved
![Page 13: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/13.jpg)
Does Routing Matters ?
• Controls target shard for each document
• Defaults to hash of a document identifier
• Can be specified explicitly (routing parameter) oras a field value (a bit less performant)
• Can take any value
• Example:
curl -XPUT localhost:9200/sematext/test/1?routing=1234 -d '{
"title" : "Test routing document"
}'
Copyright 2012 Sematext Int’l. All rights reserved
![Page 14: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/14.jpg)
Indexing the Data
ElasticSearch Cluster
Node 1 Node 2
Node 3
Shard
1
Shard
2
Shard
3
Replica
1
Replica
2
Replica
3
Indexing application
Copyright 2012 Sematext Int’l. All rights reserved
![Page 15: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/15.jpg)
How We Indexed Data
ElasticSearch Cluster
Node 1 Node 2
Shard 1 Shard 2
Node 3
Indexing application
Copyright 2012 Sematext Int’l. All rights reserved
![Page 16: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/16.jpg)
Nodes Without Data
• Nodes used only to route data and queries to
other nodes in the cluster
• Such nodes don’t suffer from I/O waits (of
course Data Nodes don’t suffer from I/O waits
all the time)
• Not default ElasticSearch behavior
• Setup by setting node.data to false
Copyright 2012 Sematext Int’l. All rights reserved
![Page 17: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/17.jpg)
Multilingual Indexing
• Detection of document's language before
sending it for indexing
• With, e.g. Sematext LangID or Apache Tika
• Set known language analyzers in configuration
or mappings
• Set analyzer during indexing (_analyzer field)
Copyright 2012 Sematext Int’l. All rights reserved
![Page 18: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/18.jpg)
Multilingual Indexing Example
curl -XPUT localhost:9200/sematext/test/10 -d '{
"title" : "Test document",
"langId" : "english"
}'
{
"test" : {
"_analyzer" : { "path" : "langId" },
"properties" : {
"id" : { "type" : "long", "store" : "yes", "precision_step" : "0" },
"title" : { "type" : "string", "store" : "yes", "index" : "analyzed" },
"langId" : { "type" : "string", "store" : "yes", "index" : "not_analyzed" }
}
}
}
Copyright 2012 Sematext Int’l. All rights reserved
![Page 19: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/19.jpg)
Multilingual Queries
• Identify language of query before its execution
(can be problematic)
• Query analyzer can be specified per query
(analyzer parameter):curl -XGET
localhost:9200/sematext/_search?q=let+AND+me&analyzer=english
Copyright 2012 Sematext Int’l. All rights reserved
![Page 20: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/20.jpg)
Query Performance Factors – Lucene
level
• Refresh interval
– Defaults to 1 second
– Can be specified on cluster or index level
– curl -XPUT localhost:9200/_settings -d '{ "index" : { "refresh_interval" : "600s" } }'
• Merge factor
– Defaults to 10
– Can be specified on cluster or index level
– curl -XPUT localhost:9200/_settings -d '{ "index" : { "merge.policy.merge_factor" : 30 } }'
Copyright 2012 Sematext Int’l. All rights reserved
![Page 21: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/21.jpg)
Let’s Talk About Routing Once Again
• Routes a query to a particular shard
• Speeds up queries depending on number of shards for a given index
• Have to be specified manualy with routingparameter during query
• routing parameter can take any value:
curl -XGET 'localhost:9200/sematext/_search?q=test&routing=2012-02-16'
Copyright 2012 Sematext Int’l. All rights reserved
![Page 22: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/22.jpg)
Querying ElasticSearch – No Routing
Shard 1 Shard 2 Shard 3 Shard 4
Shard 5 Shard 6 Shard 7 Shard 8
ElasticSearch Index
Application
Copyright 2012 Sematext Int’l. All rights reserved
![Page 23: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/23.jpg)
Shard 1 Shard 2 Shard 3 Shard 4
Shard 5 Shard 6 Shard 7 Shard 8
ElasticSearch Index
Application
Querying ElasticSearch – With Routing
Copyright 2012 Sematext Int’l. All rights reserved
![Page 24: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/24.jpg)
Performance Numbers
Queries without routing (200 shards, 1 replica)
#threads Avg response time Throughput 90% line Median CPU Utilization
1 3169ms 19,0/min 5214ms 2692ms 95 – 99%
Queries with routing (200 shards, 1 replica)
#threads Avg response time Throughput 90% line Median CPU Utilization
10 196ms 50,6/sec 642ms 29ms 25 – 40%
20 218ms 91,2/sec 718ms 11ms 10 – 15%
Copyright 2012 Sematext Int’l. All rights reserved
![Page 25: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/25.jpg)
Scaling Query Throughput – What Else ?
• Increasing the number of shards for data
distribution
• Increasing the number of replicas
• Using routing
• Avoid always hitting the same node and
hotspotting it
Copyright 2012 Sematext Int’l. All rights reserved
![Page 26: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/26.jpg)
FieldCache and OutOfMemory
• ElasticSearch default setup doesn’t limit field
data cache size
Copyright 2012 Sematext Int’l. All rights reserved
![Page 27: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/27.jpg)
FieldCache – What We Can do With It ?
• Keep its default type and set:– Maximum size (index.cache.field.max_size)
– Expiration time (index.cache.field.expire)
• Change its type:– soft (index.cache.field.type)
• Change your data:– Make your fields less precise (ie: dates)
– If you sort or facet on fields think if you can reducefields granularity
• Buy more servers :-)
Copyright 2012 Sematext Int’l. All rights reserved
![Page 28: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/28.jpg)
FieldCache After Changes
Copyright 2012 Sematext Int’l. All rights reserved
![Page 29: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/29.jpg)
Additional Problems We Encountered
• Rebalancing after full cluster restarts
– cluster.routing.allocation.disable_allocation
– cluster.routing.allocation.disable_replica_allocation
• Long startup and initialization
• Faceting with strings vs faceting on numbers on
high cardinality fields
Copyright 2012 Sematext Int’l. All rights reserved
![Page 30: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/30.jpg)
JVM Optimization
• Remember to leave enough memory to OS for
cache
• Make GC frequent ans short vs. rare and long
– -XX:+UseParNewGC
– -XX:+UseConcMarkSweepGC
– -XX:+CMSParallelRemarkEnabled
• -XX:+AlwaysPreTouch (for short performance
tests)
Copyright 2012 Sematext Int’l. All rights reserved
![Page 31: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/31.jpg)
Performance Testing
• Data
– How much data do I need ?
– Choosing the right queries
• Make changes
– One change at a time
– Understand the impact of the change
• Monitor your cluster (jstat, dstat/vmstat, SPM)
• Analyze your results
Copyright 2012 Sematext Int’l. All rights reserved
![Page 32: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/32.jpg)
ElasticSearch Cluster Monitoring
• Cluster health
• Indexing statistics
• Query rate
• JVM memory and garbage collector work
• Cache usage
• Node memory and CPU usage
Copyright 2012 Sematext Int’l. All rights reserved
![Page 33: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/33.jpg)
Cluster Health
Copyright 2012 Sematext Int’l. All rights reserved
Node restart
![Page 34: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/34.jpg)
Indexing Statistics
Copyright 2012 Sematext Int’l. All rights reserved
![Page 35: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/35.jpg)
Query Rate
Copyright 2012 Sematext Int’l. All rights reserved
![Page 36: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/36.jpg)
JVM Memory and GC
Copyright 2012 Sematext Int’l. All rights reserved
![Page 37: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/37.jpg)
Cache Usage
Copyright 2012 Sematext Int’l. All rights reserved
![Page 38: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/38.jpg)
CPU and Memory
Copyright 2012 Sematext Int’l. All rights reserved
![Page 39: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/39.jpg)
Summary
• Controlling shard and replica placement
• Indexing and querying multilingual data
• How to use sharding and routing and not to
tear your hair out
• How to test your cluster performance to find
bottle-necks
• How to monitor your cluster and find
problems right away
Copyright 2012 Sematext Int’l. All rights reserved
![Page 40: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/40.jpg)
We Are Hiring !
• Dig Search ?
• Dig Analytics ?
• Dig Big Data ?
• Dig Performance ?
• Dig working with and in open – source ?
• We’re hiring world – wide !
http://sematext.com/about/jobs.html
Copyright 2012 Sematext Int’l. All rights reserved
![Page 41: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/41.jpg)
How to Reach Us
• Rafał Kuć
– Twitter: @kucrafal
– E-mail: [email protected]
• Sematext
– Twitter: @sematext
– Website: http://sematext.com
• Graphs used in the presentation are from:
– SPM for ElasticSearch (http://sematext.com/spm)
Copyright 2012 Sematext Int’l. All rights reserved
![Page 42: Scaling massive elastic search clusters - Rafał Kuć - Sematext](https://reader033.vdocuments.net/reader033/viewer/2022051207/540e3a708d7f72747e8b4cba/html5/thumbnails/42.jpg)
Thank You For Your Attention