solr and elasticsearch, a performance study

Tom Mortimer - Technical Director27th November 2014

[email protected]/blog+44 (0) 8700 118334Twitter: @FlaxSearch

Elasticsearch and SolrCloud

a performance comparison

We design, build and support open source powered search applications

Who are Flax?


Based in Cambridge U.K., technology agnostic & independent – but open source exponents & committers

Who are Flax?



UK Authorized Partner of

Who are Flax?




Customers include Reed Specialist Recruitment, Mydeco, NLA, Gorkana, Financial Times, News UK, EMBL-EBI, Accenture, University of Cambridge, UK Government...

Who are Flax?




Customers in recruitment, government, e-commerce, news & media, bioinformatics, consulting, law...

Who are Flax?

Open source search server based on Lucene Created in 2004 by Yonik Seeley Became an Apache project in 2006 Merged with Lucene in 2011 Web API XML config, XML/JSON data formats SolrCloud features added in 2012 Uses Apache ZooKeeper for cluster management

Open source search server based on Lucene Created in 2010 by Shay Banon RESTful Web API Everything is JSON Distributed and NRT by design Own Zen Discovery module for cluster management

Both have large, dynamic communities Well-funded commercial backing Widely used in many diverse projects Elasticsearch easier to setup and configure Elasticsearch query DSL But: is Elasticsearch as tolerant of network faults?

(Jepsen tests by Kyle Kingsbury) How does performance compare?

vs.

Both have large, dynamic communities Well-funded commercial backing Widely used in many diverse projects Elasticsearch easier to setup and configure Elasticsearch query DSL But: is Elasticsearch as tolerant of network faults?

(Jepsen tests by Kyle Kingsbury) How does performance compare? Note that we don't have a preference...we use both!

vs.

Won't it be the same, as they both use Lucene? Can't you just throw hardware at it? Hardware is cheaper than developers

Why does performance matter?

Won't it be the same, as they both use Lucene? Can't you just throw hardware at it? Hardware is cheaper than developers

Well, no.


There's a lot more to them than just a web API on top of Lucene.

Several of our customers have fixed hardware budgets May have to use limited internal resources With large indexes or complex queries, need to squeeze

every last bit of performance out of the hardware


Not many found by a Google search.

http://blog.socialcast.com/realtime-search-solr-vs-elasticsearch/

Solr much faster than Elasticsearch, except for NRT searches with concurrent indexing (where situation was reversed).

But: This was over 3 years ago, before SolrCloud

What performance studies are out there?

http://blog.socialcast.com/realtime-search-solr-vs-elasticsearch/

Client with complex filtering requirements for content licensing, 10Ms of documents, limited hardware budget, no NRT requirement.

Performed tests 18 months ago on EC2. Solr was approximately 20 times faster!

More recently, Solr was 4 times faster for a project requiring geospatial filtering

What about now?

Our experience

Recent versions of Elasticsearch (1.4.0) and Solr (4.10.2) Concentrated on indexing performance, query times with

and without concurrent indexing, QPS, filters and facets.

Hardware kindly provided by BigStep.com Full Metal Cloud (real instances, not VMs) Optimised for high performance Can be faster than your own dedicated hardware!

This study

http://www.bigstep.com/


The results?

Not really very interesting

The results?

Not really very interesting SolrCloud and Elasticsearch were both very fast Similar performance with concurrent indexing or not Solr could handle higher QPS

The results?

Cluster configuration Two machines, each with 96GB RAM Two instances of SolrCloud or Elasticsearch on

each Each instance has 24GB JVM heap Four shards No replicas

Cluster configuration in BigStep

Data 40M documents created by using a Markov chain on a

seed document (on Stoicism) from gutenberg.org

“Below planets. this Below lay this the lay infinite the void infinite without void beginning, without middle, beginning, or middle, end, or this end occupied...”

Small (5-20 word) and larger (200-1000 word) docs Randomly assigned ints for “source” and “level”, to

simulate licensing filters and for facets.

Indexing Python script and requests library Single process for small index, four processes for

larger index Single process for indexing concurrent with search

Searching Python and requests Each query time logged for analysis Single process for query time testing Multiple processes to test QPS All tests performed warm

Queries consisted of three randomly chosen terms combined with OR

Filters randomly generated Facets / Elasticsearch aggregations

40M Small documents

Elasticsearch indexed them in 30 minutes Total index size was 8.8 GB (easily cacheable)

Solr indexed them in 43 minutes Total index size was 7.6 GB

40M Small documents (concurrent indexing)

Elasticsearch: 0.01s mean, 99% < 0.06sSolr: 0.01s mean, 99% < 0.10s

40M Large documents

Elasticsearch indexed them in 179 minutes Total index size was 363 GB (not completely

cacheable)

Solr indexed them in 119 minutes Total index size was 226 GB

40M Large documents (search with facets)


40M Large documents (with 10 filters)


40M Large documents (concurrent indexing)


40M Large documents (QPS)

Conclusions

SolrCloud seems to be slightly faster. However, performance was acceptable in all cases.

SolrCloud can apparently support a significantly higher number of queries per second (tested without concurrent indexing, however).

Limitations and problems Validity of generated documents? Validity of random queries? Searches did not fetch any document data Did not test highlighting, range facets, geolocation,

etc. etc... Only tested one type of cluster configuration

(Elasticsearch is very flexible about node role). Did not tune JVM parameters Did not perform profiling to identify reasons for

differences

What's next Would have also liked to have compared BigStep with

Amazon EC2. If there is any interest, I hope to address some of

these problems in the near future. We'll open source the code (next week?) onwww.github.com/flaxsearch

What to take away from this? Elasticsearch and Solr are both awesome They currently seem very close in terms of

performance (according to this limited study)

What to take away from this? Elasticsearch and Solr are both awesome They currently seem very close in terms of

performance (according to this limited study)

However, all search applications are different Solr and Elasticsearch may have quite different

performance characteristics in certain cases. Hard to predict.

If performance is important to you, it will pay to try both.

Thanks! To you, for listening To for the use of Full Metal Cloud

Any questions? - [email protected]


solr and elasticsearch, a performance study

Software