scaling real-time search and analytics with elasticsearch

Download Scaling real-time search and analytics with Elasticsearch

Post on 26-Jan-2015

104 views

Category:

Technology

0 download

Embed Size (px)

DESCRIPTION

See the video here: https://www.youtube.com/watch?v=o6lSeNatVFM A look at the elements required by Elasticsearch to turn a simple inverted index into an auto-clustering, horizontally scalable real time search and analytics engine. The talk will start from first principles, explaining how an inverted index works, how to make an inverted index suitable for real time search, how to scale that out, and how to add reliability and failover to the cluster.

TRANSCRIPT

  • 1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. Clinton Gormley @clintongormley Scaling real time search and analytics with elasticsearch

2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. 3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. elasticsearch.org/guide 4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. elasticsearch 5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. elasticsearch real-time 6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. elasticsearch real-time distributed 7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. elasticsearch real-time distributed search 8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. elasticsearch real-time distributed search analytics 9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. how to use it? 10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. how to use it? 11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. how does it work? 12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. step 1: making text searchable 13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. 14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. where content like %darling%buds% 15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. slow & inexible 16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. 17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. Term Doc1 Doc2 Doc3 breathe brings buds but by can damasked darling date day deaf death declines delight sorted list of unique terms 18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. Term Doc1 Doc2 Doc3 breathe brings buds but by can damasked darling date day deaf death declines delight where they occur 19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. Term Doc1 Doc2 Doc3 breathe brings buds but by can damasked darling date day deaf death declines delight 20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. Term Doc1 Doc2 Doc3 breathe brings buds but by can damasked darling date day deaf death declines delight 21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. inverted index 22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. inverted index term frequencies 23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. inverted index term frequencies relevance 24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. inverted index term frequencies text length relevance 25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. inverted index term frequencies text length relevance doc weight 26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. inverted index term frequencies text length term positions relevance doc weight 27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. inverted index term frequencies text length term positions relevance doc weight word proximity 28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. inverted index term frequencies text length term positions char offsets relevance doc weight word proximity 29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. inverted index term frequencies text length term positions char offsets relevance doc weight word proximity highlighting 30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. inverted index not just for text 31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. inverted index numbers, dates, bools, enums geopoints, geoshapes, etc 32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. step 2: analytics 33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. for search map values doc_ids 34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. for search map values doc_ids for analytics map doc_ids values 35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. uninvert the index 36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. uninvert the index cache values in memory called elddata 37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. uninvert the index data access from RAM very fast 38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. on-the-y analytics in the context of a users query 39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. on-the-y analytics relevant analytics for each user 40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. calculate metrics count, min, max, sum, avg, percentiles, cardinality, stddev, variance, sum of squares ! 41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. grouped by popular terms, signicant terms, ranges, dates, geolocation, etc 42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. grouped by groups can contain subgroups which contain subgroups etc 43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. step 3: building the inverted index 44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. inverted index 45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. immutable 46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. immutable cache friendly 47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. immutable cache friendly reads from RAM 48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. immutable cache friendly reads from RAM fielddata never changes 49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. immutable cache friendly reads from RAM fielddata never changes compressible 50. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. immutable cache friendly reads from RAM fielddata never changes compressible no locking 51. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. but, immutable 52. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. step 4: dynamic inverted index 53. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. in-memory buffer commit segment 54. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. commit point segment searchable 55. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. commit point segment searchable commit 56. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. commit point segment searchable 57. Copyright Elasticsearc