elastic search: beyond ordinary fulltext search (webexpo 2011 prague)

ElasticSearchBeyond Ordinary Fulltext Search

Karel Minařík

ElasticSearch

http://karmi.cz

http://karmi.cz

http://karmi.cz

ElasticSearch

AUDIENCE POLL

Does your application have a search feature?

ElasticSearch

AUDIENCE POLL

What do you use for search?

1. SELECT ... LIKE %foo%

2. Sphinx3. Apache Solr4. ElasticSearch

ElasticSearch

Search is the primary interfacefor getting information today.

http://www.apple.com/macosx/what-is-macosx/spotlight.html



http://www.apple.com/iphone/features/search.html



#uxfail???

Y U NO ALIGN???

ElasticSearch

Search is hard.Let's go write SQL queries!

How do you implement search?WHY SEARCH SUCKS?

def search @results = MyModel.search params[:q] respond_with @resultsend



ResultResultsQuery

MAGIC


ResultResultsQuery


MAGIC + /

A personal story...

670px

23px

MyModel.search "(this OR that) AND NOT whatever"

Arel::Table.new(:articles). where(articles[:title].eq('On Search')). where(["published_on => ?", Time.now]). join(comments). on(article[:id].eq(comments[:article_id])) take(5). skip(4). to_sql

Compare your search library with your ORM libraryWHY SEARCH SUCKS?

ElasticSearch

How does search work?

A collection of documentsHOW DOES SEARCH WORK?

file_1.txtThe ruby is a pink to blood-‐red colored gemstone ...

file_2.txtRuby is a dynamic, reflective, general-‐purpose object-‐oriented programming language ...

file_3.txt"Ruby" is a song by English rock band Kaiser Chiefs ...

How do you search documents?HOW DOES SEARCH WORK?

File.read('file_1.txt').include?('ruby')File.read('file_2.txt').include?('ruby')...

The inverted indexHOW DOES SEARCH WORK?

http://en.wikipedia.org/wiki/Index_(search_engine)#Inverted_indices

TOKENS POSTINGS

ruby file_1.txt file_2.txt file_3.txt

pink file_1.txt

gemstone file_1.txt

dynamic file_2.txt

reflective file_2.txt

programming file_2.txt

song file_3.txt

english file_3.txt

rock file_3.txt






pink file_1.txt

gemstone file_1.txt

dynamic file_2.txt



song file_3.txt

english file_3.txt

rock file_3.txt

MySearchLib.search "ruby"





pink file_1.txt

gemstone file_1.txt

dynamic file_2.txt



song file_3.txt

english file_3.txt

rock file_3.txt

MySearchLib.search "song"







pink file_1.txt

gemstone file_1.txt

dynamic file_2.txt



english file_3.txt

rock file_3.txt

MySearchLib.search "ruby AND song"

song file_3.txt



module SimpleSearch

def index document, content tokens = analyze content store document, tokens puts "Indexed document #{document} with tokens:", tokens.inspect, "\n" end

def analyze content # >>> Split content by words into "tokens" content.split(/\W/). # >>> Downcase every word map { |word| word.downcase }. # >>> Reject stop words, digits and whitespace reject { |word| STOPWORDS.include?(word) || word =~ /^\d+/ || word == '' } end

def store document_id, tokens tokens.each do |token| # >>> Save the "posting" ( (INDEX[token] ||= []) << document_id ).uniq! end end

def search token puts "Results for token '#{token}':" # >>> Print documents stored in index for this token INDEX[token].each { |document| " * #{document}" } end

INDEX = {} STOPWORDS = %w|a an and are as at but by for if in is it no not of on or that the then there these this to with|

extend self

end

A naïve Ruby implementation

SimpleSearch.index "file1", "Ruby is a language. Java is also a language."SimpleSearch.index "file2", "Ruby is a song."SimpleSearch.index "file3", "Ruby is a stone."SimpleSearch.index "file4", "Java is a language."

Indexed document file1 with tokens:["ruby", "language", "java", "also", "language"]

Indexed document file2 with tokens:["ruby", "song"]

Indexed document file3 with tokens:["ruby", "stone"]

Indexed document file4 with tokens:["java", "language"]

Indexing documentsHOW DOES SEARCH WORK?

Words downcased,stopwords removed.

puts "What's in our index?"

p SimpleSearch::INDEX

{ "ruby" => ["file1", "file2", "file3"], "language" => ["file1", "file4"], "java" => ["file1", "file4"], "also" => ["file1"], "stone" => ["file3"], "song" => ["file2"]}

The indexHOW DOES SEARCH WORK?

SimpleSearch.search "ruby"

Results for token 'ruby':* file1* file2* file3

Search the indexHOW DOES SEARCH WORK?



TOKENS POSTINGS


pink file_1.txt

gemstone file_1.txt

dynamic file_2.txt



song file_3.txt

english file_3.txt

rock file_3.txt

31

ElasticSearch

It is very practical to know how search works.

For instance, now you know thatthe analysis step is very important.

It's more important than the “search” step.

module SimpleSearch

def index document, content tokens = analyze content store document, tokens puts "Indexed document #{document} with tokens:", tokens.inspect, "\n" end

def analyze content # >>> Split content by words into "tokens" content.split(/\W/). # >>> Downcase every word map { |word| word.downcase }. # >>> Reject stop words, digits and whitespace reject { |word| STOPWORDS.include?(word) || word =~ /^\d+/ || word == '' } end

def store document_id, tokens tokens.each do |token| # >>> Save the "posting" ( (INDEX[token] ||= []) << document_id ).uniq! end end

def search token puts "Results for token '#{token}':" # >>> Print documents stored in index for this token INDEX[token].each { |document| " * #{document}" } end

INDEX = {} STOPWORDS = %w|a an and are as at but by for if in is it no not of on or that the then there these this to with|

extend self

end A naïve Ruby implementation

http://search-engines-book.com

Search EnginesInformation Retrieval in PracticeBruce Croft, Donald Metzler and Trevor StrohmaAddison Wesley, 2009

The Search Engine TextbookHOW DOES SEARCH WORK?

http://www.search-engines-book.com/

http://www.search-engines-book.com/

Lucene in ActionMichael McCandless, Erik Hatcher and Otis GospodneticJuly, 2010

The Baseline Information Retrieval ImplementationSEARCH IMPLEMENTATIONS

http://manning.com/hatcher3

http://www.manning.com/hatcher3/

http://www.manning.com/hatcher3/

http://elasticsearch.org



ElasticSearch

ElasticSearch is an open source, scalable, distributed, cloud-ready, highly-available full-text search engine and database with powerfull aggregation features, communicating by JSON over RESTful HTTP, based on Apache Lucene.

http://lucene.apache.org/

http://lucene.apache.org/

ElasticSearch

HTTPJSONSchema-freeIndex as ResourceDistributedQueriesFacetsMappingRuby

{ }

# Add a documentcurl -‐X POST \

"http://localhost:9200/articles/article/1" \

-‐d '{ "title" : "One" }'

INDEX TYPE ID

ELASTICSEARCH FEATURES

HTTP / JSON / Schema-free / Index as Resource / Distributed / Queries / Facets / Mapping / Ruby

DOCUMENT


# Add a documentcurl -‐X POST "http://localhost:9200/articles/article/1" -‐d '{ "title" : "One" }'

# Perform querycurl -‐X GET "http://localhost:9200/articles/_search?q=One"curl -‐X POST "http://localhost:9200/articles/_search" -‐d '{ "query" : { "terms" : { "tags" : ["ruby", "python"], "minimum_match" : 2 } }}'

# Delete indexcurl -‐X DELETE "http://localhost:9200/articles"

# Create index with settings and mappingcurl -‐X PUT "http://localhost:9200/articles" -‐d '{ "settings" : { "index" : "number_of_shards" : 3, "number_of_replicas" : 2 }},{ "mappings" : { "document" : { "properties" : { "body" : { "type" : "string", "analyzer" : "snowball" } } } }}'


http {

server {

listen 8080; server_name search.example.com;

error_log elasticsearch-‐errors.log; access_log elasticsearch.log;

location / {

# Deny access to Cluster API if ($request_filename ~ "_cluster") { return 403; break; }

# Pass requests to ElasticSearch proxy_pass http://localhost:9200; proxy_redirect off; proxy_set_header X-‐Real-‐IP $remote_addr; proxy_set_header X-‐Forwarded-‐For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host;

# Authorize access auth_basic "ElasticSearch"; auth_basic_user_file passwords;

# Route all requests to authorized user's own index rewrite ^(.*)$ /$remote_user$1 break; rewrite_log on;

return 403; }

}}



GET http://user:password@localhost:8080/_search?q=* => http://localhost:9200/user/_search?q=*

https://gist.github.com/986390

#664 Add HTTPS and basic authentication support NO.



https://github.com/elasticsearch/elasticsearch/issues/664






{ "id" : "abc123",

"title" : "ElasticSearch Understands JSON!",

"body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...",

"published_on" : "2011/05/27 10:00:00", "tags" : ["search", "json"],

"author" : { "first_name" : "Clara", "last_name" : "Rice", "email" : "[email protected]" }}

JSON

curl -‐X DELETE "http://localhost:9200/articles"; sleep 1curl -‐X POST "http://localhost:9200/articles/article" -‐d '

{ "id" : "abc123",




"author" : { "first_name" : "Clara", "last_name" : "Rice", "email" : "[email protected]" }}'curl -‐X POST "http://localhost:9200/articles/_refresh"

curl -‐X GET \ "http://localhost:9200/articles/article/_search?q=author.first_name:clara"



curl -‐X GET "http://localhost:9200/articles/_mapping?pretty=true"{ "articles" : { "article" : { "properties" : { "title" : { "type" : "string" }, // ... "author" : { "dynamic" : "true", "properties" : { "first_name" : { "type" : "string" }, // ... } }, "published_on" : { "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd", "type" : "date" } } } }}



curl -‐X POST "http://localhost:9200/articles/article" -‐d '..."published_on" : "2011/05/27 10:00:00",...

curl -‐X POST "http://localhost:9200/articles/comment" -‐d '{ "body" : "Wow! Really nice JSON support.",

"published_on" : "2011/05/27 10:05:00",

"author" : { "first_name" : "John", "last_name" : "Pear", "email" : "[email protected]" }}'curl -‐X POST "http://localhost:9200/articles/_refresh"

curl -‐X GET \ "http://localhost:9200/articles/comment/_search?q=author.first_name:john"

HTTP / JSON / Schema Free / Index as Resource / Distributed / Queries / Facets / Mapping / Ruby


DIFFERENT TYPE

curl -‐X GET \ "http://localhost:9200/articles/comment/_search?q=body:json"

curl -‐X GET \ "http://localhost:9200/articles/_search?q=body:json"

curl -‐X GET \ "http://localhost:9200/articles,users/_search?q=body:json"

curl -‐X GET \ "http://localhost:9200/_search?q=body:json"



Search single type

Search whole index

Search multiple indices

Search all indices



curl -‐X DELETE "http://localhost:9200/articles"; sleep 1

curl -‐X POST "http://localhost:9200/articles/article" -‐d '{ "id" : "abc123",




"author" : { "first_name" : "Clara", "last_name" : "Rice", "email" : "[email protected]" }}'curl -‐X POST "http://localhost:9200/articles/_refresh"

curl -‐X GET "http://localhost:9200/articles/article/abc123"



{"_index":"articles","_type":"article","_id":"1","_version":1, "_source" : { "id" : "1",




"author" : { "first_name" : "Clara", "last_name" : "Rice", "email" : "[email protected]" }}}

“The Index Is Your Database”



http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html

my_alias

index_A

index_B

curl -‐X POST 'http://localhost:9200/_aliases' -‐d '{ "actions" : [ { "add" : { "index" : "index_1", "alias" : "myalias" } }, { "add" : { "index" : "index_2", "alias" : "myalias" } } ]}'

Index Aliases





logs

The “Sliding Window” problem

logs_2010_02

logs_2010_03

logs_2010_04

curl -‐X DELETE http://localhost:9200 / logs_2010_01

“We can really store only three months worth of data.”

curl -‐X PUT localhost:9200/_template/bookmarks_template -‐d '{ "template" : "users_*",

"settings" : { "index" : { "number_of_shards" : 1, "number_of_replicas" : 3 } },

"mappings": { "url": { "properties": { "url": { "type": "string", "analyzer": "url_ngram", "boost": 10 }, "title": { "type": "string", "analyzer": "snowball", "boost": 5 } // ... } } }}'



Apply this configurationfor every matchingindex being created

http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html

Index Templates





Node 1 Node 2 Node 3 Node 4MASTER

Automatic Discovery Protocol

http://www.elasticsearch.org/guide/reference/modules/discovery/

$ cat elasticsearch.yml

cluster: name: <YOUR APPLICATION>





A

A1

A2

A3

A1'

A2'

A3'

A1''

A2''

A3''

Replicas

Shards

curl -‐XPUT 'http://localhost:9200/A/' -‐d '{ "settings" : { "index" : { "number_of_shards" : 3, "number_of_replicas" : 2 } }}'

Index is split into 3 shards, and duplicated in 2 replicas.

Improve indexing performance



SHARDS

REPLICAS

Impro

ve search perfo

rmance



Y U NO ASK FIRST???



Indexing 100 000 documents (~ 56MB), one shard, no replicas, MacBookAir SSD 2GB

# Index all at oncetime curl -‐s -‐X POST "http://localhost:9200/_bulk" \ -‐-‐data-‐binary @data/bulk_all.json > /dev/null

real 2m1.142s

# Index in batches of 1000for file in data/bulk_*.json; do time curl -‐s -‐X POST "http://localhost:9200/_bulk" \ -‐-‐data-‐binary @$file > /dev/nulldone

real 1m36.697s (-‐25sec, 80%)

# Do not refresh during indexing in batches"settings" : { "refresh_interval" : "-‐1" }for file in data/bulk_*.json; do...real 0m38.859s (-‐82sec, 32%)

HTTP / JSON / Schema Free / Distributed / Queries / Facets / Mapping / Ruby


Terms appleapple iphone

Phrases "apple iphone"

Proximity "apple safari"~5

Fuzzy apple~0.8

Wildcards app**pp*

Boosting apple^10 safari

Range [2011/05/01 TO 2011/05/31][java TO json]

Boolean

apple AND NOT iphone+apple -‐iphone(apple OR iphone) AND NOT review

Fieldstitle:iphone^15 OR body:iphonepublished_on:[2011/05/01 TO "2011/05/27 10:00:00"]

http://lucene.apache.org/java/3_1_0/queryparsersyntax.html

$ curl -‐X GET "http://localhost:9200/_search?q=<YOUR QUERY>"



curl -‐X POST "http://localhost:9200/articles/_search?pretty=true" -‐d '{ "query" : { "terms" : { "tags" : [ "ruby", "python" ], "minimum_match" : 2 } }}'



Query DSL

http://www.elasticsearch.org/guide/reference/query-dsl/

JSON



curl -‐X POST "http://localhost:9200/venues/venue" -‐d '{ "name": "Pizzeria", "pin": { "location": { "lat": 50.071712, "lon": 14.386832 } }}'

curl -‐X POST "http://localhost:9200/venues/_search?pretty=true" -‐d '{

"query" : { "filtered" : { "query" : { "query_string" : { "query" : "pizzeria" } }, "filter" : { "geo_distance" : { "distance" : "0.5km", "pin.location" : { "lat" : 50.071481, "lon" : 14.387284 } } } } }}'



Accepted formats for Geo:

[lon, lat] # Array

"lat,lon" # String

drm3btev3e86 # Geohash

Geo Search

http://www.elasticsearch.org/guide/reference/query-dsl/geo-distance-filter.html





http://blog.linkedin.com/2009/12/14/linkedin-faceted-search/

Query

Facets



curl -‐X POST "http://localhost:9200/articles/_search?pretty=true" -‐d '{ "query" : { "query_string" : { "query" : "title:T*"} }, "filter" : { "terms" : { "tags" : ["ruby"] } }, "facets" : { "tags" : { "terms" : { "field" : "tags", "size" : 10 } } }}'

# facets" : {# "tags" : {# "terms" : [ {# "term" : "ruby",# "count" : 2# }, {# "term" : "python",# "count" : 1# }, {# "term" : "java",# "count" : 1# } ]# }# }

User query

“Checkboxes”

Facets

http://www.elasticsearch.org/guide/reference/api/search/facets/index.html





curl -‐X POST "http://localhost:9200/articles/_search?pretty=true" -‐d '{ "facets" : { "published_on" : { "date_histogram" : { "field" : "published", "interval" : "day" } } }}'

http://www.elasticsearch.org/blog/2011/05/13/data-visualization-with-elasticsearch-and-protovis.html



Geo Facetscurl -‐X POST "http://localhost:9200/venues/_search?pretty=true" -‐d '{ "query" : { "query_string" : { "query" : "pizzeria" } }, "facets" : { "distance_count" : { "geo_distance" : { "pin.location" : { "lat" : 50.071712, "lon" : 14.386832 }, "ranges" : [ { "to" : 1 }, { "from" : 1, "to" : 5 }, { "from" : 5, "to" : 10 } ] } } }}'



http://www.elasticsearch.org/guide/reference/api/search/facets/geo-distance-facet.html



def analyze content # >>> Split content by words into "tokens" content.split(/\W/). # >>> Downcase every word map { |word| word.downcase }. # ... end



Remember?

curl -‐X DELETE "http://localhost:9200/articles"curl -‐X POST "http://localhost:9200/articles/article" -‐d '{ "mappings": { "article": { "properties": { "tags": { "type": "string", "analyzer": "keyword" }, "content": { "type": "string", "analyzer": "snowball" }, "title": { "type": "string", "analyzer": "snowball", "boost": 10.0 } } } }}'

curl -‐X GET 'http://localhost:9200/articles/_mapping?pretty=true'

http://www.elasticsearch.org/guide/reference/api/admin-indices-create-index.html





curl -‐X DELETE "http://localhost:9200/urls"curl -‐X POST "http://localhost:9200/urls/url" -‐d '{ "settings" : { "index" : { "analysis" : { "analyzer" : { "url_analyzer" : { "type" : "custom", "tokenizer" : "lowercase", "filter" : ["stop", "url_stop", "url_ngram"] } }, "filter" : { "url_stop" : { "type" : "stop", "stopwords" : ["http", "https", "www"] }, "url_ngram" : { "type" : "nGram", "min_gram" : 3, "max_gram" : 5 } } } } }}'




HTTP / JSON / Schema Free / Distributed / Queries / Facets / Mapping / RubyELASTICSEARCH FEATURES

Tire.index 'articles' do delete create

store :title => 'One', :tags => ['ruby'], :published_on => '2011-‐01-‐01' store :title => 'Two', :tags => ['ruby', 'python'], :published_on => '2011-‐01-‐02' store :title => 'Three', :tags => ['java'], :published_on => '2011-‐01-‐02' store :title => 'Four', :tags => ['ruby', 'php'], :published_on => '2011-‐01-‐03'

refreshend

s = Tire.search 'articles' do query { string 'title:T*' }

filter :terms, :tags => ['ruby']

sort { title 'desc' }

facet 'global-‐tags' { terms :tags, :global => true }

facet 'current-‐tags' { terms :tags }end

http://github.com/karmi/tire

https://github.com/karmi/tire


class Article < ActiveRecord::Base include Tire::Model::Search include Tire::Model::Callbacksend



Article.search do query { string 'love' } facet('timeline') { date :published_on, :interval => 'month' } sort { published_on 'desc' }end

$ rake environment tire:import CLASS='Article'



class Article include Whatever::ORM

include Tire::Model::Search include Tire::Model::Callbacksend



Article.search do query { string 'love' } facet('timeline') { date :published_on, :interval => 'month' } sort { published_on 'desc' }end

$ rake environment tire:import CLASS='Article'



$ rails new tired -‐m "https://gist.github.com/raw/951343/tired.rb"

A “batteries included” installation.Downloads and launches ElasticSearch.Sets up a Rails applicationand and launches it.When you're tired of it, just delete the folder.

Try ElasticSearch in a Ruby On Rails aplication with a one-line command

https://gist.github.com/raw/951343/tired.rb

https://gist.github.com/raw/951343/tired.rb

Thanks!d

elastic search: beyond ordinary fulltext search (webexpo 2011 prague)

Technology

search ruby ruby

search sucks

search song ruby

search params

search step

search library

search rubyresults

search works