discovering elasticsearch

Download Discovering ElasticSearch

Post on 13-Nov-2014

640 views

Category:

Technology

3 download

Embed Size (px)

DESCRIPTION

My introduction to ElasticSearch at Laracon EU 2014, where I explain the ins and outs of ElasticSearch. The talk is centred around a single example; objective, that is modelled both in ElasticSearch and plain SQL. I discuss the advantages of both as well as integration with Laravel 4.

TRANSCRIPT

  • 1. Discovering+Elas/cSearch

2. A"li%le"about"me 3. 23#years#old 4. Jess'%boyfriend 5. Zoe's&dad 6. Mountain(biker 7. Scuba&diver 8. Scuba&biker? 9. Mar$al&Arts&Instructor 10. I'm$also$a$developer.I"run"a"development"agency"in"Australia,"Webcomm 11. Finally,(I'm(a(Laracon'addictwho$has$travelled132,000&kma"third"of"the"distance"to"the"moon. 12. Flashcard @ben_corle+ h+p://github.com/bencorle+ h+p://webcomm.com.au 13. Search'isfiltering!informa)onand$determining$relevance 14. Picture(this(scenario 15. "I#want#to#find#hotels#called#Renaissance#for#under+150,#within+500m+of+Bimhuis#so#I#am#close#to#Laracon#EU#2014.#The#hotel#needs#to#have#disability+access#and#ideally#provide#Wifi#and#have#rooms#above+ground+level." 16. Requirements1. Name'of'hotel'called'Renaissance.2. Under'150.'13. Within'500m'of'Bimhuis.'14. Disability'access.1"Perceived"requirements"may"be"flexible. 17. Wants1. Wifi.&12. Rooms&above&ground&level.1"Let's"be"honest,"this"should"be"a"requirement";) 18. If!search!is!filtering!informa1on!and!determining!relevanceAnd!humans!think!with!expression!and!emo2on,!why!do!your!apps!operate!like!robots?How!can!we!tailor!our!apps!to!think!like!our!users? 19. Use$the$right$toolset;$andKnow%your%data. 20. Introducing+Elas0cSearch 21. Elas%cSearch1. Powerful+search+and+analy3cs+engine.+12. Object;based+document+store+where+every+field+is+indexed+and+searchable.3. Distributed;+ready+to+scale.+21"Elas'cSearch"may"be"used"for"much"more"than"just"a"search"engine.2"Webscalez"FTW"(trollolol). 22. SQL$vs.$Elas+cSearch 23. SQL$and$Elas+cSearch;$a$comparison SQL%is%a%rela,onal%database,%Elas,cSearch%is%a%search%engine. Where%SQL%is%great%at%filtering%on%a%binary%level,%Elas,cSearch%thrives%on%both%binary%data%and%full%text%relevance. SQL%indexes%are%always%up%to%date%with%your%primary%store,%Elas,cSearch%needs%to%be%synced. Elas,cSearch%is%very%easy%to%horizontally%scale%for%performance%and%redundancy. 24. SQL$and$Elas+cSearch;$a$comparisonSQL$uses$the$following$structure$for$its$data$store:database > table > rowElas%cSearch+has+a+different,+yet+comparable+structure:index > type > document 25. Elas%cSearch+101 26. It's%all%about%documents1. Elas'cSearch-is-document5oriented.2. Documents-are-represented-using-JSON.3. Data-can-be-in-nested-JSON-objects,-arrays-and-is-all-searchable. 27. It's%all%about%documents{"name": "Renaissance Hotel Amsterdam","company": "Marriott","location": [52.3712561, 4.9005577],"floor_levels": 6,"features": ["disability_access","wifi","smoking_allowed","pool"]} 28. Elas%cSearch+RequiresJavaYou$have$3$seconds$to$sulk$and$complain,$then$shutup. 29. Installa'on)is)easy1. Install)Java.2. Download)and)run)Elas4cSearch)in)3)bash)commands.)13. Debian)or)RPM)packages)available.4. Puppet)&)chef)scripts)available.1"h$p://www.elas.csearch.org/guide/en/elas.csearch/reference/current/_installa.on.html 30. Scaling(isn't(scary1. Near'zero*configura0on*to*build*a*cluster*of*Elas0cSearch*instances.2. Easy*to*scale*horizontally.3. Each*Elas0cSearch*instance*is*referred*to*as*a*node.4. Any*node*is*capable*of*handling*any*request*and*delega0ng*load. 31. In#very#basic#terms,#horizontal*scaling#is#addingmore%serversto#build#a#cluster,#or#cloud... 32. ...where&ver$cal(scaling&is&throwingmore%resourcesat#an#individual#server. 33. Elas%cSearch+exposes+aRESTful(API 34. Win. 35. Communica)ng+with+Elas)cSearch1. HTTP&verbs&,&GET,&POST,&PUT,&DELETE,&etc...2. Send&&&receive&JSON&payloads.:VERB /:index/:type/:document{"key": "value","complex": ["foo", "bar"]} 36. Crea%ng(a(documentPOST /myapp/hotel{"name": "Renaissance Hotel Amsterdam","company": "Marriott","location": [52.3712561, 4.9005577],"floor_levels": 6,"features": ["disability_access","wifi","smoking_allowed","pool"]} 37. Upda%ng(a(documentPUT /myapp/hotel/1 # ^1{"name": "Renaissance Hotel Amsterdam","company": "Marriott","location": [52.3712561, 4.9005577],"floor_levels": 10,"features": ["disability_access","wifi","pool","restaurant"]}1"Actually"an"upsert;"create"or"update"depending"on"existance. 38. Dele$ng'a'documentDELETE /myapp/hotel/1 39. There's'lots'more'to'documents1. Par&al(document(upda&ng.2. Document(versioning.3. Conflict(resolu&on(for(distributed(documents.4. Bulk(CRUD(methods(to(avoid(HTTP(boEleneck.See#h%p://www.elas.csearch.org/guide/en/elas.csearch/reference/current/docs.html 40. Elas%cSearch+makessearching*fun 41. Searching*in*Elas.cSearch1. Every(single(field(can(be(searchable.2. Perform(structured(queries(or(filters,(against(fields.13. Perform(full(text(queries(to(find(documents.4. Queries(and(filters(represented(using(JSON.5. Organise(results(by(relevance.1"SQL&like"approach. 42. Index1. Index&(noun)#$#refers#to#the#equivalent#of#a#database#in#an#SQL#system.2. Index&(verb)#$#refers#to#the#process#of#storing(a(document#in#an#index.3. Inverted&index#$#list#of#all#terms#inside#Elas@cSearch#and#the#documents#in#which#they#appear. 43. Analysis Character(filters"simplify"data,"such"as"changing: "&""to""and". """to""e". Data"is"split"into"terms"through"a"process"called"tokenisa1on. 44. Analysis Token&filters"tweak"and"normalise"terms,"such"as: Cast"to"lowercase. Remove&stop3words"like""a""or""the" Add"synonyms. 45. Inverted(index1. Analysis#process#extremely#configurable.12. Mul7lingual#support#(33#languages#in#total),#interchangeable#per#index.3. Any#fields#not#indexed#are#not#searchable.4. The+same+analysis+process+occurs+at+search+3me.1"h$p://www.elas.csearch.org/guide/en/elas.csearch/reference/current/analysis.html 46. Example(inverted(indexConsider)the)two)following)sentences:"The%quick%brown%fox%jumped%over%the%lazy%dog""Quick%brown%foxes%leap%over%lazy%dogs%in%summer" 47. Example(inverted(indexTerm Doc_1 Doc_2-------------------------brown | X | Xdog | X | Xfox | X | Xin | | Xjump | X | Xlazy | X | Xover | X | Xquick | X | Xsummer | | Xthe | X | X------------------------ 48. Scoring1. Term%frequency#$#the#more#o+en#a#term#appears#in#a#field,#the#more%relevant.2. Inverted%document%frequency#$#the#more#o+en#a#term#appears#in#the#inverted#index,#the#less%relevant.3. Field%length%norm#$#the#longer#the#field,#the#less#relevant#each#term#in#it#is. 49. Scoring1. Fields)are)"boostable")to)increase)relevance.2. Func5ons)(inbuilt)and)scripted))can)be)used)to)increase/decrease)relevance.3. Altering)analysis)to)fine?tune)scoring.4. Very)important)to)know%your%data. 50. Queries'and'filters1. Both'are'modular;'think'of'building(blocks.2. Both'can'be'nested'inside'one'another.3. Syntax'does'not'change,'regardless'of'posi?on'or'nes?ng.4. En?re'JSON'object'is'the'Elas/cSearch(Query(DSL. 51. Querying)in)Elas.cSearch1. There'37'queries'(as'of'August'2014).'12. Queries'are'intelligent;'they'score'all'results'according'to'a'relevance'algorithm.3. Any'nesBng'passes'relevance'back'to'parents.1"h$p://www.elas.csearch.org/guide/en/elas.csearch/guide/current/relevance9intro.html 52. Filters1. You&will&find&27&filters&(as&of&August&2014).&12. Filters&are&binary;&either&a&field&matches&or&it&doesn't.3. Filters&don't&affect&relevance&scoring.1"h$p://www.elas.csearch.org/guide/en/elas.csearch/reference/current/query;dsl;filters.html 53. Querying)in)Elas.cSearchGET /myapp/hotel/_search{"query": {"match": "Renaissance"}}This%is%a%match%query.%It%is%the%go1to%full%text%query. 54. Querying)in)Elas.cSearchGET /myapp/hotel/_search{"query": {"filtered": {"filter": {"term": {"features": "disability_access"}}}}}This%is%a%filtered%query,%containing%a%term%filter. 55. Back%to%our%scenario 56. "I#want#to#find#hotels#called#Renaissance#for#under+150,#within+500m+of+Bimhuis#so#I#am#close#to#Laracon#EU#2014.#The#hotel#needs#to#have#disability+access#and#ideally#provide#Wifi#and#have#rooms#above+ground+level." 57. Two$approaches1. Possible*with*both*SQL*and*Elas5cSearch.2. Much*easier*Elas5cSearch.3. Elas5cSearch*understands*the*concept*of*relevance.4. Elas5cSearch*can*severely*outperform*SQL. 58. SQL$approach 59. First,'we'll'build'the'obvious...select *from `hotels`where `name` like "%Renaissance%"and `price` 'myapp','type' => 'hotel','id' => '1','body' => ['name' => 'Renaissance Hotel Amsterdam','company' => 'Marriott','location' => [52.3712561, 4.9005577],'floor_levels' => 10,'features' => ['disability_access', 'wifi', 'pool', 'restaurant'],],]);You$can$create$or$update$in$the$same$request. 83. Par$ally'upda$ng'documents$client = new ElasticsearchClient();$client->update(['index' => 'myapp','type' => 'hotel','id' => '1','body' => ['floor_levels' => 11,],]); 84. Dele$ng'documents$client = new ElasticsearchClient();$client->delete(['index' => 'myapp','type' => 'hotel','id' => '1',]); 85. Searching*documents$client = new ElasticsearchClient();$client->search(['index' => 'myapp','type' => 'hotel','body' => ['query' => ['match' => 'Renaissance',],],]); 86. Create/update/delete+eloquent+documentsHotel::created(function ($hotel) {$client = new ElasticsearchClient();$client->index(['index' => 'myapp','type' => 'hotel','id' => $hotel->id,'body' => $hotel->toArray(),]);}); 87. Create/update/delete+eloquent+documentsHotel::updated(function ($hotel) {$client = new ElasticsearchClient();$client->index(['index' => 'myapp','type' => 'hotel','id' => $hotel->id,'body' => $hotel->toArray(),]);}); 88. Create/update/delete+eloquent+documentsHotel::deleted(function ($hotel) {$client = new ElasticsearchClient();$client->delete(['index' => 'myapp','type' => 'hotel','id' => $hotel->id,]);}); 89. Searching*+*two*approaches1. Acceptable#have#a#search()#method#directly#on#Eloquent#to#use#Elas6cS