use cases for elastic search percolator
TRANSCRIPT
Use Cases for Elastic Search Percolator
Search Meetup 2016
2
Percolator
Search Meetup 2015
Traditionally you design documents based on your data, store them into an index, and then define queries
to retrieve these documents.
The The The The percolator works in the opposite direction.percolator works in the opposite direction.percolator works in the opposite direction.percolator works in the opposite direction.
First you store queries into an index and then you define documents in order to retrieve these queries.
The percolator and most of its features work in realtime - once a percolate query is indexed it can immediately be used in the percolate API.
3
Search Meetup 2015
Near – time decisions for rewarding customerin PAYBACK programs
4
Search Meetup 2015
Business
target
Near – time decisions for rewarding customerin PAYBACK programs
source: https://www.flickr.com/photos/mdalmuld/9559878695
5
Search Meetup 2015
GamificationMore achievements and more emotions with games.
source: https://www.flickr.com/photos/129941959@N06/15664857154/
6
Implementation details
Elastic Elastic Elastic Elastic SearchSearchSearchSearch as real time Data Container with all events
PercolatorPercolatorPercolatorPercolator as real time Rule Engine
RedisRedisRedisRedis as queuing system
LogstashLogstashLogstashLogstash for event pushing
DropwizardDropwizardDropwizardDropwizard for RESTfull standalone application
HTML5 and AngularJS HTML5 and AngularJS HTML5 and AngularJS HTML5 and AngularJS for responsive GUI
Search Meetup 2015
Tools / frameworks
7
Architecture
Search Meetup 2015
8
Percolator
Search Meetup 2015
curl -XPUT 'localhost:9200/data/.percolator/g1_123456700' -d '{
"query" : {"match" : {
"service" : "processPurchaseEvent",
"alias" : "123456700"}
}}'
Creation of percolator query
9
Percolator
Search Meetup 2015
curl -XGET 'localhost:9200/data/logs/_percolate' -d '{
"doc" : {"service" : "processPurchaseEvent","alias" : "123456700"
.
.
.}
}'
Usage of percolator
10
Redis
Search Meetup 2015
{"matches":["g2_123456700","g1_123456700"],
"event":{
"service":"processPurchaseEvent","logstash_checksum":"cef6adb614de4623bbd809197fb79e3f",
…{"alias":"123456700","aliasType":1}}
…
}}
Data for redis
11
Java application
Search Meetup 2015
Creation of percolators
12
Logstash plugin
Search Meetup 2015
Ruby code for using percolator queries
13
Logstash plugin
Search Meetup 2015
Configuration
output {
elasticsearch { host => " HOSTNAME :9200"
protocol => "http"
document_id => "%{logstash_checksum}"
}
percolator {host => "HOSTNAME"
port => "9200"
redis_host => ["localhost"]
document_id => "%{logstash_checksum}"
pattern_index => "percolators"index => "data"
data_type => "channel"
}
}
14
Percolator details
stored per index in the .percolator type
active percolator queries are kept in memory
at percolate time, the document from the request gets parsed into a Lucene document and is stored in a in-memory Lucene index
percolator queries are executed on single document in-memory index
multi percolate and percolation of existing documents is possible
Search Meetup 2015
Under the hood
15
Alerting Use Cases
Scaling the PAYBACK platform for the US: plenti.com
16
Architecture
Search Meetup 2015
17
Alerting Use Case
easy with JSON structured logs
simplest use case: percolate on FATAL and ERROR log levels
alerts can be triggered for displaying on dashboards
also easy for enhanced use cases such as Monitoring of invariants: create alerts whenever system goes outside of expected normal ranges
shift monitoring from reactive to proactive – be able to look for exceptional behavior before it exhibits itself as a production problem
Search Meetup 2015
Using with ELK
18
Other use cases
Search Meetup 2015
Alerting based notifications
Having corresponding data in elastic search you can notify customer about
• new product release
• price change of a product (is below threshold)
• weather changes (snow, rain etc)
• stock changes
• what do you use?
19
Search Meetup 2015
Questions&
Answers
source: https://www.flickr.com/photos/21496790@N06/5065834411/