sf elasticsearch meetup - how hipchat scaled to 1b messages
DESCRIPTION
HipChat Engineer, Zuhaib Siddique, presents how HipChat uses Elasticsearch in its infrastructure to provide near real-time search and chat history to the hundreds of thousands of active chatters using HipChat. Zuhaib covers cover of the issues the HipChat team has overcome using elasticsearch with AWS and CouchDB.TRANSCRIPT
![Page 1: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/1.jpg)
Wednesday, November 20, 13
![Page 2: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/2.jpg)
Heavy Lifting
ZUHAIB S IDDIQUE • @ZUHAIB • ENGINEER
Wednesday, November 20, 13
![Page 3: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/3.jpg)
Wednesday, November 20, 13
![Page 4: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/4.jpg)
1. Persistent chat rooms
Wednesday, November 20, 13
![Page 5: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/5.jpg)
2. File sharing & Chat history
Wednesday, November 20, 13
![Page 6: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/6.jpg)
3. @mentions + notifications
Wednesday, November 20, 13
![Page 7: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/7.jpg)
4. Integrations and API
Wednesday, November 20, 13
![Page 8: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/8.jpg)
5. Runs on everything
Wednesday, November 20, 13
![Page 9: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/9.jpg)
5. Runs on everything
Wednesday, November 20, 13
![Page 10: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/10.jpg)
5. Runs on everything
Wednesday, November 20, 13
![Page 11: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/11.jpg)
Group chat and IM for teams.
Wednesday, November 20, 13
![Page 12: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/12.jpg)
1,000,000,000
messages sent
1,200,000,000
Wednesday, November 20, 13
![Page 13: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/13.jpg)
{ "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" }}
What is a HipChat Message?
Wednesday, November 20, 13
![Page 14: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/14.jpg)
In the beginning there was …
Wednesday, November 20, 13
![Page 15: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/15.jpg)
Then we added...
https://github.com/rnewson/couchdb-luceneWednesday, November 20, 13
![Page 16: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/16.jpg)
CouchDB + CouchDB-Lucene
•Started to tip over at 200M Docs
•m2.4xlarge
Wednesday, November 20, 13
![Page 17: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/17.jpg)
CouchDB + CouchDB-Lucene
m2.4xlarge m2.4xlarge
Worked well for the first 500MWednesday, November 20, 13
![Page 18: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/18.jpg)
m2.4xlarge hs1.8xlarge
Wednesday, November 20, 13
![Page 19: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/19.jpg)
But it still wasn’t big enough.
Wednesday, November 20, 13
![Page 20: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/20.jpg)
Dont F*ck the Customer
Wednesday, November 20, 13
![Page 21: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/21.jpg)
First Attempt at Scaling
Wednesday, November 20, 13
![Page 22: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/22.jpg)
Wednesday, November 20, 13
![Page 23: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/23.jpg)
Why ElasticSearch for HipChat
Wednesday, November 20, 13
![Page 24: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/24.jpg)
Why ElasticSearch for HipChat
{ "query_string" : { "default_field" : "content", "query" : "this AND that OR thus" }}
Wednesday, November 20, 13
![Page 25: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/25.jpg)
Why ElasticSearch for HipChat
Wednesday, November 20, 13
![Page 26: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/26.jpg)
Search Error Rate
Wednesday, November 20, 13
![Page 27: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/27.jpg)
Wednesday, November 20, 13
![Page 28: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/28.jpg)
Don’t Forget about CouchDB
AWS Max EBS size
Wednesday, November 20, 13
![Page 29: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/29.jpg)
Stop the CouchDB bleeding
Wednesday, November 20, 13
![Page 30: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/30.jpg)
CouchDB end of the road
Wednesday, November 20, 13
![Page 31: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/31.jpg)
Wednesday, November 20, 13
![Page 32: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/32.jpg)
(allthethings) ElasticSearch
{ "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } }}
{ "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" }}
ElasticSearch CouchDB
Wednesday, November 20, 13
![Page 33: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/33.jpg)
(allthethings) ElasticSearch
{ "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } }}
{ "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" }}
ElasticSearch CouchDB
Wednesday, November 20, 13
![Page 34: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/34.jpg)
CouchDB vs ElasticSearchCouchDB Spike
to 800ms!
Avg 100ms with ES
Wednesday, November 20, 13
![Page 35: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/35.jpg)
master
Running ElasticSearch
clientsHipChat Customer
Thanks Philip O’Toole - Loggly
Wednesday, November 20, 13
![Page 36: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/36.jpg)
$ fab create_esindex:hostname=esindex-d4
Wednesday, November 20, 13
![Page 37: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/37.jpg)
•Spin up EC2 Instance• Install Oracle Java 7•Create Provision IOPs EBS• Install and configure ElasticSearch•???•Profit!
Behind the Scenes
Wednesday, November 20, 13
![Page 38: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/38.jpg)
# Elasticsearchnodes = partial_search(:node, "role:elasticsearch_server AND chef_environment:#{node.chef_environment}", :keys => { 'hostname' => [ 'hostname' ] })if nodes.empty? Chef::Log.error "site.json - No Elasticsearch servers found"else es_nodes = nodes.map {|n| { 'host' => n['hostname'], 'port' => 9200 } } data["elasticsearch"] = { "connections" => es_nodes.sort_by {|n| n['host'] } }end
Wednesday, November 20, 13
![Page 39: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/39.jpg)
site.json"elasticsearch": { "connections": [ { “host": "esindex-c1", "port": 9200 }, ... { "host": "esindex-e3", "port": 9200 } ]}
Wednesday, November 20, 13
![Page 40: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/40.jpg)
You
Wednesday, November 20, 13
![Page 41: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/41.jpg)
Running Elasticsearch
Wednesday, November 20, 13
![Page 42: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/42.jpg)
Running ElasticSearch
Single Node OOM!
Wednesday, November 20, 13
![Page 43: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/43.jpg)
Monitor, Monitor, Monitor!
Wednesday, November 20, 13
![Page 44: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/44.jpg)
Backup
Wednesday, November 20, 13
![Page 45: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/45.jpg)
100,000,000,000?To infinity and beyond!
Wednesday, November 20, 13
![Page 46: SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages](https://reader034.vdocuments.net/reader034/viewer/2022052410/54c637704a795920538b45c1/html5/thumbnails/46.jpg)
Questions?
Wednesday, November 20, 13