mongo db parse-v17
DESCRIPTION
Parse powers over 250k mobile apps on MongoDB. Since we can have no prior knowledge of the schemas or usage patterns or scaling requirements for each app, we rely on advanced operational techniques and best practices for managing MongoDB and maintaining peak performance. In this talk we will talk about the unique challenges of this use case, such as dealing with cotenancy issues and how to automatically generate good indexes for each app. I will talk about some of the interesting operational techniques involved in managing hundreds of databases and millions of collections on MongoDB, as well as running large and growing database ecosystems on AWS. I will also talk about some of our favorite black belt tips for tuning and scaling MongoDB. And finally we will talk about how to recover from performance-impacting incidents gracefully, without making everything worse for yourself.TRANSCRIPT
![Page 1: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/1.jpg)
Charity Majors @mipsytipsy
![Page 2: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/2.jpg)
</>
Mobile Backend as a Service
Data Storage
Push Notifications
Analytics
Server-Side Code
![Page 3: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/3.jpg)
300K
250K
200K
150K
100K
50K
02011 2012 2013 2014 2015
Apps Built On Parse
270,000+
![Page 4: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/4.jpg)
250%developers
500%api requests
![Page 5: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/5.jpg)
a love^Wless hateful storyMongoDB + Ops:
![Page 6: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/6.jpg)
![Page 7: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/7.jpg)
?MongoDB + Ops:
?
![Page 8: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/8.jpg)
Reliability
Flexibility
Automation
1
2
3
![Page 9: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/9.jpg)
Reliability
Flexibility
Automation
1
2
3
![Page 10: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/10.jpg)
re · li · abil · i · ty (noun) \ri-ˌlī-ə-ˈbi-lə-tē\1. Through resiliency 2. Magic of the replica set
![Page 11: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/11.jpg)
Replica Set
![Page 12: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/12.jpg)
Pets?
![Page 13: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/13.jpg)
No Pets!
![Page 14: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/14.jpg)
![Page 15: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/15.jpg)
Arbiters
Extra Votes
Spanning AZs
Spanning DCs
Non-voting
Hidden
Client Logic
More!
![Page 16: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/16.jpg)
Planning for failure.
![Page 17: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/17.jpg)
![Page 18: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/18.jpg)
Reliability
Flexibility
Automation
1
2
3
![Page 19: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/19.jpg)
When you change a schema, everythi ng breaks !
![Page 20: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/20.jpg)
DATA MODEL FLEXIBILITY
![Page 21: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/21.jpg)
DATA MODELWORKLOAD FLEXIBILITY
![Page 22: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/22.jpg)
![Page 23: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/23.jpg)
270kapps hosted on Parse
![Page 24: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/24.jpg)
270kapps hosted on Parse
270kdifferent workloads
=
![Page 25: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/25.jpg)
270kapps hosted on Parse
270kdifferent workloads
=Every app must be individually performant!
![Page 26: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/26.jpg)
DEPLOYMENT
BACKUPS
RECOVERY
DISASTER RECOVERY
DRIVERS
PARTIAL FAILURES
PERFORMANCE TUNING
INDEXING PLAN
DESIGN FAILOVER PLAN
OPERATIONAL EXPERTISE
![Page 27: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/27.jpg)
ONEre-usable solution
![Page 28: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/28.jpg)
![Page 29: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/29.jpg)
Reliability
Flexibility
Automation
1
2
3
![Page 30: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/30.jpg)
Replica Set
![Page 31: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/31.jpg)
![Page 32: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/32.jpg)
![Page 33: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/33.jpg)
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
![Page 34: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/34.jpg)
+
![Page 35: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/35.jpg)
![Page 36: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/36.jpg)
PLEASE. SHARE YOUR TOOLS.
![Page 37: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/37.jpg)
MongoDB Chef Cookbook github.com/ParsePlatform/Ops/tree/master/chef/cookbooks/mongodb
AWS Chef Cookbook (ec2 raid recipe) github.com/opscode-cookbooks/aws
Continuous Compaction blog.parse.com/2013/03/26/always-be-compacting/
Warmup Scripts blog.parse.com/2013/03/07/techniques-for-warming-up-mongodb/
![Page 38: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/38.jpg)
Mongo Proxy github.com/xxxxxx
Flashback github.com/xxxxxx
NEW
NEW
![Page 39: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/39.jpg)
Mongo Proxy url url url url -- naitik shah (& blog post url)
NEW
![Page 40: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/40.jpg)
Flashback url url url url -- kai liu (& blog post url)
NEW
![Page 41: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/41.jpg)
a love story[*]. The End
* mutual loathing, bitterness, mistrust. Love the one you’re with.
MongoDB + Ops:
![Page 42: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/42.jpg)
“Which database do I hate the most?”
![Page 43: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/43.jpg)
“Which database do I hate the most?”
“The one I’m using right now”
![Page 44: Mongo db parse-v17](https://reader030.vdocuments.net/reader030/viewer/2022012902/540d9ff18d7f728d7e8b4a45/html5/thumbnails/44.jpg)
Charity Majors @mipsytipsy