growing up mongodb
DESCRIPTION
TRANSCRIPT
![Page 1: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/1.jpg)
Growing up MongoDB
Kiril Savino - CTO GameChanger@kirilnyc
![Page 2: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/2.jpg)
About Me.
Lead Engineer, Higher One
Lead Engineer, DoubleClick
Lead Engineer -> CTO, ShopWiki
Director Engineering, Conductor
Founder & CTO, GameChanger Media
10 years Oracle and MySQL, 4 MongoDB
![Page 3: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/3.jpg)
pre-$not
![Page 4: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/4.jpg)
About GameChanger.
![Page 5: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/5.jpg)
Growing up.
865,443,426+
3 terabytes
16-nodes
240GB RAM, 8TB SSD storage
120,000 ops/s sustained
![Page 6: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/6.jpg)
Learnings.
Schemas
Concurrency
Availability
Firefighting
![Page 7: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/7.jpg)
1. Schemas!
![Page 8: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/8.jpg)
Schema-less.
I do not think that word means what you think it means.
![Page 9: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/9.jpg)
Be abnormal.
![Page 10: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/10.jpg)
Schema-less does mean not having to separate data for modeling reasons
Focus on data usage patterns along with semantics
You're going to have to do this anyway: start now and scale up easy
![Page 11: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/11.jpg)
Go monolithic.
Learn to query, then forget and pretend MongoDB is a really fancy, full-featured KV store.
![Page 12: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/12.jpg)
Querying secondary data is a waste
Scans & indexed queries are slow
Memory fragmentation can kill you
Use MongoDB's strengths
![Page 13: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/13.jpg)
Garbage in...
![Page 14: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/14.jpg)
Validation is your problem
Don't let inconsistency linger
Know what parts of schema are flexible
![Page 15: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/15.jpg)
Index wisely.
Data size
Insert/Update speed
Bad schema smell
![Page 16: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/16.jpg)
2. Concurrency!
![Page 17: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/17.jpg)
(A)CID.
![Page 18: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/18.jpg)
Good schema design provides for basic atomicity at the document level
Obviates the need for transactions in many trivial cases
![Page 19: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/19.jpg)
Your friends.
$set/$unset
$push/$pull
$addToSet
findAndModify
![Page 20: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/20.jpg)
Two phased commits.
![Page 21: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/21.jpg)
Optimistic locking.
![Page 22: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/22.jpg)
External transactions.
![Page 23: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/23.jpg)
Eventual Consistency.
Write canonical data first
Ensure queuing of propagation
Guarantee queue entry completeness
![Page 24: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/24.jpg)
3. Availability!
“/dev/null is web scale”
![Page 25: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/25.jpg)
Durability.
Journaling.
OK?
![Page 26: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/26.jpg)
Replication.
![Page 27: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/27.jpg)
Moar = better?
![Page 28: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/28.jpg)
4. Firefighting!
![Page 29: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/29.jpg)
Test your capacity.Naïve throughput testing with real hardware
Clone prod configuration
Consider copying data or subset
Start with crude approximations
Get as close to “real load” as makes sense
![Page 30: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/30.jpg)
Model your growth.
db.stats()
db.<collection>.stats()
avg doc size, avg index size / doc
growth rates / collection
approx active portion / collection
![Page 31: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/31.jpg)
Read your logs.
{...}ntoreturn:1keyUpdates:0numYields: 136locks(micros) r:368727reslen:78 199ms
![Page 32: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/32.jpg)
Don’t scan.
![Page 33: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/33.jpg)
So...
Schema-less != no schema
ACID overrated; concurrency not
High availability is up to you
Understand the mechanics
![Page 34: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/34.jpg)
Thanks!
Kiril SavinoCTO, GameChanger Media
www.GameChanger.io@kirilnyc
kirilsavino.com/blog
![Page 35: Growing Up MongoDB](https://reader038.vdocuments.net/reader038/viewer/2022103014/54b6ea874a7959b35f8b474f/html5/thumbnails/35.jpg)
Next Sessions at 3:405th Floor:
West Side Ballroom 3&4: Advanced Replication Internals
West Side Ballroom 1&2: Building a High-Performance Distributed Task Queue on MongoDB
Juilliard Complex: WhiteBoard Q&A
Lyceum Complex: Ask the Experts
7th Floor:
Empire Complex: Managing a Maturing MongoDB Ecosystem
SoHo Complex: MongoDB Indexing Constraints and Creative Schemas