mongodb performance debugging
DESCRIPTION
TRANSCRIPT
![Page 2: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/2.jpg)
![Page 3: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/3.jpg)
20M crashes/day
2TB data / 50GB index
11 nodes / 3* availability zones
![Page 4: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/4.jpg)
I don't care if MongoDB is slowI care if my app is slow
![Page 5: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/5.jpg)
How to make my app fast again if it's slow because of the way I'm using MongoDB
![Page 6: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/6.jpg)
What does slow mean?
![Page 7: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/7.jpg)
![Page 8: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/8.jpg)
Performance over time
![Page 9: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/9.jpg)
Performance over time
![Page 10: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/10.jpg)
![Page 11: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/11.jpg)
It's slow :(
![Page 12: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/12.jpg)
![Page 13: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/13.jpg)
![Page 14: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/14.jpg)
Solution 1
Denormalize
![Page 15: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/15.jpg)
![Page 16: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/16.jpg)
It's fast :)
![Page 17: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/17.jpg)
It's slow :(
![Page 18: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/18.jpg)
db.errors. find({project_id: x}). sort({ _id: -1}). limit(30)
![Page 19: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/19.jpg)
db.errors. find({project_id: x}). sort({ _id: -1}). limit(30). explain()
![Page 20: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/20.jpg)
{ "cursor" : "BtreeCursor _id_ reverse", "isMultiKey" : false, "n" : 0, "nscannedObjects" : 227756, "nscanned" : 227756, "nscannedObjectsAllPlans" : 227756, "nscannedAllPlans" : 227756, "scanAndOrder" : false, "indexOnly" : false, "nYields" : 1779, "nChunkSkips" : 0, "millis" : 461, "indexBounds" : { "_id" : [ [ { "$maxElement" : 1 }, { "$minElement" : 1 } ] ] }, "server" : "Jaroussky.local:27017", "filterSet" : false}
![Page 21: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/21.jpg)
{"cursor" : "BtreeCursor _id_ reverse",
"nscanned" : 227756,
"indexBounds" : {"_id" : [ [{"$maxElement" : 1}, {"$minElement" : 1}] ] }}
![Page 22: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/22.jpg)
{"cursor" : "BtreeCursor _id_ reverse",
"nscanned" : 227756,
"indexBounds" : {"_id" : [ [{"$maxElement" : 1}, {"$minElement" : 1}] ] }}
![Page 23: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/23.jpg)
Solution 2
Index
![Page 24: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/24.jpg)
db.errors.ensureIndex( {project_id: 1, id: -1})
![Page 25: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/25.jpg)
{"cursor" : "BtreeCursor project_id_1__id_1","nscanned" : 6,"indexBounds" : { "project_id" : [[ ObjectId(x),ObjectId(x) ]], "_id" : [[ {"$minElement" : 1}, {"$maxElement" : 1} ]]}}
![Page 26: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/26.jpg)
It's fast :)
![Page 27: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/27.jpg)
It's slow :(
![Page 28: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/28.jpg)
Only 1 query...Indexed properly...
![Page 29: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/29.jpg)
mongostat
![Page 30: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/30.jpg)
insert query update delete getmore command flushes mapped 5 97 57 *0 95 127|0 0 320g 5 98 61 *0 113 146|0 0 320g 8 94 61 *0 95 137|0 0 320g
vsize res faults locked db idx miss % qr|qw ar|aw 641g 8.65g 7 bugsnag:47.7% 0 0|0 0|0 641g 8.66g 2 bugsnag:21.0% 0 0|0 0|0 641g 8.64g 3 bugsnag:23.4% 0 0|0 0|0
netIn netOut conn set repl time 47k 73k 145 bugsnag1 PRI 06:34:05 63k 99k 146 bugsnag1 PRI 06:34:06 98k 124k 146 bugsnag1 PRI 06:34:07
![Page 31: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/31.jpg)
command locked db time 127|0 bugsnag:47.7% 06:34:05 146|0 bugsnag:21.0% 06:34:06 137|0 bugsnag:23.4% 06:34:07
![Page 32: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/32.jpg)
![Page 33: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/33.jpg)
Solution 3
Shard
![Page 34: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/34.jpg)
![Page 35: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/35.jpg)
sh.shardCollection("errors"{ project_id: 1, _id: -1})
![Page 36: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/36.jpg)
command locked db time 83|0 bugsnag:3.1% 06:46:12 69|0 bugsnag:4.1% 06:46:13 73|0 bugsnag:2.4% 06:46:14
![Page 37: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/37.jpg)
It's fast :)
![Page 38: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/38.jpg)
It's slow :(
![Page 39: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/39.jpg)
Only 1 query...Indexed properly...
Lock % ok...
![Page 40: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/40.jpg)
iostat
![Page 41: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/41.jpg)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-szxvdi 0.00 2.00 15.00 743.00 436.00 7905.50 22.01xvdi 0.00 2.00 0.00 663.00 0.00 6706.00 20.23xvdi 0.00 1.00 0.00 726.00 0.00 5593.50 15.41
avgqu-sz await r_await w_await svctm %util 11.02 14.54 5.33 14.73 0.60 45.60 8.04 12.12 0.00 12.12 0.51 33.60 13.01 17.92 0.00 17.92 0.50 36.40
![Page 42: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/42.jpg)
rkB/s wkB/s await %util 7660 138 14.73 83.60 2248 2043 12.12 65.20 7905 436 17.92 45.60
![Page 43: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/43.jpg)
db.stats()['indexSize']= 3852071824
= 3932614656free -b (m3.medium)
![Page 44: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/44.jpg)
Solution 4
Scale
![Page 45: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/45.jpg)
db.stats()['indexSize']= 3852071824
= 15775363072free -b (r3.large)
![Page 46: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/46.jpg)
It's fast :)
![Page 47: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/47.jpg)
It's slow :(
![Page 48: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/48.jpg)
Main query seems fine...
"Quick" queries sometimes slow...
![Page 49: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/49.jpg)
mongotop
![Page 50: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/50.jpg)
ns total read write 2014-06-22T19:11:35 bugsnag.events 35ms 0ms 35ms bugsnag.errors 30ms 6ms 24msbugsnag.system.namespaces 26ms 26ms 0ms bugsnag.projects 12ms 5ms 7ms bugsnag.users 15ms 2ms 13ms bugsnag.error_aggregates 4ms 0ms 4ms bugsnag.deploys 4ms 3ms 1ms bugsnag.event_tallies 3ms 0ms 3ms
![Page 51: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/51.jpg)
ns tot r w bugsnag.events 35 0 35 bugsnag.errors 30 6 24bugsnag.projects 12 5 7 bugsnag.users 15 7 8
![Page 52: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/52.jpg)
Solution 5
Tag shards
![Page 53: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/53.jpg)
It's fast :)
![Page 54: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/54.jpg)
Solution 1
Denormalize
![Page 55: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/55.jpg)
Solution 2
Index
![Page 56: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/56.jpg)
Solution 3
Shard
![Page 57: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/57.jpg)
Solution 4
Scale
![Page 58: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/58.jpg)
Solution 5
Tag shards
![Page 59: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/59.jpg)
Solution 6
...
![Page 60: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/60.jpg)
NewRelic / Skylight
explain()
mongostat
iostat
...
![Page 61: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/61.jpg)
It will be slow
![Page 62: MongoDB Performance Debugging](https://reader034.vdocuments.net/reader034/viewer/2022050808/54b6ea984a7959ff2d8b461d/html5/thumbnails/62.jpg)
You can speed it up