wiredtiger in-memory - vs wiredtiger b-tree...∙ introducing percona memory engine for mongodb ∙...
TRANSCRIPT
![Page 1: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/1.jpg)
WiredTiger In-Memoryvs WiredTiger B-Tree
October, 5, 2016 — Movenpick Hotel — AmsterdamSveta Smirnova
![Page 2: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/2.jpg)
∙What is Percona Memory Engine for MongoDB?
∙Typical use cases
∙Advanced Memory Engine
Table of Contents
2
![Page 3: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/3.jpg)
What is Percona Memory Engine for MongoDB?
3
![Page 4: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/4.jpg)
∙ Up to 1000 times faster for OLTP wokloads∙ 10 times faster for read-only workloads∙ Stable throughput∙ No checkpointing
∙ No jitter
Extremely fast In-Memory storage
4
![Page 5: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/5.jpg)
∙ Document-level locking
∙ B-Tree
∙ Practically WiredTiger, but without diskaccess
Based on WiredTiger
5
![Page 6: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/6.jpg)
∙ Doesn’t store data on disk∙ Except small amount of statistics∙ You can control when to log statistics with
option –inMemoryStatisticsLogDelaySecs∙ Still must specify –dbpath
sveta@Thinkie:~/mongo_tests$ ls -lh single/
total 40K
drwxrwxr-x 2 sveta sveta 4,0K Eyl 29 15:00 diagnostic.data
-rw-r--r-- 1 sveta sveta 6 Eyl 29 14:58 mongod.lock
-rw-rw-r-- 1 sveta sveta 93 Eyl 29 14:55 storage.bson
∙ Data does not persist between restarts
WiredTiger without storage
6
![Page 7: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/7.jpg)
∙ –storageEngine=inMemory
∙ Can be only engine on MongoDB server∙ MongoDB restriction, applicable to all engines
∙ Heterogeneous replication and shardingsetups supported
How to enable Memory Engine?
7
![Page 8: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/8.jpg)
∙ Engine can use up to –inMemorySizeGB∙ If data exceeds this amount
∙ WT CACHE FULL error is returned for allkinds of operations that cause user data sizeto grow
INSERTCREATEUPDATE
∙ Reads are not affected
How to control memory usage
8
![Page 9: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/9.jpg)
∙ 100% Open Source
∙ Code available at GitHub
∙ Free for all Percona users and customers
Open Source
9
![Page 10: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/10.jpg)
Typical use cases for Percona Memory Engine
10
![Page 11: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/11.jpg)
∙ Session management∙ Store active sessions in memory∙ Users will receive answer almost immediately∙ Reduce application response time
dramatically
∙ Various temporary collections∙ All you used to store in memcached
Application cache
11
![Page 12: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/12.jpg)
∙ Application runtime data which does notrequire on-disk storage
∙ Intermediary results of calculations
∙ User-specific options
∙ Your idea
Transient Runtime State
12
![Page 13: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/13.jpg)
∙ Thousand-lines aggregations
∙ Temporary collections to store intermediarydata
∙ Complicated queries
Sophisticated data manipulation
13
![Page 14: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/14.jpg)
∙ Large aggregations might be slow∙ Especially if use many collections∙ Often this is not avoidable
∙ To calculate number of distinct values youneed to read whole index
∙ Fast dedicated server is great solution
Real-Time Analytics
14
![Page 15: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/15.jpg)
∙ Data sharing between multi-tier ormulti-language applications
English labels
∙ Articles∙ Pictures∙ Contact
information∙ Other content
Russian labels
Multi-tier object sharing
15
![Page 16: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/16.jpg)
∙ Are you tired to wait when data, needed forapplication test, loads?
∙ Any change in test data causes delay?∙ With Memory engine you can reduce
turnaround time for automated applicationtests.
∙ And still use same syntax
Application Testing
16
![Page 17: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/17.jpg)
Advanced Percona Memory Engine
17
![Page 18: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/18.jpg)
∙ Are you amazed with speed of the Memoryengine?
∙ But still need data to persist betweenrestarts?
∙ You can combine both Memory and WiredTiger in Replica Set or Sharded Cluster
Best of both worlds
18
![Page 19: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/19.jpg)
∙ Setup 2 or more Memory replicas whichcan be Primary
∙ Let WiredTiger to persist data on disk∙ In rare cases if all Memory replicas crash at
the same time you will loose fewtransactions∙ Number of transactions depends on the
latency between In-Memory Primary replicaand WiredTiger replica
Hidden WiredTiger, storing changes in Replica Set
19
![Page 20: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/20.jpg)
Memory
WiredTiger
Memory
Hidden WiredTiger, storing changes in Replica Set
20
![Page 21: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/21.jpg)
rs.initiate(
... {
... "_id" : "rs",
... "members" : [
... {"_id" : 0, "host" : "inMemory1", "priority" : 1},
... {"_id" : 1, "host" : "inMemory2", "priority" : 1},
... {"_id" : 2, "host" : "WiredTiger", "priority" : 0, "hidden" : true}
... ]
... }
)
Hidden WiredTiger to store on disk: example setup
21
![Page 22: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/22.jpg)
∙ Make WiredTiger Primary
∙ Move all reads to read-only Memoryreplicas
∙ Writes will be slow
WiredTiger as Primary in Replica Set
22
![Page 23: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/23.jpg)
Memory
WiredTiger
Memory
WiredTiger as Primary in Replica Set
23
![Page 24: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/24.jpg)
∙ Create Sharded Cluster using Memorynodes only
∙ Split data between nodes
∙ Create copies of data to prevent data loss
Scaling beyond the RAM of a single server
24
![Page 25: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/25.jpg)
Shard 1 Shard 2
Scaling beyond the RAM
25
![Page 26: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/26.jpg)
Shard 1 Shard 2
R1
R2
R3 R4
R5
R6
Scaling beyond the RAM: add redundancy
26
![Page 27: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/27.jpg)
∙ You can use both engines in the ShardedCluster
∙ Split data∙ Session data on Memory nodes∙ Persistent data on WiredTiger node(s)
∙ Duplicate Memory shards to avoid loosingdata
Memory and WiredTiger in Sharded Cluster
27
![Page 28: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/28.jpg)
∙ You can have sharded nodes which useMemory engine
∙ Make them parts of Replica Set
∙ Let hidden WiredTiger member to persistdata on disk
Sharded Cluster Memory and Replica Sets
28
![Page 29: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/29.jpg)
∙ User posts changing rarely are stored ondisk
∙ Session data stored using Memory engine∙ Active comments (last 24 hours) and
actively accessed posts are cached inMemory node
Example: blog application
29
![Page 30: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/30.jpg)
mongos> sh.addShardTag("shard01", "memory") // Memory node
WriteResult( "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 )
mongos> sh.addShardTag("shard02", "memory") // Memory node
WriteResult( "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 )
mongos> sh.addShardTag("shard03", "persist") // WiredTiger node
WriteResult( "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 )
mongos> sh.addShardTag("shard04", "persist") // WiredTiger node
WriteResult( "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 )
Example: tag shards
30
![Page 31: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/31.jpg)
mongos> sh.addTagRange("blog.sessions", { sid: 0 }, { sid: 1000000 }, "memory")
WriteResult({
"nMatched" : 0,
"nUpserted" : 1,
"nModified" : 0,
" id" : {
"ns" : "blog.sessions",
"min" : {
"sid" : 0
}
}
})
Example: split data
31
![Page 32: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/32.jpg)
mongos> sh.addTagRange("blog.comments", { store: "persist", cid: 0 },
... { store: "persist", cid: 1000000 }, "persist")
WriteResult({
...
" id" : {
"ns" : "blog.comments",
"min" : {
"store" : "persist",
"cid" : 0
}}})
mongos> sh.addTagRange("blog.comments", { store: "memory", cid: 0 },
... { store: "memory", cid: 1000000 }, "memory")
WriteResult({
...
Example: split data
31
![Page 33: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/33.jpg)
mongos> sh.addTagRange("blog.posts", { pid: 0 }, { pid: 1000000 }, "persist")
WriteResult({
"nMatched" : 0,
"nUpserted" : 1,
"nModified" : 0,
" id" : {
"ns" : "blog.posts",
"min" : {
"pid" : 0
}
}
})
Example: split data
31
![Page 34: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/34.jpg)
∙ Percona Memory Engine replacesWiredTiger when you need better speedand can afford loosing data
∙ Can be used in setups which combine bothhigh performance of the Memory engineand data persistence of WiredTiger
∙ Open Source
Summary
32
![Page 35: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/35.jpg)
∙ David Bennett
∙ David Murphy
∙ Fernando Ipar
∙ Denis Protyvenskyi
Special thanks
33
![Page 36: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/36.jpg)
∙ Benchmarks for Percona Memory Engine
∙ Introducing Percona Memory Engine forMongoDB
∙ Percona Server for MongoDB manual
∙ Source code for Percona Memory Engine
More informaiton
34
![Page 37: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/37.jpg)
Rate My Session!
35
![Page 38: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/38.jpg)
???
Place for your questions
36
![Page 39: WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙ Percona Server for MongoDB manual](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f7128134d1e2f3e5a276a9e/html5/thumbnails/39.jpg)
http://www.slideshare.net/SvetaSmirnova
https://twitter.com/svetsmirnova
Thank you!
37