mongosv schema workshop

Schema Design Workshop

Sridhar Nanjundeswaran

Software Engineer, 10Gensridhar@10gen.com

@snanjund

Wednesday, December 5, 12

Agenda

• Part One - Basic Schema & Patterns• Part Two - Schema Design• Part Three - Sharding• Part Four: - Replication

Why is schema design different?• RDBMS design you ask "what answers do I have"

• MongoDB you ask "what questions will I have"

• Learn Data Modeling with MongoDB• Labs to try to solve problems• Understand implications of• Replication • Sharding

Please, ask many, many questions!

Part OneBasic Schema & Patterns

So why model data?

http://bit.ly/SSs7QB

Normalization• 1970 E.F.Codd introduces 1st Normal Form (1NF)• 1971 E.F.Codd introduces 2nd and 3rd Normal Form (2NF, 3NF)• 1974 Codd & Boyce define Boyce/Codd Normal Form (BCNF)• 2002 Date, Darween, Lorentzos define 6th Normal Form (6NF)

Goals:• Avoid anomalies when inserting, updating or deleting• Minimize redesign when extending the schema• Make the model informative to users• Avoid bias towards a particular style of query

* source : wikipediaWednesday, December 5, 12

So today’s example will use...

http://bit.ly/RyIOvO

TerminologyRDBMS MongoDB

Table Collection

Row(s) JSON Document

Index Index

Join Embedding & Linking

Partition Shard

Partition Key Shard Key

Schema DesignRelational Database

Schema DesignMongoDB

linking

embedding

linking

Basic schema

Design documents that simply map to your application

> post = { author: "Hergé", date: ISODate("2011-09-18T09:56:06.298Z"), text: "Destination Moon", tags: ["comic", "movie"] }

> db.blogs.save(post)

> db.blogs.find()

{ _id: ObjectId("4c4ba5c0672c685e5e8aabf3"), author: "Hergé", date: ISODate("2011-09-18T09:56:06.298Z"), text: "Destination Moon", tags: [ "comic", "movie" ] } Notes:• ID must be unique, but can be anything you’d like• MongoDB will generate a default ID if one is not supplied

Find the document

Secondary index for “author”

// 1 means ascending, -1 means descending> db.blogs.ensureIndex( { author: 1 } )

> db.blogs.find( { author: 'Hergé' } ) { _id: ObjectId("4c4ba5c0672c685e5e8aabf3"), date: ISODate("2011-09-18T09:56:06.298Z"), author: "Hergé", ... }

Add an index, find via Index

Examine the query plan

> db.blogs.find( { author: "Hergé" } ).explain(){! "cursor" : "BtreeCursor author_1",! "nscanned" : 1,! "nscannedObjects" : 1,! "n" : 1,! "millis" : 5,! "indexBounds" : {! ! "author" : [! ! ! [! ! ! ! "Hergé",! ! ! ! "Hergé"! ! ! ]! ! ]! }}

How long it took

Number of objects returned

Query operatorsConditional operators: $ne, $in, $nin, $mod, $all, $size, $exists, $type, .. $lt, $lte, $gt, $gte, $ne...

// find posts with any tags> db.blogs.find( { tags: { $exists: true } } )

Regular expressions:// posts where author starts with h> db.blogs.find( { author: /^h/i } )

Counting: // number of posts written by Hergé> db.blogs.find( { author: "Hergé" } ).count()

Extending the Schema

http://bit.ly/PpjT1l

Extending the Schema> new_comment = { author: "Kyle", date: new Date(), text: "great book" }

> db.blogs.update( { text: "Destination Moon" }, { "$push": { comments: new_comment }, "$inc": { comments_count: 1 } } )

Extending the Schema> new_comment = { author: "Kyle", date: new Date(), text: "great book" }

> db.blogs.update( { text: "Destination Moon" }, { "$push": { comments: new_comment }, "$inc": { comments_count: 1 } } )

Increment counterAdd element to

> db.blogs.find( { author: "Hergé"} )

{ _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author : "Hergé", date : ISODate("2011-09-18T09:56:06.298Z"), text : "Destination Moon", tags : [ "comic", "movie" ], comments : [! {! ! author : "Kyle",! ! date : ISODate("2011-09-19T09:56:06.298Z"),! ! text : "great book"! } ], comments_count: 1 }

// create index on nested documents:> db.blogs.ensureIndex( { "comments.author": 1 } )

> db.blogs.find( { "comments.author": "Kyle" } )

// find last 5 posts:> db.blogs.find().sort( { date: -1 } ).limit(5)

// most commented post:> db.blogs.find().sort( { comments_count: -1 } ).limit(1)

When sorting, check if you need an index

Common Patterns

http://bit.ly/SNnt4z

Inheritance

http://bit.ly/T7MqUz

Inheritance

select * from shapes;

id type area radius length width

1 circle 3.14 1

2 square 4 2

3 rect 10 5 2

Single Table Inheritance - RDBMS

Single Table Inheritance - MongoDB> db.shapes.find() { _id: "1", type: "c", area: 3.14, radius: 1} { _id: "2", type: "s", area: 4, length: 2} { _id: "3", type: "r", area: 10, length: 5, width: 2}

missing values not stored!

// find shapes where radius > 0 > db.shapes.find( { radius: { $gt: 0 } } )

// create index> db.shapes.ensureIndex( { radius: 1 }, { sparse:true } )

index only values present!

One to Many

http://bit.ly/Oqbt8z

One to Many

One to Many relationships can specify• degree of association between objects• containment• life-cycle

One to ManyEmbedded Array

•$slice operator to return subset of comments•some queries harder

•e.g find latest comments across all blogs

blogs: { author : "Hergé", date : ISODate("2011-09-18T09:56:06.298Z"), comments : [! { author : "Kyle",! ! date : ISODate("2011-09-19T09:56:06.298Z"),! ! text : "great book" } ] }

> db.blogs.find( { author: "Hergé" }, { comment: { $slice : 10 } } )

One to ManyNormalized (2 collections)• most flexible• more queries

blogs: { _id: 1000, author: "Hergé", date: ISODate("2011-09-18T09:56:06.298Z"), comments: [! {comment : 1)} ]}

comments : { _id : 1, blog: 1000, author : "Kyle",! ! date : ISODate("2011-09-19T09:56:06.298Z")}

> blog = db.blogs.find( { text: "Destination Moon" } );> db.comments.find( { blog: blog._id } ).limit(5);

Many to Many

http://bit.ly/QTzhBF

Many - Many

Example: • Blog can have many Tags• Tag can be used by many Blogs

// Each Tag lists the "_id" of the Blogtags: { _id: 20, name: "comic", // Unique blog_ids: [ 10, 11, 12 ] }

{ _id: 30, name: "movie", // Unique blog_ids: [ 10 ] }

Many - Many

// Each Blog lists the "tag" of the Tagsblogs: { _id: 10, name: "Destination Moon", tags: [ "comic", "movie" ] }

Many - Many

// Each Blog lists the "tag" of the Tagsblogs: { _id: 10, name: "Destination Moon", tags: [ "comic", "movie" ] }

Many - Many

links via unique key, in this case "tags", could be "_id"

// Each Blog lists the "tag" of the Tagsblogs: { _id: 10, name: "Destination Moon", tags: [ "comic", "movie" ] } // All Tags for a given Blog> db.tags.find( { blog_ids: 10 } )

Many - Many

Use _id or not?

blogs: { _id: 10, name: "..." tags: [ "comic", "movie" ] }

Pros:• Single query

Cons:• Cascade any changes

blogs: { _id: 10, name: "..." tags: [ 10, 20 ] }

Pros:• Single update

Cons:• Second query required

// Each Blog lists the _id of the Tagblogs: { _id: 10, name: "Destination Moon", tag_ids: [ 20, 30 ] } // Association not stored on the Tagtags: { _id: 20, name: "comic" }

Alternative

// All Blogs for a given Tag> db.blogs.find( { tag_ids: 20 } )

Alternative

// All Blogs for a given Tag> db.blogs.find( { tag_ids: 20 } )

// All Tags for a given Blog> blog = db.blogs.findOne( { _id: 10 } )> db.tags.find({_id: {$in : blog.tag_ids}})

Alternative

Many - Many Intersection AttributesExample: • Blog can have many Tags• Tag can be used my many Blogs• When a Tag is used, record the usage date

// Each Blog lists the _id of the Tagblogs: { _id: 10, name: "...", tag_ids: [ 20, 30 ] } // Association not stored on the Tagtags: { _id: 20, name: "comic" }

// Store the interaction and usage dateusages: { blog_id: 10, // Blog _id tag_id : 20, // Tag _id usage: ISODate("2012-10-12...") }

// Find the Tags for a Blogfor(var c = db.usages.find({ blog_id: 10 }); c.hasNext(); ){ u = c.next(); t = db.tags.findOne( { _id: c.tag_id } ) printjson( u.usage );

Many - Many Normalized

// Each Blog lists the Blog Usage Objectblogs: { _id: 10, name: "Destination Moon", tags: [ { tag: "comic", usage: ISODate("2012-10-12...") } { tag: "movie", usage: ISODate("2012-09-11...") } ] }

// Find the Tags for a Blog> db.blogs.find( { _id: 10 }, { tags: 1} ) Pros:• Usage object encapsulated where used

Cons:• If updates allowed, changes will have to be cascaded

Many - Many Intersection Attributes

Summary

• Single biggest performance factor

• More choices than in an RDBMS

• Embedding, index design, shard keys

Part TwoSchema Design

Lab #1Design Schema for Twitter

• Model each users activity stream• Users

• Name, email address, display name• Tweets

• Text• Who• Timestamp

Lab #1 - Solution ATwo Collections// users - one doc per user{ _id: "alvin", email: "alvin@10gen.com", display: "jonnyeight"}

// tweets - one doc per user per tweet{ user: "bob", for: "alvin", tweet: "20111209-1231", text: "Best Tweet Ever!", ts: ISODate("2011-09-18T09:56:06.298Z")}

Lab #1 - Solution BEmbedded Tweets// users - one doc per user with all tweets{ _id: "alvin", email: "alvin@10gen.com", display; "jonnyeight", tweets: [! {! ! user: "bob",! ! tweet: "20111209-1231",! ! text: "Best Tweet Ever!", ts: ISODate("2011-09-18T09:56:06.298Z")! } ]}

Embedding

• Great for read performance

• One seek to load entire object

• One roundtrip to database

• Writes can be slow if adding to objects all the time

Linking or Embedding?

Linking can make some queries easy

// Find latest 50 tweets for "alvin"> db.tweets.find( { _id:"alvin"} ) .sort( {ts:-1} ) .limit(50)

But what effect does this have on the systems?

Collection 1

Index 1

Virtual Address Space 1

Collection 1

Index 1 This is your virtual memory size

(mapped)

Physical RAM

Collection 1

Index 1

This is your resident

memory size

Physical RAM

DiskCollection 1

Index 1

Physical RAM

DiskCollection 1

Index 1

100 ns

10,000,000 ns

Physical RAM

DiskCollection 1

Index 1

> db.tweets.find( { _id: "alvin" } ) .sort( { ts: -1 } ) .limit(10)

Linking = Many seeks + random reads

Physical RAM

DiskCollection 1

Index 1

Embedding = Large Sequential Read

> db.tweets.find( { _id: "alvin" } )

Lab #2Alternative Schema

• Display last 10 tweets from today• Efficiently use memory and Disk seeks / IOPs

Lab #2 - SolutionBuckets// tweets : one doc per user per day> db.tweets.findOne()

{ _id: "alvin-2011/12/09", email: "alvin@10gen.com", tweets: [ { user: "Bob",! tweet: "20111209-1231",! text: "Best Tweet Ever!" } , ! { author: "Joe",! tweet: "20111210-9025",! date: "May 27 2011",! text: "Stuck in traffic (again)" } ]}

Lab #2 - SolutionLast 10 Tweets

> db.tweets.find( { _id: "alvin-2011/12/09" }, { tweets: { $slice : 10 } } ) .sort( { _id: -1 } ) .limit(1)

Lab #2 - SolutionAdding a Tweet> tweet = { user: "Bob",! tweet: "20111209-1231",! text: "Best Tweet Ever!" }

> db.tweets.update( { _id : "alvin-2011/12/09" }, { $push : { tweets : tweet } );

Lab #2 - SolutionGetting All Tweets> cursor = db.tweets.find ( { _id : /^alvin/ } ).sort( { _id : -1 } )

> while ( cursor.hasNext() ) { doc = cursor.next(); for ( var i=0; i<doc.tweets.length; i++ ) printjson( doc.tweets[i] )}

Lab #2 - SolutionDeleting a Tweet> db.tweets.update( { _id: "alvin-20111209" }, { $pull: { tweets: { tweet: "20111209-1231" } })

Physical RAM

DiskCollection 1

Index 1

> db.tweets.find( { _id: "alvin-2011/12/09" }, { tweets: { $slice : 10 } } ) .sort( { _id: -1 } ) .limit(1)

Bucket = 1 seek + 1 sequential read

http://bit.ly/Oqc8Xs

Hierarchical information

Full Tree in Document

{ retweet: [ { who: “Kyle”, text: “...”, retweet: [ {who: “James”, text: “...”, retweet: []} ]} ]}

Pros: Single Document, Performance, Intuitive

Cons: Hard to search, Partial Results, 16MB limit

Array of Ancestors// Store all Ancestors of a node { _id: "a" } { _id: "b", tree: [ "a" ], retweet: "a" } { _id: "c", tree: [ "a", "b" ], retweet: "b" } { _id: "d", tree: [ "a", "b" ], retweet: "b" } { _id: "e", tree: [ "a" ], retweet: "a" } { _id: "f", tree: [ "a", "e" ], retweet: "e" }

// find all direct retweets of "b"> db.tweets.find( { retweet: "b" } )

// find all retweets of "e" anywhere in tree> db.tweets.find( { tree: "e" } )

// find tweet history of f:> tweets = db.tweets.findOne( { _id: "f" } ).tree> db.tweets.find( { _id: { $in : tweets } } )

Trees as Paths

Store hierarchy as a path expression• Separate each node by a delimiter, e.g. “/”• Use text search for find parts of a tree

{ retweets: [ { _id: "a", text: "initial tweet", path: "a" }, { _id: "b", text: "reweet with comment", path: "a/b" }, { _id: "c", text: "reply to retweet", path : "a/b/c"} ] }

// Find the conversations "a" started > db.tweets.find( { path: /^a/i } )

http://bit.ly/QeNsPX

Queues & Workflows

Lab #3Following Requests• Users are allowed to "follow" another user

• User send a "follow" request• Follower approves or not• Requests are timed out after 7 days

• The approval is an async process

Lab #3 - SolutionQueues & Workflows• Need to maintain order and state• Ensure that updates are atomic

> db.approvals.insert( { inprogress: false, approved: false, priority: 1, text: "Hey Jim, want to follow you!" } );// find highest priority approval and mark as in-progressjob = db.approvals.findAndModify({ query: { inprogress: false }, sort: { priority: -1 }, update: { $set: { inprogress: true, started: new Date() } }, new: true})

Lab #3 - SolutionQueues & Workflows• Need to maintain order and state• Ensure that updates are atomic

> db.approvals.insert( { inprogress: false, approved: false, priority: 1, text: "Hey Jim, want to follow you!" } );// find highest priority approval and mark as in-progressjob = db.approvals.findAndModify({ query: { inprogress: false }, sort: { priority: -1 }, update: { $set: { inprogress: true, started: new Date() } }, new: true})

Lab #3 - SolutionQueues & Workflows

{ inprogress: true, priority: 1, approved: False, started: ISODate("2011-09-18T09:56:06.298Z") ... }

updated

Lab #3 - SolutionQueues & Workflows• Follower approves request

// update approval after receiving approval> job = db.approvals.update( { _id: "1234" }, { $set: { approved: true } } )

• System times out request after 7 days

var limit=new Date();limit.setDate(limit.getDate()-7);

> job = db.approvals.update( { inprogress: true, started: { $gt: limit} }, { $set: { approved: false } } )

Lab #4Voting

Twitter meets Stack Overflow

• Users can "vote" for a tweet• A user can "vote" once and only once• Need to display current votes

Lab #4 - SolutionVotes// One document per voter per tweet> db.votes.insert( { tweet: "20111209-1231", voter: "alvin" } );

// Unique index guarantees the user can't vote twice> db.votes.ensureIndex( { tweet: 1, voter: 1 }, { unique: true } );

// Count will return the number of votes cast> db.votes.find({ tweet: "20111209-1231" }).count()

Count or Not?

• Indexes in MongoDB are not counting• The count has to be computed via a index scan

// One summary document per tweet, no "voter" key> db.votes.update( { tweet: "20111209-1231", voter: { $exists: false } }, { "$inc": { count: 1 } }, true, false );

// Return the count for the no "voter" document> db.votes.find( { tweet: "20111209-1231", voter: { $exists: false } }, { count: 1, _id: 0} )

Lab #5Time Series• Records votes by

• Day, Hour, Minute• Show time series of votes cast

Lab #5 - Solution ATime Series// Time series buckets, hour and minute sub-docs{ _id: "20111209-1231", ts: ISODate("2011-12-09T00:00:00.000Z") daily: 67, hourly: { 0: 23, 1: 14, 2: 19 ... 23: 72 }, minute: { 0: 0, 1: 4, 2: 6 ... 1439: 0 }}

Lab #5 - Solution ATime Series// Add one to the last minute before midnight> db.votes.update( { _id: "20111209-1231", ts: ISODate("2011-12-09T00:00:00.037Z") }, { $inc: { daily: 1 }, $inc: { "hourly.23": 1 }, $inc: { "minute.1439": 1 } )

What is the cost of updating the minute before midnight?

• Sequence of key/value pairs• NOT a hash map• Optimized to scan quickly

• 1439 skips

BSON Storage

...0 1 2 3 1439

• Can skip sub-documents

• 23 skips (hours) + 59 skips (minutes) = 82 skips

BSON Storage

... ...59

1380 143960 ... 119

Lab #5 - Solution BTime Series// Time series buckets, each hour a sub-document{ _id: "20111209-1231", ts: ISODate("2011-12-09T00:00:00.000Z") daily: 67, minute: { 0: { 0: 0, 1: 7, ... 59: 2 }, ... 23: { 0: 15, ... 59: 6 } }}

// Add one to the last second before midnight> db.votes.update( { _id: "20111209-1231" }, ts: ISODate("2011-12-09T00:00:00.000Z") }, { $inc: { daily: 1 }, $inc: { "minute.23.59": 1 } })

Lab #6Inventory

• User has a number of "votes" they can use

Lab #6 - SolutionInventory // Number of votes and who voted for { _id: "alvin", votes: 42, voted_for: [] }

// Subtract a vote and add the voted for tweet // "20111209-1231" > db.user.update( { _id: "alvin", votes : { $gt : 0}, voted_for: { $ne: "20111209-1231" }}, { "$push": { voted_for: "20111209-1231"}, "$inc": { votes: -1} } )

Lab #6 - SolutionInventory // After vote > db.votes.findOne() { _id: "alvin", votes: 41, voted_for: ["20111209-1231"] }

decremented

Lab #7Statistic Buckets• Record referring web sites on customer sign up• Independent counter for each web site

Lab #7 - Solution AStatistic Buckets{ _id: "alvin", referrers: [ { domain: "www.google.co.uk", count: 4 }, { domain: "www.yahoo.com", count: 1 }, ] }

> db.referers.update( { "referrers.domain": "www.google.co.uk" }, { $inc: { "referrers.$.count": 1 } } )

{ _id: "alvin", referrers: [ { domain: "www.google.co.uk", count: 5 }, { domain: "www.yahoo.com", count: 1 }, ] }

Lab #7 - Solution AStatistic Buckets

> db.referers.update( { "referrers.domain": "www.bing.com" }, { $inc: {"referrers.$.count": 1 } }, false, true ) What happens if a new referring site is used?

Lab #7 - Solution BStatistic Buckets// Need to replace dots with underscores{ _id: "alvin", referrers: { "www_google_co_uk": 4, "www_yahoo_com": 1 }, }

// simple $inc will add www_bing_com if not present> db.referers.update( { _id: "alvin" }, { $inc: { "referrers.www_bing_com": 1 } }, true, false);

Part ThreeSharding

What is Sharding

• Ad-hoc partitioning

• Consistent hashing• Amazon Dynamo

• Range based partitioning• Google BigTable• Yahoo! PNUTS• MongoDB

MongoDB Sharding

• Automatic partitioning and management

• Range based

• Convert to sharded system with no downtime

• Fully consistent

• No code changes required

Sharding - Range distribution

shard01 shard02 shard03

sh.shardCollection("mydb.tweets", {_id: 1} , false)

Sharding - Range distribution

a-i j-r s-z

Sharding - Splits

a-i ja-jz s-z

Sharding - Splits

a-i ja-ji s-z

Sharding - Auto Balancing

a-i ja-ji s-z

Sharding - Auto Balancing

a-i ja-ji s-z

Sharding for caching

shard01

300 GB

96 GB Mem3:1 Data/Mem

Aggregate Horizontal Resources

a-i j-r s-z

100 GB 100 GB 100 GB

Sharding Features• Shard data without no downtime • Automatic balancing as data is written• Commands routed (switched) to correct node

• Inserts - must have the Shard Key• Updates - can have the Shard Key• Queries

• With Shard Key - routed to nodes• Without Shard Key - scatter gather

• Indexed / Sorted Queries• With Shard Key - routed in order• Without Shard Key - distributed sort merge

Lab #8Sharding Twitter Pictures

User can upload pictures to Twitter feed

{ photo_id : ???? , data : <binary> }

What should photo_id be?How will photo_id be sharded?

Lab #8Sharding Key

{ photo_id : ???? , data : <binary> }

What’s the right key?• auto increment• MD5( data )• month() + MD5( data )

• Only have to keep small portion in ram• Right shard "hot" • Time Based

• ObjectId• Auto Increment

Right balanced access

• Have to keep entire index in ram• All shards "warm"

• Hash

Random access

• Have to keep some index in ram• Some shards "warm"

•Month + Hash

Segmented access

Lab #9Single Identities// Shard by _idids:{ _id : "alvin", email: "alvin@10gen.com", addresses: [ { state : "CA", country: "USA" }, { country: "UK" } ] }

How would the following queries be executed?

> db.ids.find( { _id: "alvin"} )> db.ids.find( { email: "alvin@10gen.com" } )

Sharding - Routed Query

a-i ja-ji s-z

find( { _id: "alvin"} )

Sharding - Routed Query

a-i ja-ji s-z

find( { _id: "alvin"} )

Sharding - Scatter Gather

a-i ja-ji s-z

find( { email: "alvin@10gen.com" } )

Sharding - Scatter Gather

a-i ja-ji s-z

find( { email: "alvin@10gen.com" } )

Lab #9Multiple Identities

User can have multiple identities• twitter name• email address• facebook name• etc.

What is the best sharding key & schema design?

Lab #9 - Solution AMultiple Identities

// Shard by _id{ _id: "alvin", email: "alvin@10gen.com", fb: "alvin.richards", // facebook li: "alvin.j.richards", // linkedin tweets: [ ... ] }

Lookup by _id hits 1 node Lookup by email, li or fb is scatter gather Cannot create a unique index on email, li or fb

Lab #9 - Solution BMultiple Identitiesidentities{ _id: { _id: "alvin"}, info: "1200-42"}{ _id: { em: "alvin@10gen.com"}, info: "1200-42"}{ _id: { li: "alvin.j.richards"}, info: "1200-42"}

tweets{ _id: "1200-42", tweets: [ ... ]}

• Shard identities on { _id: 1}• Can create unique index on _id• Shard info on { _id: 1 }

Sharding - Multiple Identities

idscollection

tweetscollection

em: a-q em: r-z _id: a-z

li: s-z

li: a-c

li: d-r_id: "Min"-"1100"

_id: "1100"-"1200"

_id: "1200"-"Max"

shard01 shard02 shard03em: a-q em: r-z _id: a-z

li: s-z

li: a-c

li: d-r_id: "Min"-"1100"

_id: "1100"-"1200"

_id: "1200"-"Max"

ids.find({ _id: {"em","alvin@10gen.com })

idscollection

tweetscollection

ids.find({ _id: {"em","alvin@10gen.com })

tweets.find({ _id: "1200-‐42" })

idscollection

tweetscollection

em: a-q em: r-z _id: a-z

li: s-z

li: a-c

li: d-r_id: "Min"-"1100"

_id: "1100"-"1200"

_id: "1200"-"Max"

Part FourReplication

Types of outage• Planned

• Hardware upgrade• O/S or file-system tuning• Relocation of data to new file-system / storage• Software upgrade

• Unplanned• Hardware failure• Data center failure• Region outage• Human error• Application corruption

Replica Sets

• Data Protection• Multiple copies of the data• Spread across Data Centers, AZs

• High Availability• Automated Failover• Automated Recovery

Replica Sets

Primary

Secondary

Asynchronous Replication

Replica Sets

Primary

Secondary

Replica Sets

Primary

Secondary

Automatic Election of new Primary

Replica Sets

Recovering

Primary

Secondary

New primary serves data

Replica Sets

Secondary

Primary

Secondary

Elections

During an election• Most up to date• Highest priority• Less than 10s behind failed Primary

Types of Durability with MongoDB• Fire and forget• Wait for error • Wait for fsync• Wait for journal sync • Wait for replication

Network Ack- Old Default

Driver Primary

apply in memory

Get last error - New default

Driver Primary

getLastError apply in memory

Wait for Journal Sync

Driver Primary

apply in memory

j:trueWrite to journal

getLastError

Wait for replication

Driver Primary

apply in memory

Secondary

replicate

getLastError

Tunable Data DurabilityMemory Journal Secondary Other Data Center

networkACK

w=1j=true

w="majority"w=n

w="myTag"

Less More

Eventual ConsistencyUsing Replicas for ReadsRead preference• primary (only)• primaryPreferred• secondary (only)• secondaryPreferred• nearest

Immediate Consistency

PrimaryThread #1

Insert

Update

Eventual Consistency

Primary SecondaryThread #1

Insert

Update

Thread #2

reads v1

v1 does not exist

✔ reads v2

✔reads v1

Lab #10Replication

Primary, Secondary or both?

• Show the latest "votes" for a tweet and/or user• Changing your profile picture• Showing your thumbnail with a tweet

Summary

• Schema design is different in MongoDB

• Basic data design principals stay the same

• Focus on how the application manipulates data

• Rapidly evolve schema to meet your requirements

• Consider sharding early

• Understand the impact of eventual consistency

@mongodb

conferences, appearances, and meetupshttp://www.10gen.com/events

http://bit.ly/mongo> Facebook | Twitter | LinkedIn

http://linkd.in/joinmongo

download at mongodb.org

mongosv schema workshop

herg db

schema db

btreecursor author

tags db

schema new

document db

descending db

new date

Documents

mongosv 2011

mongosv 2011 - replication

high dimensional indexing using mongodb (mongosv 2012)

european spatial data infrastructure conceptual schema...

{ontology: resource} x {matching : mapping} x {schema :...

mongosv 2012- mongo performance tuning (3)

xml schema xml-schema xml schema -...

13.06.02gml 2.01 Überblick: -einführung (einschließlich...

sql workshop guide - oracle cloud...changes in oracle...

-wp4 workshop- draft proposal for a fabric global schema

v13.00.00 retail market prioritisation process · v13.00.00...

schema delle posizioni (lallai 2) schema a. schema delle...

monitoring mongodb (mongosv)

mongodb, hadoop and humongous data - mongosv 2012

gus plugin system michael saffitz genomics unified schema...

evaluation of the small subset of euroroads according to iso...

module 4: designing a schema policy. overview identifying...

icat schema current schema organization what’s there but...

mongosv - schema design

aem communities 6.1 - mongosv '15