Application Development SeriesBack to BasicsInteracting with the database

Daniel Roberts@dmroberts

#MongoDBBasics

2

• Recap from last session

• MongoDB Inserts & Queries– ObjectId– Returning documents – cursors– Projections

• MongoDB Update operators– Fixed Buckets– Pre Aggregated Reports

• Write Concern– Durability vs Performance trade off

Agenda

3

• Virtual Genius Bar

– Use the chat to post questions

– EMEA Solution Architecture / Support team are on hand

– Make use of them during the sessions!!!

Q & A

Recap from last time….

5

• Looked at the application architecture– JSON / RESTful– Python based

Architecture

Client-sideJSON

(eg AngularJS) (BSON)

Pymongo driver

Python web app

HTTP(S) REST

6

• Schema design– Modeled

• Articles• Comments• Interactions• Users

Schema and Architecture

7

Modeling Articles

• Posting articles• insert

• Get List of articles• Return Cursor

• Get individual article

{ '_id' : ObjectId(...),

'text': 'Article content…',

'date' : ISODate(...),

'title' : ’Intro to MongoDB',

'author' : 'Dan Roberts',

'tags' : ['mongodb',

'database',

'nosql’]

}

Articles collection

METHODSdef get_article(article_id)def get_articles():def create_article():

8

Modeling Comments

• Storing comments

• Quickly retrieve most recent comments

• Add new comments to document

• ‘Bucketing’

{‘_id’ : ObjectId(..),

‘article_id’ : ObjectId(..), ‘page’ : 1, ‘count’ : 42 ‘comments’ : [

{ ‘text’ : ‘A great article, helped me understand schema design’, ‘date’ : ISODate(..), ‘author’ : ‘johnsmith’ }, …}

Comments collection

METHODSdef add_comment(article_id):def get_comments(article_id):

9

Modeling Interactions

• Used for reporting on articles

• Create “pre-aggregated” reports

{‘_id’ : ObjectId(..),

‘article_id’ : ObjectId(..), ‘section’ : ‘schema’,

‘date’ : ISODate(..),‘daily’: { ‘views’ : 45,

‘comments’ : 150 } ‘hours’ : { 0 : { ‘views’ : 10 }, 1 : { ‘views’ : 2 }, … 23 : { ‘views’ : 14,

‘comments’ : 10 } }}

Interactions collection

METHODS def add_interaction(article_id, type):

Inserting / Querying

11

>db.articles.insert({'text': 'Article content…’,

'date' : ISODate(...), 'title' : ’Intro to MongoDB’, 'author' : 'Dan Roberts’, 'tags' : [ 'mongodb',

'database', 'nosql’

]});

• Driver generates ObjectId() for _id – if not specified– 12 bytes - 4-byte epoch, 3-byte machine id, a 2-byte process id, and a 3-byte

counter.

Inserting documents

12

$gt, $gte, $in, $lt, $lte, $ne, $nin

• Use to query documents

• Logical: $or, $and, $not, $nor Element: $exists, $type

• Evaluation: $mod, $regex, $where Geospatial: $geoWithin, $geoIntersects, $near, $nearSphere

Comparison Operators

db.articles.find( { 'title' : ’Intro to MongoDB’ } )

db.articles.find( { ’date' : { ‘$lt’ : {ISODate("2014-02-

19T00:00:00.000Z") }} )

db.articles.find( { ‘tags’ : { ‘$in’ : [‘nosql’, ‘database’] } } );

13

• Find returns a cursor– Use to iterate over the results– cursor has many methods

Cursors

>var cursor = db.articles.find ( { ’author' : ’Dan Roberts’ } )>cursor.hasNext()true>cursor.next(){ '_id' : ObjectId(...),

'text': 'Article content…’, 'date' : ISODate(...), 'title' : ’Intro to MongoDB’, 'author' : 'Dan Roberts’, 'tags' : [ 'mongodb', 'database’, 'nosql’ ]

}

14

• Return only the attributes needed– Boolean 0 or 1 syntax select attributes– Improved efficiency

Projections

>var cursor = db.articles.find( { ’author' : ’Dan Roberts’ } , {‘_id’:0, ‘title’:1})>cursor.hasNext()true>cursor.next(){ "title" : "Intro to MongoDB" }

Updates

16

$each, $slice, $sort, $inc, $push$inc, $rename, $setOnInsert, $set, $unset, $max, $min

$, $addToSet, $pop, $pullAll, $pull, $pushAll, $push

$each, $slice, $sort

Update Operators

>db.articles.update(

{ '_id' : ObjectId(...)},

{ '$push' : {'comments' : ‘Great

article!’ }}

)

{ 'text': 'Article content…’ 'date' : ISODate(...),

'title' : ’Intro to MongoDB’,

'author' : 'Dan Roberts’,'tags' : ['mongodb',

'database’,'nosql’ ],’comments' :

[‘Great article!’ ]

}

17

Push to a fixed size array with…

$push, $each, $slice

Update Operators

>db.articles.update(

{ '_id' : ObjectId(...)},

{ '$push' : {'comments' :

{

'$each' : [‘Excellent’], '$slice' : -3}}, })

{ 'text': 'Article content…’ 'date' : ISODate(...),

'title' : ’Intro to MongoDB’,

'author' : 'Dan Roberts’,'tags' : ['mongodb',

'database’,'nosql’ ],’comments' :

[‘Great article!’, ‘More please’, ‘Excellent’ ]

}

18

• Push 10 comments to a document (bucket).

• Automatically create a new document.

• Use {upsert: true} instead of insert.

Update Operators - Bucketing

>db.comments.update(

{‘c’: {‘$lt’:10}}, {

‘$inc’ : {c:1}, '$push' : {

'comments' : ‘Excellent’}},{ upsert : true }

)

{‘_id’ : ObjectId( … )‘c’ : 3,’comments' :

[‘Great article!’,

‘More please’, ‘Excellent’ ]

}

19

Analytics – Pre-Aggregated reports

• Used for reporting on articles

• Create “pre-aggregated” reports

{‘_id’ : ObjectId(..),

‘article_id’ : ObjectId(..), ‘section’ : ‘schema’,

‘date’ : ISODate(..),‘daily’: { ‘views’ : 45,

‘comments’ : 150 } ‘hours’ : { 0 : { ‘views’ : 10 }, 1 : { ‘views’ : 2 }, … 23 : { ‘views’ : 14,

‘comments’ : 10 } }}

Interactions collections

METHOD def add_interaction(article_id, type):

20

• Use $inc to increment multiple counters.

• Single Atomic operation.

• Increment daily and hourly counters.

Incrementing Counters

>db.interactions.update({‘article_id’ : ObjectId(..)}, {

‘$inc’ : {

‘daily.views’:1,

‘daily.comments’:1

‘hours.8.views’:1

‘hours.8.comments’:1 })

{‘_id’ : ObjectId(..),

‘article_id’ : ObjectId(..), ‘section’ : ‘schema’,

‘date’ : ISODate(..),‘daily’: { ‘views’ : 45,

‘comments’ : 150 } ‘hours’ : { 0 : { ‘views’ : 10 }, 1 : { ‘views’ : 2 }, … 23 : { ‘views’ : 14,

‘comments’ : 10 } }}

21

• Increment new counters

Incrementing Counters 2

>db.interactions.update({‘article_id’ : ObjectId(..)}, {

‘$inc’ : {

‘daily.views’:1,

‘daily.comments’:1,

‘hours.8.views’:1,

‘hours.8.comments’:1,

‘referrers.bing’ : 1})

{‘_id’ : ObjectId(..),

‘article_id’ : ObjectId(..), ‘section’ : ‘schema’,

‘date’ : ISODate(..),‘daily’: { ‘views’ : 45,

‘comments’ : 150 } ‘hours’ : {

…..}‘referrers’ : {

‘google’ : 27}

}

22

• Increment new counters

Incrementing Counters 2

>db.interactions.update({‘article_id’ : ObjectId(..)}, {

‘$inc’ : {

‘daily.views’:1,

‘daily.comments’:1,

‘hours.8.views’:1,

‘hours.8.comments’:1,

‘referrers.bing’ : 1})

{‘_id’ : ObjectId(..),

‘article_id’ : ObjectId(..), ‘section’ : ‘schema’,

‘date’ : ISODate(..),‘daily’: { ‘views’ : 45,

‘comments’ : 150 } ‘hours’ : {

…..}‘referrers’ : {

‘google’ : 27,‘bing’ : 1

}}

Durability

24

Durability

• With MongoDB you get to choose• In memory• On disk• Multiple servers

• Write Concerns• Report on success of write operations• getLastError called from driver

• Trade off• Latency of response

25

Unacknowledged

26

MongoDB Acknowledged

Default Write Concern

27

Wait for Journal Sync

28

Replica Sets

• Replica Set – two or more copies

• “Self-healing” shard

• Addresses many concerns:

- High Availability

- Disaster Recovery

- Maintenance

29

Wait for Replication

Summary

31

• Interacting with the database

– Queries and projections– Inserts and Upserts

– Update Operators– Bucketing– Pre Aggregated reports

• basis for fast analytics

Summary

32

– Indexing• Indexing strategies• Tuning Queries

– Text Search

– Geo Spatial

– Query Profiler

Next Session – 6th March