mongodb & mongomapper 4 real

4 real

railswaycon 2010, berlin.

jan krutisch <jan.krutisch@mindmatters.de>http://jan.krutisch.de/

Montag, 31. Mai 2010

http://www.cashbits.de/

mongodb you say?

document database

NoSQLincluded!

no fixed schemano migrations

rich data structure

id title descr pos_lat pos_lng

{ "_id" : ObjectId("4c00245062610475a005afcd"), "address" : "Bernstorffstr. 174\n22767 Hamburg\nDE", "description" : null, "position" : { "lat" : 53.5600912, "lng" : 9.9596977 }, "tags" : [ "hausarzt", "naturheilverfahren", "akupunktur", "allgemeinmedizin" ], "title" : "Dr. med. Lilo Eisenbarth", "loxicon_id" : 808261 }

✗Montag, 31. Mai 2010

‣ Binary serialized JSON‣ http://bsonspec.org/‣ Goals: Lightweight, Traversable, Efficient‣ Format for Datastorage and Wire

rich queries

Queries‣ expressed BSON query documents‣ very flexible‣ relatively simple query expressions‣ pretty close to SQL conceptually‣ examples will follow‣ on top: map/reduce for aggregation

Scaling

‣ Master > Slave replication‣ Replica Pairs (with Arbiter)‣ Replica Sets (Target: 1.6)‣ Autosharding (Target: 1.6)

A few words on durability

‣ MongoDB only fsyncs every <n> seconds‣ There‘s a desaster waiting to happen!‣ When in production, replicate!‣ This is not as bad as it sounds.

Installation/Hosting

OS X:$ brew install mongodb

Ubuntu/Debian:theres an apt for that

excuse the pun

http://www.mongodb.org/display/DOCS/DownloadsMontag, 31. Mai 2010

http://mongohq.com/Montag, 31. Mai 2010

http://mongomachine.com/Montag, 31. Mai 2010

basic usage

$ mongo

> use testswitched to db test

db.quotes.save({ text: "You can observe a lot just by watching.", from: "Yogi Berra", created_at: new Date() });

db.quotes.save({ text: "Silence is one of the hardest arguments to refute.", from: "Josh Billings", created_at: new Date() });

let‘s query

db.quotes.find();// returns all records in collection.

db.quotes.find({from: "Yogi Berra"});{ "_id" : ObjectId("4c0022551496fc2051e93695"), "text" : "You can observe a lot just by watching.", "from" : "Yogi Berra", "created_at" : "Fri May 28 2010 22:06:45 GMT+0200 (CEST)" }

$lt <$gt >$lte <=$gte >=$ne !=

db.quotes.find({from: {"$ne": "Yogi Berra"}});{ "_id" : ObjectId("4c0022551496fc2051e93696"), "text" : "Silence is one of the hardest arguments to refute.", "from" : "Josh Billings", "created_at" : "Fri May 28 2010 22:06:45 GMT+0200 (CEST)" }

$in IN (2,3,4)$nin NOT IN$all [2,3] ~ [1,2,3]

db.quotes.find({from:{ "$in":["Yogi Berra","Josh Billings"]}});{ "_id" : ObjectId("4c0022551496fc2051e93695"), "text" : "You can..."...}{ "_id" : ObjectId("4c0022551496fc2051e93696"), "text" : "Silence..."...}

db.arrays.save({list: [1,2,3]});db.arrays.save({list: [4,5,6]});db.arrays.save({list: [3,4,5]});

db.arrays.find({list:{ "$in":[3,4]}});{ "_id" : ObjectId("4c0025461496fc2051e93697"), "list" : [ 1, 2, 3 ] }{ "_id" : ObjectId("4c0025501496fc2051e93698"), "list" : [ 4, 5, 6 ] }{ "_id" : ObjectId("4c0025591496fc2051e93699"), "list" : [ 3, 4, 5 ] }

db.arrays.find({list:{ "$all":[3,5]}});{ "_id" : ObjectId("4c0025591496fc2051e93699"), "list" : [ 3, 4, 5 ] }

$mod yah, RLY$size okay$exists NOT NULL$type huh?

db.arrays.find({list:{ "$mod":[4,0]}});{ "_id" : ObjectId("4c0025501496fc2051e93698"), "list" : [ 4, 5, 6 ] }{ "_id" : ObjectId("4c0025591496fc2051e93699"), "list" : [ 3, 4, 5 ] }

db.arrays.find({list:{ "$size":3}}); { "_id" : ObjectId("4c0025461496fc2051e93697"), "list" : [ 1, 2, 3 ] }{ "_id" : ObjectId("4c0025501496fc2051e93698"), "list" : [ 4, 5, 6 ] }{ "_id" : ObjectId("4c0025591496fc2051e93699"), "list" : [ 3, 4, 5 ] }

db.arrays.find({list: {"$exists": true}});[...]

db.arrays.find({list: {"$type": 1}});[...]

...and...

db.quotes.find({from: /^Yog/});[...]

db.quotes.find("this.from == 'Yogi Berra'");[...]

db.quotes.find({"$where": "this.from == 'Yogi Berra'"});[...]

sort()

db.quotes.find().sort({from:1}) {"from" : "Josh Billings" ... }{"from" : "Yogi Berra" ...}

db.quotes.find().sort({from:-1}){"from" : "Yogi Berra" ...}{"from" : "Josh Billings" ... }

limit()

skip() // == OFFSET

count()

db.quotes.find().count(); 2

Indices

Indexing

‣ Same concept as SQL-Indices‣ You want them. (Same concept as with...)‣ Sort order, unique, compound, geospatial

db.quotes.ensureIndex({from: 1});

db.quotes.ensureIndex({from: -1});

db.quotes.ensureIndex({text: 1}, {unique: true});

db.quotes.ensureIndex({from: 1, text: 1});

db.quotes.dropIndexes();

db.quotes.dropIndex({from: 1, text: 1});

db.quotes.reIndex();

map/reduce, we can haz it, too

function() { this.tags.forEach(function(z) { emit(z, {count: 1}); }); }

function(key, values) { var total = 0; values.forEach(function(v) { total += v }); return {count: total} }

(it‘s not fast...)

one more thing

GridFS

GridFS file storage

‣ Binary fields in BSON limited to 4MB‣ GridFS API fixes that, files stored as chunks‣ Use the language drivers

I‘m in u‘r rubies,querying teh MongoDB!

ruby integration‣ mongo gem‣ bson/ bson_ext gem

‣ mongo_mapper‣ mongoid

Basic driver usage

require 'rubygems'require 'mongo'

db = Mongo::Connection.new.db("test")doc = { :text => "You can observe a lot just by watching.", :from => "Yogi Berra", :created_at => Time.now}db['quotes'].insert(doc)db['quotes'].find.each do |row| puts row.inspectend

{ "_id"=>$oid4bffe2896261046e79000001, "from"=>"Yogi Berra", "created_at"=>Fri May 28 15:34:33 UTC 2010, "text"=>"You can observe a lot just by watching."}

require 'rubygems'require 'mongo'

db = Mongo::Connection.new.db("test")

100.times do |i| db['numbers'].insert({"i" => i})end

db['numbers'].find("i" => {"$lt" => 2}).each do |row| puts row.inspectend

# {"_id"=>$oid4bffe4396261046f25000001, "i"=>0}# {"_id"=>$oid4bffe4396261046f25000002, "i"=>1}

db['text_entries'].drop_index("tags_1")db['text_entries'].create_index("tags")db['text_entries'].index_information

GridFS usage

db = Mongo::Connection.new.db("test")

grid = Mongo::Grid.new(db)

id = grid.put("You can put Strings in here", :filename => 'test.txt')

file = grid.get(id)puts file.filenameputs file.read

grid.delete(id)

grid.put( File.open("/Users/jankrutisch/Dropbox/Photos/IMGP8989.jpg"))

fs = Mongo::GridFileSystem.new(db)

fs.open("test.txt", "w") do |f| f.write "You can put stuff in here"end

fs.open("test.txt", "r") do |f| puts f.readend

fs.delete("test.txt")

mongo_mapper‣ By John Nunemaker (@jnunemaker)‣ works‣ a few quirks‣ almost completely undocumented‣ Some stuff is still missing

# config/initializers/mongo_mapper.rbFile.open(File.join(Rails.root, 'config/mongodb.yml'), 'r') do |f| @settings = YAML.load(f)[Rails.env]end

MongoMapper.connection = Mongo::Connection.from_uri(@settings["connection"]) if @settings["connection"]

MongoMapper.database = @settings["database"]

class Loop include MongoMapper::Document key :name key :public, Boolean key :message_id key :plays_and_downloads, Integer

belongs_to :user timestamps!end

@loops = Loop.all(! :user_id => {"$exists" => true},! :order => 'created_at DESC', ! :limit => 10)

„created_at DESC“ ?!?

mongoid

mongoid‣ By Durran Jordan (Hashrocket)‣ Two major versions:‣ 1.x (currently 1.9) for Rails 2.3 compatibility‣ 2.x (currently 2.x beta) for Rails 3 compatibility

‣ Good documentation‣ API is better (?)

File.open(File.join(RAILS_ROOT, 'config/mongodb.yml'), 'r') do |f| @settings = YAML.load(f)[RAILS_ENV]end

Mongoid::Config.instance.from_hash(@settings)

class Loop include Mongoid::Document include Mongoid::Timestamps

field :name field :public, :type => Boolean field :message_id field :plays_and_downloads, :type => Integer

belongs_to_related :userend

Criteria API

‣ A bit like Arel‣ chainable method calls‣ Named scopes

Embedded Documents

class Person include Mongoid::Document field :first_name field :last_name embeds_one :address embeds_many :phonesend

class Address include Mongoid::Document field :street field :city field :state field :post_code embedded_in :person, :inverse_of => :addressend

GridFS

acts_as_attachment

class Loop include Mongoid::Document include Mongoid::Timestamps

include Mongoid::Grid field :name field :public, :type => Boolean field :message_id field :plays_and_downloads, :type => Integer

attachment :nanend

using MongoDB 4 real

Installation is easy

(if you‘re using Ubuntu)

setting up replication

$ mongod --masterormaster = true # mongodb.conf

$ mongod --slave --source slaveserver.example.com

slave = truesource = slaveserver.example.com

OpLog size!

„security“

memory usage?

limits?

stability?

bonus level

you‘ve made it this far!

MongoDB explain()ed

db.text_entries.find({tags: "restaurant"}).limit(10).explain();{! "cursor" : "BtreeCursor tags_1",! "indexBounds" : [! ! [! ! ! {! ! ! ! "tags" : "restaurant"! ! ! },! ! ! {! ! ! ! "tags" : "restaurant"! ! ! }! ! ]! ],! "nscanned" : 210,! "nscannedObjects" : 210,! "n" : 10,! "millis" : 0,! [...]}

My personal impression

Extremely easy to grasp

Normalisation suXXorZ

Migrations hurt

Seems to be very fast

Some issues for small projects

thanks for listening.

Moi‣ jan.krutisch@mindmatters.de‣ http://jan.krutisch.de/‣ http://github.com/halfbyte/‣ http://twitter.com/halfbyte‣ http://www.mindmatters.de/

Pointers

‣ http://www.mongodb.org/‣ http://www.mongoid.org/‣ http://github.com/jnunemaker/mongo_mapper

mongodb & mongomapper 4 real

Technology

mongodb days uk: mongodb and spark

mongodb and using mongodb with .net

mongodb: what, why, when. solutions architect, mongodb inc....

mongodb evenings minneapolis: medtronic's mongodb journey

mongodb world 2016: mongodb + google cloud

mongodb são paulo - utilizando mongodb com .net

real time data analytics with mongodb and fluentd at wish

mongodb world 2016: mongodb & ibm

philipp krenn @xeraa for real...→ mongodb...

realtime analytics with mongodb - mongodb meetup nyc

a real systemwithjms-rest-protobuf-mongodb

building real time systems on mongodb using the oplog at...

mongodb europe 2016 - mongodb atlas

developing a real time application on the cloud using node...

real value in real time: mongodb-based analytics at...

mongomapper - mapping ruby to and from mongo

nosql, mongodb y mongomapper

real time ecommerce analytics with mongodb at gilt groupe...

mongodb training | mongodb online training | mongodb...

nosql concepts mongodb concepts mongodb demos agenda