mongodb & mongomapper 4 real

4 real

railswaycon 2010, berlin.

jan krutisch <[email protected]>http://jan.krutisch.de/

Montag, 31. Mai 2010

mailto:[email protected]


http://jan.krutisch.de


http://www.cashbits.de/


http://www.cashbits.de

http://www.cashbits.de

mongodb you say?


document database


document database

NoSQLincluded!


no fixed schemano migrations


rich data structure


id title descr pos_lat pos_lng


{ "_id" : ObjectId("4c00245062610475a005afcd"), "address" : "Bernstorffstr. 174\n22767 Hamburg\nDE", "description" : null, "position" : { "lat" : 53.5600912, "lng" : 9.9596977 }, "tags" : [ "hausarzt", "naturheilverfahren", "akupunktur", "allgemeinmedizin" ], "title" : "Dr. med. Lilo Eisenbarth", "loxicon_id" : 808261 }


✗Montag, 31. Mai 2010

BSON

‣ Binary serialized JSON‣ http://bsonspec.org/‣ Goals: Lightweight, Traversable, Efficient‣ Format for Datastorage and Wire


http://bsonspec.org

http://bsonspec.org

rich queries


Queries‣ expressed BSON query documents‣ very flexible‣ relatively simple query expressions‣ pretty close to SQL conceptually‣ examples will follow‣ on top: map/reduce for aggregation


Scaling

‣ Master > Slave replication‣ Replica Pairs (with Arbiter)‣ Replica Sets (Target: 1.6)‣ Autosharding (Target: 1.6)


A few words on durability

‣ MongoDB only fsyncs every <n> seconds‣ There‘s a desaster waiting to happen!‣ When in production, replicate!‣ This is not as bad as it sounds.


Installation/Hosting


OS X:$ brew install mongodb


Ubuntu/Debian:theres an apt for that


Ubuntu/Debian:theres an apt for that

excuse the pun


http://www.mongodb.org/display/DOCS/DownloadsMontag, 31. Mai 2010

http://www.mongodb.org/display/DOCS/Downloads


also


http://mongohq.com/Montag, 31. Mai 2010



http://mongomachine.com/Montag, 31. Mai 2010

http://mongomachine.com/

http://mongomachine.com/

basic usage


$ mongo


> use testswitched to db test

db.quotes.save({ text: "You can observe a lot just by watching.", from: "Yogi Berra", created_at: new Date() });

db.quotes.save({ text: "Silence is one of the hardest arguments to refute.", from: "Josh Billings", created_at: new Date() });


let‘s query


db.quotes.find();// returns all records in collection.

db.quotes.find({from: "Yogi Berra"});{ "_id" : ObjectId("4c0022551496fc2051e93695"), "text" : "You can observe a lot just by watching.", "from" : "Yogi Berra", "created_at" : "Fri May 28 2010 22:06:45 GMT+0200 (CEST)" }


$lt <$gt >$lte <=$gte >=$ne !=


db.quotes.find({from: {"$ne": "Yogi Berra"}});{ "_id" : ObjectId("4c0022551496fc2051e93696"), "text" : "Silence is one of the hardest arguments to refute.", "from" : "Josh Billings", "created_at" : "Fri May 28 2010 22:06:45 GMT+0200 (CEST)" }


$in IN (2,3,4)$nin NOT IN$all [2,3] ~ [1,2,3]


db.quotes.find({from:{ "$in":["Yogi Berra","Josh Billings"]}});{ "_id" : ObjectId("4c0022551496fc2051e93695"), "text" : "You can..."...}{ "_id" : ObjectId("4c0022551496fc2051e93696"), "text" : "Silence..."...}

db.arrays.save({list: [1,2,3]});db.arrays.save({list: [4,5,6]});db.arrays.save({list: [3,4,5]});

db.arrays.find({list:{ "$in":[3,4]}});{ "_id" : ObjectId("4c0025461496fc2051e93697"), "list" : [ 1, 2, 3 ] }{ "_id" : ObjectId("4c0025501496fc2051e93698"), "list" : [ 4, 5, 6 ] }{ "_id" : ObjectId("4c0025591496fc2051e93699"), "list" : [ 3, 4, 5 ] }

db.arrays.find({list:{ "$all":[3,5]}});{ "_id" : ObjectId("4c0025591496fc2051e93699"), "list" : [ 3, 4, 5 ] }


$mod yah, RLY$size okay$exists NOT NULL$type huh?


db.arrays.find({list:{ "$mod":[4,0]}});{ "_id" : ObjectId("4c0025501496fc2051e93698"), "list" : [ 4, 5, 6 ] }{ "_id" : ObjectId("4c0025591496fc2051e93699"), "list" : [ 3, 4, 5 ] }

db.arrays.find({list:{ "$size":3}}); { "_id" : ObjectId("4c0025461496fc2051e93697"), "list" : [ 1, 2, 3 ] }{ "_id" : ObjectId("4c0025501496fc2051e93698"), "list" : [ 4, 5, 6 ] }{ "_id" : ObjectId("4c0025591496fc2051e93699"), "list" : [ 3, 4, 5 ] }

db.arrays.find({list: {"$exists": true}});[...]

db.arrays.find({list: {"$type": 1}});[...]


...and...


db.quotes.find({from: /^Yog/});[...]

db.quotes.find({from: /^Yog/});[...]

db.quotes.find("this.from == 'Yogi Berra'");[...]

db.quotes.find({"$where": "this.from == 'Yogi Berra'"});[...]


sort()


db.quotes.find().sort({from:1}) {"from" : "Josh Billings" ... }{"from" : "Yogi Berra" ...}

db.quotes.find().sort({from:-1}){"from" : "Yogi Berra" ...}{"from" : "Josh Billings" ... }


limit()


skip() // == OFFSET


count()


db.quotes.find().count(); 2


Indices


Indexing

‣ Same concept as SQL-Indices‣ You want them. (Same concept as with...)‣ Sort order, unique, compound, geospatial


db.quotes.ensureIndex({from: 1});

db.quotes.ensureIndex({from: -1});

db.quotes.ensureIndex({text: 1}, {unique: true});

db.quotes.ensureIndex({from: 1, text: 1});

db.quotes.dropIndexes();

db.quotes.dropIndex({from: 1, text: 1});

db.quotes.reIndex();


map/reduce, we can haz it, too


function() { this.tags.forEach(function(z) { emit(z, {count: 1}); }); }

function(key, values) { var total = 0; values.forEach(function(v) { total += v }); return {count: total} }


(it‘s not fast...)


one more thing


GridFS


GridFS file storage

‣ Binary fields in BSON limited to 4MB‣ GridFS API fixes that, files stored as chunks‣ Use the language drivers


I‘m in u‘r rubies,querying teh MongoDB!


ruby integration‣ mongo gem‣ bson/ bson_ext gem

‣ mongo_mapper‣ mongoid


Basic driver usage


require 'rubygems'require 'mongo'

db = Mongo::Connection.new.db("test")doc = { :text => "You can observe a lot just by watching.", :from => "Yogi Berra", :created_at => Time.now}db['quotes'].insert(doc)db['quotes'].find.each do |row| puts row.inspectend

{ "_id"=>$oid4bffe2896261046e79000001, "from"=>"Yogi Berra", "created_at"=>Fri May 28 15:34:33 UTC 2010, "text"=>"You can observe a lot just by watching."}


require 'rubygems'require 'mongo'

db = Mongo::Connection.new.db("test")

100.times do |i| db['numbers'].insert({"i" => i})end

db['numbers'].find("i" => {"$lt" => 2}).each do |row| puts row.inspectend

# {"_id"=>$oid4bffe4396261046f25000001, "i"=>0}# {"_id"=>$oid4bffe4396261046f25000002, "i"=>1}


db['text_entries'].drop_index("tags_1")db['text_entries'].create_index("tags")db['text_entries'].index_information


GridFS usage


db = Mongo::Connection.new.db("test")

grid = Mongo::Grid.new(db)

id = grid.put("You can put Strings in here", :filename => 'test.txt')

file = grid.get(id)puts file.filenameputs file.read

grid.delete(id)

grid.put( File.open("/Users/jankrutisch/Dropbox/Photos/IMGP8989.jpg"))


fs = Mongo::GridFileSystem.new(db)

fs.open("test.txt", "w") do |f| f.write "You can put stuff in here"end

fs.open("test.txt", "r") do |f| puts f.readend

fs.delete("test.txt")


ODMs


mongo_mapper‣ By John Nunemaker (@jnunemaker)‣ works‣ a few quirks‣ almost completely undocumented‣ Some stuff is still missing


# config/initializers/mongo_mapper.rbFile.open(File.join(Rails.root, 'config/mongodb.yml'), 'r') do |f| @settings = YAML.load(f)[Rails.env]end

MongoMapper.connection = Mongo::Connection.from_uri(@settings["connection"]) if @settings["connection"]

MongoMapper.database = @settings["database"]


class Loop include MongoMapper::Document key :name key :public, Boolean key :message_id key :plays_and_downloads, Integer

belongs_to :user timestamps!end


@loops = Loop.all(! :user_id => {"$exists" => true},! :order => 'created_at DESC', ! :limit => 10)


„created_at DESC“ ?!?


mongoid


mongoid‣ By Durran Jordan (Hashrocket)‣ Two major versions:‣ 1.x (currently 1.9) for Rails 2.3 compatibility‣ 2.x (currently 2.x beta) for Rails 3 compatibility

‣ Good documentation‣ API is better (?)


File.open(File.join(RAILS_ROOT, 'config/mongodb.yml'), 'r') do |f| @settings = YAML.load(f)[RAILS_ENV]end

Mongoid::Config.instance.from_hash(@settings)


class Loop include Mongoid::Document include Mongoid::Timestamps

field :name field :public, :type => Boolean field :message_id field :plays_and_downloads, :type => Integer

belongs_to_related :userend


Criteria API

‣ A bit like Arel‣ chainable method calls‣ Named scopes


Embedded Documents


class Person include Mongoid::Document field :first_name field :last_name embeds_one :address embeds_many :phonesend

class Address include Mongoid::Document field :street field :city field :state field :post_code embedded_in :person, :inverse_of => :addressend


GridFS


acts_as_attachment


class Loop include Mongoid::Document include Mongoid::Timestamps

include Mongoid::Grid field :name field :public, :type => Boolean field :message_id field :plays_and_downloads, :type => Integer

attachment :nanend


using MongoDB 4 real


Installation is easy


(if you‘re using Ubuntu)


setting up replication


$ mongod --masterormaster = true # mongodb.conf

$ mongod --slave --source slaveserver.example.com

slave = truesource = slaveserver.example.com


OpLog size!


„security“


memory usage?


limits?


stability?


bonus level


you‘ve made it this far!


MongoDB explain()ed


db.text_entries.find({tags: "restaurant"}).limit(10).explain();{! "cursor" : "BtreeCursor tags_1",! "indexBounds" : [! ! [! ! ! {! ! ! ! "tags" : "restaurant"! ! ! },! ! ! {! ! ! ! "tags" : "restaurant"! ! ! }! ! ]! ],! "nscanned" : 210,! "nscannedObjects" : 210,! "n" : 10,! "millis" : 0,! [...]}


My personal impression


Extremely easy to grasp


Normalisation suXXorZ


Migrations hurt


Seems to be very fast


Some issues for small projects


I ♥


thanks for listening.


Moi‣ [email protected]‣ http://jan.krutisch.de/‣ http://github.com/halfbyte/‣ http://twitter.com/halfbyte‣ http://www.mindmatters.de/






http://github.com/halfbyte/

http://github.com/halfbyte/

http://twitter.com/halfbyte

http://twitter.com/halfbyte

http://www.mindmatters.de

http://www.mindmatters.de

Pointers

‣ http://www.mongodb.org/‣ http://www.mongoid.org/‣ http://github.com/jnunemaker/mongo_mapper


http://www.mongodb.org

http://www.mongodb.org

http://www.mongoid.org

http://www.mongoid.org

http://github.com/jnunemaker/mongo_mapper

http://github.com/jnunemaker/mongo_mapper

mongodb & mongomapper 4 real

Technology