btree nosql oak

31
CouchDB Apache Monday, November 2, 2009 how couchdb treats the disks append only btree

Upload: chris-anderson

Post on 01-Nov-2014

3.625 views

Category:

Technology


0 download

DESCRIPTION

An introduction to CouchDB's append-only storage model.

TRANSCRIPT

Page 1: Btree Nosql Oak

CouchDBApache

Monday, November 2, 2009

how couchdb treats the disksappend only btree

Page 2: Btree Nosql Oak

Replication

JSON

Map Reduce

HTTPMonday, November 2, 2009

intro:offline appssimple developmenteasy to deploy

Page 3: Btree Nosql Oak

! !

Stuart Langridge - Canonical

Monday, November 2, 2009

15 million desktops by the end of the year.solve the data island problem

Page 4: Btree Nosql Oak

Monday, November 2, 2009

raindrop - next gen messaging platformname: your own personal piece of the cloud

Page 5: Btree Nosql Oak

Monday, November 2, 2009

Page 6: Btree Nosql Oak

Monday, November 2, 2009

Couchapps -- easiest way to deploy an Ajax application that needs to store data.-- capable of scaling up with CouchDB. (etags etc)

Page 7: Btree Nosql Oak

http://www.flickr.com/photos/mcpig/872293700/

“Ground Computing”@jhuggins

Monday, November 2, 2009

app runs near the user. lower latency, lower cost.

Page 8: Btree Nosql Oak

Monday, November 2, 2009

Page 9: Btree Nosql Oak

http://www.flickr.com/photos/shane-h/280084650

Offline by Default

Monday, November 2, 2009

reliabletelephone poles / like akamai but for dynamic database backed applications. - same application on the client and server or the cloud

Page 10: Btree Nosql Oak

http://jacobian.org/writing/of-the-web/

http://apod.nasa.gov/apod/ap050930.html

“Of the Web”

Monday, November 2, 2009

Let me tell you something: Django may be built for the Web, but CouchDB is built of the Web. I've never seen software that so completely embraces the philosophies behind HTTP. ... this is what the software of the future looks like. Jacob Kaplan-Moss -- October, 2007

perfect spot - discovered not invented

When you can move data via replication, it

changes the physics of the web

so data is useful independent of location.

Page 11: Btree Nosql Oak

LocklessStorage

Monday, November 2, 2009

html5 story js ruby transactions?

-- getting CouchDB right involves going all the way to core lockless storage engine.

performance, map reduce, and deployment

Page 12: Btree Nosql Oak

B-Tree

Monday, November 2, 2009

data structuresorted by keysminimize disk seeks

Page 13: Btree Nosql Oak

Monday, November 2, 2009

logical btree

Page 14: Btree Nosql Oak

Monday, November 2, 2009

indexes - efficientkey lookupsrange scans

Page 15: Btree Nosql Oak

Append

OnlyMonday, November 2, 2009

robust / reliablenever touch bytes that are on diskcan’t be inconsistent

Page 16: Btree Nosql Oak

Monday, November 2, 2009

append only - in actionupdate J to a new value (rewrite j’s leaf node)

Page 17: Btree Nosql Oak

Monday, November 2, 2009

rewrite the path along the tree to Jup the parent path until we write the root.

Page 18: Btree Nosql Oak

Monday, November 2, 2009

all pointers point backwards into the file

Page 19: Btree Nosql Oak

Monday, November 2, 2009

update cost?

only has O(log n) cost where n is the # of keys

Page 20: Btree Nosql Oak

Monday, November 2, 2009

another update - add 5.

rewrite path to Z node to add 5

Page 21: Btree Nosql Oak

Monday, November 2, 2009

time follows the blue arrow. top half is actually an accumulation of updates like the bottom.note: batch efficiency -- save block updates

Page 22: Btree Nosql Oak

Monday, November 2, 2009

2 day test runcouchdb over long haul4 writers, 16 readers, random access, small keyspace - lots of 409s etc.

Page 23: Btree Nosql Oak

Incremental

ReduceMonday, November 2, 2009

takes advantage of the write pattern we’ve seensetup main storage doc count example

Page 24: Btree Nosql Oak

Monday, November 2, 2009

rereduce = falsecount # of keys in the block

Page 25: Btree Nosql Oak

Monday, November 2, 2009

rereduce = truesum # of docs in child nodesflexible range querying

Page 26: Btree Nosql Oak

Monday, November 2, 2009

intermediate reductions stored during a view calculationupdating J doesn’t change any counts, adding the 5 key does

Page 27: Btree Nosql Oak

Monday, November 2, 2009

we are rewriting the parent nodes anyway, so it makes sense to run the reduce calculation

Page 28: Btree Nosql Oak

Concurrent Access

Monday, November 2, 2009

readers see a consistent viewwe can support many readersreads don’t block writeswrites don’t block reads

Page 29: Btree Nosql Oak

Monday, November 2, 2009

concurrent readerslist all docs, then a write comes in, you don’t want to be surprised by an extra docreaders can be as long lived as they want without blocking writers or seeing inconsistencies

Page 30: Btree Nosql Oak

Local Web Platform

Monday, November 2, 2009

user interviews

Page 31: Btree Nosql Oak

http://www.longnow.org/projects/clock/orrery/Monday, November 2, 2009

append-onlyno fixup phasereliable and concurrentsequential write pattern (few writer seeks)hot backupscompaction