introduction to riak and ripple (kc.rb)
TRANSCRIPT
Introduction toRiak & Ripple
Sean CribbsDeveloper Advocatebasho
Introduction toRiak & Ripple
Sean CribbsDeveloper Advocate
Rubyist, Evangelist, Support Monkey
basho
Riak @10,000 ft.
Riak @10,000 ft.•Amazon Dynamo-inspired
replicated, distributed, fault-tolerant, masterless, scalable, operations-friendly
Riak @10,000 ft.•Amazon Dynamo-inspired
replicated, distributed, fault-tolerant, masterless, scalable, operations-friendly
•Key-Value / Document
Riak @10,000 ft.•Amazon Dynamo-inspired
replicated, distributed, fault-tolerant, masterless, scalable, operations-friendly
•Key-Value / Document
• Schema-less, content-agnostic
Riak @10,000 ft.•Amazon Dynamo-inspired
replicated, distributed, fault-tolerant, masterless, scalable, operations-friendly
•Key-Value / Document
• Schema-less, content-agnostic
•Web-friendlyHTTP, JSON, Javascript
Dynamo-like Scalability
Dynamo-like Scalability
•To get...
Dynamo-like Scalability
•To get...
•more storage,
Dynamo-like Scalability
•To get...
•more storage,
•more throughput,
Dynamo-like Scalability
•To get...
•more storage,
•more throughput,
•lower latency...
Dynamo-like Scalability
•To get...
•more storage,
•more throughput,
•lower latency...
•...add more machines.aka horizontal & linear
Key-Value++
Key-Value++•Data objects are identified by keys,
and have metadata
Key-Value++•Data objects are identified by keys,
and have metadata
•Keys are grouped in buckets
Key-Value++•Data objects are identified by keys,
and have metadata
•Keys are grouped in buckets
•Links enable lightweight relationships
Key-Value++•Data objects are identified by keys,
and have metadata
•Keys are grouped in buckets
•Links enable lightweight relationships
•Query with MapReduce
Sans-Schema
Sans-Schema•Buckets created on the fly
Sans-Schema•Buckets created on the fly
•Values are opaque
Sans-Schema•Buckets created on the fly
•Values are opaque
•Content-Type matters
Sans-Schema•Buckets created on the fly
•Values are opaque
•Content-Type matters
•The application defines the semantics: more flexibility, more responsibility
Web-Friendly
Web-Friendly•HTTP is primary interface
Web-Friendly•HTTP is primary interface
•JSON is used for structured data
Web-Friendly•HTTP is primary interface
•JSON is used for structured data
•Javascript is used for MapReduce functions
Web-Friendly•HTTP is primary interface
•JSON is used for structured data
•Javascript is used for MapReduce functions
•Plays well with Varnish, Squid, HAProxy, F5, etc.
Web-Friendly•HTTP is primary interface
•JSON is used for structured data
•Javascript is used for MapReduce functions
•Plays well with Varnish, Squid, HAProxy, F5, etc.[see also Webmachine]
Use Cases
Use Cases•Document storage
(sparse structure)
Use Cases•Document storage
(sparse structure)
•Distributed file storage (small size, large soon)
Use Cases•Document storage
(sparse structure)
•Distributed file storage (small size, large soon)
•Session storage
Use Cases•Document storage
(sparse structure)
•Distributed file storage (small size, large soon)
•Session storage
•Distributed cache
Use Cases•Document storage
(sparse structure)
•Distributed file storage (small size, large soon)
•Session storage
•Distributed cache
•Browser-only/Mobile Apps
Getting Started
Getting Started•Download a package from
http://downloads.basho.com
Getting Started•Download a package from
http://downloads.basho.com
•Set the node name, IPs
Getting Started•Download a package from
http://downloads.basho.com
•Set the node name, IPs
•Start up the node
Getting Started•Download a package from
http://downloads.basho.com
•Set the node name, IPs
•Start up the node
•Join a cluster
Getting Started•Download a package from
http://downloads.basho.com
•Set the node name, IPs
•Start up the node
•Join a cluster
•Start storing/retrieving values
Riak in Ruby
Riak in Ruby•Three gems
Riak in Ruby•Three gems
•riak-client (basic ops)
Riak in Ruby•Three gems
•riak-client (basic ops)
•ripple (ODM)
Riak in Ruby•Three gems
•riak-client (basic ops)
•ripple (ODM)
•riak-sessions (Rack/Rails session stores)
Riak in Ruby•Three gems
•riak-client (basic ops)
•ripple (ODM)
•riak-sessions (Rack/Rails session stores)
•All HTTP - Protobuffs coming
require ‘riak’
require ‘riak’• Make a client object
client = Riak::Client.new
require ‘riak’• Make a client object
client = Riak::Client.new
• Get a bucketbucket = client.bucket(‘foo’) # Riak::Bucket
require ‘riak’• Make a client object
client = Riak::Client.new
• Get a bucketbucket = client.bucket(‘foo’) # Riak::Bucket
• Get an object from the bucketobj = bucket.get(‘bar’) # Riak::RObject
require ‘riak’• Make a client object
client = Riak::Client.new
• Get a bucketbucket = client.bucket(‘foo’) # Riak::Bucket
• Get an object from the bucketobj = bucket.get(‘bar’) # Riak::RObject
• Initialize a new objectobj = bucket.new(‘baz’)
Riak::RObject
Riak::RObject•Get/set object key
obj.key = “bar”
Riak::RObject•Get/set object key
obj.key = “bar”
•Get/set content-typeobj.content_type = ‘application/json’
Riak::RObject•Get/set object key
obj.key = “bar”
•Get/set content-typeobj.content_type = ‘application/json’
•Get/set the object body dataobj.data = {“name” => “Sean”}
Riak::RObject•Get/set object key
obj.key = “bar”
•Get/set content-typeobj.content_type = ‘application/json’
•Get/set the object body dataobj.data = {“name” => “Sean”}
•Store the objectobj.store
More RObject
More RObject•Get object’s bucket
obj.bucket
More RObject•Get object’s bucket
obj.bucket
•Delete the objectobj.delete # freezes the object
More RObject•Get object’s bucket
obj.bucket
•Delete the objectobj.delete # freezes the object
•Detect/extract siblings (more later)obj.conflict? && obj.siblings# Array<RObject>
More RObject•Get object’s bucket
obj.bucket
•Delete the objectobj.delete # freezes the object
•Detect/extract siblings (more later)obj.conflict? && obj.siblings# Array<RObject>
•Follow/traverse links (more later)obj.walk(:tag => “friend”)
Riak::Bucket
Riak::Bucket•Set the replication factor
bucket.n_val = 5
Riak::Bucket•Set the replication factor
bucket.n_val = 5
•Set default request quorumsbucket.r = 3 # w,dw,rw # number or one/all/quorum
Riak::Bucket•Set the replication factor
bucket.n_val = 5
•Set default request quorumsbucket.r = 3 # w,dw,rw # number or one/all/quorum
•List keysbucket.keys # pass block to stream
Riak::Bucket•Set the replication factor
bucket.n_val = 5
•Set default request quorumsbucket.r = 3 # w,dw,rw # number or one/all/quorum
•List keysbucket.keys # pass block to stream
•Set consistency flag (allow sibling generation)obj.allow_mult = true
Links: LightweightRelationships
Link Header
Link: </riak/demo/test1>; riaktag=”userinfo”
Link Header
Link: </riak/demo/test1>; riaktag=”userinfo”
bucket
Link Header
Link: </riak/demo/test1>; riaktag=”userinfo”
bucket
key
Link Header
Link: </riak/demo/test1>; riaktag=”userinfo”
bucket
key
tag
Links in Ruby API
Links in Ruby API•Create a link
Riak::Link.new(“/riak/bucket/key”, “tag”)
Links in Ruby API•Create a link
Riak::Link.new(“/riak/bucket/key”, “tag”)
•Read an object’s linksobj.links # Set<Riak::Link>
Links in Ruby API•Create a link
Riak::Link.new(“/riak/bucket/key”, “tag”)
•Read an object’s linksobj.links # Set<Riak::Link>
•Convert an object to a linkobj.links << obj2.to_link(“next”)obj.store
Link-Walking
Link-Walking•Asks Riak to traverse a sequence of
links via special URL
Link-Walking•Asks Riak to traverse a sequence of
links via special URL
•Filter by bucket/tag
Link-Walking•Asks Riak to traverse a sequence of
links via special URL
•Filter by bucket/tag
•Can return results from intermediate traversals
Link-Walking•Asks Riak to traverse a sequence of
links via special URL
•Filter by bucket/tag
•Can return results from intermediate traversals
•Response is nested multipart/mixed(riak-client handles this for you)
L/W Example
L/W ExampleGET /riak/demo/test1/_,friend,1
Start at demo/test1, follow all links tagged “friend” and return the objects
obj = client[‘demo’][‘test1’]obj.walk(:tag => “friend”,:keep => true)
L/W ExampleGET /riak/demo/test1/_,friend,1
Start at demo/test1, follow all links tagged “friend” and return the objects
obj = client[‘demo’][‘test1’]obj.walk(:tag => “friend”,:keep => true)
L/W Example
L/W ExampleGET /riak/demo/test1/_,_,_/_,_,1
Start at demo/test1, follow all links, follow all links from those, return everything from the last set
obj.walk({},{:keep => true})
Map-Reduce
Map-Reduce
list of keys
Map-Reduce
list of keys
map map map map map
Map-Reduce
list of keys
map map map map map
reduce
Map-Reduce
list of keys
map map map map map
reduce
results
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Riak object
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Riak object
{ bucket:”foo”, key:”bar”, vclock:”...”, values:[ {metadata:{...}, data:”....”} ]}
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in JavascriptKey-specific
data
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in JavascriptPhase-
specific data
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Parse JSON data
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Check field equality
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Return nothing if unequal
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Return the object if all equal
Map
function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}
in Javascript
Reduce
function(values,arg) { return values.sort(function(a,b){ return a[arg] - b[arg]; });}
in Javascript
Reduce
function(values,arg) { return values.sort(function(a,b){ return a[arg] - b[arg]; });}
in Javascript
Map results
Reduce
function(values,arg) { return values.sort(function(a,b){ return a[arg] - b[arg]; });}
in Javascript
Reduce
function(values,arg) { return values.sort(function(a,b){ return a[arg] - b[arg]; });}
in Javascript
Phase-specific data
Reduce
function(values,arg) { return values.sort(function(a,b){ return a[arg] - b[arg]; });}
in Javascript
Reduce
function(values,arg) { return values.sort(function(a,b){ return a[arg] - b[arg]; });}
in Javascript
Custom sort on
Reduce
function(values,arg) { return values.sort(function(a,b){ return a[arg] - b[arg]; });}
in Javascript
Reduce
function(values,arg) { return values.sort(function(a,b){ return a[arg] - b[arg]; });}
in Javascript
Reduce potentially called multiple times!!
Example Query
Example Query{“inputs”: “goog”, “query”: [{“map”:{“language”:”javascript”, “name”: “App.findHighGreater”, “arg”: 600.0, “keep”: false}, {“reduce”:{“language”:”javascript”, “name”: “Riak.reduceMax”, “keep”: true}]}
Example Query{“inputs”: “goog”, “query”: [{“map”:{“language”:”javascript”, “name”: “App.findHighGreater”, “arg”: 600.0, “keep”: false}, {“reduce”:{“language”:”javascript”, “name”: “Riak.reduceMax”, “keep”: true}]}
Riak::MapReduce.new(c).add(‘goog’).map(‘App.findHighGreater’, :arg => 600.0).reduce(“Riak.reduceMax”, :keep => true).run
Built-in Functions
Built-in Functions•Riak.mapValues
Built-in Functions•Riak.mapValues
•Riak.mapValuesJson
Built-in Functions•Riak.mapValues
•Riak.mapValuesJson
•Riak.mapByFields
Built-in Functions•Riak.mapValues
•Riak.mapValuesJson
•Riak.mapByFields
•Riak.reduceSum
Built-in Functions•Riak.mapValues
•Riak.mapValuesJson
•Riak.mapByFields
•Riak.reduceSum
•Riak.reduceSort
Built-in Functions•Riak.mapValues
•Riak.mapValuesJson
•Riak.mapByFields
•Riak.reduceSum
•Riak.reduceSort
•Riak.reduceMin/reduceMax
Ripple - ODM
Document Modelsclass Person include Ripple::Document
property :name, String, :presence => true many :addresses many :friends, :class => Personend
class Address include Ripple::EmbeddedDocument
property :street, Stringend
Rails Setup
Rails Setup
# Gemfilegem 'curb' # Faster HTTPgem 'yajl-ruby' # Faster JSONgem 'ripple'
Rails Setup
# Gemfilegem 'curb' # Faster HTTPgem 'yajl-ruby' # Faster JSONgem 'ripple'
Rails Setup
# Gemfilegem 'curb' # Faster HTTPgem 'yajl-ruby' # Faster JSONgem 'ripple'
$ gem install curb yajl-ruby ripple
Rails Setup (cont.)
Rails Setup (cont.)
# config/ripple.ymldevelopment: host: 127.0.0.1 port: 8098
Rails Setup (cont.)
# config/ripple.ymldevelopment: host: 127.0.0.1 port: 8098
# config/application.rbrequire 'ripple/railtie'
Ripple Roadmap
Ripple Roadmap•Testing server (edge on Github)
Ripple Roadmap•Testing server (edge on Github)
•Protocol Buffers support
Ripple Roadmap•Testing server (edge on Github)
•Protocol Buffers support
•Streaming MapReduce
Ripple Roadmap•Testing server (edge on Github)
•Protocol Buffers support
•Streaming MapReduce
•Ripple-specific built-ins
Ripple Roadmap•Testing server (edge on Github)
•Protocol Buffers support
•Streaming MapReduce
•Ripple-specific built-ins
•Better ActionView support (form_for)
Happy FunDemo Time
Plug
basho
PlugInterested in learning about support, consulting, or Enterprise features?
basho
PlugInterested in learning about support, consulting, or Enterprise features?Email [email protected] or go to http://www.basho.com/contact.html to talk with us.
basho
PlugInterested in learning about support, consulting, or Enterprise features?Email [email protected] or go to http://www.basho.com/contact.html to talk with us.
basho
PlugInterested in learning about support, consulting, or Enterprise features?Email [email protected] or go to http://www.basho.com/contact.html to talk with us.
www.basho.com
[email protected]@seancribbs
basho
Questions