social gaming & gambling summit - london chris anderson

@jchrisChris Anderson

NoSQL Landscape& Grid Compu7ng

1Saturday, October 6, 12

• 2.2 Billion internet users• 50% Americans use smartphones

• Your app can grow overnight

• Are you ready?

Growth is the New Reality

Saturday, October 6, 12

Instagrowth: Android Launch

• Instagram gained nearly 1 million users overnight when they expanded to Android

Example

1 Instagram=

7.5M MAU*

Instagrowth: Android Launch

Example

Draw Something -‐ Social Game

35 million monthly active users in 1 monthabout 5 Instagrams

(Instagram today is waaaay more than 1 Instagram)

Goes Viral 3 Weeks aOer Launch

191715131197533/12826242220181614121082/6

Draw Something by OMGPOPDaily Ac)ve Users (millions)

35+M MAUat peak

By Contrast, at 1/2 an Instagram

The Simpson’s: Tapped OutDaily Ac)ve Users (millions)

GETTING IT RIGHT

Scalable Data Layer

●On-‐demand cluster sizing●Grow or shrink with workload

●Easy node provisioning●All nodes are the same

●MulA-‐master Cross-‐Datacenter ReplicaAon●For a fast and reliable user experience worldwide

●EffecAve Auto-‐sharding●Should avoid cluster hot spots

Old School Hits a Scale Wall

Application Scales OutJust add more commodity web servers

Database Scales UpGet a bigger, more complex server

Expensive & disruptive sharding, doesn’t perform at web scale

Tradi^onal MySQL + Memcached Architecture

●Run as many MySQL machines as you need

●Data sharded evenly across the machines using client code

●Memcached used to provide faster response time for users and reduce load on the database Memcached Tier

MySQL Tier

App Servers

www.example.com

Limita^ons of MySQL + Memcached

● To scale you need to start using MySQL more simply

● Scale by hand

● Replication / Sharding is a black art

● Code overhead to manage keeping memcache and mysql in sync

● Lots of components to deploy

Learn From Others -‐ This Scenario Costs Time and Money. Scaling SQL is poten^ally disastrous when going Viral: very risky ^me for major code changes and migra^ons... you have no Time when skyrocke^ng up.

NoSQL Architectural Promise

Couchbase Database Servers

App Servers

www.example.com

•High Performance data access

• Scale Up/Down Horizontally

• 24x7x365 Always-On Availability

• Flexible Schema Document Model

NOSQL TAXONOMY

The Key-Value Store – the foundation of NoSQL

101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101

101100101000100010011101101100101000100010011101

101100101000100010011101101100101000100010011101101100101000100010011101

OpaqueBinaryValue

Memcached – the NoSQL precursor

101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101

101100101000100010011101101100101000100010011101

101100101000100010011101101100101000100010011101101100101000100010011101

OpaqueBinaryValue

memcached

In-‐memory onlyLimited set of opera^onsBlob Storage: Set, Add, Replace, CASRetrieval: GetStructured Data: Append, Increment

“Simple and fast.”

Challenges: cold cache, disrup^ve elas^city

Redis – More “Structured Data” commands

101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101

101100101000100010011101101100101000100010011101

101100101000100010011101101100101000100010011101101100101000100010011101

“Data Structures”BlobListSetHash…

In-‐memory onlyVast set of opera^onsBlob Storage: Set, Add, Replace, CASRetrieval: Get, Pub-‐SubStructured Data: Strings, Hashes, Lists, Sets,Sorted lists

Example opera7ons for a SetAdd, count, subtract sets, intersec^on, is member?, atomic move from one set to another

NoSQL catalog

Key-‐Value

memcached redis

Data Structure Document Column Graph

ory on

Membase – From key-‐value cache to database

Disk-‐based with built-‐in memcached cacheCache refill on restartMemcached compa^ble (drop in replacement)Highly-‐available (data replica^on)Add or remove capacity to live cluster

“Simple, fast, elas^c.”

membaseKey

101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101101100101000100010011101

101100101000100010011101101100101000100010011101

101100101000100010011101101100101000100010011101101100101000100010011101

OpaqueBinaryValue

NoSQL catalog

Key-‐Value

memcached

membase

ory on

Database

ory/disk)

Couchbase – document-‐oriented database

{ “string” : “string”, “string” : value, “string” : { “string” : “string”, “string” : value }, “string” : [ array ]}

Auto-‐shardingDisk-‐based with built-‐in memcached cacheCache refill on restartMemcached compa^ble (drop in replace)Highly-‐available (data replica^on)Add or remove capacity to live cluster

When values are JSON objects (“documents”):Create indices, views and query against the views

JSONOBJECT

(“DOCUMENT”)

Couchbase

NoSQL catalog

Key-‐Value

memcached

membase

ory on

Database

ory/disk)

membase couchbase

MongoDB – Document-‐oriented database

{ “string” : “string”, “string” : value, “string” : { “string” : “string”, “string” : value }, “string” : [ array ]}

Disk-‐based with in-‐memory “caching”BSON (“binary JSON”) format and wire protocolMaster-‐slave replica^onAuto-‐shardingValues are BSON objectsSupports ad hoc queries – best when indexed

BSONOBJECT

(“DOCUMENT”)

MongoDB

NoSQL catalog

Key-‐Value

memcached

membase

mongoDB

couchbase

ory on

Database

ory/disk)

Cassandra – Column overlays

Disk-‐based systemClustered External caching required for low-‐latency reads“Columns” are overlaid on the dataNot all rows must have all columnsSupports efficient queries on columnsRestart required when adding columnsGood cross-‐datacenter support

CassandraColumn 1

Column 2

Column 3 (not present)

NoSQL catalog

Key-‐Value

memcached

membase

mongoDB

couchbase cassandra

ory on

Database

ory/disk)

Neo4j – Graph database

Disk-‐based systemExternal caching required for low-‐latency readsNodes, rela^onships and pathsProper^es on nodesDelete, Insert, Traverse, etc.

NoSQL catalog

Key-‐Value

memcached

membase

mongoDB

couchbase cassandra

ory on

Database

ory/disk)

The Landscape

ScaleCouchbase

Cassandra

MongoDB

CouchDB

SimpleDB

memcached

RDBMSDatomic

Datomic -‐ immutable func^onal data

Hello Couchbase Server 2.0

Couchbase Server 2.0 beta

Couchbase handles real world scale

(Really) High Performance

Latencyless than 1/2 ms

Throughputgrows linearly with cluster size

5 Nodes -- 1.75M operations per second

Cisco and Solarflare benchmark of Couchbase Server

How fast?

35hrp://www.slideshare.net/renatko/couchbase-‐performance-‐benchmarking

Latency

COMPLEXITY IS THE ENEMY

Couchbase Server Basic Opera^on

COUCHBASE CLIENT LIBRARY

§Docs distributed evenly across servers in the cluster

§Each server stores both ac)ve & replica docs§Only one server ac^ve at a ^me

§Client library provides app with simple interface to database

§Cluster map provides map to which server doc is on§App never needs to know

§ App reads, writes, updates docs§ Mul^ple App Servers can access same document at same ^me

SERVER 1

SERVER 2

SERVER 3

Doc 8 Doc 6

Ac^ve Docs Ac^ve Docs Ac^ve Docs

CLUSTER MAP CLUSTER MAP

APP SERVER 1 APP SERVER 2

COUCHBASE SERVER CLUSTER

SERVER 1

SERVER 2

SERVER 3

Doc 8 Doc 6

Doc 8 Doc 2

Replica Docs Replica Docs Replica Docs

SERVER 1

SERVER 2

SERVER 3

Read/Write/Update

Doc 8 Doc 6

Doc 8 Doc 2

Add Nodes to the Cluster

§ Two servers added to cluster§ One-‐click opera^on

§ Docs automa^cally rebalanced across cluster§ Even distribu^on of docs§ Minimum doc movement

§ Cluster map updated§ App database calls now distributed over larger # of servers

Ac^ve Docs

Replica Docs

COUCHBASE CLIENT LIBRARYCLUSTER MAP

APP SERVER 1

APP SERVER 2

SERVER 1

SERVER 2

SERVER 3

Doc 8 Doc 6

Doc 8 Doc 2

Ac^ve Docs

Replica Docs

APP SERVER 1

APP SERVER 2

SERVER 1

SERVER 2

SERVER 3

Doc 8 Doc 6

Doc 8 Doc 2

SERVER 4 SERVER 5

Ac^ve Docs Ac^ve Docs

Replica Docs Replica Docs

Ac^ve Docs

Replica Docs

APP SERVER 1

APP SERVER 2

SERVER 1

SERVER 2

SERVER 3

Doc 7 Doc 8

DOCDOC

DOC DOC

Doc 5DOC

SERVER 4 SERVER 5

Read/Write/Update Read/Write/Update

Ac^ve Docs

Replica Docs

APP SERVER 1

APP SERVER 2

SERVER 1

SERVER 2

SERVER 3

Doc 7 Doc 8

DOCDOC

DOC DOC

Doc 5DOC

SERVER 4 SERVER 5

Fail Over Node

SERVER 1

SERVER 2 SERVER 3

Doc 7 Doc 8

Doc 6DOC

DOCDOC

DOC DOC

DOCDoc 5DOC

SERVER 4 SERVER 5

§ App servers happily accessing docs on Server 3

§ Server fails§ App server requests to server 3 fail§ Cluster detects server has failed

§ Promotes replicas of docs to ac)ve§ Updates cluster map

§ App server requests for docs now go to appropriate server

§ Typically rebalance would follow

Ac^ve Docs

Replica Docs

DOCDoc 9

Fail Over Node

SERVER 1

SERVER 2 SERVER 3

Doc 7 Doc 8

Doc 6DOC

DOCDOC

DOC DOC

DOCDoc 5DOC

SERVER 4 SERVER 5

Ac^ve Docs

Replica Docs

DOCDoc 9

Fail Over Node

SERVER 1

SERVER 2 SERVER 3

Doc 7 Doc 8

Doc 6DOC

DOCDOC

DOC DOC

DOCDoc 5DOC

SERVER 4 SERVER 5

Ac^ve Docs

Replica Docs

DOCDoc 9

Fail Over Node

SERVER 1

SERVER 2 SERVER 3

Doc 7 Doc 8

Doc 6DOC

DOCDOC

DOC DOC

DOCDoc 5DOC

SERVER 4 SERVER 5

Ac^ve Docs

Replica Docs

DOCDoc 9

Fail Over Node

SERVER 1

SERVER 2 SERVER 3

Doc 7 Doc 8

Doc 6DOC

DOCDOC

DOC DOC

DOCDoc 5DOC

SERVER 4 SERVER 5

●Suddenly, disk writes all began to time out●Many services experienced outages:●FourSquare, Reddit, Quora, among others

●With memory buffered writes, a scalable data layer keeps working●When EBS came back online, Couchbase wrote all the updated data to disk without missing a beat.

War Story: EBS Outage

Cross Data Center Replica^on

§Data close to users§Mul^ple loca^ons for disaster recovery§Independently managed clusters serving local data

US DATA CENTER EUROPE DATA CENTER ASIA DATA CENTER

Replica7on Replica7on

Replica7on

Built for Produc^on

JSON DOCUMENT DATABASE

Document Database as Aggregate Database

hrp://mar^nfowler.com/bliki/AggregateOrientedDatabase.html

Document Database

This synergy between the programming model and the distribution model is very valuable. It allows the database to use its knowledge of how the application programmer clusters the data to help performance across the cluster.

hrp://mar^nfowler.com/bliki/AggregateOrientedDatabase.html

o::1001{

uid: ji22jd,customer: Ann,line_items: [

{ sku: 0321293533, quan: 3, unit_price: 48.0 },{ sku: 0321601912, quan: 1, unit_price: 39.0 },{ sku: 0131495054, quan: 1, unit_price: 51.0 }

],payment: { type: Amex, expiry: 04/2001,

last5: 12345 }}

Developers <3 JSON

LET’S GET POST-‐RELATIONAL!

JSON Documents

• Maps more closely to external API• CRUD Opera^ons, lightweight schema

• Stored under an iden^fier key

{ “fields” : [“with basic types”, 3.14159, true], “like” : “your favorite language”}

client.set(“mydocumentid”, myDocument);mySavedDocument = client.get(“mydocumentid”);

Meta + Document Body

{ "brewery": "New Belgium Brewing", "name": "1554 Enlightened Black Ale", "abv": 5.5, "descrip7on": "Born of a flood...", "category": "Belgian and French Ale", "style": "Other Belgian-‐Style Ales", "updated": "2010-‐07-‐22 20:00:20"}

{ "id" : "beer_Enlightened_Black_Ale”, ...{

Documentuser data,

can be anything

unique ID

Metadataidentifier,

expiration, etc

“vintage” date format from an SQL dump >_<

{ "brewery": "New Belgium Brewing", "name": "1554 Enlightened Black Ale", "abv": 5.5, "descrip7on": "Born of a flood...", "category": "Belgian and French Ale", "style": "Other Belgian-‐Style Ales", "updated": "2010-‐07-‐22 20:00:20", “ra7ngs” : { “525” : 5, “30” : 4, “1044” : 2 }, “comment_ids” : [ “f1e62”, “6ad8c” ]}

Add comments to the beer

{ "type": "comment", "about_id": "beer_Enlightened_Black_Ale", "user_id": 525, "text": "tastes like college!", "updated": "2010-‐07-‐22 20:00:20"}

link to comments

link to beer

{ "id": "f1e62"}

How to: look up comments from a beer

• SERIALIZED LOOP

figure hrp://www.ibm.com/developerworks/webservices/library/ws-‐sdoarch/

beer = client.get(“beer:A_cold_one”);beer.comment_ids.each { |id| comments.push(client.get(id));}

• FAST MULTI-‐KEY LOOKUPbeer = client.get(“beer:A_cold_one”);comments = client.multiGet(beer.comment_ids)

• ASYNC VIEW QUERYcomments = client.query(“myapp”,“by_comment_on”, {:key => “beer:A_cold_one”});

Emergent Schema

JSON.org

Github API

Twiqer API

"Capture the user's intent"

• The database can handle it• Your app controls the schema

Audience par^cipa^on! *that means you

npm install twitterfightnpm start twitterfight

@jchrishrp://www.couchbase.com/

Chris Anderson

Thank You!

INCREMENTAL MAP-‐REDUCEFOR REALTIME ANALYTICS

What do you mean “Incremental?”

like:CREATE INDEX city ON brewery city;

QUERY PATTERN:FIND BY ATTRIBUTE

Find documents by a specific arribute

• Lets find beers by brewery_id!

The index defini^on

The result set: beers keyed by brewery_id

QUERY PATTERN:BASIC AGGREGATIONS

Use a built-‐in reduce func^on with a group query

• Lets find average abv for each brewery!

We are reducing doc.abv with _stats

Group reduce (reduce by unique key)

QUERY PATTERN:TIME-‐BASED ROLLUPSWITH KEY GROUPING

group_level=3 -‐ daily results -‐ great for graphing

• Daily, hourly, minute or second rollup all possible with the same index.

• hrp://crate.im/posts/couchbase-‐views-‐reddit-‐data/

GEO INDEX & FULL TEXT INTEGRATION

GeoCouch R-‐Tree Index

• Op^mized for bulk loading of large data sets• Simplified query model (bounding box, nearest neighbor)

Elas^c Search Adapter

• Elas^c Search is good for ad-‐hoc queries and faceted browsing• Our adapter is aware of changing Couchbase topology• Indexed by Elas^c Search aOer stored to disk in Couchbase

@jchrishrp://www.couchbase.com/

Chris Anderson

Thank You!

social gaming & gambling summit - london chris anderson

ons of mysql

available data replica

scalable data layer

mysql machines

keyvalue cache

internet users

onal mysql

nosql taxonomy14saturday

Documents

new gambling options with digital gaming platforms

e-gaming & e-gambling entertainment or full-time job

social gaming & gambling summit - london harry holmwood

problem gambling | arizona department of gaming

social responsibility and problem gambling dr mark...

the convergence of gaming and gambling

spectrum model of social casino gaming/gambling ... ·...

gaming law and gambling law - carolina academic press

e- gaming & e- gambling entertainment or full-time job

another side to gambling/gaming in bc

gaming machines amendment (gambling harm minimisation

arkhipov mobile gaming as gambling risks 08_02_2014

social gaming & the convergence of gambling & … gaming &...

worldpay social gaming and gambling whitepaper chapter one

gaming, gambling & the new mobile age

the law of gambling and regulated gaming · the law of...

gambling and gaming convergence: epistemological shifts...

native gaming and gambling - university of calgary

responsible conduct of gambling - liquor & gaming nsw

college student gambling: examining the effects of gaming