couchbase 101: couchbase connect 2014

Couchbase 101Dipti Borkar | Sr. Director | Solutions Engineering

Couchbase

What is Couchbase?

Couchbase Server

©2014 Couchbase, Inc. 3

General purpose

Elastic scalability Consistent high

performance

Always

available

Flexible, global

deployment

Enterprise grade

administration

Real time big

data

Data

mobility

Developer

focused

The Most complete, scalable & highest performing NoSQL database

Couchbase Server

Couchbase offers a full range of

Data Management solutions

High Availability

Cache

Key Value Document Mobile

device

SSN: 400 658 9993

Pass: ******


Key Capabilities


• Developer focused

JSON Support

Indexing/Querying

Incremental Map-Reduce

• Elastic Scalability

Shared-nothing

architecture with single

node type

Cross-data center

replication (XDCR)

Push button scale out

• Consistent High Performance

Built-in Object level cache

Fine grained locking

Hash Partitioning

• Always available

Zero downtime administration and

upgrades

Streaming and rack aware

replication

Comprehensive cluster-wide

monitoring

Couchbase Server 3.0


Cluster-wide Diagnostics Tool

Improved Crash Reporting

Better Serviceability

Improved Monitoring for Warm-up

Stats Enhancements

Improved Checkpointing with XDCR

Expended Options with Couchbase-cli Parallelized Warm-up

Database Change Protocol (DCP)

Increased Connection LimitsExtended Documentation

Enhanced Event Logging

Extended XDCR Resiliency

Better failover resiliency with DCP

Enhanced SSD Performance

Streamlined Build

Side by side support for DCP and TAP

Improved Error Reporting for Apps

Built-in OS Tuning for Linux Flavors

… and More

Improved Resume-ability with Intra Cluster Replication

View Engine rewrite in C

Web.Config & App.Config support with .NET in SDK 2.0

Unified New App Model with 2.0 SDK

Java SDK 2.0 built on top of RxJava

N1QL Preview Support in the 2.0 SDKs

Faster Warm-up time under Metadata Ejection

Access log for monitoring port 8091

CRAM & MD5 Support in .Net SDK

Client Side Log4Net Integration in .Net SDK

New Cluster object for Cluster Operations in SDK 2.0

Replica Read in SDK 2.0 with .Net and PHP

Added Support for Debian v7

Faster View indexing

Faster XDCR Synchronization

Faster ReplicateTo

Faster PersistTo

Faster Rebalance

Encrypted Data Access

Encrypted Admin Access

Encrypted View Access

Graceful Failover

Delta Node Recovery

Incremental Backup and Restore

Lower Latency XDCR

Auto tuning IO subsystem

Additional Configuration on .Net pooling interval

New repository for Yum for Ubuntu

Community Edition of 3.0 Release with Enterprise Edition

Key Concepts

©2014 Couchbase, Inc.

Couchbase can act as a

Key-Value Store Document Store

2014-06-23-10:15am : 75F

2014-06-23-11:30am : 77F

2014-06-23-02:00pm : 82F

0001:

{firstname: “Dipti”,

lastname: “Borkar”,

language: “English”,

time_zone: “PST”,

zip: 94403

}

Key - UTF-8 string up to 250 bytes

Value - can be 0 bytes – 20 MB (best practice < 1 MB)©2014 Couchbase, Inc. 9

Similar to primary keys in relational databases

Documents are sharded based on the document ID

ID based document lookup is extremely fast

Must be unique

Fundamentals

JSON

Binary - integers, strings, booleans

Common binary values include serialized objects, compressed XML, compressed text

Document ID or Key

Value

CAS Value (unique identifier for concurrancy)

TTL

Flags (optional client library metadata)

Revision #

Metadata


10

Can Represent Complex Objects and Data Structures

Very simple notation, lightweight, compact, readable

The most common API return type for Integrations

Facebook, Twitter, you name it, return JSON

Native to Javascript (can be useful)

Can be inserted straight into Couchbase (faster development)

Serialization and Deserialization are very fast

Benefits of JSON


Storing and retrieving documents

Couchbase Cluster

Server Nodes

User/application data

Which live on

Data Buckets

DocumentsRead from / Written to

That form a

Clients

Servers

Dynamically scalable

Based on hash partitioning


User Objectstring uid

string firstname

string lastname

int age

array favorite_colors

string email

u::[email protected]{ “uid”: 123456,

“firstname”: “John”,“lastname”: “Smith”,“age”: 22,“favorite_colors”: [“blue”, “black”],“email”: “[email protected]”

}

User Objectstring uid

string firstname

string lastname

int age

array favorite_colors

string email

u::[email protected]{ “uid”: 123456,

“firstname”: “John”,“lastname”: “Smith”,“age”: 22,“favorite_colors”: [“blue”, “black”],“email”: “[email protected]”

}

add()

get()

Objects Serialized to JSON and Back


Within each server – Single Node Type


Heart

beat

Pro

cess m

onitor

Glo

bal sin

gle

ton s

uperv

isor

Configura

tion m

anager

on each node

Rebala

nce o

rchestr

ato

r

Node h

ealth m

onitor

one per clusa

vB

ucket sta

te a

nd r

eplic

atio

n m

anager

http

RE

ST

man

ag

em

en

t A

PI/

Web

UI

HTTP

8091Erlang port mapper

4369Distributed Erlang

21100 - 21199

Erlang/OTP

storage interface

Couchbase EP Engine

11210Memcapable 2.0

Moxi

11211Memcapable 1.0

Memcached

Persistence Layer

8092Query API

Qu

ery

En

gin

e

Data Manager Cluster Manager

Couchbase Basic Operations


Single Node Operations - Write


33 2Managed Cache

Dis

k Q

ueu

e

Disk

Replication Queue

App Server

Memory-to-Memory

Replication to other

node

Doc

Doc Doc

Managed Cache

Disk

Single Node Operations - Read


Managed Cache

Doc 1

Get Doc 1

Doc 1Doc 1

App Server

Dis

k Q

ueu

e

Replication Queue

Memory-to-Memory


node

Disk

Managed Cache

Single Node Operations – Cache Ejection


Doc 1

Doc 1

Doc 2Doc 3Doc 4Doc 5Doc 6


App Server

Dis

k Q

ueu

e

Replication Queue

Memory-to-Memory


node

Single Node Operations – Cache Miss


33 2

Dis

k Q

ueu

e

Disk

Replication Queue

App Server

Memory-to-Memory


node

Doc 1



Doc 1

Doc 1Doc 1

Managed Cache

Get Doc 1

Cluster-wide Operations


Each bucket has active and replica data sets

Each data set has 1024 Virtual Bucket (vBuckets)

Documents gets logically mapped to vBuckets

Document IDs always get hashed to the same virtual bucket

Virtual buckets to do not have a fixed physical server location

Mapping between the virtual buckets and physical server is called the cluster map

Each virtual bucket contains 1/1024th portion of the data set

Auto sharding – Bucket and vBuckets

vB

Data buckets

vB

1 ….. 1024

Virtual buckets

©2014 Couchbase, Inc.21

Cluster Map

Hash function (KEY)

vB1 vB2 vB3 vB4 vB5 vB6

Ph

ys

ica

l

se

rve

rs

A B C

Add nodeWhen more scalability

required

Lo

gic

al

Pa

rtit

ion

s

Cluster Map

New Cluster Map


read/write/update

Active

SERVER 1

Active

SERVER 2

Active

SERVER 3

APP SERVER 1

COUCHBASE Client Library

CLUSTER MAP


CLUSTER MAP

APP SERVER 2

Shard

5

Shard

2

Shard

9

Shard

Shard

Shard

Shard

4

Shard

7

Shard

8

Shard

Shard

Shard

Shard

1

Shard

3

Shard

6

Shard

Shard

Shard

Replica Replica Replica

Shard

4

Shard

1

Shard

8

Shard

Shard

Shard

Shard

6

Shard

3

Shard

2

Shard

Shard

Shard

Shard

7

Shard

9

Shard

5

Shard

Shard

Shard

Multi-Node Operations

• Docs distributed evenly across servers

• Each server stores both active and replica docs- Only one server active at a time

• Client library provides app with simple interface to database

• Cluster map provides map to which server doc is on- App never needs to know

• App reads, writes, updates docs

• Multiple app servers can access same document at same time


SERVER 4 SERVER 5

Replica

Active

Replica

Active

read/write/update

APP SERVER 1


CLUSTER MAP


CLUSTER MAP

APP SERVER 2

Active

SERVER 1

Shard

9

Shard

Replica

Shard

4

Shard

1

Shard

8

Shard

Shard

Shard

Active

SERVER 2

Shard

8

Shard

Replica

Shard

6

Shard

3

Shard

2

Shard

Shard

Shard

Active

SERVER 3

Shard

6

Shard

Replica

Shard

7

Shard

9

Shard

5

Shard

Shard

Shard

read/write/update

Shard

5

Shard

2

Shard

Shard

Shard

4

Shard

7

Shard

Shard

Shard

1

Shard

3

Shard

Shard

Adding Nodes

• Two servers added withone-click operation

• Docs automatically rebalance across cluster- Even distribution of docs- Minimum doc movement

• Cluster map updated

• App database calls now distributed over larger number of servers


Failover

SERVER 4 SERVER 5

Replica

Active

Replica

Active

App Server 1


CLUSTER MAP


CLUSTER MAP

App Server 2

Active

SERVER 1

Shard 5

Shard 2

Shard 9Shard

Shard

Shard

Replica

Shard 4

Shard 1

Shard 8Shard

Shard

Shard

Active

SERVER 2

Shard 4

Shard 7 Shard 8

Shard

Shard Shard

Replica

Shard 6

Shard 3 Shard 2

Shard

Shard Shard

Active

SERVER 3

Shard 1

Shard 3

Shard 6Shard

Shard

Shard

Replica

Shard 7

Shard 9

Shard 5Shard

Shard

Shard

• App servers accessing Shards

• Requests to Server 3 fail

• Cluster detects server failedo Promotes replicas of

Shards to activeo Updates cluster map

• Requests for docs now go to appropriate server

• Typically rebalance would follow

Shard 1 Shard 3

Shard


Cross Datacenter Replication (XDCR)

US DATA

CENTER

EUROPE

DATA

CENTER

ASIA DATA

CENTER

http://blog.groosy.com/wp-content/uploads/2011/10/internet-map.jpg©2014 Couchbase, Inc. 26



• Replicates data continuously FROM source cluster TO remote clusters

• Supports unidirectional and bidirectional operation

• Application can read and write from both clusters (active – active replication)

• Replication throughput scales out linearly

• Simplified Administration via console, REST, and CLI



Unidirectional Replication

• Hot spare / Disaster Recovery

• Development/Testing copies

• Replicate to indexing cluster

• Integrate to Connector e.g. Solr,

ElasticSearch

• Integrate to custom consumer



Bidirectional Replication

• Multiple Active Masters

• Data locality

• Disaster Recovery

33 2

XDCR after Write

2

Managed Cache

Dis

k Q

ueu

e

Disk

Replication Queue

App Server

Couchbase Server Node

Doc 1

Doc 1

XDCR Queue

Doc 1Doc 1

(New in 3.0)

Memory-to-Memory

Replication to

remote clusterMemory-to-Memory


node


ACTIVE

SERVER 1

RAM

DISK

Doc

Doc 2

Doc 9

Doc Doc Doc

ACTIVE

SERVER 2

RAM

DISK

Doc

Doc

Doc

Doc Doc Doc

ACTIVE

SERVER 3

RAM

DISK

Doc

Doc

Doc

Doc Doc Doc

Cross Data Center Replication (XDCR)

COUCHBASE SERVER CLUSTER

NYC DATA CENTER

COUCHBASE SERVER CLUSTER

SF DATA CENTER

ACTIVE

SERVER 1

RAM

DISK

Doc

Doc 2

Doc 9

Doc Doc Doc

ACTIVE

SERVER 2

RAM

DISK

Doc

Doc

Doc

Doc Doc Doc

ACTIVE

SERVER 3

RAM

DISK

Doc

Doc

Doc

Doc Doc Doc

{ } { } { }


Index and Query

Distributed indexing and querying

Secondary indexes of JSON document content

Flexible querying of indexes

Incremental Map-Reduce

Distributed simple real-time analytics

Only considers changes due to updated data

Full Text Search

Robust integration with ElasticSearch / Solr cluster

Flexible full text search and faceted search

Indexing and Querying Features


33 2

View processing after write

2

Managed Cache

Dis

k Q

ueu

e

Disk

Replication Queue

App Server

Couchbase Server Node

Doc 1

Doc 1

To other node

View engine Doc 1Doc 1


Active

SERVER 1

Shard

5

Shard

2

Shard

Shard

Replica

Shard

4

Shard

1

Shard

Shard

Shard

1

Active

SERVER 3

Shard

5

Shard

2

Shard

Shard

Replica

Shard

4

Shard

1

Shard

Shard

Shard

1

Active

SERVER 2

Shard

5

Shard

2

Shard

Shard

Replica

Shard

4

Shard

1

Shard

Shard

Shard

1

APP SERVER 1


CLUSTER MAP


CLUSTER MAP

APP SERVER 2

Couchbase Server Architecture - Views

• Indexing work is distributed amongst nodes

• Large data set possible

• Parallelize the effort

• Each node has index for data stored on it

• Queries combine the results from required nodes


Live cluster


Q & A


[email protected]

@dborkar


couchbase 101: couchbase connect 2014

Data & Analytics