overview of couchbase server architecture – couchbase connect 2016

©2016 Couchbase Inc. 1

Couchbase Server Architecture

©2016 Couchbase Inc. 2©2016 Couchbase Inc.

Look Who’s Talking

David HaikneyDirector of Engineering,

EMEA

Dave RigbyTechnical Lead, Data Service

©2016 Couchbase Inc. 3©2016 Couchbase Inc.

What To Expect

•Whistle-Stop Tour with demos•Key Architectural Concepts•Exploration of each Service•Where to hear more @ Connect!

©2016 Couchbase Inc. 4

Key Concepts

©2016 Couchbase Inc.©2016 Couchbase Inc.

Concept 1 – Your Choice of Services

STORAGE

Couchbase Server 1

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

Data Service

Index Service

Query Service

Full Text Search

©2016 Couchbase Inc.©2016 Couchbase Inc.

Concept 1 – Your Choice of Services

STORAGE

Couchbase Server 1

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

Data Service

Index Service

Query Service

STORAGE

Couchbase Server 2

Managed Cache

Cluster ManagerCluster Manager

STORAGE

Couchbase Server 3

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

STORAGE

Couchbase Server 4

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

STORAGE

Couchbase Server 5

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

STORAGE

Couchbase Server 6

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

Full Text Search

©2016 Couchbase Inc.©2016 Couchbase Inc.

Concept 1 – Your Choice of Services

STORAGE

Couchbase Server 1

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

Data Service STORAGE

Couchbase Server 2

Managed Cache

Cluster Manager

Data Service STORAGE

Couchbase Server 3

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager

Data Service STORAGE

Couchbase Server 4

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager

Index Service

STORAGE

Couchbase Server 5

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager

Index Service

STORAGE

Couchbase Server 6

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager

Query Service

FTS Service

©2016 Couchbase Inc. 8©2016 Couchbase Inc.

Find Out More!

Preparing for success: the top deployment 10 do's and don'ts

• Today 11:00 am - 11:50 am

• Operations Track• Great America 1

Karthik SekarSolutions Architect,

Couchbase

Ian McCloyTechnical Support Manager,

EMEA, Couchbase

©2016 Couchbase Inc. 9©2016 Couchbase Inc.

Find Out More!

Picking the right API for the right job

• Wednesday 2:00 pm - 2:50 pm

• Development Track• Great America K Michael Nitschinger

JVM Engineer, Couchbase

©2016 Couchbase Inc. 10©2016 Couchbase Inc.

Application

Concept 2 – Automatic Sharding

Cluster

Key = connect2016

HASH(“connect2016”)

Server A Server CServer B

VB Server

0 A

1 B

2 C

…

27 B

1023 C

{…

}

©2016 Couchbase Inc. 11©2016 Couchbase Inc.

Application

Concept 2 – Automatic Sharding

Cluster

Key = connect2016

HASH(“connect2016”)

Server A Server CServer B

VB Server

0 A

1 B

2 C

…

27 D

1023 C

{…

}

Server D

©2016 Couchbase Inc. 12©2016 Couchbase Inc.

Concept 3 – Exceptional Data Streaming

• Database Change Protocol (DCP)

• High Performance / In-Memory

• De-Duplication• Ordered, predictable and

consistent• Restartable

Data Service

Index Service

Map/Reduce

DCP

Cross Data Center Cluster

Hadoop

Client/AppNotifications

(future)

Backup/Export XDCR

Full Text Search

DataReplication

©2016 Couchbase Inc. 13

STORAGE

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

Data Service

Index Service

Query Service

Full Text Search

©2016 Couchbase Inc. 14

Cluster Manager

©2016 Couchbase Inc.©2016 Couchbase Inc.

The Cluster Manager

STORAGE

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

Data Service

Index Service

Query Service

Full Text Search

The control plane of the server:• Cluster membership

• Status & health monitoring

• Service layout• Data placement

• Rebalance• Failover

• Authentication• Admin APIs

Implemented in Erlang

©2016 Couchbase Inc.©2016 Couchbase Inc.

The Cluster Manager

The control plane of the server:• Cluster membership

• Status & health monitoring

• Service layout• Data placement

• Rebalance• Failover

• Authentication• Admin APIs

Implemented in Erlang

Generic distributed facilities• Global configuration (gossip replication)• Distributed node discovery

Generic local facilities

Master services• Cluster-wide

operations • Data placement • Rebalance• Auto-failover

Admin Console & REST API

Per-node services • Heartbeats• Babysitter

Bucket services• DCP management• Stats collection & monitoring

Authentication

• Local configuration store• Logging and Other Services• Vclock, uuid• Work queue

• Misc

©2016 Couchbase Inc. 17

Data Service

©2016 Couchbase Inc. 18©2016 Couchbase Inc.

The Data Service

The (low-level) data plane of the server: • Key/Value access• Map/Reduce Views [*]

KV-Engine:

• Evolution of memcached; adding persistence, replication, enhanced data access APIs

• Asynchronous networking supports ~10K clients.

• Memory-centric architecture.• Disk IO performed via background threads.

C++11

STORAGE

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

Data Service

Index Service

Query Service

Full Text Search

©2016 Couchbase Inc.©2016 Couchbase Inc.

Data Service Architecture

Database Engine (ep-engine)

Dispatcher Request / Response

Scheduler

Reader IO

Aux IONon IO

…

KV-Bucket

…

KV-EngineItem Pager

Expiry Pager

Checkpoint Manager

ManagedCache

Partition Hash Tables

(Active and Replica)

Flusher

Batch Reader

Partition Hash Tables

(Active and Replica) Writer IO

DCP Streaming Authentication

©2016 Couchbase Inc. 20

Journey of a K/V operation

©2016 Couchbase Inc. 21©2016 Couchbase Inc.

Journey of a K/V operation

STORAGE

Couchbase Server 1

Managed Cache

Cluster Manager

Data Service

(1x replica)STORAGE

Couchbase Server 0

Managed Cache

Cluster Manager

Data Service

(1x replica)

SDK Application

DCP Replication

'tweets::Dhaikney'-> SD.upsert('hashtags', 'Connect2016')

©2016 Couchbase Inc. 22

Indexing Service

©2016 Couchbase Inc. 23©2016 Couchbase Inc.

Index Service

Three “indexing” services:• Incremental Map / Reduce Views

• Javascript map() & reduce() functions applied to all mutations• Supports geo-spatial views• Co-located with Data service

• Global Secondary Indexes (GSI)• Full-Text Search – more later

GSI: efficient indexes for secondary lookups and ad-hoc query processing

STORAGE

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

Data Service

Index Service

Query Service

Full Text Search

©2016 Couchbase Inc.©2016 Couchbase Inc.

Data Service

Projector & Router

Indexing Service

Bucket 1

DCP Stream

Projector and Router: 1 Projector and Router per node1 stream of changes per buckets per supervisor

Bucket 2

©2016 Couchbase Inc.©2016 Couchbase Inc.

Data Service

Projector & Router

Indexing Service

Query ServiceIndex Service

SupervisorIndex maintenance &

Scan coordinator

Index#2Index#1

Query Processorcbq-engine

Bucket 1

DCP StreamIndex#4Index#3

Bucket#2

Bucket#1

Projector and Router: 1 Projector and Router per node1 stream of changes per buckets per supervisor

ForestDBStorage Engine

Supervisor1 Supervisor per nodeMany indexes per Supervisor

Bucket 2

Memory-Optimized

Indexes

©2016 Couchbase Inc. 26©2016 Couchbase Inc.

Find Out More!

Memory-optimized indexes: how they work

Today, 4:00 pm - 4:50 pm• Great America 2

Sarath LakshmanSenior Software Engineer, Couchbase

©2016 Couchbase Inc. 27

Query Service

©2016 Couchbase Inc.©2016 Couchbase Inc.

Query Service

Client

FetchParse Plan Join FilterPre-Aggregate

Offset Limit ProjectSortAggregateScan

Query Service

Index Service

Data Service

©2016 Couchbase Inc. 29©2016 Couchbase Inc.

Find Out More!

How to understand and use the query optimizer

Today, 11:00 am - 11:50 am• Great America 2

Keshav MurthyDirector, Query Development, Couchbase

©2016 Couchbase Inc. 30©2016 Couchbase Inc.

Find Out More!

Going beyond SELECT, top 10 ways to use N1QL

Wednesday 10:00 am - 10:50 am

• Great America KPrasad VarakurProduct Manager, Couchbase

©2016 Couchbase Inc. 31

Full Text Search

©2016 Couchbase Inc. 32©2016 Couchbase Inc.

Full Text Search

“Googling for your JSON documents”

• Index Fields or Documents

• Lexical Analysis and Stemming

• Flexible Query Capabilities

• Available and Scalable

STORAGE

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

Data Service

Index Service

Query Service

Full Text Search

©2016 Couchbase Inc. 33©2016 Couchbase Inc.

Full Text Search

“Googling for your JSON documents”

• Index Fields or Documents

• Lexical Analysis and Stemming

• Flexible Query Capabilities

• Available and Scalable

Config{ Node Definitions: … Index Definitions: … Plan Indexes: …}

Planner Janitor PIndex

bleve

DCPFeed

REST Interface

Manager

DCPFeed

PIndex

bleve

©2016 Couchbase Inc. 34©2016 Couchbase Inc.

Find Out More!

Full-text search: how it works and what it can do

Wednesday, 1:00 pm - 1:50 pm

Great America 2 Steve YenCo-Founder, Couchbase

Marty SchochSenior Software Engineer,

Couchbase

©2016 Couchbase Inc. 35

New! Analytics Service

©2016 Couchbase Inc.©2016 Couchbase Inc.

New: Analytics Service!

STORAGE

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

Data Service

Index Service

Query Service

Full Text Search

Analytics Service

Sneak Peak: Couchbase AnalyticsWednesday, 2pm –

2:50pm• Great America 2 Till Westman, Senior Director,

Server Development, Couchbase

©2016 Couchbase Inc. 37©2016 Couchbase Inc.

Summary

• The architecture of a next generation database• The addition of new features whilst retaining our principles:

• Performance• Scalability• Availability • Ease of use

• More technical meat available at Connect• Ask questions and get involved!

©2016 Couchbase Inc. 38

Thank You!