couchbase 105 - cross datacenter replication and elasticsearch

61

Upload: couchbase

Post on 20-Aug-2015

1.525 views

Category:

Technology


0 download

TRANSCRIPT

Technical  Evangelist

twi0er:  @scalabl3email:  [email protected]

Jasdeep  Jaitla

Couchbase  105:    XDCR  and  elasEcsearch

(XDCR)

CROSS  DATA  CENTER  REPLICATION

(XDCR)

Replication Use Cases

• Unidirectional - Create Backup Cluster- Create Geographic Cluster, Chained Propagation- Create Development/Test Cluster with Active Data- Integrate with Elastic Search- Integrate with Custom Consumer

• Bi-Directional - Populate Multiple Clusters with Data Modified in each Cluster

Active-Active Replication• XDCR configured Per Bucket • Within Bucket, Each Partition (vbucket) Replicated Directly - Data is replicated from Active Partition -> Active Partition- Each Partition (1024) has a separate append only file!

- XDCR process:๏ Queues documents for replication๏ Transmits Documents via HTTP in batches๏ Keeps track of Checkpoint Intervals if Replication Streams are

Disconnected๏ Upon disconnection between clusters, automatically handles

reconnection attempts

8091

8092

8091

8092

XDCR Ports-Connections

MAPMAP

8091

8092

8091

8092

XDCR Ports-Connections

MAPMAP

HTTP

Retrieve  Target  Cluster  Map  &  Cluster  Map

8091

8092

8091

8092

XDCR Ports-Connections

MAPMAP

HTTP

Retrieve  Target  Cluster  Map  &  Cluster  Map

HTTP

Data  Replica4on

Partitions

1023

3210

102210211020

Partitions

1023

3210

102210211020

Active-to-Active

MAP MAP

16 GB RAM

512

Partitions

6 IO Workers

16 GB RAM

6 IO Workers512

Partitions

8 GB RAM

512

Partitions

3 IO Workers

8 GB RAM

3 IO Workers512

Partitions

Replication Follows Cluster Map

MAP

MAP

MAP

MAP

16 GB RAM

512

Partitions

6 IO Workers

16 GB RAM

6 IO Workers512

Partitions

8 GB RAM

512

Partitions

3 IO Workers

8 GB RAM

3 IO Workers512

Partitions

Replication Follows Cluster Map

MAP

MAP

MAP

MAP

16 GB RAM

512

Partitions

6 IO Workers

16 GB RAM

6 IO Workers512

Partitions

8 GB RAM

512

Partitions

3 IO Workers

8 GB RAM

3 IO Workers512

Partitions

Replication Follows Cluster Map

MAP

MAP

MAP

MAP

16 GB RAM

512

Partitions

6 IO Workers

16 GB RAM

6 IO Workers512

Partitions

8 GB RAM

512

Partitions

3 IO Workers

8 GB RAM

3 IO Workers512

Partitions

Replication Follows Cluster Map

MAP

MAP

MAP

MAP

Different  Configura4on  OK

8 GB RAM

512

Partitions

3 IO Workers256

8 GB RAM

Partitions

3 IO Workers

256

8 GB RAM

Partitions

3 IO Workers

256

8 GB RAM

Partitions

3 IO Workers

256

8 GB RAM

Partitions

3 IO Workers

8 GB RAM

3 IO Workers512

Partitions

Cluster Sizes Don’t Have to Match

8 GB RAM

512

Partitions

3 IO Workers256

8 GB RAM

Partitions

3 IO Workers

256

8 GB RAM

Partitions

3 IO Workers

256

8 GB RAM

Partitions

3 IO Workers

256

8 GB RAM

Partitions

3 IO Workers

8 GB RAM

3 IO Workers512

Partitions

Cluster Sizes Don’t Have to Match

Smaller  Cluster

Cluster Sizes Don’t Have to Match

256

8 GB RAM

Partitions

3 IO Workers

256

8 GB RAM

Partitions

3 IO Workers

256

8 GB RAM

Partitions

3 IO Workers

256

8 GB RAM

Partitions

3 IO Workers

8 GB RAM

512

Partitions

3 IO Workers

8 GB RAM

3 IO Workers512

Partitions

Cluster Sizes Don’t Have to Match

256

8 GB RAM

Partitions

3 IO Workers

256

8 GB RAM

Partitions

3 IO Workers

256

8 GB RAM

Partitions

3 IO Workers

256

8 GB RAM

Partitions

3 IO Workers

8 GB RAM

512

Partitions

3 IO Workers

8 GB RAM

3 IO Workers512

Partitions

Larger  Cluster

Chaining XDCR

Partitions

1023

3210

102210211020

Partitions

1023

3210

102210211020

Partitions

1023

3210

102210211020

Chaining XDCR

Partitions

1023

3210

102210211020

Partitions

1023

3210

102210211020

Partitions

1023

3210

102210211020

All  Writes

Chaining XDCR

Partitions

1023

3210

102210211020

Partitions

1023

3210

102210211020

Partitions

1023

3210

102210211020

All  Writes

Unidirec4onal  XDCR

Chaining XDCR

Partitions

1023

3210

102210211020

Partitions

1023

3210

102210211020

Partitions

1023

3210

102210211020

All  Writes

Unidirec4onal  XDCR 1.5  

Chaining XDCR

Intra-Cluster Node

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

Storage to XDCR Replication

Remote    Cluster

Intra-Cluster Node

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

Storage to XDCR Replication

storage ops

Remote    Cluster

Intra-Cluster Node

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

Storage to XDCR Replication

storage ops

Remote    Cluster

Intra-Cluster Node

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

Storage to XDCR Replication

storage ops

Remote    Cluster

Intra-Cluster Node

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

XDCR: Eventually Consistent

Remote    Cluster

Intra-Cluster Node

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

XDCR: Eventually Consistent

storage ops

Remote    Cluster

Time  1

Intra-Cluster Node

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

XDCR: Eventually Consistent

storage ops

Remote    Cluster

Time  1

Time  2

Intra-Cluster Node

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

XDCR: Eventually Consistent

storage ops

Remote    Cluster

Time  1

Time  2

Time  3

Bi-Directional Conflicts

Cluster 2

App Servers

Cluster 1

App Servers

What happens when you write to the same key in two clusters?

Bi-Directional Conflicts

Cluster 2

App Servers

Cluster 1

App Servers

replace(“key”, data)replace(“key”, data)

What happens when you write to the same key in two clusters?

Bi-Directional Conflicts

Cluster 2

App Servers

Cluster 1

App Servers

replace(“key”, data)replace(“key”, data)

What happens when you write to the same key in two clusters?

(conflict)

CONFLICT  RESOLUTION

Conflict Resolution Rules• Completely Automatic, Currently Not Customizable

!

- Rule 1: Revision #/Count๏ Whichever cluster has higher revision number will replicate to other cluster๏ If document is mutated in cluster with lower revision number, the document

will not be put in replication queue

!

- Rule 2: Metadata๏ If Revision Count equal, use CAS, Flags, and TTL๏ Basically Random

!

• It is not Time/NTP based

Bi-Directional Caution!

Avoid (as in don’t) update the same document in multiple clusters with bi-directional XDCR, as it may not have expected behavior due to

the conflict resolution rules.!

Separate Key spaces (use DC prefix) so that you avoid conflicts on individual documents.

Recommended

DC1::user:a9838-­‐s92-­‐s00  

DC2::user:293ba-­‐293-­‐922

The Document Life• Storage operation (set/add/replace/incr/decr)

• Stored in RAM Cache

๏ Can be Read by get operation

• Added to Disk Write Queue/Replication Queue

• Pushed to Replica Node RAM Cache (and DWQ)

๏ Exists after Failover

• Persisted to Active Partition Disk

• Added to XDCR Replication Queue

• Indexed by View Indexers (Design Document)

๏ Can be Queried in View

• Pushed to XDCR Remote Cluster RAM Cache

๏ Can be Read on Remote Cluster get operation

• Indexed by Elastic Search

๏ Can be Queried in Elastic Search

INTERFACE  DEMO

Q  &  A

ELASTIC  SEARCH

Integrated via XDCR

Integrated via XDCR

Replica Couchbase Intra-Cluster Machine

Architecture

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

Replica Couchbase Intra-Cluster Machine

Architecture

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

storage ops

Replica Couchbase Intra-Cluster Machine

Architecture

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

storage ops

Replica Couchbase Intra-Cluster Machine

Architecture

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

storage ops

Replica Couchbase Intra-Cluster Machine

Architecture

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

storage ops

Replica Couchbase Intra-Cluster Machine

Architecture

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

storage ops

Replica Couchbase Intra-Cluster Machine

Architecture

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

View Engine

Indexers

XDCR Replication Queue

Application Server

storage ops

Using Elastic SearchApplication Workflow

Applica>on

Using Elastic SearchApplication Workflow

Applica>on

result  set

Using Elastic SearchApplication Workflow

Applica>on

docs

INTERFACE  DEMO

Q  &  A