MySQLMulti-Master Replication

Challenge

Michael Naumov

About Wix 1. We provide a platform to build your own and free awesome web site for any device 2. About 40M Registered Users 3. Around 1.3M new registrations every month

4. About 500 employees with 4 offices around the world.

Why do we need Multi-Master topology?

The Vision: Multiple, active DBMS servers with exactly the same data 1. Simple high availability model 2. Operate systems over multiple sites 3. Access geographically “close” data

BUT!It is not so easy to do as its seems

MySQL consistency in stand alone install

Transaction #1 Transaction #2update users update usersset follower=10 set follower=100where id=101 where id=101

Stop

Consistency in a Distributed environment

Transaction #1 Transaction #2update users update usersset follower=10 set follower=100where id=101 where id=101

Communication latency

Transaction #1 In US Transaction #2 in EU

Async replication =No application latency

SemiSync/Sync replication =Application latency

CAP Theorem

So what are the solutions out there?

MySQL Native Replication

How Does Native MySQL M/M Replication Work

Master 1 Master 2

Binlog Binlog

• Statement replication = send client SQL

• Row replication = send changed rows

• Use row replication for multi-master

Row vs. Statement Replication

Auto-Increment Key Offsets

• Set different keys on each server to avoid primary key collisions

[my.cnf]server-id=1auto-increment-offset = 1auto-increment-increment = 4...

[my.cnf]server-id=2auto-increment-offset = 2auto-increment-increment = 4...

MySQL Multi-Master Topologies

Master 1

Master 2 Master 3 Master 2

Master 1

Master-master Circular

Replication Replication

MySQL Native Replication Summary

• Built in with well-known capabilities

• MySQL replication by default is asynchronous. In addition to the built-in asynchronous replication, MySQL 5.5 and above supports semi-synchronous replication.

• Very limited topology support

• Very limited conflict avoidance

• Not a good choice for multi-master if there are writes to more than 1 master

MySQL Cluster (NDB)

How Does NDB Work?

AccessLayer

Storage Layer

Mysql Client Mysql Client

NDB1

NDB2NDB4

NDB3

NDB Eventual Consistency Algorithm

• NDB has built-in cross-cluster conflict detection based on epochs and primary keys

• Updates to primary always succeed

• Update on secondary may be rolled back if primary has a conflicting update

• MySQL Cluster resends updates from the primary to “cure” conflicts on the secondary's clusters

NDB Supported Topologies

NDBCluster 2

Secondary

NDB Cluster 2

Secondary

NDBCluster 3

Secondary

NDBCluster 1Primary

Master-master CircularReplication Replication

NDBCluster 1Primary

MySQL NDB Cluster Summary

• Allows active/active operation on 2 clusters

• Fully synchronous, no action can be returned to client until transactions on all nodes are really accepted.

• NDB is mainly an in memory database and also if it support table on disk the cost of them not always make sense.

• Use horizontal partition to equally distribute data cross node, but none of them has the whole dataset

• Covers failure of individual MySQL nodes inside the cluster by replication factor

• Detects conflicts automatically on rows

Galera

How Does Galera Work?

Galera Replicator

Master 1 Master 2 Master 3

wsrep API* wsrep API wsrep API

wsrep API (write set replication API), defines the interface between Galera replication and the DBMS

Galera approach is Data Centric

• Data does not belong to a node – Node belong to data. Using global transaction Id’s

• Connect to any nodes for writes

• No headache for auto increment. Galera will do it for you

• Galera replicate the full dataset across ALL nodes.

• Galera replicate data synchronously from one node to cluster on the commit, but apply them on each node by a FIFO queue (multi thread).

Galera Summary

• Galera require InnoDB to work.

• Galera data replication overhead, increase with the number of nodes present in the cluster.

• Galera do not offers any parallelism between the nodes when retrieving the data; clients rely on the single node they access.

• Synchronous replication

• Percona XtraDB Cluster is based on Galera

Tungsten

How Does Tungsten Replication Work

Master Alpha

Master Bravo

RemoteBravoSlave

TungstenReplicator Alpha

TungstenReplicator Bravo

LocalAlpha

Master

LocalBravo

Master

RemoteAlphaSlave

Tungsten Failure Handling

• Replication stops and resumes automatically when network link goes down

• Replication stops on replicator or DBMS failure and recovers after operator restart

• Conflicts can break replication. Auto increment keys should be manually configured on each node

• Have his own filters for data replication

Tungsten Supported Topologies

All Masters

Star

Snowflake

Tungsten Summary

• Allows active/active operation on N clusters

• Transfer is asynchronous

• Links can be down for days or weeks if required

• It is the application’s responsibility to ensure there are no conflicts

• Tungsten Replicator can replicate data from MySQL to MongoDB, Oracle, NuoDB, Vertica and others

• Tungsten allows replication from Oracle by using Change Data Capture (CDC). Destination DBMS can be MySQL or Oracle

Solution Comparison

Native MySQL (5.6)

MySQL NDB Galera Tungsten

InnoDB + - + +

Asynchronous + - + +

Statement based + + - +

Row Based + - + +

Semi-synchronous + - - -

Synchronous - + + -

Global TRX Id + + + +

Time delay replication + - - +

Thank you all

@NaumovMichaelhttp://goo.gl/OqaigN