open source india - mysql labs: multi-source replication

1 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |

MySQL Labs : Multi Source Replication

Shivji Kumar Jha, Software Developer,MySQL Replication Team


Safe Harbour Statement

The following is intended to outline our general product direction. It is intended for

information purposes only, and may not be incorporated into any contract.

It is not a commitment to deliver any material, code, or functionality, and should not

be relied upon in making purchasing decisions. The development, release, and

timing of any features or functionality described for Oracle’s products remains at the

sole discretion of Oracle.


MySQL Labs features are not fit for production.

They are provided solely for testing purposes, to try the latest bug fixes and generally to keep up with the development.

Please, do not use these binaries in production.

Instead, install them on a spare server.

If you are looking for production ready binaries, please visit MySQL Downloads.

http://dev.mysql.com/downloads/


Agenda

Background: Why Use Replication?

Multi-source Replication

– Introduction

– Use cases

– Internals

– Monitoring

Reading More about Multi-source Replication


Read scale-out

M SS

S

S

M

write clients read clientsread clients

write clients

Morereads?More

slaves!

Background: What is Replication Used For?


M M?

write clients write clients

Morewrites?More

Masters?

What about Write scale-out?

M?

M?

Enter MySQL Fabric ...


Redundancy: If master crashes, promote slave to master

C

B

A


C

B

ACrash

C

B

A

B is thenew master

Uh Oh! Whew!


On-line Backup and Reporting


M

S

write clients

business intelligent client apps reporting client apps big queries client apps


Image fromwww.ginkgomaps.com


CB

BAAC

Image fromwww.ginkgomaps.com


Agenda



– Introduction

– Use cases

– Internals

– Monitoring



Multi-Source Replication: Introduction

A Slave can have more than one master.

S

M4

M1

M3

M2



M1db1db1db1




M1

M2

db1db1db1

db2




M4

M1

M3

M2

db1db1db1

db4

db3

db2




S db1db1db1 db4db3db2

M4

M1

M3

M2

db1db1db1

db4

db3

db2



Agenda



– Introduction

– Use cases

– Internals

– Monitoring



Multi-Source Replication: Use Cases

The main use cases of Multi-source replication are related to data aggregation.


Business IntelligenceData analytics

Backupetc

SM2

M1

M3

Database 2

Database 3

Database 1, 2, 3

Database 1




SM

M

M

Shard 2

Shard 3

Full table

Shard 1


Agenda



– Introduction

– Use cases

– Internals

– Monitoring



Multi-Source Replication: More details..

S

M4

M1

M3

M2

Slave can have more than one master.

– Receive transactions from several MySQL servers simultaneously.

– Apply transactions from different masters simultaneously.

– No conflict detection or resolution.


Insert...

Insert...

B

binary log

Insert...

relay log

Insert...

A

binary log

Client

Network

Multi-Source Replication: What is a Channel?

Senderthread

Receiverthread Applier

thread

A channel is an an abstraction for a sender-receiver-applier path.

Channel


S

relay log1M1 binary log Network

Multi-Source Replication: How Many Channels?

Number of channels on slave = Number of sources.



binary log



Multi-Source Replication: Slave Appliers

Each channel has its own single-threaded slave applier.



binary logS


binary log

M1 binary log Network

Multi-Source Replication: Slave Appliers

Each channel has its own multi-threaded slave applier.

SM2

M3

Network relay log2

relay log3Network relay log3

binary log

binary log

relay log1


Adding / Configuring A Channel

Each channel can be configured individually.

To add or alter a channel configuration use:

CHANGE MASTER TO master_def …

FOR CHANNEL=”<channel_name>”


The FOR CHANNEL=”<channel_name>” clause is consistent across different replication commands.

– start slave [...] FOR CHANNEL=”<channel_name>"

– stop slave [thread_types] FOR CHANNEL=”<channel_name>”

– reset slave [all] FOR CHANNEL=”<channel_name>”

– flush relay logs FOR CHANNEL=”<channel_name>”

– show relay log events FOR CHANNEL=”<channel_name>”

...

Replication Commands on a Channel


The FOR ALL CHANNELS clause.

– START SLAVE [thread_type] FOR ALL CHANNELS

– STOP SLAVE [thread_type] FOR ALL CHANNELS

Working With All Channels


Compatibility With Other Replication Features

Multi-Threaded Slaves Yes

Global Transaction Identifiers Yes

Semi-Synchronous Replication Yes

Crash Safe Slaves Yes

Filters Yes


Agenda



– Introduction

– Use cases

– Internals

– Monitoring



Replication Monitoring: P_S Replication Tables

Connection Configuration

Connection Status

Execution Configuration

Execution Status

Slave Status

Applier / CoordinatorStatus

WorkersStatus


We have six performance schema tables for replication (MySQL-5.7.2):

– replication_connection_configuration

– replication_connection_status

– replication_execute_configuration

– replication_execute_status

– replication_execute_status_by_coordinator

– replication_execute_status_by_worker

Consistent semantics across tables. Lets explore one of them.

Replication Monitoring: P_S Replication Tables

One row per CHANNEL

One row per CHANNEL per WORKER


mysql> select * from performance_schema.replication_connection_status\G

*************************** 1. row ***************************CHANNEL_NAME : CHANNEL1SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12bTHREAD_ID : 13SERVICE_STATE : ONRECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4LAST_ERROR_NUMBER : 0LAST_ERROR_MESSAGE :LAST_ERROR_TIMESTAMP : 0000-00-00 00:00:00*************************** 2. row ***************************CHANNEL_NAME : CHANNEL2...

Replication Monitoring: Connection Status

One row per CHANNEL





Receiver thread & its service state





Set of transactions receivedthrough this channel



*************************** 1. row ***************************CHANNEL_NAME : CHANNEL1SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12bTHREAD_ID : 13SERVICE_STATE : ONRECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4LAST_ERROR_NUMBER : 1045LAST_ERROR_MESSAGE : error connecting to master '[email protected]:13000' ...LAST_ERROR_TIMESTAMP : 2013-11-04 13:37:23*************************** 2. row ***************************CHANNEL_NAME : CHANNEL2...


Oops! There was an error in the receiver thread on this channel



*************************** 1. row ***************************CHANNEL_NAME : CHANNEL1SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12bTHREAD_ID : NULLSERVICE_STATE : OFFRECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4LAST_ERROR_NUMBER : 1045LAST_ERROR_MESSAGE : error connecting to master '[email protected]:13000' ...LAST_ERROR_TIMESTAMP : 2013-11-04 13:37:23*************************** 2. row ***************************CHANNEL_NAME : CHANNEL2...


Receiver thread on this channel stopped serving...


Agenda



– Introduction

– Use cases

– Internals

– Monitoring



Read More About Multi Source Replication

Read more about Multi-source replication on Rith's blog:http://on-mysql-replication.blogspot.in/2013/09/mysql-labs-multi-source-replication.html

Read more about Multi-source replication from the design notes:http://dev.mysql.com/worklog/task/?id=1697

Read a feature preview of Multi-source replication on Rith's blog:http://on-mysql-replication.blogspot.in/2013/09/feature-preview-mysql-multi-source-replication.html


Read more about replication P_S tables on Shiv's blog:http://shivjijha.blogspot.com/2013/09/Monitoring-Replication-with-the-NEW-performance-schema-tables.html

Read More About Multi Source Replication

Official MySQL documentation for replication P_S tables:http://dev.mysql.com/doc/refman/5.7/en/performance-schema-replication-tables.html

http://shivjijha.blogspot.com/2013/09/Monitoring-Replication-with-the-NEW-performance-schema-tables.html

http://shivjijha.blogspot.com/2013/09/Monitoring-Replication-with-the-NEW-performance-schema-tables.html

open source india - mysql labs: multi-source replication

Technology

open source india bangalore

oracle andor

use replication

clients copyright

clients8 copyright

sole discretion of oracle

new master7 copyright

mysql downloads