mysql highav availability

MySQL High Availability Solutions Best Practices

baruch osoveskiy

[email protected]

About MeBaruch osoveskiy

• Senior Consultant @ brillix.co.il

• Linux-Unix SYSADMIN form 1997

• DBA on Oracle and MySQL Since 2000

• Working with unstructured Data (“big data”) since 2000 like text and spatial images etc..

[email protected]

https://github.com/barucho

http://www.slideshare.net/baruchosoveskiy

mailto:[email protected]

https://github.com/barucho


• In the end will be Q&A slide –

you can ask question I will try to answer

• The Demo will be in github :

https://github.com/barucho/MySQLHA2016.git

• You can send me question :

[email protected]

You can find more presentations on


• I write in ildba.co.il (not so often)

Some of our Customers

High Availability

The Problem

Redundancy

▪ multiple copies of data in event of a disaster or bad DML/DDL run in Production

Scaling

▪ to increase throughput (more network, disk, cpu)

High Availability

▪ get databases back online quickly

High Availability , Redundancy, Scaling ?

CAP …

Consistency▪ All nodes see the same data at the same time

• Availability▪ Every request receives a response

about whether it succeeded or not (COMMIT ?)

• Partition Tolerance▪ The system continues to operate

despite arbitrary partitioning due to

network failures

What is my expected UP-TIME (9`s)?

Manual Failover

▪ human usually can make a better decision as to whether failover is necessary.

▪ Avoid ping-ping between nodes!

▪ easy to configure

Automatic Failover

▪ More 9’s due to minimized outages.

▪ No need to wait on a DBA/ops.

▪ Very Complicate to configure (need cluser/scripts)

Automatic or Manual Failover?

The Solutions

Galera Cluster High Availability, Scaling-improve read performance

• Active Active MySQL* cluster • On Commit Changes are collected into a write

set.• Write set is set to all nodes for certification.• PKs are used to determine if the write set can be

applied.• If certification fails, the write set is dropped and

the transaction is rolled back.• All nodes will reach the same transaction

eventually.

Corosync-pacemaker DRDB active/passiveHigh Availability

• Corosync/pacemaker cluster or a co-ordinatingthe startup and recovery of inter-related services across a set of machines.

• DRDB shared storage systems by networked mirroring

• Only one mysqld process can run at a time • No active slave.• DRDB can have performance issues in high IO

load • Complicate to Configure

Master Slave

DRDB

corosync

pacemaker

DRDB

MySQL MySQL

corosync pacemaker DRDB active/passive MySQL

quorum

Master Slave

DRDB

corosync

pacemaker

DRDB

MySQL MySQL


quorumMaster

Master Slave

DRDB

corosync

pacemaker

DRDB

MySQLMySQL


quorum

MySQL

• One or more mysqld connected to one or more data nodes

• The data on the data node is shard and replicated

• Simple and automatic sharding and replication

• To use MySQL cluster you have to use NDB storage engine.

MySQL Cluster / NDB / shardingHigh Availability, Scaling

mysqld

MySQL Cluster / NDB / sharding

mysqld mysqld mysqld

A1

B2

B1

A2

NDB NDB NDB

NDB API

SQL

REST API

February 2015: MySQL Cluster 7.4 - 200 Million NoSQL QPS

MySQL Cluster 7.4 delivers massively concurrent NoSQL access –

200 Million reads per second using the FlexAsync benchmark.

This was achieved with 32 (out of a maximum 48) data nodes,

each running on a server with 2x Intel Haswell E5-2697 v3 CPUs.https://www.mysql.com/why-mysql/benchmarks/mysql-cluster/

INNODB vs NDB

Feature InnoDB 1.1 NDB 7.5

Transactions All standard types READ COMMITTED

MVCC Yes No

Data Compression Yes No

Large Row Support (> 14K) Supported for VARBINARY, VARCHAR, BLOB, and TEXT columns

Supported for BLOB and TEXT columns only

High Availability (HA) Requires additional software Yes (Designed for 99.999% uptime)

Time for Node Failure Recovery 30 seconds or longer Typically < 1 second

NoSQL Access to Storage Engine Yes Yes

MySQL GTID Replication

What is MySQL Replication

Master SlaveNETWORK

What is MySQL Replication

Master SlaveNETWORKSlave

SlaveSlave

Slave

Why MySQL Replication

• High availability – can promote slave to master

• Scale Out – add multiple read slaves • Analytics – can run Analytics on live data.• Backup – run cold backup on slave.• For DEV and Pre-Prod

How MySQL Replication Works

Replication Formats• SBR - Statement Based Replication

• Nondeterministic function problem • RBR - Row Based Replication

• High network usage • Higher IO usage

• Mixed Mode


Not safe Nondeterministic functionFOUND_ROWS(), GET_LOCK(), IS_FREE_LOCK(), IS_USED_LOCK(), LOAD_FILE(), MASTER_POS_WAIT(), RELEASE_LOCK(), ROW_COUNT(), SESSION_USER(), SLEEP(), SYSDATE(), SYSTEM_USER(), USER(), UUID(), and UUID_SHORT().


Where the Changes Stored • Binary Log – Master (blue-bin.000002) • Relay Log – Slave (white-relay-bin.000003)


Replication Threads• Master Server

• Binlog Dump Thread• Slave Server

• IO Thread• SQL Thread


Method of Sending the Data (Replication Type)• Asynchronous Replication• Semi-Synchronous Replication (5.5 )• Group Replication (plugin)


Master

TRX1TRX2TRX3

.

Slave

1

Binarylog

DML+

COMMIT

Binlog Dump Thread


Master

TRX1TRX2TRX3

.

Slave

1

2

Binarylog

IO Thread

DML+

COMMIT


Master

TRX1TRX2TRX3

.

Slave

TRX1TRX2TRX3

.

1

2

3

Binarylog

RelayLog

IO Thread

DML+

COMMIT


Master

TRX1TRX2TRX3

.

Slave

TRX1TRX2TRX3

.

1

2

34

Binarylog

RelayLog

IO Thread

SQL Thread

DML+

COMMIT


Master

TRX1TRX2TRX3

.

Slave

TRX1TRX2TRX3

.

1

2

34

5

Binarylog

RelayLog

IO Thread

SQL Thread

DML+

COMMIT

MySQL 5.6 Replication

What's New

• Multi threaded Slave• Relay log recovery • Binary Log Group Commit• Crash-safe Replication – Binlog Checksums• Global Transaction ID

You can read more in:http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html

http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html

Multi-Threaded Slaves

Replication performance is improved by using multiple execution threads to apply replication events

slave-parallel-workers=2



Binary Log Group Commit

• Commit a group of transactions to the binary log.• Ensures durability with good performance.• Designed to handle seamlessly temporary peeks on the

workload.

enable by default in 5.6



Relay log recovery

• slave can automatically recover from a failure and resume replicating DML updates

• the slave can roll back replication to the last successfully committed transaction

relay-log-recovery =1


And in Bug #13893363


Global Transaction ID

Position (pre 5.6)

• Transactions are not guaranteed to execute in the order of the Binary logs

• The show slave status is not accurate

Position

Master

Slave1

Slave2

BinlogFile:mysql-bin.0010Position:1238574



Position - What's the problem

• The same event can have different Binary log file and position on different slaves

• Hard to find inconsistency in replication• Very problematic to reconfigure replication


• Unique identifier of a transaction across all servers in replication topology

The GTID look like :

GTID = source_id:transaction_idWhere transaction_id = transaction_start-transaction_stop

eg :3E11FA47-71CA-11E1-9E33-C80AA9429562:1-5


Master

Slave1

Slave2

BinlogFile:mysql-bin.00103E11FA47-71CA-11E1-9E33-C80AA9429562:1-5



Global Transaction IDHow to enable GUID on MySQL replication• Enable read only mode on Slaves

set global read_only=1• Compare the binlog position on master and slave

show slave status show master status

• Change configuration on all servers and restart bin-logguid_mode=ONEnforce-guid-consistency

• Initialize GUID on all servers change master to MASTER_AUTO_POSITION = 1;

• Disable read only mode on Slaves

Replication Setupmaster.my.cnf

[mysqld]...server-id=1report-host=bluebinlog-format=MIXEDlog-binlog-slave-updates=truegtid-mode=on enforce-gtid-consistency=true master-info-repository=TABLErelay-log-info-repository=TABLEsync-master-info=1binlog-checksum=CRC32master-verify-checksum=1

Replication Setupslave.my.cnf[mysqld]...server-id=2report-host=whitebinlog-format=MIXEDlog-slave-updates=truelog-bingtid-mode=on enforce-gtid-consistency=true master-info-repository=TABLErelay-log-info-repository=TABLEsync-master-info=1slave-parallel-workers=2binlog-checksum=CRC32master-verify-checksum=1slave-sql-verify-checksum=1binlog-rows-query-log_events=1

MySQL Replication Types

Group Replication

• The replication group is a set of servers that interact with each other through message passing

• Each server in the group may execute transactions independently

• RW transaction only commits after that operation is coordinated within the group

• when a transaction is ready to commit, the server atomically broadcasts the write values (row events) and write set (unique identifiers of the rows that were updated). This establishes a global total order for that transaction

Group Replication

• A highly available distributed MySQL database service

• Removes the need for manually handling server fail-over

• Provides distributed fault tolerance

• Enables Active/Active update anywhere setups

• Automates reconfiguration (adding/removing nodes, crashes, failures)

• Automatically detects and handles conflicts

• Automated master monitoring and failover.

• MHA consists of MHA Manager and MHA Node packages. MHA Manager runs on a manager server, and MHA Node runs on each MySQL server

• MHA is a free opensource tool

• Can be configured to power off old Master to avoid split brain.

MySQL Replication and MHA

MaxScale (open source database proxy)

• Open Source; Developed by mariadb

• “simple” database proxy

• Provides plugins

• Content aware

MaxScale (open source database proxy)Redundancy, High Availability, Scaling (read)

Master Slave

MaxScleMaxScle

MaxScle

Client

Read Write Read Only

SlaveSlave

MySQL ReplicationSlave

Delay Replication

MySQL Router

• Open Source; Developed by Oracle MySQL HA

• Simple layer 4 packet router • No packet inspection

• Have Framework for plugins

• Natively integrated with MySQL Fabric

• Support MySQL InnoDB Cluster (Lab)

MySQL Router - plugins

• Routing • First available • Round robin

• Fabric cache

• Logger

fa fa

MySQL Router

Slave

MySQL Router

Client


SlaveMySQL

Replication

Fabric repository

RO HA GROUP

Metadata

RW HA GROUP

More database routers/Load balancer

• HAProxy

• F5 BigIP

• ProxySQL

• ELB (AWS)

MySQL INNODB Cluster

MySQL InnoDB Cluster: Vision “A single product — MySQL — with high availability and scaling features baked in; providing an integrated end-to-end solution that is easy to use.”

Redundancy, High Availability, Scaling


A minimum of three instances are required to create an InnoDB

cluster that is tolerant to the failure of one instance.

MySQL Replication demo

3306 3307

MaxScle

Client


3308MySQL

Replication

Thank [email protected]

mysql highav availability

Technology