mysql replication best practices 105-232-931

baruch osoveskiy

baruch@dbaces.com

Baruch osoveskiy

Senior Consultant in dbaces.com

Linux-Unix SYSADMIN form 1997

DBA on Oracle and MySQL Since 2000

Working with unstructured Data (“big data”)

since 2000 like text and spatial

Oracle and Microsoft Partners

Oracle Aces and Ace Directors

What we do◦ Remote DBA services

◦ Database consulting and projects

◦ Oracle identity and access management

◦ Training

Platforms◦ Oracle DB

◦ Microsoft SQL Server

◦ MySQL

◦ PostgreSQL

One webinar each week

Webinar recording will be available on site a few

days after webinar

Check our webinars schedule @

http://www.dbaces.com/training/upcoming-webinars

subscribe to get updates @ www.dbaces.com

MySQL replication Introduction

New features in MySQL 5.6 replication

How to Create MySQL replication with hot-

backups

How To monitor MySQL replication

Master SlaveNETWORK

Master SlaveNETWORKSlave

SlaveSlave

Slave

• High availability – can promote slave to master

• Scale Out – add multiple read slaves • Analytics – can run Analytics on live

data.• Backup – run cold backup on slave.• For DEV and Pre-Prod

Replication Formats• SBR - Statement Based Replication

• Nondeterministic function problem • RBR - Row Based Replication

• High network usage • Higher IO usage

• Mixed Mode

Not safe Nondeterministic functionFOUND_ROWS(), GET_LOCK(), IS_FREE_LOCK(), IS_USED_LOCK(), LOAD_FILE(), MASTER_POS_WAIT(), RELEASE_LOCK(), ROW_COUNT(), SESSION_USER(), SLEEP(), SYSDATE(), SYSTEM_USER(), USER(), UUID(), and UUID_SHORT().

Where the Changes Stored • Binary Log – Master (blue-bin.000002) • Relay Log – Slave (white-relay-bin.000003)

Replication Threads• Master Server

• Binlog Dump Thread• Slave Server

• IO Thread• SQL Thread

Method of Sending the Data (Replication Type)• Asynchronous Replication• Semi-Synchronous Replication (5.5 )

Master

TRX1TRX2TRX3

.

Slave

1

Binarylog

DML+

COMMIT

Binlog Dump Thread

Master

TRX1TRX2TRX3

.

Slave

1

2

Binarylog

IO Thread

DML+

COMMIT

Master

TRX1TRX2TRX3

.

Slave

TRX1TRX2TRX3

.

1

2

3

Binarylog

RelayLog

IO Thread

DML+

COMMIT

Master

TRX1TRX2TRX3

.

Slave

TRX1TRX2TRX3

.

1

2

34

Binarylog

RelayLog

IO Thread

SQL Thread

DML+

COMMIT

Master

TRX1TRX2TRX3

.

Slave

TRX1TRX2TRX3

.

1

2

34

5

Binarylog

RelayLog

IO Thread

SQL Thread

DML+

COMMIT

Pros• Fast• slave can be disconnectedCons• Potential loss of transactions and data integrity

Master

TRX1TRX2TRX3

.

Slave

1

Binarylog

DML

Binlog Dump Thread

Master

TRX1TRX2TRX3

.

Slave

1

Binarylog

DMLIO Thread

Master

TRX1TRX2TRX3

.

Slave

1

Binarylog

DMLIO Thread

2

34

IO Thread

TRX1TRX2TRX3

.

RelayLog

Master

TRX1TRX2TRX3

.

Slave

TRX1TRX2TRX3

.

1

2

34

Binarylog

RelayLog

IO Thread

IO ThreadDML

Master

TRX1TRX2TRX3

.

Slave

TRX1TRX2TRX3

.

1

2

34

Binarylog

RelayLog

IO Thread

IO ThreadDML

COMMIT

Master

TRX1TRX2TRX3

.

Slave

TRX1TRX2TRX3

.

1

2

34

Binarylog

RelayLog

IO Thread

IO ThreadDML

COMMIT

5SQL Thread

Pros• data integrity

• Commit ensure the transaction will copy to slave

Cons• Performance Implications

• The transaction need to Wait for the slave to commit

• Problematic in high DML load

What's New

• Multi threaded Slave• Relay log recovery • Binary Log Group Commit• Crash-safe Replication – Binlog Checksums• Global Transaction ID

You can read more in:http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html

Replication performance is improved by using multiple execution threads to apply replication events

slave-parallel-workers=2

You can read more in:http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html

• slave can automatically recover from a failure and resume replicating DML updates

• the slave can roll back replication to the last successfully committed transaction

relay-log-recovery =1

You can read more in:http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html

And in Bug #13893363

• Commit a group of transactions to the binary log.• Ensures durability with good performance.• Designed to handle seamlessly temporary peeks

on the workload.

enable by default in 5.6

You can read more in:http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html

Master and slave can have different data • Non deterministic queries. • Wrong use of replication filters. • Write executed on slave outside the.

replication (slave is not read only)• Binlogs or relay log corruption.

How to check ? Before 5.6 you can use only :• show slave status • pt-table-checksum

• utilty from precona that can check table checksum

• First enable checksum on the master

binlog_checksum = CRC32;master_verify_checksum = on;

• Then start binlog verify on the slave

slave_sql_verify_checksum = on;

* From >= 5.6.6 enable by default

mysql> show slave status

…Last_Error : 1538 Last_Error : relay log read error…… …..# mysqlbinlog --verify-binlog-checksum mysql_18676-relay-bin.000002

[...]ERROR: Error in Log_event::read_log_event(): 'Event crc check failed! Most likely there is event corruption.', data_len: 103, event_type: 2ERROR: Could not read entry at offset 283: Error in log format or read error.[...]

Pros• Can help to find error in the bin log/relay log • Notify on a bin log corruptionCons• More CPU for the CRC • The MySQL will keep writing to the corrupt bin log

• Transactions are not guaranteed to execute in the order of the Binary logs

• The show slave status is not accurate

Master

Slave1

Slave2

BinlogFile:mysql-bin.0010Position:1238574

BinlogFile:mysql-bin.0020Position:1043

BinlogFile:mysql-bin.0140Position:8373

• The same event can have different Binary log file and position on different slaves

• Hard to find inconsistency in replication• Very problematic to reconfigure replication

• Unique identifier of a transaction across all servers in replication topology

The GTID look like :

GTID = source_id:transaction_idWhere transaction_id = transaction_start-transaction_stop

eg :3E11FA47-71CA-11E1-9E33-C80AA9429562:1-5

Master

Slave1

Slave2

BinlogFile:mysql-bin.00103E11FA47-71CA-11E1-9E33-C80AA9429562:1-5

BinlogFile:mysql-bin.00203E11FA47-71CA-11E1-9E33-C80AA9429562:1-5

BinlogFile:mysql-bin.01403E11FA47-71CA-11E1-9E33-C80AA9429562:1-5

How to enable GUID on MySQL replication• Enable read only mode on Slaves

set global read_only=1• Compare the binlog position on master and slave

show slave status show master status

• Change configuration on all servers and restart bin-logguid_mode=ONEnforce-guid-consistency

• Initialize GUID on all servers change master to MASTER_AUTO_POSITION = 1;

• Disable read only mode on Slaves

Master Blue

192.168.1.110

SlaveWhite

192.168.1.111

Master Blue

192.168.1.110

SlaveWhite

192.168.1.111

Data files from

Master Blue

TRX1TRX2TRX3

.

Binarylog

Master Blue

192.168.1.110

SlaveWhite

192.168.1.111Replication

blue.my.cnf

[mysqld]...server-id=1report-host=bluebinlog-format=MIXEDlog-binlog-slave-updates=truegtid-mode=on enforce-gtid-consistency=true master-info-repository=TABLErelay-log-info-repository=TABLEsync-master-info=1binlog-checksum=CRC32master-verify-checksum=1

white.my.cnf[mysqld]...server-id=2report-host=whitebinlog-format=MIXEDlog-slave-updates=truelog-bingtid-mode=on enforce-gtid-consistency=true master-info-repository=TABLErelay-log-info-repository=TABLEsync-master-info=1slave-parallel-workers=2binlog-checksum=CRC32master-verify-checksum=1slave-sql-verify-checksum=1binlog-rows-query-log_events=1

On Master:

[mysqld]rpl_semi_sync_master_enabled=1rpl_semi_sync_master_timeout=1000 # 1 second

On each Slave:

[mysqld]rpl_semi_sync_slave_enabled=1

we need to create the replication user on master

mysql> GRANT REPLICATION SLAVE ON *.* TO replication@192.168.1.111 IDENTIFIED BY 'reppwd';FLUSH PRIVILEGES;

create Hot backup with * Oracle Enterprise Backup

#mysqlbackup --port=3306 --protocol=tcp --user=root --password --backup-dir=/data/backup backup-and-apply-log

* Also can use Percona XtraBackup

• copy the backup directory to the slave via scp or other method

on the master:#tar –cvfZ /data/backup.taz /data/backup#scp /data/backup.taz 192.168.1.111:/data/

on the slave:#tar –xvfZ /data/backup.taz

• Start the Slave #service start mysql• Start the replication process that will start the

io_Thread and the sql_Threadmysql>CHANGE MASTER TO MASTER_HOST='192.168.1.110', MASTER_USER='replication', MASTER_PASSWORD='reppwd', MASTER_AUTO_POSITION=1;START SLAVE;

show slave status\G*************************** 1. row********Slave_IO_State: Waiting for master to send eventMaster_Host: 192.168.1.110Master_User: replicationMaster_Log_File: blue-bin.000002Read_Master_Log_Pos: 936Relay_Log_File: white-relay-bin.000003Relay_Log_Pos: 1146Relay_Master_Log_File: blue-bin.000002Slave_IO_Running: YesSlave_SQL_Running: Yes

show slave status\G*************************** 1. row********Last_Errno: 0Last_Error: Skip_Counter: 0Master_UUID: 2671c08b-7cf0-11e2-ac39-00163ebee7c2

Master_Info_File: mysql.slave_master_infoSQL_Delay: 0SQL_Remaining_Delay: NULL

Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update itRetrieved_Gtid_Set: 2671c08b-7cf0-11e2-ac39-00163ebee7c2:1-4Executed_Gtid_Set: 2671c08b-7cf0-11e2-ac39-00163ebee7c2:1-4

baruch@dbaces.com