mysql replication @ gnugroup

79

Upload: jayant-chutke

Post on 22-Jan-2018

71 views

Category:

Data & Analytics


3 download

TRANSCRIPT

What is replication?Allows 2 or more databases to maintain state

between themMySQL 3.23 and up supports asynchronous

replicationOne server acts as a master for one or more slavesSlaves pull data to be replicated from server’s

binary logs and execute the relevant statements locally

3

MySQL Replication Why? How?

1. High Availability Snapshots (Backup) Possibility of fail-over 1. Client program mysqldump

2. Load-balancing/Scale- With log coordinates out 2. Using backup Query multiple servers InnoDB, NDB

3. Off-site processing Don’t disturb master

Binary log 1. Replication

Asynchronous pushing to slave

2.Point-in-time recovery Roll-forward

Replication basics…

One master supports multiple slaves Each node on the network is assigned a unique identifying number Each slave attempts to connect to the master and fetches data to be replicated “Master” keeps logs of all changes – called “binary logs” or “binlogs” “Slave” connects to the master through normal MySQL protocol (TCP port 3306)

MasterSlave

SlaveSlaveSlaves

Replication basics…

Clients perform data modification on master server

INSERT, SELECT, DELETE, LOAD DATAEXECUTE in MySQL 5.0 and above

Master SlaveClient

DATA DATAINSERT INTO …

Replication basics…

Immediately following execution of command on master, the command is written to the local binary log

Additionally, the master records its unique ID (to prevent endless loops in circular replication scenarios) and the timestamp for use with statements which use NOW(), etc.

Master SlaveClient

DATADATA

INSERT INTO …

Binary Log

Replication basics…

If the slave is online, the command is transmitted to the slave in parallel (well, immediately following) to being written in the local binary log

Otherwise, when the slave next connects it will receive a list of all pending statements from the master server’s binary log

The slave’s replication IO thread stores the command in the local relay log

Master SlaveClient

DATADATA

INSERT INTO …

INSERT INTO …

Relay Log

Replication Thread

Replication basics…

Once the data is received in the slave’s relay log, the slave SQL thread executes the command locally, bringing the slave up-to-date with the master

In MySQL 3.23, the IO and SQL threads were just one thread. In later versions this was changed to boost performance

Master SlaveClient

DATADATA

INSERT INTO …

INSERT INTO …

Relay Log

Replication Thread

DATA

Replication basics…(contd)

Replication works with all tables types• Any “critical” reads must be done on the master –replication is asynchronous, there may be a delay• Master will rotate binary logs automatically forevery 1G of log records• You must purge any old, unused, logs yourself

Terminology

Synchronous replication Master

• A transaction is not committed until the data MySQL has been replicated (and applied) Server

• Safer, but slower • This is available in MySQL Cluster

Replication

Asynchronous replication • A transaction is replicated after it has been

MySQL committed Server • Faster, but you can in some cases loose

transactions if master fails Slave

• Easy to set up between MySQL servers

Terminology Master MySQL Server

• Changes data • Has binlog turned on Master

• Pushes binlog events to slave after slave has requested them MySQL Server

Slave MySQL Server • Main control point of replication

Replication • Asks master for replication log • Gets binlog event from master

MySQL Binary log Server

• Log of everything executed Slave • Divided into transactional components • Used for replication and point-in-time recovery

Simple Replication SetupModify my.cnf to include a unique server-id for

each nodeOn master server, ensure that log-bin (binary

logging) is enabled in my.cnfOn slave, configure login credentials on master,

either via my.cnf or CHANGE MASTER TO statement

Copy initial data snapshot from master to slaveConfigure initial binary log position on slaveStart replication with SLAVE START command

server-id – The unique server ID of this MySQL• log-bin – Enable logging of changes to binarylogs• log-slave-updates – Log updates that arrive onthe slave thread to the binary logs as well(required if this master is also a slave of anothermachine)• binlog-do-db – Disables logging of any changes,except to the specified databases• binlog-ignore-db – Log all changes, as usual,except for the specified databases

my.cnf

-------------

[mysqld]

server-id = 1

log-bin

ON MASTER SIDE

1 ) grant slave users on the master server :Each slave must connect to the master using a standard MySQL user name and password, so there must be a user account on the master that the slave can use to connect. In this scenario with existing slave, user for replication already exist on the master server.

PermissionsSlaves need REPLICATION SLAVE permission on

master for basic usageIf LOAD TABLE FROM MASTER or LOAD DATA

FROM MASTER statements are used, slave will also need SUPER and RELOAD privileges

Configuration - grants on master

GRANT REPLICATION SLAVE on *.* TO ‘rep_user’@’slave-host’ IDENTIFIED BY ‘this-is-the-password’

ON MASTER SIDE

2) Locking databases:Take a section of mysql and lock the tables with the below command.USE exampledb; database name

mysql> FLUSH TABLES WITH READ LOCK;

Do not close the section. Kept it open

ON MASTER SIDE

3) Position Noting:Use the below command and note down the position of master

SHOW MASTER STATUS;

MASTER mysql> SHOW MASTER STATUS;+---------------------------+----------+--------------+------------------+| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |+---------------------------+----------+--------------+------------------+| vmware-mirimar-bin.000002 | 79 | | |+---------------------------+----------+--------------+------------------+

ON MASTER SIDE

 4) Backup the database: Backup the databases using the following command

#> mysqldump -u root -p --databases trains  > /tmp/NNNNNN.sql

ON MASTER SIDE

 5) unlocking databases;On successfully compellations of backup activity up can unlock databases on the activity section left open previouslymysql> UNLOCK TABLES;

my.cnf

-------------

[mysqld]

server-id = 2

master-user = someuser                ---> this setting not recommended in cnf file.

master-password = secret              ---> this setting not recommended.

master-host = ip.of.master             ---> this setting not recommended.

ON SLAVE SIDE

 1) restore backup;

Restore the database

Mysql> \. /tmp/NNNNNN.sql 

NNNN is DB name

Restore the backup onto the slave

Master

Slave

ON SLAVE SIDE 2) Defining Position: CHANGE MASTER TO MASTER_HOST='IP', MASTER_USER='slave_user', MASTER_PASSWORD='<some_password>', MASTER_LOG_FILE='mysql-bin.NNNNN',MASTER_LOG_POS=NN;

ON SLAVE SIDE 3) Start Slave:start slave;

SLAVE mysql> SHOW SLAVE STATUS\G*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: vmware-mirimar Master_User: someuser Master_Port: 3306 Connect_Retry: 60 Master_Log_File: vmware-mirimar-bin.000002 Read_Master_Log_Pos: 79 Relay_Log_File: vmware1-mirimar-relay-bin.000002 Relay_Log_Pos: 250 Relay_Master_Log_File: vmware-mirimar-bin.000002 Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 79 Relay_Log_Space: 250

Seconds_Behind_Master: 0

REPLICATION

IS

DONE

More optional configuration on the slave read-only

log-slave-updates skip-slave-start

Replication Topologies

Master with Slave

Master

Slave

Master with Slave

binary log 

Master

TCP connection 

Slave

Replication is independent of Storage Engines You can replicate between any pair of engines 

InnoDB to InnoDB 

MyISAM to MyISAM 

InnoDB to MyISAM 

MEMORY to MyISAM 

etc...

The binary log is not the InnoDB transaction log (or the Falcon log, or ...) 

Master with Many Slaves

Master

Slave Slave Slave Slave

Chain

Master/ Master Slave

Slave

log_slave_updates = 1

Chain - Server 2 goes down...

Master/ Master Slave X 

... Server 3 is still up, but out of sync

Master/ Master Slave X 

Each server has a unique "server_id"

server_id=3 server_id=1

Master/ Master Slave

Slave

server_id=2

... and every event in a binary log file contains the server id number of the server where the event originated. 

Ring server_id=2

Master/ Slave

Master/ Slave

server_id=1 Master/ Slave

server_id=3

The ring topology is not a recommended configuration

Master/ Slave

Master/ Slave

Master/ X

Pair of Masters

Master/ Master/ Slave Slave

The pair is a “special case” of the ring topology used for high availability.

The two most common topologies for MySQL Replication

Master

Master/ Master/ Slave Slave

Slave

Slave

Slave

The "Relay Slave" The master has to handle only one

Master TCP connection.

Relay Slave log_slave_updates

Slave Slave Slave Slave Slave

And now introducing... the blackhole storage engine

Master

engine = blackhole The relay slave Relay manages replication Slave

logs, but not actual data.

Slave Slave Slave Slave Slave

Replication Commands A quick run-through of the commands

SHOW MASTER STATUS Used on master

Requires SUPER or REPLICATION CLIENT privileges

Gives log file and position master is writing to Also

shows database filters used

mysql> SHOW MASTER STATUS; +---------------+----------+--------------+------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | +---------------+----------+--------------+------------------+ | mysql-bin.003 | 73 | test | manual,mysql | +---------------+----------+--------------+------------------+

SHOW BINARY LOGS Used on master

Requires SUPER privileges

Will display a list of binary logs on the server

Use it before using PURGE BINARY LOGS

mysql> SHOW BINARY LOGS; +---------------+-----------+ | Log_name | File_size | +---------------+-----------+ | binlog.000015 | 724935 | | binlog.000016 | 733481 | +---------------+-----------+

SHOW BINLOG EVENTS Used on master

Requires REPLICATION SLAVE privileges

Show events in binary log

Also check mysqlbinlog utility

mysql> SHOW BINLOG EVENTS FROM 390 LIMIT 1\G *** 1. row ***

Log_name: slave-bin.000001 Pos: 390

Event_type: Query Server_id: 2

End_log_pos: 476 Info: use `test`; create table t1 (a int) 1

row in set (0.00 sec)

SHOW SLAVE HOSTS Used on master

Requires REPLICATION SLAVE privileges Shows list of

slaves currently registered with the master Only slaves

started with report-host option are visible

mysql> SHOW SLAVE HOSTS; +-----------+-----------+------+-----------+ | Server_id | Host | Port | Master_id | +-----------+-----------+------+-----------+ | 2 | 127.0.0.1 | 9308 | 1 | +-----------+-----------+------+-----------+ 1 row in set (0.00 sec)

PURGE BINARY LOGS Used on master

Requires SUPER privileges

Removes log files before a certain log file or date

MASTER can be used in place of BINARY

Alternative is to use variable EXPIRE_LOGS_DAYS

SHOW SLAVE STATUS Used on slave

Requires SUPER or REPLICATION CLIENT privileges

Shows some interesting information:

If the slave threads are running What

position the I/O thread read last What

position the SQL thread executed last

Error message and code, if thread stopped due to an error

SHOW SLAVE STATUS (5.1) mysql> SHOW SLAVE STATUS\G

*** 1. row *** Slave_IO_State: Last_Errno: 0

Master_Host: 127.0.0.1 Last_Error: Master_User: root Skip_Counter: 0 Master_Port: 10190 Exec_Master_Log_Pos: 0

Connect_Retry: 1 Relay_Log_Space: 102 Master_Log_File: Until_Condition: None

Read_Master_Log_Pos: 4 Until_Log_File: Relay_Log_File: slave-relay-bin.000001 Until_Log_Pos: 0 Relay_Log_Pos: 4 Master_SSL_Allowed: No

Relay_Master_Log_File: Master_SSL_CA_File: Slave_IO_Running: No Master_SSL_CA_Path: Slave_SQL_Running: No Master_SSL_Cert: Replicate_Do_DB: Master_SSL_Cipher:

Replicate_Ignore_DB: Master_SSL_Key: Replicate_Do_Table: Seconds_Behind_Master: NULL

Replicate_Ignore_Table: Last_IO_Errno: 0 Replicate_Wild_Do_Table: Last_IO_Error:

Replicate_Wild_Ignore_Table: Last_SQL_Errno: 0 Last_SQL_Error:

1 row in set (0.00 sec)

START SLAVE and STOP SLAVE Used on slave

Used to start or stop the slave threads Defaults to

affecting both I/O and SQL thread ... but individual

threads can be started or stopped START SLAVE

SQL_THREAD

START SLAVE IO_THREAD

RESET SLAVE Used on slave

Removes all info on replication position

Deletes master.info, relay-log.info and all relay logs

Relay logs are unconditionally removed!

... even if they have not been fully applied

SET GLOBAL SQL_SLAVE_SKIP_COUNTER Used on slave

Global server variable

Requires SUPER privileges

Slave SQL thread shall not be running

Slave will skip events when starting

Useful when recovering from slave stops

Might leave master and slave with different data in tables

... so be careful when you use it

Use Cases

Use Cases, Part 1 - Basic Replication

Intensive Reads High Availability

Master

Master/ Master/ Slave Slave

Slave

Slave Slave

“Specialist" slaves - backups and reporting

Master

Slave Slave

reports Slave

Slave Slave

backups

“Specialist" slaves - per-application

friends: 10 GB Master

messages: 30 GB

Slave Slave Slave Slave

“message board” queries “friends list” queries

“Specialist" slaves - Blackhole Engine

Master

Slave Slave

Slave Slave “message board” queries

(friends table in black “friends list” queries hole) (message table in black hole)

Things to think about in basic replication

Initial snapshot of slaves

load balancing of clients

Failover of clients to new master

HA + Scale out?

Master/ Master/ Slave Slave

Slave

Slave Slave

Any better?

Master/ Master/ Slave Slave

Proxy Master

Slave Slave

Slave

Use Cases, Part 3 - Multiple Data Centers secure

San Jose New York tunnel

rep Active

Master Master

wr wr

wr wr

Slave Slave

rd rd app app

Slave Slave

( Jeremy Cole - MySQL Users Conf 2006 )

After Failover secure

San Jose New York tunnel

rep Active Master

Master wr wr

wr wr

Slave Slave

rd rd app app

Slave Slave

( Jeremy Cole - MySQL Users Conf 2006 )

Internal ThreadsSince MySQL 4.0, replication slaves run two threadsIO thread continuously receives updates from master

and writes to local relay logSQL thread continuously executes statements in relay

log

IO thread isolationIsolating IO thread means that slave won’t have to

wait for long-executing statements to finish executing before retrieving data from master

Also, slave will continue reading data from master if a statement creates a data conflict

SQL thread isolationSQL thread isolation allows for replication in an

environment without a continuous link between slave and masters

If master fails (or slave simply has no access), the IO thread will try to reconnect endlessly (waiting 60 seconds between attempts)

SQL thread will continue processing relay logs even while IO thread is unable to connect to master

Master ThreadAdditionally, the master server runs the Binlog Dump

threadThis thread is simply dedicated to scanning the binary

logs on the master and sending updates to the connected slave

If this thread isn’t running, it means that replication isn’t running – more accurately, that no slaves are currently connected

Status files2 status files for replication’s useTheir use is to record the state of replication between

server shutdown and startupmaster.info records information about the slave’s

master serverrelay-log.info records information about the local

relay logs

Information in master.infoMaster log fileRead master log posMaster HostMaster UserPassword (will not be shown in SHOW SLAVE

STATUS)Master PortConnect RetryIn MySQL 4.1+, SSL options are stored if SSL is

used

Information in relay-log.infoRelay log fileRelay log posRelay master-log posExec master-log pos

Backup master Master backups can be accomplished with

mysqldump Care must be taken to ensure the following 2

special considerations:1. Consistent snapshot of master date (via lock tables

for MyISAM or single transaction for InnoDB)2. Recording of binary log information, for use on slaves

(master-data)

Backup master filesIf a file-system level backup is required, care should be

taken to manually record binary log name and position via SHOW MASTER STATUS statement.

To ensure consistency between backup and binary log position, the tables should be locked via FLUSH TABLES WITH READ LOCK immediately before backup (and SHOW MASTER STATUS)

LEAVE THE CLIENT CONNECTED!!!After backup finishes, execute UNLOCK TABLES to release

the read lock

Backup slaveSame idea as master file system backupInstead of recording position, it’s enough to backup

the master.info and relay-log.info filesInstead of acquiring global read lock, it’s enough to

STOP SLAVE before backup and START SLAVE once backup finishes

DO Make sure server-id is set on all machines Enable log-bin on your master If a slave will also be a master, set log-

slaveupdates Take backups on a slave Architect your replication setup for growth!

DO NOTDon’t use master-* lines in my.cnf; use CHANGEMASTER instead Don’t take your backups on the master

Thank You!

For more information:

www.gnugroup.org