mysql high availability

MYSQL HA SOLUTIONSadzmely mansor

[email protected]

MYSQL HA

• Possible HA solutions with MySQL

• replications

• clusters

• Combinations :: Third party solutions

• LVS/Heartbeat

• DRBD

MYSQL REPLICATION

MYSQL :: REPLICATION

• Natively support ONE WAY - ASYNCHRONOUS replication

• One Master + N number of Slaves

• Asynchronous ::

• does not take place in real time

• no guarantee data from master replicated to slaves


• Semi-Synchronous Replication :: MySQL 5.5+

• Master waits until at least on semi-synchronous slave acknowledges a commit transaction

• slave acknowledge only after event has been written to relay log and flushed to disk

• if timeout occurs without any semi-synchronous slave acknowledgment - master reverts to asynchronous


• Semi-Synchronous Replication :: MySQL 5.5+

• when one semi-synchronous slave catch-up - master returns to semi-synchronous

• semi-synchronous must be enabled on both master and slave sides

• missing / not enabled on either master or at least one slave - master uses asynchronous replication


• With Asynchronous Replication

• master writes event to its binary log

• slave request them when they are ready

• no guarantee any event will ever reach any slave


• With Fully Synchronous Replication

• master commits a transaction

• all slaves also will have to committed the transaction before the master return to the session that perform the transaction

• might be a lot of delay to complete a transaction


• With Semi-Synchronous Replication - is in between asynchronous and fully synchronous

• master commits a transaction

• wait for at least one slave acknowledge the transaction

• no need to wait for all slaves acknowledgment


• Asynchronous

• High Speed

• Less Data Integrity


• Fully Synchronous

• Less Speed

• Higher Data Integrity


• Semi - Synchronous

• Speed

• Data Integrity


• Advantages

• operate across different platforms

• in asynchronous mode

• you can start or stop at any time

• suitable over slower links or partial links

• across geographical boundaries - DRC, etc


• Advantages

• one master - many slaves

• suitable for read intensive application such as Web Services by spreading load across multiple servers


• Disadvantages

• data can only be written to master

• no guarantee master and slaves is consistent at a given point of time

• asynchronous - small delay

• application must be replication - aware (write only to master, and read from slaves)


• Recommended Uses

• Scale-out (horizontal scaling) solutions that require large number of reads but fewer writes

• Logging/data analysis of live data

• by replicating to slave, will not disturb/degrade/affecting master operation


• Recommended Uses

• Online backup (availability)

• Offline backup

• after a while having a reliable snapshots, take the slave down

• run again to catch up


• How It Works

• master writes updates to its binary log files

• serve as record of updates to be sent to slave

• slave connects to master

• determines last position of last successful update

• receives new updates taken since last update

MYSQL - REPLICATIONLAB EXERCISE


• # mkdir /etc/mysql

• /etc/mysql/master.cnf

• # mysql_install_db --defaults-file=/etc/mysql/master.cnf

• # mysqld --defaults-file=/etc/mysql/master.cnf &

[mysqld]server-id=1log-bin=black-bin.logdatadir=/home/mysql/master/datasync_binlog=1user=mysql


• # mysql -u root

mysql> CREATE USER [email protected]> GRANT REPLICATION SLAVE ON -> *.* TO [email protected] -> IDENTIFIED BY ‘password’;


• # mkdir /etc/mysql

• /etc/mysql/slave.cnf

• # mysqld --defaults-file=/etc/mysql/slave.cnf &

[mysqld]server-id=2master-host=192.168.0.31master-user=replmaster-password=passwordrelay-log-index=slave-relay-bin.indexrelay-log=slave-relay-bindatadir=/home/mysql/slave/data


• # mysql -u root

• replication error at this point, why?

• binlog-ignore-db=dbname

• exclude database from replication

mysql> start slave;mysql> show slave status\G


• mysql> show slave status\G Slave_IO_Running: YesSlave_SQL_Running: No...... Last_Error: Error ‘Duplicate entry’...


• purging slave relay logmysql> stop slave;mysql> reset slave;mysql> show slave status\G

...Read_Master_Log_Pos: 4...


• purging master binary log

• look at master position value

mysql> reset master;

mysql> show master status;


• restart slave - reconnect to mastermysql> start slave;mysql> show slave status\G

...Read_Master_Log_Pos: 98... Slave_IO_Running: Yes Slave_SQL_Running: Yes...


• testing the replication - at master

# mysql -u root

mysql> create database dummy;mysql> use dummy;mysql> create table profile (id int(3), name varchar(30));mysql> insert into profile values (1,‘Abdullah’);mysql> insert into profile values (2,‘John Doe’);mysql> select * from profile;


• testing the replication - at slave

# mysql -u root

mysql> show databases;mysql> use dummy;mysql> show tables;mysql> select * from profile;


• testing the replication - at slave# rm -rf /home/mysql/slave/data/dummy

# mysql -u root

mysql> flush tables;mysql> show databases;mysql> show slave status\Gmysql> stop slave;mysql> change master to master_log_pos=98;mysql> start slave;mysql> show slave status\Gmysql> show databases;mysql> use dummy;mysql> select * from profile;


• at master - mysqlbinlog command can be used to view master binary log and position related

• purging binary log

• oldest kept log

# mysqlbinlog black-bin.000001 | more

# mysql -pmysql> purge master logs to “black-bin.000201”;


• Exercise

• add another slave to master

• set server-id=3

• make sure at master grant permission to same replication user but with different IP address


• How to recover in this kind of situation?

• master crashed

• you are running a single slave

• slave used as read and write during master recovery

• master server now ready - how to sync?


• Conclusion from this exercise

• learn how to create a simple master-slave replication

• understand how mysql replication work internally via binary log

• how to recover when master or slave fails


Master => Slave


Master => Slaves


Master => Slave => Slaves


Master => Slave (Multi-Source)


Master => Master


Circular (Multi-Master)

• In Multi-Master Environment

• Never Load Balance Multi Master

• Only Write to Single Master, in case of failure failover to another master. Never write to multiple master

• possibilities out of sync

• no “conflict resolution” in MySQL replication



• Configuring a Slave as “Master” ready

• make sure log-bin configured in configuration file

• ready any time for slave connection

• create user and grant acess

[mysqld]...log-bin=black-bin.logsync_binlog=1...

AUTOMATIC FAILOVERWITH MYSQL REPLICATION

AUTOMATIC FAILOVERINTRODUCTION TO LINUX VIRTUAL SERVER (LVS)

• project started 1998

• mission to provide high scaleability, reliability and serviceability

• advanced IP load balancing software

• included in standard kernel since 2.4

• used to build high available and scaleable network services

• web, email, media services, VOIP, etc

AUTOMATIC FAILOVERINTRODUCTION TO LINUX VIRTUAL SERVER (LVS)

• 3 types of LVS load balancing

• Network Address Translation (NAT)

• IP Tunneling

• redirecting to a different IP address

• Direct Routing

AUTOMATIC FAILOVERINTRODUCTION TO LINUX HEARTBEAT

• Using Linux HA project also known as Linux Heartbeat

• implements a heartbeat protocol

• message sent at a regular interval to one or more nodes

• not received - node assumed failed

• trigger a fail over action

AUTOMATIC FAILOVERINTRODUCTION TO LINUX HEARTBEAT

• Heartbeat Method/Medium

• serial interface (/dev/ttyS[0-9])

• broadcast

• multicast - group (224.0.0.0 - 239.0.0.0)

• unicast - udp

AUTOMATIC FAILOVERUSING LINUX LVS/HEARTBEAT + MYSQL

Linux LVS / Heartbeat

virtual IP 192.168.0.1

Replication

Master Slave

192.168.0.31 192.168.0.34

Linux LVS

AUTOMATIC FAILOVERUSING LINUX LVS/HEARTBEAT + MYSQL



Replication

Master Slave

192.168.0.31 192.168.0.34

Linux LVS + Heartbeat (HA)

AUTOMATIC FAILOVERUSING LINUX LVS/HEARTBEAT + WEB + MYSQL

DB/MySQL


Replication

Master Slave

Linux LVS + Heartbeat (HA)

HTTP/WEB


DB Server :192.168.0.1

AUTOMATIC FAILOVERLAB EXERCISE

USING LVS DIRECT ROUTING

AUTOMATIC FAILOVERUSING LINUX LVS/HEARTBEAT - DIRECT ROUTING

LVS Server IP192.168.0.101

LVS Server IP192.168.0.102

Heart Beat IP172.16.0.101

Heart Beat IP172.16.0.102

DB Virtual IP192.168.0.1

Replication

Master Slave

Master DB IP192.168.0.11

NO ARP lo:0 IP192.168.0.1

Slave DB IP192.168.0.12

NO ARP lo:0 IP192.168.0.1

DB Virtual IP192.168.0.1

Active LVS Passive LVS

DB Connectionrequest viaVirtual IP

192.168.0.1

DB Server Reply Directly to Connection/Requestor

by passing LVS

AUTOMATIC FAILOVERCONFIGURING HEARTBEAT

• yum install heartbeat

• /etc/ha.d/

• resource.d/

• ha.cf

• authkeys

• haresources


• Heartbeat medium : : broadcast

• /etc/ha.d/ha.cf

# udpport port# bcast devudpport 694bcast eth0


• Heartbeat medium : : multicast

• /etc/ha.d/ha.cf# mcast [dev] [mcast group] [port] [ttl] [loop]# # [dev] device to send/recv heartbeat on# [mcast group] multicast group to join - Class D# [port] udp port to sendto/recvfrom# [ttl] ttl value for outbound heartbeat# how far multicast packet to propogate# [loop] toggle loopback for outbound multicast# heartbeats - loopback to interface it # was sent on

mcast eth0 225.0.0.1 694 1 0


• Heartbeat medium : : unicast

• /etc/ha.d/ha.cf# ucast [dev] [peer-ip-addr]# # [dev] device to send/recv heartbeat on# [peer-ip-addr] IP address of peer to send packets to

udpport 694ucast eth0 172.16.0.102


• /etc/ha.d/ha.cf :: main heartbeat configuration file

debugfile /var/log/ha-debuglogfile /var/log/ha-loglogfacility local0keepalive 2deadtime 5# udpport 694# bcast eth0mcast eth0 225.0.0.1 694 1 0# ucast eth0 172.16.0.102auto_failback onnode primemaster.xjutsu.comnode secondmaster.xjutsu.com


• /etc/ha.d/authkeys :: authentication method and password type/password between heartbeat nodes

• chmod 600 authkeys

auth 11 md5 secretpassword


• /etc/ha.d/haresources :: node resources monitored/loaded

# node-name resource1::options resourceN::options# # node-name: from uname -n# resource: managed - script refer to /etc/ha.d/resources.d# options: parameters/arguments passed to resource script

primemaster.xjutsu.com ldirectord::ldirectord.cf \ IPaddr2::192.168.0.1/24/eth0:0/192.168.0.255

AUTOMATIC FAILOVERCONFIGURING LDIRECTORD - LOAD BALANCER

• /etc/ha.d/ldirectord.cfchecktimeout=10checkinterval=20autoreload=yeslogfile=”/var/log/ldirectod.log”quiescent=yesvirtual=192.168.0.1:3306 fallback=192.168.0.12:3306 gate real=192.168.0.11:3306 gate service=mysql scheduler=wlc protocol=tcp checktype=connect## man ipvsadm for detail scheduler options

AUTOMATIC FAILOVERCONFIGURING LDIRECTORD - LOAD BALANCER

• The linux directors must be able to route traffics to real server

• by enabling IPV4 packet forwarding

• /etc/sysctl.conf

net.ipv4.ip_forward = 1

# sysctl -p

AUTOMATIC FAILOVERCONFIGURING REAL MYSQL SERVERS (MASTER/SLAVE)

• ip aliasing on localhost device

• interface lo:0 (/etc/sysconfig/netwrok-scripts/ifcfg-lo:0)

DEVICE=lo:0IPADDR=192.168.0.1NETMASK=255.255.255.255ONBOOT=yesNAME=loopback

AUTOMATIC FAILOVERCONFIGURING REAL MYSQL SERVERS (MASTER/SLAVE)

• disable ARP broadcast

• /etc/sysctl.confnet.ipv4.conf.all.arp_ignore = 1net.ipv4.conf.eth0.arp_ignore = 1net.ipv4.conf.all.arp_announce = 2net.ipv4.conf.eth0.arp_announce = 2

# sysctl -p# ifup lo:0

AUTOMATIC FAILOVERLDIRECTORD + HEARTBEAT

• starting the service

• /etc/init.d/heartbeat [start, stop, restart]

• ifconfig

• virtual interface eth0:0 automatically initialized

• ipvsadm -L

• list virtual server table

HIGH AVAILABILITYWITH MYSQL REPLICATION + DRBD


• Distributed Replicated Block Device (DRBD)

• mirroring a device via network

• network based raid 1

• integrated into linux kernel starting from 2.6.33


Dedicated Interface

Active Server Passive Server

MySQL Datadrbd device formated with cluster

filesystem (GFS2/OCFS2) making two way read/write

possible with DRBD

HIGH AVAILABILITYWITH MYSQL REPLICATION + DRBD + LVS/HEARTBEAT

Dedicated Interface

Active Server Passive Server

MySQL Datadrbd device formated with cluster

filesystem (GFS2/OCFS2)making two way read/write

possible with DRBD



MYSQL CLUSTER

MYSQL CLUSTERWHAT IS MYSQL CLUSTER?

• Relational Database Technology

• enables clustering of in-memory and disk-based tables

• with shared nothing technology

• protect against single point of failure

• if node fail, others nodes can be used to reconstruct datas

• shared disk is not required

MYSQL CLUSTERWHAT IS MYSQL CLUSTER?


• synchronous replication with two phase commit mechanism

• guarantee that data is written to multiple nodes upon committing the data

MYSQL CLUSTERCORE CONCEPTS


• every part of the cluster is a node

• three types of cluster nodes

• Management Node

• Data Node

• SQL/Application Node

MYSQL CLUSTERCORE CONCEPTS

MYSQL CLUSTERCORE CONCEPTS - MANAGEMENT NODE

• Manage the other nodes within the cluster

• providing configuration data

• starting / stoping nodes

• should be started first before other nodes

• started with the command ndb_mgmd

MYSQL CLUSTERCORE CONCEPTS - DATA NODE

• stores cluster data

• started with the command ndbd

• starting from version 7.0 ndbmtd can also be used for data node process

• multi threaded data node daemon

• at least 2 data nodes - 1 possible but no replica

MYSQL CLUSTERCORE CONCEPTS - SQL NODE

• node that access the cluster data

• a traditional mysql server that uses NDBCLUSTER storage engine

• mysql started with --ndbcluster and --ndb-connectstring options

• like a API node which accesses MySQL cluster data

MYSQL CLUSTERNODES, NODE GROUPS, REPLICAS + PARTITIONS

• Partition

• is a portion of data stored by the cluster

• each node is responsible for keeping at least one copy of any partitions assigned to it


• (Data) Node - an ndbd process

• stores a replica

• copy of the partition assigned to the node group

• which the node is member


• Node Group

• consist of one or more nodes

• stores partitions or sets of replicas

MYSQL CLUSTERHARDWARE, SOFTWARE + NETWORKING REQUIREMENTS

• can run on commodity hardware

• However data nodes required large amount of RAM

• all live data storage is done is memory

• can reduce RAM requirement by using Disk Data Tables

• only for non indexed columns of NDB tables

• multiple faster CPU can enhance performance

MYSQL CLUSTERHARDWARE, SOFTWARE + NETWORKING REQUIREMENTS

• communication between nodes via TCP/IP networking

• minimum expected for each host is a standard 100Mbps ethernet controller

• recommend that MySQL to be run on its own subnet

• not sharing with machines not forming part of the cluster

• software requirement is simple, what needed is production release of MySQL 5.1.51-ndb-7.0.21 or 5.1.51-ndb-7.1.10

MYSQL CLUSTERTYPICAL USE CASES

• subscriber databases

• DNS/DHCP for broadband

• Telecoms Application Servers

• Service Delivery Platforms

• AAA Databases

• deploying FreeRadius with MySQL database

MYSQL CLUSTERLAB EXERCISE

MYSQL CLUSTERINSTALL IN ALL SERVERS

• locate the tar ball downloaded

• extract using tar command

• tar xvf mysql-cluster-gpl-7.1.9-linux-x86_64-glibc23.tar.gz

• mv mysql-cluster-gpl-7.1.9-linux-x86_64-glibc23 /usr/local

• ln -s mysql-cluster-gpl-7.1.9-linux-x86_64-glibc23 mysqlc

• short name

MYSQL CLUSTERCONFIGURE

• create data and configuration folder in mysql cluster manager server

• mkdir /etc/mysqlc

• create and set ownership for data folder in all servers

• /home/mysqlc

• chown mysql.mysql /home/mysqlc


• example setup with 5 nodesmanagement node - 192.168.1.5data node = 192.168.1.101data node = 192.168.1.102api node = 192.168.1.5api node = 192.168.1.5


• configure the node cluster manager - /etc/mysqlc/config.ini[ndb_mgmd]NodeId=1Hostname=192.168.1.5

[ndbd default]NoOfReplicas=2Datadir=/home/mysqlc

[ndbd]NodeId=3Hostname=192.168.1.101[ndbd]NodeId=4Hostname=192.168.1.102

[mysqld]NodeId=5Hostname=192.168.1.5[mysqld]NodeId=6Hostname=192.168.1.5


• start/reload the node cluster manager

• start the data nodes (in all configured nodes)

• display cluster status from management node

# ndb_mgmd -f /etc/mysqlc/config.ini --initial --configdir=/etc/mysqlc

# ndb_mgmd -f /etc/mysqlc/config.ini --reload

# ndbd -c 192.168.1.5:1186

# ndb_mgm -e show

MYSQL CLUSTERCONFIGURE SQL NODE

• /etc/mysqlc/my.cnf

• create mysql default database

• start SQL/API node

[mysqld]basedir=/usr/local/mysqlcdatadir=/home/mysqlcndbclusterndb-connectstring=”192.168.1.5:1186”socket=/home/mysqlc/mysql.socklog-error=error.logmemlock

# cd /usr/local/mysqlc# scripts/mysql_install_db --no-defaults --datadir=/home/mysqlc

# /usr/local/mysqlc/bin/mysqld --defaults-file=/etc/mysqlc/my.cnf &

MYSQL CLUSTERTESTING SQL/API NODE

• in API server start mysql CLI# /usr/local/mysqlc/bin/mysql -h 127.0.0.1# /usr/local/mysqlc/bin/mysql -S /home/mysqlc/mysql.sock

mysql> create database clusterdb1;mysql> use clusterdb1;mysql> create table dummy1 (id int(10) ket, name varchar(100)) engine=ndbcluster;mysql> insert into dummy1 values (1, ‘John Doe’);

# in mgm server - ndb_desc to view partition informationndb_desc -c 192.168.1.5 -d clusterdb1 dummy1 -p

mysql high availability

Documents