jailbreaking mysql replication

Jailbreaking MySQL replication

Giuseppe MaxiaQA DirectorContinuent, Inc@datacharmer

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Saturday, October 1, 11

about me -Giuseppe Maxia§ a.k.a. The Data Charmer§ QA Director, Continuent, Inc§ Long time hacking with database features§ Formerly, database consultant, designer, coder.§ A passion for QA§ An even greater passion for open source§ ... and community§ Passionate blogger§ http://datacharmer.blogspot.com

Saturday, October 1, 11

http://datacharmer.blogspot.com

























In jail?


BINARY LOG

REPLICATION

MySQLDBMS

trans

actio

n

trans

actio

n

trans

actio

n

trans

actio

n

trans

actio

ntra

nsac

tion

trans

actio

ntra

nsac

tion

trans

actio

ntra

nsac

tion

trans

actio

n

MySQLDBMS

transactiontransaction






MySQL replicationis single threaded


MySQLDBMS

MySQLDBMS

MySQLDBMS

MySQLDBMS

master

slave slaveslave

MySQLDBMS

MySQLDBMS

MySQLDBMS

MySQLDBMS

master master master

slave

single source multi source (fan-in)

multiple sources?Saturday, October 1, 11

MySQLDBMS

MySQLDBMS

MySQLDBMS

MySQLDBMS

mastermaster

master master

from this from this

multiple masters?


MySQLDBMS

MySQLDBMS

MySQLDBMS

master

slaveslave

MySQLDBMS

MySQLDBMS

MySQLDBMS

master

slaveslave

MySQLDBMS

MySQLDBMS

MySQLDBMS

master

slaveslave

Seamless failover?


All these examples tell us:

Nice dream, but MySQL can’t do it


Enter Tungsten Replicator


Tungsten Replicator 2.0

• What is it?


http://code.google.com/p/tungsten-replicator



Open Source



100% GPL v2Open Source


What can it do?

• Easy failover

• Multiple masters

• Multiple sources to a single slave

• Parallel replication

• Replicate to Oracle and PostgreSQL database


© Continuent 2010

MySQL to foreign services

BinLogs

Master DB

(Binlogs enabled) Slave DB

MySQL setup to run as

MySQL master

Data is applied to PostgreSQL or

Oracle or to NoSQL entities


© Continuent 2010

MySQL to Oracle replication

/ Today at 2:45pm/ Moscone West 2016/ Full session about MySQL to Oracle replication/ with demos


Tungsten Replicator in a nutshell

binlog THL

slavemaster

host1 host2

THL

trep_commit_seqnotrep_commit_seqnotrep_commit_seqnoorigin seqno eventid

trep_commit_seqnotrep_commit_seqnotrep_commit_seqnoorigin seqno eventid

global transaction ID


Main components

• Transaction History Logs (THL)

• roughly corresponding to MySQL relay logs

• have a lot of metadata

• Service database

• contains metadata for latest transactions

• Metadata is committed together with data

• Makes slaves crash proof


A failover scenario1: MySQL native replication


© Continuent 2011

1. one Master, two slaves

/ Loading the “employees” test database


© Continuent 2011

2. Master goes away

/ Stop replication

/ Slaves are updated at different levels

# 2select count(*) from titles 333,145

# 3

select count(*) from titles 443,308


© Continuent 2011

3. Look into Slave #2 binary logs

/ find the last transaction


© Continuent 2011

4. Look into Slave #3 binary logs

/ find the transaction that was last in slave #2/ (This can actually take you a LOOOONG TIME)

/ Get the position immediately after this transaction / (e.g. 134000 in file mysql-bin.000018)


© Continuent 2011

5. promote Slave #3 to master

/ in slave #2

CHANGE MASTER TO master_host=‘slave_3_IP’,

master_user=‘slavename’,master_password=‘slavepassword’,master_log_file=‘mysql-bin.000018’,master_log_pos=134000;


A failover scenario1: Tungsten Replicator


© Continuent 2011

1. one master, two slaves

/ loading the ‘employees’ test database


© Continuent 2011

2. Master goes away

/ Stop replication

/ Slaves are updated at different levels

# 2select count(*) from titles 333,145

# 3

select count(*) from titles 443,308


© Continuent 2011

3. no need to find the last transaction

/ simply change roles

trepctl -host slave3 setrole -role mastertrepctl -host slave2 setrole -role slave \

-uri thl://slave3

trepctl -host slave3 online State: ONLINE

trepctl -host slave2 online State: GOING-ONLINE:SYNCHRONIZING


© Continuent 2011

4. Check that the slave has synchronized

# new master select seqno from tungsten.trep_commit_seqno;78

# new slave select seqno from tungsten.trep_commit_seqno;64


© Continuent 2011

4. Tell the replicator to hurry up

# new mastertrepctl -node slave3 flushMaster log is synchronized with database at log sequence number: 78

# new slavetrepctl host slave2 wait -applied 78ONLINE

select seqno from tungsten.trep_commit_seqno;78


© Continuent 2011

4. ... and we’re done

# new masterselect count(*) from employees.titlescount(*)443308

# new slave: count(*)443308


Failover

• DEMO


master/slave with an attitude


Parallel replication


parallel replication

From here ...


parallel replication

To here.


Parallel replication facts

• Sharded by database

• Good choice for slave lag problems

• Bad choice for single database projects


Testing parallel replication


sysbench

sysbench

sysbench

sysbench

sysbench

sysbench

sysbench

sysbench

sysbench

sysbench

db1

db2

db3

db4

db5

db6

db7

db8

db9

preparation (1)db0


before the test (2)

binary logs

MySQL slave

Tungsten slave

OFFLINE

STOPPED

replicator alpha

direct: alpha(slave)


starting the test

binary logs

MySQL slave

Tungsten slave

OFFLINE

STOPPED

replicator alpha


Concurrent sysbenchon 30 databasesrunning for 1 hour

Slaves will have 1 hour lag


measuring results

binary logs

MySQL slave

Tungsten slave

ONLINE

START

replicator alpha


Recording catch-up time


MySQL native replication

slave catch up in 00:59:30


Tungsten parallel replication

slave catch up in 00:25:40


Parallel replication made simpler

FROM HERE ....Saturday, October 1, 11


TO HERESaturday, October 1, 11

parallel replicationdirect slave facts



• No need to install Tungsten on the master




• Tungsten runs only on the slave





• Replication can revert to native slave with two commands (trepctl offline; start slave)






• Native replication can continue on other slaves






• Native replication can continue on other slaves

• Failover (either native or Tungsten) becomes a manual task


Multiple masters


multiple masters

• Tungsten Replicator recipe: use more services


Bi-directional replication


Bi-directional replication with slaves


Multiple sources


multiple sources

• Tungsten Replicator recipe is still valid: use more services


Multiple source replication


Multiple masters replication: 3 nodes


Multiple masters replication: 4 nodes


Tungsten in practiceInstallation


Installation

• Check the requirements

• Get the binaries

• Expand the tarball

• Run ./tools/tungsten-installer


REQUIREMENTS

• Java JRE or JDK (Sun/Oracle or Open-jdk)

• Ruby 1.8 (only during installation)

• ssh access to the same user in all nodes

• sudo access in all nodes (only for some backups)

• MySQL user with all privileges


Installation types

• multiple hosts

• master / slave

• slave - direct

• sandbox

• master/slave

• slave-direct


master/slave

binlog

THL

THL

slave

slave

master

host1host2

host3

THL


master/slave - sandbox

binlog

master

host1

THL

slave

THL

slave

THL


slave direct

binlogTHL

slavemaster

relay log

host1host2

host3

THL

slave

relay log


slave direct - sandbox

binlog THL

slavemaster

relay log

THL

slave

relay log

host1


sample installation

• 4 servers

• qa.r1.continuent.com r1




• MySQL 5.1

• tungsten-replicator-2.0.4


Installation (1)# starting at node 4, but any would do

for N in 1 2 3 4do ssh r$N mkdir tinstalldone

cd tinstalltar -xzf /path/to/tungsten-replicator-2.0.4.tar.gzcd tungsten-replicator-2.0.4


Installation (2)export TUNGSTEN_BASE=$HOME/tinstall

./tools/tungsten-installer \ --master-slave \ --master-host=r1 \ --datasource-user=tungsten \ --datasource-password=secret \ --service-name=dragon \ --home-directory=$TUNGSTEN_BASE \ --cluster-hosts=r1,r2,r3,r4 \ --start

1 2 3 4 5 6 7 8 9


Installation (2)export TUNGSTEN_BASE=$HOME/tinstall

./tools/tungsten-installer \ --master-slave \ # installation mode --master-host=r1 \ # who’s the master --datasource-user=tungsten \ # mysql username --datasource-password=secret \ # mysql password --service-name=dragon \ # name of the service --home-directory=$TUNGSTEN_BASE \ # where we install --cluster-hosts=r1,r2,r3,r4 \ # hosts in cluster --start # start replicator after installing


What does the installation do

host4

1: Validate all servers

✔✗✔

✔

host1

✔✗✔

✔

host2

✔✗✔

✔

host3

✔✗✔

✔

Report all errorsSaturday, October 1, 11


host4

1 (again): Validate all servers

✔

✔

✔

host1

✔

✔

✔

host2

✔

✔

✔

host3

✔

✔

✔

✔✔ ✔ ✔



2: install Tungsten in all servers

host3

$HOME/ tinstall/ config/ releases/ relay/ logs/ tungsten/

host4host1

host2


examplessh r2 chmod 444 $HOME/tinstall./tools/tungsten-installer \ --master-slave --master-host=r1 \ --datasource-user=tungsten \ --datasource-password=secret \ --service-name=dragon \ --home-directory=$HOME/tinstall \ --thl-directory=$HOME/tinstall/logs \ --relay-directory=$HOME/tinstall/relay \ --cluster-hosts=r1,r2,r3,r4 --start

ERROR >> qa.r2.continuent.com >> /home/tungsten/tinstall is not writeable


examplessh r2 chmod 755 $HOME/tinstall./tools/tungsten-installer \ --master-slave --master-host=r1 \ --datasource-user=tungsten \ --datasource-password=secret \ --service-name=dragon \ --home-directory=$HOME/tinstall \ --thl-directory=$HOME/tinstall/logs \ --relay-directory=$HOME/tinstall/relay \ --cluster-hosts=r1,r2,r3,r4 --start

# no errors


Installation - direct./tools/tungsten-installer \ --direct \ --master-host=r1 \ --slave-host=r4 \ --master-user=tungsten \ --slave-user=tungsten \ --master-password=secret \ --slave-password=secret \ --service-name=Castor \ --channels=1 \ --home-directory=$TUNGSTEN_BASE \ --start-and-report


MySQL Sandbox• Free software (GPL)• One (unix) host• Many database servers• Single or multiple sandboxes • Customized scripts to use the servers• Standard or circular replication• Installs IN SECONDS

http://mysqlsandbox.net




installation - direct - sandbox./tools/tungsten-installer \ --direct \ --master-host=127.0.0.1 \ --master-port=7101 \ --slave-host=127.0.0.1 \ --slave-port=7102 \ --master-user=msandbox --slave-user=msandbox \ --master-password=msandbox --slave-password=msandbox \ --master-log-directory=$SANDBOX/tr_dbs/node1/data \ --service-name=Castor \ --channels=5 \ --home-directory=$TUNGSTEN_BASE \ --start


installation - master/slave - sandbox

./tools/tungsten-installer \ --master-slave \ --cluster-hosts=127.0.0.1 \ --master-host=127.0.0.1 \ --datasource-port=7001 \ --datasource-user=msandbox \ --datasource-password=msandbox \ --home-directory=$TUNGSTEN_BASE/db1 \ --datasource-log-directory=$SBOX/tr/node1/data \ --service-name=tsandbox ...



$ ./tungsten-sandbox -hUSAGE: ./tungsten-sandbox [flags] args

flags:

-n,--nodes: how many nodes to install (default: 3)

-m,--mysql_version: which MySQL version to use (default: '5.1.56')

-t,--tungsten_base: where to install the sandbox (default: '$HOME/tsb2')

-d,--group_dir: sandbox group directory name (default: 'tr_dbs')

-s,--service: how the service is named (default: 'tsandbox')

-P,--base_port: port base for MySQL sandbox nodes (default: 710)

-l,--thl_port: port for the THL service (default: 1211)

-r,--rmi_port: port for the RMI service (default: 1010)

-v,--[no]version: show Tungsten sandbox version (default: false)

-h,--[no]help: show this help (default: false)



tungsten-sandbox.sh --nodes=3 --mysql_version=5.1.57

# Will install 3 nodes (1 master and two slaves)# with Tungsten replicator, using MySQL 5.1.57

# It will also create utility scripts


an installed tungsten sandboxinside $HOME/tsb2# MySQL sandbox commandssend_kill_all start_all status_all stop_all clear_all use_alln1 n2 n3 # invoke node 1, 2, and 3

# home of the replicators db1 db2 db3

# runs replicator [args] for all nodesreplicator_all # runs trepctl [args] for all nodestrepctl_all


inside a single sandbox nodeinside $HOME/tsb2/db1

replicator # shortcut to the replicatorthl # shortcut to the thl commandtrepctl # shortcut to the trepctl commandshow_log # shows the replicator log


Tungsten in practiceAdmin


Main components

• THL (Tungsten generated Transaction History Log)

• service database


Tools

• replicator

• trepctl

• thl


replicator

• It’s the service provider

• You launch it once when you start

• You may restart it when you change config


trepctl

• Tungsten Replicator ConTroLler

• It’s the driving seat for your replication

• You can start, update, and stop services

• You can get specific info


thl

• Transaction History List

• Gives you access to the Tungsten relay logs


http://bit.ly/tr20_cookbook




© Continuent 2011© Continuent 2011

Tungsten Replicator vs. Enterprise

Feature Replicator EnterpriseSimple admin with trepctl X XMySQL Row/Statement 5.0/5.1/5.5 X XMulti-master replication X XParallel apply (2.0) X XDistributed management with cctrl XAutomated failover and recovery XZero-downtime upgrade XSQL routing and load-balancing XFloating IP support X


Conclusion and Q&A


© Continuent 2011

Worldwide560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009e-mail: [email protected]

Contact Information

Continuent Web Site:http://www.continuent.com

Tungsten Projecthttp://tungsten-replicator.org

Getting started with Tungsten Replicator:http://datacharmer.blogspot.com


mailto:[email protected]

mailto:[email protected]

http://www.continuent.com


http://tungsten-replicator.org

http://tungsten-replicator.org



jailbreaking mysql replication

Technology

change master

database continuent

transaction continuent

mysql relay logs

file mysqlbin

mysql mastermaster dbdata

demos continuent

synchronizing continuent