synchronise your data between mysql and mongodb

©Continuent 2013

Synchronize your data between MySQL

and MongoDB using Tungsten Replicator

Giuseppe Maxia, Director of QAContinuent, Inc

1Tuesday, October 15, 13

©Continuent 2013

About me

2

• Giuseppe Maxia, a.k.a. "The Data Charmer"

• Director of Quality Assurance, Continuent, Inc

• 25+ years development and DB experience

• Long timer MySQL community member.

• Oracle ACE Director

• Blog: http://datacharmer.blogspot.com

• Twitter: @datacharmer


http://datacharmer.blogspot.com


©Continuent 2013

Introducing Continuent

3

• The leading provider of clustering and replication for open source DBMS

• Our Product: Continuent Tungsten

• Clustering - Commercial-grade HA, performance scaling and data management for MySQL

• Replication - Flexible, high-performance data movement


©Continuent 2013

A Review of Tungsten Replicator

4


©Continuent 2013

Tungsten Replicator Overview

5

Master

(Transactions + Metadata)

Slave

THL

DBMSLogs

Replicator

(Transactions + Metadata)

THLReplicator

Download transactions via network

Apply using JDBC


©Continuent 2013

Master Replication Service

6

Extract Filter Apply

StageExtract Filter Apply

Stage

Pipeline

MySQLMaster

TransactionHistory Log

In-MemoryQueue

Slave ReplicatorsBinlog

tcp/ip


©Continuent 2013

Slave Replication Service

7

Extract Filter Apply



Stage

Pipeline

TransactionHistory Log

In-MemoryQueue

SlaveDBMS

MasterReplicator

tcp/

ip


©Continuent 2013

star

master-slave Heterogeneous

fan-in slave all-masters

MySQL

Oracle

Oracle

MySQL Oracle

Oracle

MySQL MySQL


©Continuent 2013

MongoDB in a nutshell

9


©Continuent 2013

What is MongoDB

10

• A non-relational database

• A document-oriented database

• Schema-free

• Open source

• High performance

• Scalable

• Developer-friendly (sort of)


©Continuent 2013

What is MongoDB good for?

• Storing large amount of unrelated data

• Data that can't be constrained in a schema

• Complement to relational data

• IT IS NOT a drop-in replacement for a relational database

11


©Continuent 2013

Relational vs. document

12

personpersonperson

p_id name age

1 Joe 30

2 Fred 23

3 Jack 26

4 Sue 25

5 Pete 32

departmentdepartmentd_id name

1 sales2 dev3 supportpers_deptpers_dept

p_id d_id

1 2

2 2

3 1

4 3

5 1Relational


©Continuent 2013


13

personpersonperson

p_id name age

1 Joe 30

2 Fred 23

3 Jack 26

4 Sue 25

5 Pete 32



p_id d_id

1 2

2 2

3 1

4 3

5 1Relational


©Continuent 2013


14

personpersonperson

p_id name age

1 Joe 30

2 Fred 23

3 Jack 26

4 Sue 25

5 Pete 32



p_id d_id

1 2

2 2

3 1

4 3

5 1Relational


©Continuent 2013


15

personpersonperson

p_id name age

1 Joe 30

2 Fred 23

3 Jack 26

4 Sue 25

5 Pete 32



p_id d_id

1 2

2 2

3 1

4 3

5 1Relational


©Continuent 2013


16

personpersonperson

p_id name age

1 Joe 30

2 Fred 23

3 Jack 26

4 Sue 25

5 Pete 32



p_id d_id

1 2

2 2

3 1

4 3

5 1Relational


©Continuent 2013


17

personpersonperson

p_id name age

1 Joe 30

2 Fred 23

3 Jack 26

4 Sue 25

5 Pete 32



p_id d_id

1 2

2 2

3 1

4 3

5 1Relational


©Continuent 2013


18

personpersonperson

_id name age department

1 Joe 30 dev

2 Fred 23 dev

3 Jack 26 sales

4 Sue 25 support

5 Pete 32 sales

Document


©Continuent 2013

How MongoDB keeps data

• three levels:

• dbs

• collections

• documents

19


©Continuent 2013

MongoDB insertion demo

> show collections>

> db.person.insert( {_id: 1, name: "Joe", age: 30, department: "dev"})> show collectionspersonsystem.indexes

20


©Continuent 2013

MongoDB insertion demo

> db.person.insert( {_id: 2, name: "Fred", age: 23, department: "dev"})> db.person.insert( {_id: 3, name: "Jack", age: 26, department: "sales"})> db.person.insert( {_id: 4, name: "Sue", age: 25, department: "support"})> db.person.insert( {_id: 5, name: "Pete", age: 30, department: "sales"})> db.person.find(){ "_id" : 1, "name" : "Joe", "age" : 30, "department" : "dev" }{ "_id" : 2, "name" : "Fred", "age" : 23, "department" : "dev" }{ "_id" : 3, "name" : "Jack", "age" : 26, "department" : "sales" }{ "_id" : 4, "name" : "Sue", "age" : 25, "department" : "support" }{ "_id" : 5, "name" : "Pete", "age" : 30, "department" : "sales" }

21


©Continuent 2013

MySQL to MongoDB basics

22


©Continuent 2013

Replication from MySQL to MongoDB

23

• Requires ROW-based-replication

• Replication happens by table

• There is no consolidation into "documents"

• DDL commands are ignored

• Statement commands are ignored

• Column names become document attributes

• enum and set columns are converted to strings


©Continuent 2013

First example of replication

# MySQL

create schema oneschema;use oneschema ;create table myfirst( num int not null primary key, dt datetime, ts timestamp, going enum('yes', 'no'));

# MongoDB> show dbslocal 0.078125GBtest 0.203125GBtungsten_mysql2mongodb 0.203125GB

# NOTICE: no "oneschema"

24


©Continuent 2013

Inserting data

# MySQLinsert into myfirst values (1, '2003-04-26 09:15:00', null, 'yes');Query OK, 1 row affected (0.01 sec)

select * from myfirst;+-----+---------------------+---------------------+-------+| num | dt | ts | going |+-----+---------------------+---------------------+-------+| 1 | 2003-04-26 09:15:00 | 2013-10-14 19:39:38 | yes |+-----+---------------------+---------------------+-------+1 row in set (0.00 sec)

25


©Continuent 2013

Checking results in MongoDB

# MongoDB> show dbslocal 0.078125GBoneschema 0.203125GBtest 0.203125GBtungsten_mysql2mongodb 0.203125GB

> use oneschemaswitched to db oneschema> show collectionsmyfirstsystem.indexes

> db.myfirst.find(){ "_id" : ObjectId("525c2c5af5d9ca820fcee01d"), "num" : "1", "dt" : "2003-04-26 11:15:00.0", "ts" : "2013-10-14 19:39:38.0", "going" : "yes" }

26


©Continuent 2013

Another interesting insertion

#MySQLcreate table t1(_id int not null primary key, c char(10));insert into t1 values (1, 'abc');select * from t1;+-----+------+| _id | c |+-----+------+| 1 | abc |+-----+------+1 row in set (0.00 sec)

# MongoDB> db.t1.find(){ "_id" : "1", "c" : "abc" }

27


©Continuent 2013

More insertions

# MySQLinsert into t1 values (2,'def'), (3,'ghi'), (4,'jkl'), (5, 'mno');Query OK, 4 rows affected (0.01 sec)Records: 4 Duplicates: 0 Warnings: 0

select * from t1;+-----+------+| _id | c |+-----+------+| 1 | abc || 2 | def || 3 | ghi || 4 | jkl || 5 | mno |+-----+------+5 rows in set (0.00 sec)

28


©Continuent 2013

More insertions

# MongoDB

> db.t1.find(){ "_id" : "1", "c" : "abc" }{ "_id" : "2", "c" : "def" }{ "_id" : "3", "c" : "ghi" }{ "_id" : "4", "c" : "jkl" }{ "_id" : "5", "c" : "mno" }

29


©Continuent 2013

Update and delete as seen on master

update t1 set c = 'ZZZ' where _id = 3;Query OK, 1 row affected (0.00 sec)Rows matched: 1 Changed: 1 Warnings: 0

delete from t1 where _id=2;Query OK, 1 row affected (0.00 sec)

select * from t1;+-----+------+| _id | c |+-----+------+| 1 | abc || 3 | ZZZ || 4 | jkl || 5 | mno |+-----+------+4 rows in set (0.00 sec)

30


©Continuent 2013

e!ects of update and delete on the slave

# MongoDB

> db.t1.find(){ "_id" : "1", "c" : "abc" }{ "_id" : "3", "c" : "ZZZ" }{ "_id" : "4", "c" : "jkl" }{ "_id" : "5", "c" : "mno" }

31


©Continuent 2013

Overview of Tungsten installer

32


©Continuent 2013

Overview of Installation Process

1. Set up hosts

2. Prepare MySQL replicas

3. Download software

4. Install using tpm

33

Amazon Setup:https://docs.continuent.com/wiki/display/TEDOC/

Preparing+EC2+Servers


https://docs.continuent.com/wiki/display/TEDOC/Preparing+EC2+Servers




©Continuent 2013

How tungsten-installer Works for Basic Master/Slave Deployment

34

db1

db2

db3

Staging copy of files

check prereqscopy codeconfigure


©Continuent 2013

Tungsten master/slave replication

35

host1 host2

host3

alpha alpha

alpha

THL THL

THL

installer


©Continuent 2013

Bi-directional replication

36

host1 host2

alphaalpha

bravobravo

Install all master and slave services on all hosts at once

installer


©Continuent 2013

4 nodes all-masters

37

host1

host4

host2

host3

alphaalpha

bravobravo

charlie

charlie

delta

delta

bravobravo

alpha

delta

delta

charlie

charlie

alpha


©Continuent 2013

Tungsten security layer

• Tra!c encryption:

• all data in transit (transaction history logs, or THL) is encrypted using SSL

• all administrative tra!c is encrypted with SSL

• Transparent to the user

• Independent of the database server (works also for heterogeneous replication)

38


©Continuent 2013

Tungsten replicator without security

39

replicatorservices

host1

host4

host2

host3

alpha

alpha

alpha alpha

master

slave

THL

THL

THLTHL

plain text


©Continuent 2013

Tungsten Replicator with security

40

replicatorservices

host1

host4

host2

host3

alpha

alpha

alpha alpha

master

slave

THL

THL

THLTHL

SSL

SSL SSL

SSL


©Continuent 2013

master/slave using tpm

42

./tools/tpm install alpha \ --topology=master-slave \ --home-directory=/opt/continuent/replicator \ --replication-user=tungsten \ --replication-password=secret \ --master=host1 \ --slaves=host2,host3,host4 \ --start


©Continuent 2013

Installing Master/Slave Replication with MongoDB

43

host1

host4

host2

host3

alpha

alpha

alpha

alpha


©Continuent 2013

master/slave with MongoDB

44

./tools/tpm configure mysql2mongodb \ --enable-heterogenous-service=true \ --topology=master-slave \ --master=host1 \ --replication-user=tungsten \ --replication-password=secret \ --slaves=host2,host3,host4 \ --home-directory=$MYSQL_DEPLOY \ --start-and-report

./tools/tpm configure mysql2mongodb \ --hosts=host4 \ --datasource-type=mongodb \ --replication-port=$MONGODB_PORT

./tools/tpm install


©Continuent 2013

Installing Fan-In Replication

45

host1

host4

host2

host3

alphabravo

charliebravo

charlie

alpha


©Continuent 2013

fan-in using tpm

46

./tools/tpm install many_towns \ --replication-user=tungsten \ --replication-password=secret \ --home-directory=/opt/continuent/replication \ --masters=host1,host2,host3 \ --slaves=host4 \ --master-services=alpha,bravo,charlie \ --topology=fan-in \ --start


©Continuent 2013

Installing Fan-In Replication with MongoDB

47

host1

host4

host2

host3

alpha

bravo

charlie

bravo

charlie

alpha


©Continuent 2013

fan-in with MongoDB

48

./tools/tpm configure mysql2mongodb \ --enable-heterogenous-service=true \ --topology=fan-in \ --masters=host1,host2,host3 \ --master-services=alpha,bravo,charlie \ --slaves=host4 \ --replication-user=tungsten \ --replication-password=secret \ --home-directory=$MYSQL_DEPLOY \ --datasource-type=mysql \ --start-and-report


./tools/tpm install


©Continuent 2013

Install Multi-Master replication

49

host1 host2

host3

alphaalpha

bravobravo

charlie

charlie

bravo

alpha

charlie


©Continuent 2013

multi-master using tpm

50

../tools/tpm install musketeers \ --reset \ --topology=all-masters \ --home-directory=/opt/continuent/replicator \ --replication-user=tungsten \ --replication-password=secret \ --masters=host1,host2,host3 \ --master-services=alpha,bravo,charlie \ --start


©Continuent 2013

Install Multi-Master replication with Mongodb

51

host1

host4

host2

host3

alphaalpha

bravobravo

charlie

charlie

bravobravo

alpha

charlie

charlie

alpha


©Continuent 2013

multi-master with MongoDB

52

./tools/tpm configure mysql2mongodb \ --enable-heterogenous-service=true \ --topology=all-masters \ --masters=host1,host2,host3 \ --slaves=host1,host2,host3,host4 \ --master-services=alpha,bravo,charlie \ --replication-user=tungsten \ --replication-password=secret \ --home-directory=$MYSQL_DEPLOY \ --datasource-type=mysql \ --start-and-report


./tools/tpm install


©Continuent 2013

MongoDB or TokuMX

53

• TokuMX is a drop-in replacement for MongoDB

• Open source project, developed by TokuTek

• https://github.com/Tokutek/mongo

• it includes

• better indexing

• row-level locking (MongoDB locks at db level)

• transactions

• better compression


https://github.com/Tokutek/mongo

https://github.com/Tokutek/mongo

©Continuent 2013

Tungsten Replicator is Open Source

56

• Project home:

http://code.google.com/p/tungsten-replicator/

• Log bugs, "nd builds, post in replicator discussion group

• Documentation:

https://docs.continuent.com/wiki/display/TEDOC/Tungsten+Documentation+Home

https://docs.continuent.com/wiki/display/TEDOC/Deploying+MongoDB+Replication












©Continuent 2012.

Continuent Web Page:http://www.continuent.com

Tungsten Replicator 2.1:http://code.google.com/p/tungsten-replicator

Our Blogs:http://scale-out-blog.blogspot.comhttp://datacharmer.blogspot.comhttp://www.continuent.com/news/blogs

560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009e-mail: [email protected]


http://www.continuent.com

http://www.continuent.com

http://code.google.com/p/tungsten-replicator

http://code.google.com/p/tungsten-replicator

http://scale-out-blog.blogspot.com

http://scale-out-blog.blogspot.com



http://continuent-tungsten.blogspot.com

http://continuent-tungsten.blogspot.com

mailto:[email protected]

mailto:[email protected]

synchronise your data between mysql and mongodb

Technology

toolstpm configure

null primary

document person

transactions

installing

1 row

servicesalpha

installing