synchronise your data between mysql and mongodb

©Continuent 2013

Synchronize your data between MySQL

and MongoDB using Tungsten Replicator

Giuseppe Maxia, Director of QAContinuent, Inc

1Tuesday, October 15, 13

©Continuent 2013

About me

• Giuseppe Maxia, a.k.a. "The Data Charmer"

• Director of Quality Assurance, Continuent, Inc

• 25+ years development and DB experience

• Long timer MySQL community member.

• Oracle ACE Director

• Blog: http://datacharmer.blogspot.com

• Twitter: @datacharmer

©Continuent 2013

Introducing Continuent

• The leading provider of clustering and replication for open source DBMS

• Our Product: Continuent Tungsten

• Clustering - Commercial-grade HA, performance scaling and data management for MySQL

• Replication - Flexible, high-performance data movement

©Continuent 2013

A Review of Tungsten Replicator

©Continuent 2013

Tungsten Replicator Overview

Master

(Transactions + Metadata)

DBMSLogs

Replicator

(Transactions + Metadata)

THLReplicator

Download transactions via network

Apply using JDBC

©Continuent 2013

Master Replication Service

Extract Filter Apply

StageExtract Filter Apply

Pipeline

MySQLMaster

TransactionHistory Log

In-MemoryQueue

Slave ReplicatorsBinlog

tcp/ip

©Continuent 2013

Slave Replication Service

Extract Filter Apply

StageExtract Filter Apply

Pipeline

TransactionHistory Log

In-MemoryQueue

SlaveDBMS

MasterReplicator

©Continuent 2013

master-slave Heterogeneous

fan-in slave all-masters

Oracle

MySQL Oracle

Oracle

MySQL MySQL

©Continuent 2013

MongoDB in a nutshell

©Continuent 2013

What is MongoDB

• A non-relational database

• A document-oriented database

• Schema-free

• Open source

• High performance

• Scalable

• Developer-friendly (sort of)

©Continuent 2013

What is MongoDB good for?

• Storing large amount of unrelated data

• Data that can't be constrained in a schema

• Complement to relational data

• IT IS NOT a drop-in replacement for a relational database

©Continuent 2013

Relational vs. document

personpersonperson

p_id name age

1 Joe 30

2 Fred 23

3 Jack 26

4 Sue 25

5 Pete 32

departmentdepartmentd_id name

1 sales2 dev3 supportpers_deptpers_dept

p_id d_id

5 1Relational

©Continuent 2013

personpersonperson

p_id name age

1 Joe 30

2 Fred 23

3 Jack 26

4 Sue 25

5 Pete 32

p_id d_id

5 1Relational

©Continuent 2013

personpersonperson

p_id name age

1 Joe 30

2 Fred 23

3 Jack 26

4 Sue 25

5 Pete 32

p_id d_id

5 1Relational

©Continuent 2013

personpersonperson

p_id name age

1 Joe 30

2 Fred 23

3 Jack 26

4 Sue 25

5 Pete 32

p_id d_id

5 1Relational

©Continuent 2013

personpersonperson

p_id name age

1 Joe 30

2 Fred 23

3 Jack 26

4 Sue 25

5 Pete 32

p_id d_id

5 1Relational

©Continuent 2013

personpersonperson

p_id name age

1 Joe 30

2 Fred 23

3 Jack 26

4 Sue 25

5 Pete 32

p_id d_id

5 1Relational

©Continuent 2013

personpersonperson

_id name age department

1 Joe 30 dev

2 Fred 23 dev

3 Jack 26 sales

4 Sue 25 support

5 Pete 32 sales

Document

©Continuent 2013

How MongoDB keeps data

• three levels:

• dbs

• collections

• documents

©Continuent 2013

MongoDB insertion demo

> show collections>

> db.person.insert( {_id: 1, name: "Joe", age: 30, department: "dev"})> show collectionspersonsystem.indexes

©Continuent 2013

MongoDB insertion demo

> db.person.insert( {_id: 2, name: "Fred", age: 23, department: "dev"})> db.person.insert( {_id: 3, name: "Jack", age: 26, department: "sales"})> db.person.insert( {_id: 4, name: "Sue", age: 25, department: "support"})> db.person.insert( {_id: 5, name: "Pete", age: 30, department: "sales"})> db.person.find(){ "_id" : 1, "name" : "Joe", "age" : 30, "department" : "dev" }{ "_id" : 2, "name" : "Fred", "age" : 23, "department" : "dev" }{ "_id" : 3, "name" : "Jack", "age" : 26, "department" : "sales" }{ "_id" : 4, "name" : "Sue", "age" : 25, "department" : "support" }{ "_id" : 5, "name" : "Pete", "age" : 30, "department" : "sales" }

©Continuent 2013

MySQL to MongoDB basics

©Continuent 2013

Replication from MySQL to MongoDB

• Requires ROW-based-replication

• Replication happens by table

• There is no consolidation into "documents"

• DDL commands are ignored

• Statement commands are ignored

• Column names become document attributes

• enum and set columns are converted to strings

©Continuent 2013

First example of replication

# MySQL

create schema oneschema;use oneschema ;create table myfirst( num int not null primary key, dt datetime, ts timestamp, going enum('yes', 'no'));

# MongoDB> show dbslocal 0.078125GBtest 0.203125GBtungsten_mysql2mongodb 0.203125GB

# NOTICE: no "oneschema"

©Continuent 2013

Inserting data

# MySQLinsert into myfirst values (1, '2003-04-26 09:15:00', null, 'yes');Query OK, 1 row affected (0.01 sec)

select * from myfirst;+-----+---------------------+---------------------+-------+| num | dt | ts | going |+-----+---------------------+---------------------+-------+| 1 | 2003-04-26 09:15:00 | 2013-10-14 19:39:38 | yes |+-----+---------------------+---------------------+-------+1 row in set (0.00 sec)

©Continuent 2013

Checking results in MongoDB

# MongoDB> show dbslocal 0.078125GBoneschema 0.203125GBtest 0.203125GBtungsten_mysql2mongodb 0.203125GB

> use oneschemaswitched to db oneschema> show collectionsmyfirstsystem.indexes

> db.myfirst.find(){ "_id" : ObjectId("525c2c5af5d9ca820fcee01d"), "num" : "1", "dt" : "2003-04-26 11:15:00.0", "ts" : "2013-10-14 19:39:38.0", "going" : "yes" }

©Continuent 2013

Another interesting insertion

#MySQLcreate table t1(_id int not null primary key, c char(10));insert into t1 values (1, 'abc');select * from t1;+-----+------+| _id | c |+-----+------+| 1 | abc |+-----+------+1 row in set (0.00 sec)

# MongoDB> db.t1.find(){ "_id" : "1", "c" : "abc" }

©Continuent 2013

More insertions

# MySQLinsert into t1 values (2,'def'), (3,'ghi'), (4,'jkl'), (5, 'mno');Query OK, 4 rows affected (0.01 sec)Records: 4 Duplicates: 0 Warnings: 0

select * from t1;+-----+------+| _id | c |+-----+------+| 1 | abc || 2 | def || 3 | ghi || 4 | jkl || 5 | mno |+-----+------+5 rows in set (0.00 sec)

©Continuent 2013

More insertions

# MongoDB

> db.t1.find(){ "_id" : "1", "c" : "abc" }{ "_id" : "2", "c" : "def" }{ "_id" : "3", "c" : "ghi" }{ "_id" : "4", "c" : "jkl" }{ "_id" : "5", "c" : "mno" }

©Continuent 2013

Update and delete as seen on master

update t1 set c = 'ZZZ' where _id = 3;Query OK, 1 row affected (0.00 sec)Rows matched: 1 Changed: 1 Warnings: 0

delete from t1 where _id=2;Query OK, 1 row affected (0.00 sec)

select * from t1;+-----+------+| _id | c |+-----+------+| 1 | abc || 3 | ZZZ || 4 | jkl || 5 | mno |+-----+------+4 rows in set (0.00 sec)

©Continuent 2013

e!ects of update and delete on the slave

# MongoDB

> db.t1.find(){ "_id" : "1", "c" : "abc" }{ "_id" : "3", "c" : "ZZZ" }{ "_id" : "4", "c" : "jkl" }{ "_id" : "5", "c" : "mno" }

©Continuent 2013

Overview of Tungsten installer

©Continuent 2013

Overview of Installation Process

1. Set up hosts

2. Prepare MySQL replicas

3. Download software

4. Install using tpm

Amazon Setup:https://docs.continuent.com/wiki/display/TEDOC/

Preparing+EC2+Servers

©Continuent 2013

How tungsten-installer Works for Basic Master/Slave Deployment

Staging copy of files

check prereqscopy codeconfigure

©Continuent 2013

Tungsten master/slave replication

host1 host2

alpha alpha

THL THL

installer

©Continuent 2013

Bi-directional replication

host1 host2

alphaalpha

bravobravo

Install all master and slave services on all hosts at once

installer

©Continuent 2013

4 nodes all-masters

alphaalpha

bravobravo

charlie

bravobravo

charlie

Tungsten security layer

• Tra!c encryption:

• all data in transit (transaction history logs, or THL) is encrypted using SSL

• all administrative tra!c is encrypted with SSL

• Transparent to the user

• Independent of the database server (works also for heterogeneous replication)

Tungsten replicator without security

replicatorservices

alpha alpha

master

THLTHL

plain text

Tungsten Replicator with security

replicatorservices

alpha alpha

master

THLTHL

SSL SSL

host1 host2

alpha alpha

THL THL

Installing Master/Slave Replication ...

master/slave using tpm

./tools/tpm install alpha \ --topology=master-slave \ --home-directory=/opt/continuent/replicator \ --replication-user=tungsten \ --replication-password=secret \ --master=host1 \ --slaves=host2,host3,host4 \ --start

Installing Master/Slave Replication with MongoDB

master/slave with MongoDB

./tools/tpm configure mysql2mongodb \ --enable-heterogenous-service=true \ --topology=master-slave \ --master=host1 \ --replication-user=tungsten \ --replication-password=secret \ --slaves=host2,host3,host4 \ --home-directory=$MYSQL_DEPLOY \ --start-and-report

./tools/tpm configure mysql2mongodb \ --hosts=host4 \ --datasource-type=mongodb \ --replication-port=$MONGODB_PORT

./tools/tpm install

Installing Fan-In Replication

alphabravo

charliebravo

charlie

fan-in using tpm

./tools/tpm install many_towns \ --replication-user=tungsten \ --replication-password=secret \ --home-directory=/opt/continuent/replication \ --masters=host1,host2,host3 \ --slaves=host4 \ --master-services=alpha,bravo,charlie \ --topology=fan-in \ --start

Installing Fan-In Replication with MongoDB

charlie

fan-in with MongoDB

./tools/tpm configure mysql2mongodb \ --enable-heterogenous-service=true \ --topology=fan-in \ --masters=host1,host2,host3 \ --master-services=alpha,bravo,charlie \ --slaves=host4 \ --replication-user=tungsten \ --replication-password=secret \ --home-directory=$MYSQL_DEPLOY \ --datasource-type=mysql \ --start-and-report

./tools/tpm install

Install Multi-Master replication

host1 host2

alphaalpha

bravobravo

charlie

multi-master using tpm

../tools/tpm install musketeers \ --reset \ --topology=all-masters \ --home-directory=/opt/continuent/replicator \ --replication-user=tungsten \ --replication-password=secret \ --masters=host1,host2,host3 \ --master-services=alpha,bravo,charlie \ --start

Install Multi-Master replication with Mongodb

alphaalpha

bravobravo

charlie

bravobravo

charlie

multi-master with MongoDB

./tools/tpm configure mysql2mongodb \ --enable-heterogenous-service=true \ --topology=all-masters \ --masters=host1,host2,host3 \ --slaves=host1,host2,host3,host4 \ --master-services=alpha,bravo,charlie \ --replication-user=tungsten \ --replication-password=secret \ --home-directory=$MYSQL_DEPLOY \ --datasource-type=mysql \ --start-and-report

./tools/tpm install

MongoDB or TokuMX

• TokuMX is a drop-in replacement for MongoDB

• Open source project, developed by TokuTek

• https://github.com/Tokutek/mongo

• it includes

• better indexing

• row-level locking (MongoDB locks at db level)

• transactions

• better compression

DEMO:MongoDB

and multi master installation

Joining the Community

Tungsten Replicator is Open Source

• Project home:

http://code.google.com/p/tungsten-replicator/

• Log bugs, "nd builds, post in replicator discussion group

• Documentation:

https://docs.continuent.com/wiki/display/TEDOC/Tungsten+Documentation+Home

https://docs.continuent.com/wiki/display/TEDOC/Deploying+MongoDB+Replication

Continuent Web Page:http://www.continuent.com

Tungsten Replicator 2.1:http://code.google.com/p/tungsten-replicator

Our Blogs:http://scale-out-blog.blogspot.comhttp://datacharmer.blogspot.comhttp://www.continuent.com/news/blogs

560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009e-mail: sales@continuent.com

synchronise your data between mysql and mongodb

toolstpm configure

null primary

document person

transactions

installing

1 row

servicesalpha

installing

Technology

automating tls configuration verification - danneman ›...

mysql replication options - percona · mysql replication...

slides: introducing the new clustercontrol 1.2.10 for mysql,...

one tool to rule them all- seamless sql on mongodb, mysql...

from mysql to mongodb（mongodb2011北京交流会）

webinar slides: clustercontrol 1.4: the mysql replication &...

open source backups for mongodb - percona · open source...

slides: polyglot persistence for the mongodb, mysql &...

indexing big data 30,000 foot view of databases big data...

redis, mongodb 그리고 mysql 과 함께하는 모바일...

mysql vs...

mysql for mongodb dbas - percona€¦ · mysql for mongodb...

mapping sql to mongodb · 2020. 7. 21. · mapping sql to...

how resonances synchronise on thresholds

big fast data with mongodb - squarespace · mongodb redis...

from mysql to mongodb at sluggy.com

storing and accessing lifelog data with mysql & nosql...

what’s new in mysql and mongodb ecosystem - …€™s new...

mongodb is the new mysql

mongodb + scala - meetupfiles.meetup.com/1652053/scala +...