replicate from oracle to oracle, oracle to mysql, and oracle to analytics

©Continuent 2014

Replicate Oracle to Oracle, Oracle to MySQL ,

and Oracle to Analytics

Linas Virbalas, Senior Software Engineer Robert Hodges, CEO

©Continuent 2014

Introducing Continuent

���2

• The leading provider of clustering and replication for open source DBMS

• Our Product: Continuent Tungsten

• Clustering - Commercial-grade HA, performance scaling and data management for MySQL

• Replication - Flexible, high-performance data movement

©Continuent 2014

What is Tungsten Replicator?

���3

Tungsten Replicator: a high-performance

open source database replication engine

!GPL V2 license - 100% open source

Download from https://code.google.com/p/tungsten-replicator/ Annual support subscription available from Continuent

“GoldenGate without the Price Tag”®

©Continuent 2014

Tungsten Replicator Overview

���4

Master

(Transactions + Metadata)

Slave

THL

DBMS Logs

Replicator

(Transactions + Metadata)

THLReplicator

Download transactions via network

Apply using JDBC

©Continuent 2014 ���5

Extract Filter Apply

StageExtract Filter Apply

StageExtract Filter Apply

Stage

Pipeline

Master DBMS

Transaction History Log

QueueSlave

DBMS

Tungsten Replication Service

©Continuent 2014

Filters and Parallel Apply

���6

Extract Filter Apply

StageExtract Filter Apply

StageStage

Pipeline

Master DBMS

Transaction History Log

In-Memory Queue

Slave DBMS

Extract Filter ApplyExtract Filter ApplyExtract Filter Apply

©Continuent 2014

Multiple Services per Replicator

Service frommysql

Service fromoracle

frommysql

fromoracle

Replicator

Replicator

Replicator

Replicator

Aggregated

©Continuent 2014

star

master-slave Heterogeneous

fan-in slave all-masters

MySQL

Oracle

Oracle

MySQL Oracle

Oracle

MySQL MySQL

©Continuent 2014©Continuent 2014

Heterogeneous Replication

©Continuent 2014©Continuent 2014

Steps to Heterogeneous Replication

1. Prepare (translate) schema for the slave DBMS

2. Set up replication

3. Provision initial data

4. Continue real-time replication

ddlscan

tpm

Parallel ApplyMySQL Blackhole

Tungsten Replicator

Parallel Extract

©Continuent 2014©Continuent 2014

Replicating from MySQL to Oracle

©Continuent 2014©Continuent 2014

Use Case: Real-Time Sales/Order Fulfillment

Real-Time Replication

Web-based Sales Purchase Ordering

3M Order Items

~1M Xacts/ Day

©Continuent 2014©Continuent 2014

1. Translating schema for the slave

©Continuent 2014©Continuent 2014

Translating Schema

���14

• Goal - set up heterogeneous replication

Replication

©Continuent 2014©Continuent 2014

Translating Schema

���15

• Beginning - how to convert tables?

empty

Tables

©Continuent 2014©Continuent 2014

Translating Schema

���16

• Beginning - how to convert tables?

empty

Tables

•Data types? •Column lengths? •Naming conventions? •Reserved words?

©Continuent 2014©Continuent 2014

Translating Schema

���17

• ddlscan automates these conversions

empty

Tables

ddlscan

©Continuent 2014©Continuent 2014

ddlscan

���18

• Part of Tungsten Replicator, GPL v2

• Translates schema with replication in mind

• Provides errors and warnings

• Can rename schema/tables/columns

©Continuent 2014©Continuent 2014

Usage (MySQL to Oracle Example)

���19

$ cd tungsten-replicator/bin !

$ ./ddlscan \ -db test \ -template ddl-mysql-oracle.vm \ -user tungsten \ -pass secret

©Continuent 2014©Continuent 2014

Translating Schema

���20

• ddlscan looks into source schema

empty

Tables

©Continuent 2014©Continuent 2014

Translating Schema

���21

• ddlscan translates and renders DDL commands

empty

Tables

oracle-ddl.sql

©Continuent 2014©Continuent 2014

Result of ddlscan

���22

CREATE TABLE 3colors /* ERROR: table starts with a number, use rename option and a filter */

( id NUMBER(10, 0), color VARCHAR2(1) /* ENUM('R','G','B') */, enabled NUMBER(1) /* BIT(1) - no constraints for 0 or 1 */, acolumnthatdoesntfittooracleat NUMBER(10, 0)

/* WARN: truncated column name exceeding 30 characters (acolumnthatdoesntfittooracleatall) */

! /* ERROR: table must have a primary key! */ ); !CREATE TABLE talks ...

©Continuent 2014©Continuent 2014

Translating Schema

���23

• You run resulting SQL file on Oracle

empty

Tables

oracle-ddl.sql

©Continuent 2014©Continuent 2014

Translating Schema

���24

• Tables are ready!

Tables Translated tables (empty)

©Continuent 2014©Continuent 2014

2. Set Up Replication

©Continuent 2014©Continuent 2014

Replicator Installation

���26

./tools/tpm configure my_ora \ --enable-heterogenous-service=true \ --install-directory=/opt/mysql_to_ora/continuent \ --members=alpha,bravo \ --master=alpha !./tools/tpm configure my_ora --hosts=alpha \ --replication-user=tungsten \ --replication-password=secret !./tools/tpm configure my_ora --hosts=bravo \ --replication-user=tungsten_frommysql \ --replication-password=secret \ --datasource-type=oracle \ --datasource-oracle-service=ORCL \ --svc-applier-filters=dropstatementdata !./tools/tpm install my_ora !./tools/tpm start my_ora

!

©Continuent 2014©Continuent 2014

Replication Under the Hood

���27

MySQL Binlog

binlog_format=row

Service my_ora

Tungsten Master Replicator

MySQLExtractor Special Filters •Transform

ENUM to string •Transform SET

to string •Column names

and signed flag

Service my_ora

Tungsten Slave Replicator

Special Filters •Drop DDL •Map names to upper case •Rename too long or reserved

objects •Optimize updates to remove

unchanged columns •Ignore extra tables

OracleApplierMySQLExtractor

©Continuent 2014©Continuent 2014

3. Provisioning an Oracle slave

©Continuent 2014©Continuent 2014

table structure

table data

table structure

Provisioning (2)

���29

table data

table structuretable data

table structure table structuretable structureddlscan

©Continuent 2014©Continuent 2014

table structure

Provisioning (3)

���30

table structuretable structure

MySQL Sandbox

CONVERT to BLACKHOLE

table data

table structuretable data

table structuretable data

table structure

©Continuent 2014©Continuent 2014

Provisioning (4)

���31

MySQL Sandbox

table data

table structuretable data

table structuretable data

table structure

not replicating

backup

©Continuent 2014©Continuent 2014

Provisioning (5)

���32

MySQL Sandbox

backup

bin logs

THL

1

(ROW)

Master Tungsten

Replicator

Slave Tungsten

Replicator

2 3 4

©Continuent 2014©Continuent 2014

4. Continue Real-Time Replication

©Continuent 2014©Continuent 2014

Provisioning (6)

���34

MySQL Sandbox

table data

table structuretable data

table structuretable data

table structure

bin logs

THL

Master Tungsten

Replicator

Slave Tungsten

Replicator

©Continuent 2014

Replicating from Oracle to MySQL

©Continuent 2014

Use Case: Web Content Publishing

Real-Time Publication

Backend Office Web-Based Catalog

©Continuent 2014

How Does Tungsten Do That?

���37

Service ora_my

Tungsten Master Replicator

OracleCDCExtractor No Special Filters

Service ora_my

Tungsten Slave Replicator

Special Filters •Map names to lower case •Ignore extra tables •Heartbeat table renaming

demo schema

demo_pub schema

Oracle CDC (Synchronous or

Async Hotlog)

MySQLApplierOracleCDCExtractor

©Continuent 2014

Replicating from Oracle to Oracle

©Continuent 2014

How Does Tungsten Do That?

���39

Service ora_ora

Tungsten Master Replicator

OracleCDCExtractor No Special Filters

Service ora_ora

Tungsten Slave Replicator

Special Filters No Special Filters

demo schema

demo_pub schema

Oracle CDC (Synchronous or

Async Hotlog)

Tungsten Slave Replicator

OracleApplierOracleCDCExtractor

©Continuent 2014©Continuent 2014

DEMO

DEMO.* demo.*

DEMO2.* demo2.*

©Continuent 2014

Replicating from Oracle to Vertica

©Continuent 2014

Use Case: Web Content Publishing

Near Real-Time Publication

Backend Office Analytic Reports

Feeds from other data sources

©Continuent 2014

How Does Tungsten Do That?

���43

Service ora_ver

Tungsten Master Replicator

OracleCDCExtractor No Special Filters

Service ora_ver

Tungsten Slave Replicator

Special Filters •Map names to lower case •Ignore extra tables •Rename data tables (if required)

demo schema

demo_pub schema

Oracle CDC (Synchronous or

Async Hotlog)

SimpleBatchApplierOracleCDCExtractor

©Continuent 2014

Batch Loading--The Gory Details

���44

Replicator

Transactions from master

CSV FilesCSV FilesCSV Files

Staging TablesStaging TablesStaging Tables

Base Tables

Base Tables

Base Tables

Merge Script

(or) COPY

directly to base tables

COPY to stage tables SELECT to

base tables

Service ora_ver

©Continuent 2014

DEMO

©Continuent 2014

Coming Attractions

©Continuent 2014

New Replication Features in 2014

• Parallel extraction for fast provisioning

• Select data as of specific SCN

• Pull data across in multiple streams

• Matches parallel apply on slave side

!

• Hadoop data loading

• Direct loading from MySQL/Oracle to HDFS

• View data using Hive

• Generation of materialized views

���47

©Continuent 2014 ���48

Continuent Web Page: http://www.continuent.com

!Documentation:

https://docs.continuent.com !

Tungsten Replicator: http://code.google.com/p/tungsten-replicator

Our Blogs: http://flyingclusters.blogspot.com http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://continuent-tungsten.blogspot.com

560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: sales@continuent.com