how to migrate data from mongodb to postgres with torodb · who we are how to migrate data from...

HowtomigratedatafromMongoDBtoPostgreswithToroDB

Whoweare

30/06/2017 Page2HowtomigratedatafromMongoDBtoPostgreswithToroDB

ExpertsAtYourService>Over50specialistsinITinfrastructure> Certified,experienced,passionate

BasedInSwitzerland> 100%self-financedSwisscompany>OverCHF8mio.Turnover

LeadingInInfrastructureServices>Morethan150customersinCH,D&F>Over50SLAsdbiFlexService contracted

dbi services is hiring ([email protected])

Aboutme


MehdiBadaConsultant

+41799287548

mehdi.bada[at]dbi-services.com

Agenda

1.Introduction

2.MongoDB

3.ToroDB

4.Migration: from MongoDB to PostgreSQL

5.Conclusion



1

2

3

4

5

Introduction

30/06/2017HowtomigratedatafromMongoDBtoPostgreswithToroDB Page6

Introduction


1

2

3

4

5

MongoDB

> Overview> DataModel> Highavailability> Horizontalscalability> Limitations


MongoDBOverview

OpenSource

Horizontalscalability

Schema-less

Documentoriented

Highavailability

Indexes

Richqueries

FullCRUDsupport


MongoDBOverview

Conceptmapping

RDBMS MongoDB

Tables Collections

Rows/records Documents

Queries returnrows Queriesreturncursor

Join Embedded document

Partition Shard


MongoDBDataModel

Dataarestoredasdocuments

MongoDBstoresBSONdocuments(BinaryJSON)

Analogoustoadatabaserow

KeysandValues> Key:String> Valuestypes:

> String,number,Boolean,null,array,object

Specialkey:_id> Uniqueidentifier> Objectid:

{“_id”:ObjectId(“56e92b9cfdf7bc92bbb3b51f”),“first_name”:“Mike”,“surname”:“Brody”,“city”:”New-York”,“year”:1987,

}

Date MAC@ PID Counter

12byteHexString


MongoDBDataModel

Relational MongoDBschema

{“first_name” :“Durand”,“surname” :“Mike",“city” :“Geneva",“country” :“Switzerland”,“cars”:[{“model”:“Ferrari”,“year”:2013}

}

Pers_Id Surname First_name City

0 Mike Durand Geneva

1 Pat Millner London

2 Ortega Alvaro New-York

Car_Id Model Year Pers_Id

0 Ferrari 2013 0

1 Peugeot 2005 0

2 BMW 2016 1


MongoDBHighavailability

Whichmechanismensurehighavailabilityofyourdata?>MongoDBReplication

HAisachievedthroughautomaticfailover

MongoDBreplicationallows:> Highavailability(HA)

> DisasterRecovery(DR)> Dataduplicationacrossmultipledatabaseservers/storages

> FunctionalSegregation> Topologyofreplicasetscanbeusedfor

> Backups,Analytics,Reporting,DR,Readoperations…



Replicationarchitecture

Application

mongodSecondary

mongodSecondary

mongodPrimary

driver

Replication

Reads Writes



Automaticfailoverprocess mongodPrimary

mongodSecondary

mongodSecondary

Electionofanewprimary

Heartbeat

mongodSecondary

mongodPrimary

Heartbeat

Replication

Newprimaryiselected


MongoDBHorizontalscalability

Verticalscalability

IncreasingCPU,RAM,I/O

ScalingwithMongoDB>MongoDBSharding

mongodConfigSvr

Shard0

…


MongoDBHorizontalscalability

ShardingArchitectureApplication

mongodPrimary

mongodSecondary

mongos(queryrouter)

Shard1

…

mongodPrimary

mongodSecondary

mongodConfigSvr

mongodConfigSvr

Sharding


MongoDBLimitations

NoACIDtransaction> Atomictransactionsonlyworkwithinthesamedocument

[...{user:567,products: [

{id:47,units:7,

},...

]

[...{user:567,orders: [

...{

orderId:24658,products: [

...]

}...]

}...]

CART ORDERS

ThisoperationisnotAtomic!!


MongoDBLimitations

MongoDBHighavailabilityisnotsafe!!> Datalossdependingtheconsistencylevelyouchoose

MongoDBconsistencylevels>Unacknowledged:Unsafe- 42%ofdataloss> Acknowledged:Unsafe> Journaled:Unsafe> Fsynced:Unsafe> ReplicaAcknowledged:Unsafe>Onlymajorityissafe

https://aphyr.com/posts/322-jepsen-mongodb-stale-reads


MongoDBLimitations

BIqueryperformancesissues

MongoDBaggregationframeworkisveryslow>Needtoscanmultipledocuments> LotsofI/Orequiredtoanswerthequery

Aggregationonarelationaldesignis“100xfaster”

SolutionforMongoDBBIqueries?> Implementarelationalschema!!


1

2

3

4

5

ToroDB

> WhatisToroDB?> Howitworks?> WhyToroDB?


ToroDBWhatisToroDB?

ThefirstdatabasethatmergesthescalabilityofaNoSQLwiththereliabilityofSQL


ToroDBWhatisToroDB?

Opensource,document-oriented,JSONdatabasethatrunsontopofPostgreSQL

BIconnectorforMongoDB

JSONdocumentsarestoredrelationally> Significantstorage> I/Osavings

MongoDBdataispersistingintablesandrowswithinaSQLdatabase

Protocols

Abstract Doc

Data2Relational

Backends

MongoDB,Couchbase,Cassandra...

TransformingDocsintoTablesandRows

PostgreSQL,DB2,Oracle,Greenplum...


ToroDBHowitworks?

ToroDBtransformsdocumentstorelationaltables> Dataisstoredintables> ToroDBanalyzeseveryincomingdocumentandseparatesmetadata(schema)formdata(tuples)> 1+tablesperMongoDBcollections> ToroDBcreatesaRDBMScatalogschemaperMongoDBdatabase> Dynamicandimplicitschemageneration

FullcompatibilitywithMongoDB> APIprograms,clients> CRUDoperationsincludingUPDATE


ToroDBHowitworks?

ToroDBcanworkasasecondarynodeonaMongoDBreplicaset


ToroDBWhyToroDB?

NativeSQLBIConnector

Apps:WritedatawithMongoAPI,querywithSQL!

DataIntegrationPlatform:SQLandNoSQLappsinthesameRDBMS


1

2

3

4

5

Migration:fromMongoDBtoPostgreSQL

> Overview> Prerequisites> Configuration> Demo


Migration:fromMongoDBtoPostgreSQLOverview



ToroDBStampededusesMongoDBreplicasetoplog tokeeptrackofthemodificationsinMongoDB



Duringthereplication ToroDBStampededtransformsJSONdocumentsintoarelationalschema

ColumnTypeEncoding


Migration:fromMongoDBtoPostgreSQLPrerequisites

Runtimedependencies

Technology Description

MongoDB InstallandstartMongoDBinstances withthereplication features(dbiservicesbestpracticesinstallation)

Replicasetsconfiguration ToroDBStampedereceivesdatafromaMongoDBreplicaset.Asingle-nodereplicasetissufficient.

PostgreSQL InstallandstartaPostgreSQL instance.Createadedicateduseranddatabase.(dbiservicesbestpractices)

Java ToroDBStampedeiswritteninJavasoaJavaRuntimeEnvironment(JRE)requiredtorunit.Java8isrecommended.


Migration:fromMongoDBtoPostgreSQLConfiguration

Architecture

MongoEnv PostgresEnv

ToroDBStampededschemageneration

PostgreSQLinstancewithDMKinstalled

MongoDBinstanceswithDMKinstalled

192.168.56.102 192.168.56.140



Afterinstallation,export$TOROHOME variable

CreateandadapttheToroDBconfigurationfile(YAML)

mehdi@MacBook-Pro:/u00/app/torodb/ export TOROHOME /u00/app/torodb/torodb-stampede-1.0.0-beta2”

mehdi@MacBook-Pro:/u00/app/torodb/ echo $TOROHOME/u00/app/torodb/torodb-stampede-1.0.0-beta2

mehdi@MacBook-Pro:/u00/app/torodb/ /u00/app/torodb/torodb-stampede-1.0.0-beta2/bin/torodb-stampede -l > ../conf/torodb.yaml

mehdi@MacBook-Pro:/u00/app/torodb/ vi ../conf/torodb.yaml



ToroDBStampededreadsdatabasescredentialsfromthe.toropassfile

Createthe.toropassfileinthehomedirectory

mehdi@MacBook-Pro:/u00/app/torodb/ echo ”<host>:<port>:<database>:<user>:<PASSWD>" > "$HOME/.toropass"

mehdi@MacBook-Pro:/u00/app/torodb/ chmod 0400 ”~/.toropass”



CustomconfigurationforToroDBStampededonstartup> Commandlineoptions> Configurationfile

Recommendedtouseaconfigurationfile

Printthecurrentconfiguration(YAML)

mehdi@MacBook-Pro:/u00/app/torodb/ ./torodb-stampede -c myconfiguration.yml

mehdi@MacBook-Pro:/u00/app/torodb/ ./torodb-stampede -l


Migration:fromMongoDBtoPostgreSQLDemo

Steps Description

1 Java8 (JRE)InstallationandConfiguration:http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jre-8u131-linux-x64.tar.gz

2 PostgreSQL InstallationandConfiguration:- Modify,ifneeded,PostgreSQLinstanceconfiguration:/u02/pgdata/PG1/postgresql.conf

https://www.torodb.com/stampede/docs/1.0.0-beta2/configuration/postgresql-configuration-tips/- Createusertorodbanddatabasetorod(withpassword)- Testtheconnectionwithnewuseranddatabases- Adaptpg_hba.conf fornewconnections

3 MongoDBInstallationandConfiguration:- ConfigureandinitializeMongoDBreplication:replset=torodb- Checkreplicationconfig.(primaryandsecondary)- ImportdataintoMongoDB

4 ToroDBStampedeInstallationandConfiguration:- torodb.yaml configurationfilecreation- .toropass filecreationforcreation- StartToroDBStampeded


Migration:fromMongoDBtoPostgreSQLDemo


1

2

3

4

5

Conclusion

> AdvantagesvsDrawbacks


ConclusionAdvantagesvsDrawbacks

FastandPowerful

Dynamicschemageneration

OnlysupportsPostgreSQLasRDBMSbackend

Dynamicchangesintheschema

Opensourceandfree!!

Cross-platform:Linux,Windows

Needimprovementsfordifferentschemainthesamemongocollection

Anyquestions?

Pleasedoask!WewouldlovetoboostyourIT-Infrastructure

Howaboutyou?

ZürichBasel

Delémont

Nyon
