agile oracle to postgresql migrations (pgconf.eu 2013)
DESCRIPTION
Migrating an Oracle database to Postgres is never an automated operation. And it rarely (never?) involve just the database. Experience brought us to develop an agile methodology for the migration process, involving schema migration, data import, migration of procedures and queries up to the generation of unit tests for QA. Pitfalls, technologies and main migration opportunities will be outlined, focusing on the reduction of total costs of ownership and management of a database solution in the middle-long term (without reducing quality and business continuity requirements).TRANSCRIPT
![Page 1: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/1.jpg)
Higher impact of license costs
![Page 2: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/2.jpg)
Agile Oracle to PostgreSQL migrations
PGConf.EU 2013, Dublin, Nov 1st
Gabriele Bartolini
2ndQuadrant Italia / PostgreSQL [email protected]
@_GBartolini_
![Page 3: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/3.jpg)
Gabriele Bartolini•Co-Founder and Manager of 2ndQuadrant Italia
• Data Architect, Business critical environments
• Data warehousing
•Co-Founder Italian PostgreSQL Users Group
•Co-Founder PostgreSQL Europe
•PostgreSQL Contributor and Advocate
![Page 4: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/4.jpg)
Outline• Introduction
•The migration project
•Open source tools
•Conclusions
![Page 5: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/5.jpg)
Target•Chief Technical Officers
•Chief Information Officers
•Oracle users
• Evaluate moving to PostgreSQL 9.3
![Page 6: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/6.jpg)
Goals•Describe the migration process
•Spot the major sources of costs
•Help you plan a migration in the mid/long term
![Page 7: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/7.jpg)
Part IIntroduction
![Page 8: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/8.jpg)
Oracle vs Postgres
(*) not sure it is in the TODO List, yet
![Page 9: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/9.jpg)
Upgrade automaticallyWould you ever do an upgrade to a higher version of
Oracle in a completely automated way?
![Page 10: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/10.jpg)
PostgreSQL•Hot Standby (~ Data Guard)
•Barman (~ RMAN)
•Londiste/PgQ (GoldenGate)
•PostGIS (Spatial)
•SE Linux integration (~ Database Vault)
•Memcache integration (~ In-Memory Database Cache)
•Partitioning (*)
•Future: Multi-master replication (~ RAC)
![Page 11: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/11.jpg)
License•Very simple and clear
•Applies to every part (module, product, option) of Postgres
• server, client, libraries, additional modules
• 100% open source
•No need to hire a lawyer to understand it
•Highly Permissive
• Not subject to monopoly nor acquisitions
•Fosters market competition in professional services offer
![Page 12: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/12.jpg)
Oracle DBA, do not be afraid!•Transferrable knowledge
•PostgreSQL focused professional training
•100% open source
![Page 13: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/13.jpg)
Part IIThe Migration project
![Page 14: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/14.jpg)
Migration = projectbudget, scope, quality, time
![Page 15: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/15.jpg)
Budget•Crucial variable
• Usually triggers the migration process
•Cost analysis drives the migration project
• short term
• mid term
![Page 16: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/16.jpg)
Quality•PostgreSQL reliability
•Tests
![Page 17: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/17.jpg)
Components•Applications
•Database
•Processes
•Human resources
![Page 18: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/18.jpg)
Migration cost components
25,00% 50,00%
Dev Testing Processes Training Licensing
fictit
ious
dat
a
![Page 19: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/19.jpg)
Comparison with Oracle upgrade
25,00% 50,00%
Dev Testing Processes Training Licensing
fictit
ious
dat
a
![Page 20: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/20.jpg)
External applications•Force your supplier to provide Postgres support for their
application
• You set the rules
•Typical case of the public sector
• e.g.: Land registry
• Financial crisis and competition are on your side
![Page 21: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/21.jpg)
Agile approach• Requirements change
• Every context is different from the other
• No one-size-fits-all scenario
• Be dynamic:
• SCRUM (with the customer)
• Kanban (internally)
• DevOps
![Page 22: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/22.jpg)
![Page 23: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/23.jpg)
Focus•Applications
•Database
•Processes
•Human resources
![Page 24: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/24.jpg)
Database migration•Schema
•Queries
•Data
•Stored procedures
•Tests
![Page 25: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/25.jpg)
Schema conversion•Tables
• Data type mapping
• Indexes
•Views
•Constraints
![Page 26: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/26.jpg)
Schema major differences•USER = schema/database
•UPPERCASE = lowercase
•Partitioning (*)
•Tablespaces
•SYNONYMs
• search_path / auto updatable views / foreign tables
![Page 27: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/27.jpg)
Queries•ANSI/ISO SQL
•PostgreSQL has no hints (optimiser)
•Major differences:
• NULL = NULL
• Postgres: NULL
• Oracle: TRUE
• OUTER JOINs
• (+)
• use LEFT/RIGHT JOIN
• DUAL
• ROWNUM and ROWID
• ...
![Page 28: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/28.jpg)
Data•Might require specialised
consulting activities
•Major solutions:
• ETL tools (e.g. Kettle)
• External scripts (e.g. ora2pg or custom)
• COPY t FROM stdin;
• Foreign tables with Oracle FDW
• INSERT INTO t SELECT * FROM ft;
• Cut-over time
• > 0
• ~ 0
![Page 29: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/29.jpg)
Stored Procedures•Most critical and complex component of the whole migration
•Package = extension (schema)
•Two major ways:
• 1:1 conversion (not always possible)
• complete rewriting (optimised for PostgreSQL)
• It might required specialised consulting activities
![Page 30: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/30.jpg)
PL/SQL vs PL/pgSQL•Syntactical differences
•PROCEDURE = FUNCTION that returns VOID
•TRIGGER = TRIGGER FUNCTION + TRIGGER
•Lack of autonomous transactions
•Lack of global variables
• ...
![Page 31: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/31.jpg)
Test•Vital component for a successful migration process
• It measures the QUALITY of the project
• It can be reused in the future for PostgreSQL upgrades
• strategic investment in the long period
![Page 32: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/32.jpg)
Part IIIOpen source tools
![Page 33: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/33.jpg)
ora2pg•URL: http://ora2pg.darold.net/
•Written in Perl - GNU GPL 3
•Reads the information schema from Oracle
•Generates DDL instructions for PostgreSQL
•Highly configurable (mapping data type conversions)
•Basic support for PL/SQL (regexp)
![Page 34: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/34.jpg)
orafce•URL: http://orafce.projects.pgfoundry.org/
•C and SQL - BSD licensed
•Set of functions, objects and modules that are part of Oracle:
• NVL function (COALESCE)
• DBMS_ALERT, DBMS_OUTPUT, UTL_FILE, ...
• DUAL
![Page 35: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/35.jpg)
Oracle FDW•URL: http://oracle-fdw.projects.pgfoundry.org/
•SQL/MED
• uses the API for Foreign Data Wrapper
• CREATE FOREIGN TABLE
• Currently READ ONLY
•Useful for ETL
![Page 36: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/36.jpg)
Example of Oracle FDW...CREATE FOREIGN TABLE oracle_table ( id integer NOT NULL, ...) SERVER oradb OPTIONS (schema 'ORAUSER', table 'ORATAB');...-- pushes down the WHERE clauseSELECT * FROM oracle_table WHERE id=10;
![Page 37: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/37.jpg)
PgTap•Unit tests suite for PostgreSQL
•Extensions (functions)
•Produces TAP output
•Test Driven Development in databases
![Page 38: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/38.jpg)
Vagrant•Open Source software
•Creation and configuration of virtual environments
•Provisioning: Puppet, Chef
•Useful for local testing
![Page 39: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/39.jpg)
PostgreSQL•Every major release of Postgres reduces the technology gap
with Oracle
•5 years ago, Postgres did not have hot standby, streaming/synchronous/cascading replication, window functions, CTEs, column permissions, foreign tables, serialisable isolation level, XML, JSON, updatable and materialised views ...
• Is it called “OraGIS” or “PostGIS”?
•Postgres does have transactional DDL
![Page 40: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/40.jpg)
Master Standby
App
Business Continuity stack
DR
server C
repmgr
Barmanserver A server B
![Page 41: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/41.jpg)
If you really miss it ...CREATE USER scott WITH PASSWORD ‘tiger’;CREATE DATABASE scott WITH OWNER scott;\c scott scottCREATE SCHEMA scott;
![Page 42: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/42.jpg)
Part IVConclusions
![Page 43: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/43.jpg)
0
30.000
60.000
90.000
120.000
150.000
180.000
210.000
240.000
270.000
300.000
Costs in Euro
35.000
70.000
140.000
16.000
32.000
64.000
7.5001 CPU Socket 2 CPU Socket 4 CPU Socket PostgreSQL*
License Support
Cost analysis of a database solution on a period of 3 years for 1 server
* PostgreSQL with 24/7 support from us
![Page 44: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/44.jpg)
015.00030.00045.00060.00075.00090.000
105.000120.000135.000150.000
Costs in euro70.000
16.000 16.000
50.000
2.500 2.500
Anno 1 Anno 2 Anno 3
Oracle PostgreSQL
Cost analysis of a migration to PostgreSQL in a 3 year period
* PostgreSQL with 24/7 support from us
![Page 45: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/45.jpg)
Conclusions•Final solution: 100% open
source
• reduced TCO
• No vendor lock-in
•Migration must be focused on costs evaluation
•Supervised process (not automated)
• If a feature is missing, you can simply sponsor it
• If needed, professional companies can help you before, during and after the migration process (you won’t be alone)
![Page 47: Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)](https://reader034.vdocuments.net/reader034/viewer/2022052522/554a159db4c9058c5d8b4e58/html5/thumbnails/47.jpg)
Thank you!Licenza Creative Commons BY-NC-SA 3.0
http://creativecommons.org/licenses/by-nc-sa/3.0/it/deed.it