best practices: migrating a postgres production database to the cloud

© 2014 EnterpriseDB Corporation. All rights reserved. 1

Migrating a Production Postgres Database to the Cloud Webinar: January 21, 2015


Welcome!

Today’s focus:

Migrating a Production Database into the Cloud with Minimal Disruption to Customers

Agenda: −  Business drivers (see the blog) − How to perform the migration (see the cookbook)


•  Your application is successful, but on a system that can’t keep up with the growth of the business

•  Migrating the application to the cloud will solve your performance problems, but you’re not sure how to make the transition

•  You need to: −  Make sure no data is lost during the transition −  Minimize application unavailability during the transition

•  This webinar will tell you how to meet those requirements!

The Business Need


•  This webinar presents the high level flow and general technical steps of a migration – please read the Migration Cookbook for full technical details

•  Attendees are generally aware of −  How to use the Amazon AWS console −  Linux command line tools −  Networking basics

Assumptions


•  Postgres Plus Cloud Database isn’t a managed service; it allows root access to your private database instances

•  The source database can be configured to provide streaming replication

•  The transfer rate between source and target databases is high enough to exceed the rate of change to the source database

•  source and target databases are the same type and version (in this case, Postgres 9.3)

Technical Enablers and Requirements


The migration is conducted in four phases:

1.  Prepare the source and target databases −  Create and configure

2.  Move existing data from the source to target database −  Bulk move with pg_basebackup

3.  Move still-occurring transactions from source to target databases −  Streaming replication

4.  Cutover application from the source to target database −  Quiesce application, point to new database, restart operations

Overview of the Technical Solution


Phase 1:

Prepare the Source and Target Databases


•  Create the target database using the Postgres Plus Cloud Database console

•  It will be a standby node for receiving streaming replication from the source database

•  Actions: −  Create a standard PPCD database cluster −  Use the same database version as the source database −  Configure to match the requirements for your new production

database −  Make the master database operate as a standby database

Step 1a: Create the Target Database


•  Allow ssh connections between the source and target databases

•  The ssh connection will be used to conduct streaming replication from the source to target databases

•  Actions: −  Connect to the AWS management console and use the

Services menu to navigate to the EC2 Dashboard −  Select the Security Groups option −  Select the security group for your new database cluster −  Add a rule to open port 22. −  See the Cookbook and section 13.3 the PPCD Getting Started

Guide for more details

Step 1b, Target: Allow Incoming SSH


•  The Postgres Plus Cloud Database uses a load balancer for regular use of the database

•  During the migration, the load balancer must be stopped to not interfere with pg_basebackup

•  Actions: −  Login to the PPCD console used to create the database

cluster −  Select the clusters tab, then select the new database −  In the Details panel, deselect the Monitor Load Balancer

Health checkbox (we will re-enable it at the end of the migration process)

Step 1c, Target: Disable Load Balancing


•  Preserve original database certificates and configurations so they can be restored at the end of the migration process

•  Actions: −  ssh into the target database’s instance −  Create a temporary directory −  Move specific files from the /opt/PostgreSQL/9.3/data/

directory to the new temporary directory

Step 1d, Target: Back up Database Files


•  Copy the source system’s ssh private key to the target database instance

•  Create an ssh tunnel between the source and target databases to migrate ongoing transactions still occurring in the source database.

•  Actions: −  scp the ssh private key for the source instance to the target

database instance −  ssh into the target database instance −  Create an ssh tunnel between the source and target database

instances # ssh –i p_key -L5439:127.0.0.1:5432 deploy@source_ip

Step 1e, Target: Create an SSH Tunnel


•  The source database must be configured to act as a master node in a replication scenario

•  Actions: −  ssh into the target database instance −  Su to the proper identity to edit the configuration files −  Edit the pg_hba.conf file, and add a new line:

host replication postgres 127.0.0.1/32 trust

−  Edit the postgresql.conf file and add the following parameters: max_wal_senders = 5 wal_level = hot_standby wal_keep_segments = 50

−  Restart the database

Step 1f: Prepare the Source Database


Phase 2:

Move Existing Data from the Source Database to the

Target Database


•  Transfer existing data from the source to target databases using pg_basebackup

•  Actions: −  Ssh to the target cloud database and take the identity of the

postgres user and load Postgres environment variables −  Use the pg_basebackup command to copy data from the

source master node to the target standby node

$ pg_basebackup --xlog --pgdata=/mnt/pcs/data.new --port=5439 --host=127.0.0.1 --verbose --write-recovery-conf

Step 2a: Migrate Existing Data from the Source to Target Databases


Phase 3:

Move Still-Occurring Transactions from the

Source Database to the Target Database


•  Restore the database configuration and certificate files saved in step 1d above to a new /data directory

•  Actions: −  ssh to the target database instance −  Become superuser −  Copy files:

# cp –af vr /tmp/data/* /mnt/pcs/data.new

Step 3a: Restore the Target Database’s Data Files


•  Configure the target database to act as a standby node to receive streaming replication

•  Actions: −  ssh to the target database instance −  Edit the /mnt/pcs/data.new/postgresql.conf file −  Add the following line to the bottom of the file:

hot_standby = on

−  Verify that the max_connections parameter in the target database is at least one larger than the max_connections parameter in the source database

−  If the source postgresql.conf file has been modified, make sure to update the target postgresql.conf file to match

Step 3b: Prepare the Target Database


•  Prepare the target database to be started in standby mode

•  Actions: −  On the target database as superuser, stop active services of

the target database: # /etc/init.d/pcs_node_manager stop

# /etc/init.d/pgpool-3.0 stop

# /etc/init.d/postgresql-9.3 stop

−  Immediately rename the data directory: # mv /mnt/pcs/data /mnt/pcs/data.old &&

mv /mnt/pcs/data.new /mnt/pcs/data

Step 3c: Stop Services, Rename the Data Directory


•  Restart the target database as a standby to receive still-occurring transactions from the source database.

•  Actions: −  Restart the target data database and related services:

# /etc/init.d/postgresql-9.3 restart

# /etc/init.d/pcs_node_manager restart

# /etc/init.d/pgpool-3.0 restart

Step 3d: Restart Cloud Database Services


•  Verify that the source master database is replicating still-occurring transaction data to the target standby database

•  Actions: −  ssh to the target database instance, and load Postgres

environment variables −  Invoke psql, and query whether the database is in recovery (if

it is in standby mode, the result will be “t” for true) # psql

postgres=# select pg_is_in_recovery(); pg_is_in_recovery ------------------- t (1 row)

Step 3d: Confirm Streaming Replication is Enabled


Phase 4:

Cutover Application from the Source Database to the

Target Database


•  Once streaming replication has moved most of the remaining data from the source database to the target database, the application must be quiesced to prevent additional data from entering the source database.

•  This is the most critical time in the migration scenario: the application service outage starts when the application is quiesced and ends after the next two steps restart the application.

•  Quiesce the application and let any final data drain

•  Verify that all data has been moved to the target database

Step 4a: Quiesce the Application


•  Once you have confirmed that all data has been received by the target database, promote it to be the master database.

•  Once the target database is promoted to master, the source database cannot send additional data to it.

•  Actions: −  ssh to the target database instance −  Become the postgres user and load Postgres environment

variables −  Promote the standby to be the master:

$ pg_ctl -D /opt/PostgreSQL/9.3/data promote

Step 4b: Promote the Cloud Database Instance to be the Master


•  The final step of the migration is to point the application at the new master database and un-quiesce the application.

•  Actions: −  These steps are application specific −  Application database-specific information, particularly the IP

address of the new database, should be updated with the target database information

−  Instruct the application to restart customer interactions

Step 4c: Point the Application at the New Database


Congratulations!

You’ve just migrated your production Postgres

database into the public cloud!


•  Cookbook: How to Move a Production Postgres Database Into a Cloud Database Cluster [www.enterprisedb.com/move-paas-to-cloud]

•  Blog: Beating the Cloud Migration Busters [blogs.enterprisedb.com/2015/01/12/beating-the-cloud-migration-busters/]

•  EnterpriseDB Postgres Plus Cloud Database [www.enterprisedb.com/cloud]

•  Gartner Cloud Database Magic Quadrant [www.enterprisedb.com/gartner-magic-quadrant]

Resources


Thank You!

best practices: migrating a postgres production database to the cloud

Software

source database source

source database actions

database version

master database

database bulk

cloud database console

production postgres

enterprisedb corporation