Denish PatelDatabase Architect, OmniTI

Deploying PostgreSQL on Amazon EC2 A case study

1

Tuesday, May 7, 13

Story teller?

✤ Denish Patel

✤ denish@omniti.com

✤ Database Architect at OmniTI

✤ Heterogeneous Environment

✤ PostgreSQL, Oracle , MySQL, NoSQL

✤ Scalability, Performance & Simplicity

✤ OmniTI is hiring Database Engineer!2

Tuesday, May 7, 13

Situation

✤ Startup

✤ Ruby on Rails Shop

✤ Entire IT architecture on Amazon Cloud

✤ 2 Identical Database Environments on EC2

✤ 4 Production Instances

✤ 50GB Each PostgreSQL 9.0 Master - Slave ( 2 Pairs)

✤ 2 QA and 2 Dev environments3

Tuesday, May 7, 13

Situation

✤ Weekly flake out issues on EC2 instances

✤ Weekly backups using pg_dump

✤ No file system level backup

✤ Master-slave wasn’t working for months!

✤ Schema mismatch between 2 environments

✤ Almost NO monitoring in-place for DBs

4

Tuesday, May 7, 13

Situation

✤ Prod , QA and Dev ...

✤ M1 Extra large instances (m1.xlarge)

✤ 15 GB RAM, 4 Virtual Cores with 2 EC2 compute units each

✤ Configuration Management tool - Puppet

5

Tuesday, May 7, 13

The Amazon Cloud

✤ Business Benefits

✤ Offloading heavy lifting

✤ Lowers cost

✤ Reduces time to market

6

Tuesday, May 7, 13

Amazon Data Centers

7

Tuesday, May 7, 13

Amazon Instances

✤ High CPU

✤ High IO

✤ High Memory

8

Tuesday, May 7, 13

Amazon Storage Types

9

Tuesday, May 7, 13

Network and Security

✤ Amazon Security Groups

✤ Keep your Amazon EC2 firewall permissions as restrictive as possible.

✤ Elastic IP Address (EIP)

10

Tuesday, May 7, 13

Deployment Strategies

✤ Sizing Instances and Storage Types

✤ Configuration Management

✤ High Availability

✤ Replication

✤ Failover

✤ Operations

✤ Upgrade, Backups, Maintenance tools

✤ Monitoring11

Tuesday, May 7, 13

Sizing Instances/Storage

✤ Understand your database , schema and data

✤ Env 1: 50GB , Env 2: 25GB

✤ 2 large archival tables

✤ Working Set about 20-30GB in total

✤ Both environments are inter-related.

✤ Decided to consolidate both environment into single cluster

12

Tuesday, May 7, 13

Sizing Instances/Storage

✤ m2.2xlarge: 34.2GB RAM, 4 Virtual Core

✤ Storage

✤ EBS

✤ RAID 10

✤ Not EBS optimized ( Cost Savings!)

✤ 850 GB

13

Tuesday, May 7, 13

Deployment Phase #1

us-east-1AV1

AV2

us-west-1

AppServers db-master db-slave-

backup

db-slaveStreaming Replication

WAL shipping (OmniPITR)

14

Tuesday, May 7, 13

Configuration Management

✤ Automation

✤ Repeatability

✤ Maintainability

✤ Security

15

Tuesday, May 7, 13

Configuration Management

16

Tuesday, May 7, 13

Puppetizing Postgres

✤ DB server role

✤ PostgreSQL installation

✤ Installing common tools

✤ 3 Roles : prod-master role , prod-slave role, prod-slave-backup role

✤ PostgreSQL Configuration Files: postgresql.conf, pg_hba.conf

✤ Scripts, crontabs

17

Tuesday, May 7, 13

High Availability

✤ Replication

✤ Failover

18

Tuesday, May 7, 13

Replication

✤ Streaming replication

✤ WAL shipping , OmniPITR

✤ Different AV zones

✤ At least 1 slave in different Region

19

Tuesday, May 7, 13

Failover

✤ EIP - about 3 minutes to propagate changes after failover

✤ Failover tools

✤ Semi-automated script

✤ pgbouncer

✤ puppet

20

Tuesday, May 7, 13

Deployment Phase #2

us-east-1AV1

AV2

us-west-1

AppServers

db-masterdb-slave-backup

db-slaveStreaming Replication

WAL shipping (OmniPITR)

pgbouncer21

Tuesday, May 7, 13

pgbouncer

✤ Session Pooling

✤ Supports all Postgres features

✤ Transactions Pooling

✤ Does NOT support PREPARE transactions

22

Tuesday, May 7, 13

Operations

23

Tuesday, May 7, 13

Database Upgrade

✤ Upgraded from PostgreSQL 9.0 to PostgreSQL 9.2

✤ Minimal outage upgrade using Mimeo

✤ http://www.pateldenish.com/2012/10/an-easy-way-to-reduce-outage-window-for-postgresql-upgrade.html

24

Tuesday, May 7, 13

Beware!!

✤ Sept 24th, 2012

✤ PostgreSQL 9.1.0 to 9.1.5 & 9.2.0

✤ VACUUM all tables in your database & Rebuild Indices

✤ http://wiki.postgresql.org/wiki/20120924updaterelease

✤ April 4th, 2013

✤ PostgreSQL 9.0.* to 9.1.* & 9.2.*

✤ Security release

✤ http://www.postgresql.org/support/security/faq/2013-04-04/25

Tuesday, May 7, 13

Backups

✤ Online Backups

✤ Daily, db-slave , OmniPITR

✤ Test the backups!!!!

✤ Schema Backups

✤ pg_extractor

✤ Offsite Backup

✤ sending backups to Amazon S326

Tuesday, May 7, 13

Tools

✤ tracking schema in git : pg_extractor

✤ pg_bloat_report

✤ Pg log errors reporting : tail_n_mail

✤ Log analysis: pgbadger

✤ log_min_duration_statement = 50 ms

✤ log_line_prefix= '%t [%r] [%p]: [%l-1] user=%u,db=%d,e=%e '27

Tuesday, May 7, 13

Monitoring

28

Tuesday, May 7, 13

Database Checks

29

Tuesday, May 7, 13

Database Checks

30

Tuesday, May 7, 13

System Checks

31

Tuesday, May 7, 13

Business Metrics

32

Tuesday, May 7, 13

Histograms!

33

Tuesday, May 7, 13

References

✤ http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Storage.html

✤ OmniTI labs tools

✤ https://github.com/omniti-labs/pgtreats

✤ OmniPITR

✤ https://github.com/omniti-labs/omnipitr

34

Tuesday, May 7, 13