my experience with embedding postgresql

© 2013 VMware, Inc. All rights reserved

My experience with embedding PostgreSQL

Postgres Conf EU, Dublin, Ireland October 2013

Jignesh Shah,

Sr Manager, VMware


About me

•  Sr Manager, VMware Data –  Leads Postgres team at VMware –  Blog at hKp://jkshah.blogspot.com

•  Previously Staff Engineer, Sun Microsystems –  Team Member of the first published SpecJAppServer 2004 benchmark with PostgreSQL

–  Performance of PostgreSQL on Solaris/Sun systems

•  Working with PostgreSQL circa 2005


Agenda

•  Business reasons for embedding database •  Technical architecture journey


The IniXal Problem

•  >30 # products and product components •  >20 requires DB

•  Customers have to get their own DB license •  Increases the Total Cost of Deployment

– As database load increases, CPU cores required increases, so does the total license cost of DB


SQL Server Express

•  Currently shipped for some Windows based products – However limits not enough – Max 10GB database usage

•  Standard and Enterprise EdiXons get expensive preKy quickly

•  Not Available on Linux


Proposal: PostgreSQL?

•  PostgreSQL is free and open sourced •  Long history of stability •  Friendly BSD License

•  Plus I am at a Postgres Conference J


IniXal External Feedback

•  Actually got pushback from customers, partners –  PostgreSQL not part of their database standards – No in-‐house PostgreSQL DBA skills –  Lack of integrated tools , monitor, ease-‐of-‐use –  Large enterprises have to convince their managed hosXng partners to support the database also

•  AKendees here are going to be educated about it –  but we are worried about the rest of the world


IniXal Internal Feedback

•  DB Support Matrix complexity increases – Every new DB support increases N plaborm test – Large scale tests are serialized due to resources


Plan: Embedded Default DB

•  Embed PostgreSQL as part of the product •  Make it the default choice opXon for most deployment

•  Do not expose database directly to endusers •  ApplicaXon will manage the DB •  Support it for producXon deployment

•  Customer can sXll select to use an external database of their choice or save DB License cost with the embedded one


Ground Work: MigraXon

•  Products used various databases – SQL Server – Oracle – MySQL

•  Few had about 1500 stored procedures •  Some used exoXc features

•  For example MySQL used InMemory Engine, InnodB engine, MyISAM engine in the same workload


MigraXon

•  Leveraged contractors to migrate stored procedures

•  Performance bake-‐off tests to convince product teams

•  Integrated into the build infrastructure to ease pulling components


Example: vCenter Server Appliance


Plan: Backup & Recovery •  DB not the only thing to be backed up •  ConfiguraXon files •  ApplicaXon Status

•  All data to be backed up went into a virtual disk backed up with vADP

•  pg_dump / pg_restore used in few cases


IniXal Embedded DB Deployments

•  Each Product team is different –  has different levels of DB Development skills

•  Each architect comes up with their own design – Postgres Service designed (Start/Stop) – own DB Layout design – own DB Security design – Some Tuned their DB – Many did not


Early Results

•  YES ! Customer started seeing benefit •  However

–  Experience of PostgreSQL different on each product –  Random security holes idenXfied due to lack of knowledge of PostgreSQL nuances

–  Performance configuraXon was an issue – Upgrade to new versions of PostgreSQL was an akerthought (What no in-‐place upgrade?)

–  Legal work required for OSS increased –  Resulted in PostgreSQL version sXckiness


Some encounters

•  Portable binaries – Shrink wrapped sokware (not compiled on same systems) – OSS Libraries used for compiling PostgreSQL have to be shipped

– Manage security patches for OSS Libraries

•  Need Standard drivers like ODBC, JDBC


MulX-‐Plaborm Requirement

•  As company grew, acquisiXons grew •  Some products only worked on Windows •  Few used exclusively .NET framework


SoluXon: Embedded DBaaS

•  Provide End to End Experience with PostgreSQL


InstallaXon

•  Extend installaXon to setup a service •  Configurable to set up as internal service or external service (mostly embedded)

•  Security handled as part of service setup to follow best pracXcesfor embedded

•  Security based on access and minimum roles defined


InstallaXon

•  RPM installaXon – Dependent OSS Libraries to make RPM portable –  Server and client RPMs –  Service Setup configurator to setup a default service

•  User setup •  initdb setup •  pg_hba setup •  Password handling

–  Provide tools to quickly setup DB Owner, DB for various users of the service


Installer on Windows

•  PostgreSQL Installer on Windows •  Service setup on Windows •  ODBC Support •  Npgsql Support for .NET Plaborms


Backup & Recovery

•  While other backups strategy are available – Nothing beats standard database backups – Design cron jobs for criXcal data – Design Archive On Demand for semng up replicaXon quickly

– Most customer do not want to hack with backup/restore scripts.

– Should work out of the box


Upgrades

•  Seamless upgrade to new version – RPM Upgrades needs to be transparent for minor versions

– RPM upgrades for major version needs to migrate data from the older version in the default service


Upgrades

•  Problems sXll not completely resolved – Reduce downXme for major release upgrades – Generally migraXon since in-‐place upgrade does not provide ways to roll back the upgrade


Performance

•  Default PostgreSQL semng are conservaXve •  Most deployments use default values •  Started doing auto-‐tuning

– Modified PostgreSQL core engine with intelligence – For checkpoint segments, shared_memory

•  Added tool to auto tune postgresql.conf – Based on memory share for DB Service – Product workload dependent


Performance related tools

•  Provided tools to understand what’s happening with Postgres – pg_top with changes

•  Changes already contributed back to community


Example: pg_top


Performance related future work

•  ExisXng tools sXll meant for expert users •  Lot of demand for simple performance reports •  Looking into pg_statspack


Availability

•  Created a watchdog for PostgreSQL to restart PostgreSQL if database cannot be accessed for long

•  If restart fails then it can throw alerts

•  Also created plugins for monitoring the service


Related Problems: Crashes

•  PostgreSQL started throwing errors with “Out of Memory” errors in stress tests

•  MulXple applicaXons can fight for memory •  Made Performance semngs less aggressively


Supportability

•  When customers hit a problem – Hard to get access to the system

•  Created support package tool to collect relevant informaXon – Collect config, status in a tar file for analysis


Usability

•  Experts love command line uXliXes •  Novices love GUI, WebUI •  It needs to match the other GUIs VMware ship


Pluggable UI

•  Matching VMware’s GUI Plaborm


Embedded DBaaS Architecture

vSphere Plaborm

Linux Based VM

PostgreSQL

Service

Configurator

Perf-‐Tuner

Pg_top

ODBC

JDBC

WebUI Watchdog

Monitoring Support

package tool

VMware

Product


Virtual Appliances


ReplicaXon between vApps

•  Provide simple uXliXes to quickly – archive_command – create_replicaXon_user –  run_as_replica – show_replica_status – promote_replica_to_primary


In Closing

•  Free Sokware is nice •  Experience – What business users care

– User Experience – DBA Experience – Support Experience – etc

•  Improve the Experience with the technology


Thanks. QuesXons?

Follow us on twitter: @vPostgres

Data Platform Blog: http://blogs.vmware.com/dataplatforms

Download VMware vFabric Postgres

hKp://vmware.com/go/download-‐vfabric-‐postgres

my experience with embedding postgresql

Technology