my experience with embedding postgresql

© 2013 VMware, Inc. All rights reserved

My  experience  with  embedding  PostgreSQL  

Postgres  Conf  EU,  Dublin,  Ireland  October  2013  

 Jignesh  Shah,    

Sr  Manager,  VMware  

© 2013 VMware, Inc. All rights reserved

About  me  

•  Sr  Manager,  VMware      Data  –  Leads  Postgres  team  at  VMware  –  Blog  at  hKp://jkshah.blogspot.com    

•  Previously  Staff  Engineer,  Sun  Microsystems  –  Team  Member  of  the  first  published  SpecJAppServer  2004  benchmark  with  PostgreSQL  

–  Performance  of  PostgreSQL  on  Solaris/Sun  systems  

•  Working  with  PostgreSQL  circa  2005  

© 2013 VMware, Inc. All rights reserved

Agenda  

•  Business  reasons  for  embedding  database  •  Technical  architecture  journey  

© 2013 VMware, Inc. All rights reserved

The  IniXal  Problem  

•  >30    #  products  and  product  components  •  >20      requires    DB  

•  Customers  have  to  get  their  own  DB  license    •  Increases  the  Total  Cost  of  Deployment  

– As  database  load  increases,  CPU  cores  required  increases,  so  does  the  total  license  cost  of  DB  

© 2013 VMware, Inc. All rights reserved

SQL  Server  Express  

•  Currently  shipped  for  some  Windows  based  products  – However  limits  not  enough    – Max  10GB  database  usage  

•  Standard  and  Enterprise  EdiXons  get  expensive  preKy  quickly  

•  Not  Available  on  Linux    

© 2013 VMware, Inc. All rights reserved

Proposal:  PostgreSQL?  

•  PostgreSQL  is  free  and  open  sourced  •  Long  history  of  stability  •  Friendly  BSD  License  

•  Plus  I  am  at  a  Postgres  Conference  J  

© 2013 VMware, Inc. All rights reserved

IniXal  External  Feedback  

•  Actually  got  pushback  from  customers,  partners  –  PostgreSQL  not  part  of  their  database  standards  – No  in-­‐house  PostgreSQL  DBA  skills  –  Lack  of  integrated  tools  ,  monitor,  ease-­‐of-­‐use  –  Large  enterprises  have  to  convince  their  managed  hosXng  partners  to  support  the  database  also  

•  AKendees  here  are  going  to  be  educated  about  it    –  but  we  are  worried  about  the  rest  of  the  world  

© 2013 VMware, Inc. All rights reserved

IniXal  Internal  Feedback    

•  DB  Support  Matrix  complexity  increases  – Every  new  DB  support  increases  N  plaborm  test  – Large  scale  tests  are  serialized  due  to  resources  

© 2013 VMware, Inc. All rights reserved

Plan:  Embedded  Default  DB  

•  Embed  PostgreSQL  as  part  of  the  product  •  Make  it  the  default  choice  opXon  for  most  deployment  

•  Do  not  expose  database  directly  to  endusers  •  ApplicaXon  will  manage  the  DB  •  Support  it  for  producXon  deployment  

•  Customer  can  sXll  select  to  use  an  external  database  of  their  choice  or  save  DB  License  cost  with  the  embedded  one  

© 2013 VMware, Inc. All rights reserved

Ground  Work:  MigraXon  

•  Products  used  various  databases  – SQL  Server  – Oracle    – MySQL  

•  Few  had  about  1500  stored  procedures  •  Some  used  exoXc  features  

•  For  example    MySQL  used  InMemory  Engine,  InnodB  engine,  MyISAM  engine  in  the  same  workload  

© 2013 VMware, Inc. All rights reserved

MigraXon  

•  Leveraged  contractors  to  migrate  stored  procedures  

•  Performance  bake-­‐off  tests  to  convince  product  teams  

•  Integrated  into  the  build  infrastructure  to  ease  pulling  components  

© 2013 VMware, Inc. All rights reserved

Example:  vCenter  Server  Appliance  

© 2013 VMware, Inc. All rights reserved

Plan:  Backup  &  Recovery  •  DB  not  the  only  thing  to  be  backed  up  •  ConfiguraXon  files  •  ApplicaXon  Status  

•  All  data  to  be  backed  up  went  into  a  virtual  disk  backed  up  with  vADP  

•  pg_dump  /  pg_restore  used  in  few  cases  

© 2013 VMware, Inc. All rights reserved

IniXal  Embedded  DB  Deployments  

•  Each  Product  team  is  different  –   has  different  levels  of  DB  Development  skills  

•  Each  architect  comes  up  with  their  own  design    – Postgres  Service  designed  (Start/Stop)  – own  DB  Layout  design  – own  DB  Security  design  – Some  Tuned  their  DB  – Many  did  not  

© 2013 VMware, Inc. All rights reserved

Early  Results  

•  YES  !  Customer  started  seeing  benefit  •  However  

–  Experience  of  PostgreSQL  different  on  each  product  –  Random  security  holes  idenXfied  due  to  lack  of  knowledge  of  PostgreSQL  nuances  

–  Performance  configuraXon  was  an  issue  – Upgrade  to  new  versions  of  PostgreSQL  was  an  akerthought  (What  no  in-­‐place  upgrade?)  

–  Legal  work  required  for  OSS  increased  –  Resulted  in  PostgreSQL  version  sXckiness  

© 2013 VMware, Inc. All rights reserved

Some  encounters  

•  Portable  binaries  –  Shrink  wrapped  sokware  (not  compiled  on  same  systems)  – OSS  Libraries  used  for  compiling  PostgreSQL  have  to  be  shipped    

– Manage  security  patches  for  OSS  Libraries  

•  Need  Standard  drivers  like  ODBC,  JDBC  

© 2013 VMware, Inc. All rights reserved

MulX-­‐Plaborm  Requirement  

•  As  company  grew,  acquisiXons  grew  •  Some  products  only  worked  on  Windows  •  Few  used  exclusively  .NET  framework  

© 2013 VMware, Inc. All rights reserved

SoluXon:  Embedded  DBaaS  

•  Provide  End  to  End  Experience  with  PostgreSQL  

© 2013 VMware, Inc. All rights reserved

InstallaXon  

•  Extend  installaXon  to  setup  a  service  •  Configurable  to  set  up  as  internal  service  or  external  service  (mostly  embedded)  

•  Security  handled  as  part  of  service  setup  to  follow  best  pracXcesfor  embedded  

•  Security  based  on  access  and  minimum  roles  defined  

© 2013 VMware, Inc. All rights reserved

InstallaXon  

•  RPM    installaXon  – Dependent  OSS  Libraries  to  make  RPM  portable  –  Server  and  client  RPMs  –  Service  Setup  configurator  to  setup  a  default  service  

•  User  setup  •  initdb  setup  •  pg_hba  setup    •  Password  handling  

–  Provide  tools  to  quickly  setup  DB  Owner,  DB  for  various  users  of  the  service  

© 2013 VMware, Inc. All rights reserved

Installer  on  Windows  

•  PostgreSQL  Installer  on  Windows  •  Service  setup  on  Windows  •  ODBC  Support    •  Npgsql  Support  for  .NET  Plaborms  

© 2013 VMware, Inc. All rights reserved

Backup  &  Recovery  

•  While  other  backups  strategy  are  available  – Nothing  beats  standard  database  backups  – Design  cron  jobs  for  criXcal  data    – Design  Archive  On  Demand  for  semng  up  replicaXon  quickly  

– Most  customer  do  not  want  to  hack  with  backup/restore  scripts.    

– Should  work  out  of  the  box  

© 2013 VMware, Inc. All rights reserved

Upgrades  

•  Seamless  upgrade  to  new  version  – RPM  Upgrades  needs  to  be  transparent  for  minor  versions  

– RPM  upgrades  for  major  version  needs  to  migrate  data  from  the  older  version  in  the  default  service  

© 2013 VMware, Inc. All rights reserved

Upgrades  

•  Problems  sXll  not  completely  resolved  – Reduce  downXme  for  major  release  upgrades  – Generally  migraXon  since  in-­‐place  upgrade  does  not  provide  ways  to  roll  back  the  upgrade  

© 2013 VMware, Inc. All rights reserved

Performance  

•  Default  PostgreSQL  semng  are  conservaXve  •  Most  deployments  use  default  values  •  Started  doing  auto-­‐tuning  

– Modified  PostgreSQL  core  engine  with  intelligence  – For  checkpoint  segments,    shared_memory  

•  Added  tool  to  auto  tune  postgresql.conf    – Based  on  memory  share  for  DB  Service  – Product  workload  dependent  

© 2013 VMware, Inc. All rights reserved

Performance  related  tools  

•  Provided  tools  to  understand  what’s  happening  with  Postgres  – pg_top  with  changes  

•  Changes  already  contributed  back  to  community  

© 2013 VMware, Inc. All rights reserved

Example:  pg_top  

© 2013 VMware, Inc. All rights reserved

Performance  related  future  work  

•  ExisXng  tools  sXll  meant  for  expert  users  •  Lot  of  demand  for  simple  performance  reports  •  Looking  into  pg_statspack  

© 2013 VMware, Inc. All rights reserved

Availability  

•  Created  a  watchdog  for  PostgreSQL  to  restart  PostgreSQL  if  database  cannot  be  accessed  for  long  

•  If  restart  fails  then  it  can  throw  alerts  

•  Also  created  plugins  for  monitoring  the  service  

© 2013 VMware, Inc. All rights reserved

Related  Problems:  Crashes  

•  PostgreSQL  started  throwing  errors  with  “Out  of  Memory”  errors  in  stress  tests    

•  MulXple  applicaXons  can  fight  for  memory  •  Made  Performance  semngs  less  aggressively  

© 2013 VMware, Inc. All rights reserved

Supportability  

•  When  customers  hit  a  problem  – Hard  to  get  access  to  the  system  

•  Created  support  package  tool  to  collect  relevant  informaXon  – Collect  config,  status  in  a  tar  file  for  analysis  

© 2013 VMware, Inc. All rights reserved

Usability  

•  Experts  love  command  line  uXliXes  •  Novices  love  GUI,  WebUI  •  It  needs  to  match  the  other  GUIs  VMware  ship  

© 2013 VMware, Inc. All rights reserved

Pluggable  UI  

•  Matching  VMware’s  GUI  Plaborm  

© 2013 VMware, Inc. All rights reserved

Embedded  DBaaS  Architecture  

vSphere  Plaborm  

Linux  Based  VM  

PostgreSQL  

Service    

Configurator  

Perf-­‐Tuner    

Pg_top  

ODBC  

JDBC  

WebUI  Watchdog  

Monitoring  Support  

package  tool  

VMware  

Product  

© 2013 VMware, Inc. All rights reserved

Virtual  Appliances  

© 2013 VMware, Inc. All rights reserved

Virtual  Appliances  

© 2013 VMware, Inc. All rights reserved

ReplicaXon  between  vApps  

•  Provide  simple  uXliXes  to  quickly  – archive_command  – create_replicaXon_user  –  run_as_replica  – show_replica_status  – promote_replica_to_primary  

© 2013 VMware, Inc. All rights reserved

In  Closing  

•  Free  Sokware  is  nice  •  Experience  –  What  business  users  care  

– User  Experience  – DBA  Experience  – Support  Experience  – etc  

•  Improve  the  Experience  with  the  technology  

© 2013 VMware, Inc. All rights reserved

Thanks.  QuesXons?  

Follow us on twitter: @vPostgres

Data Platform Blog: http://blogs.vmware.com/dataplatforms

Download  VMware  vFabric  Postgres    

hKp://vmware.com/go/download-­‐vfabric-­‐postgres