integrating hybrid cloud database-as-a-service with cloud foundry’s service broker

© 2013 EDB All rights reserved 8.1. 1

Integration:EDB Postgres Cloud Management & Pivotal Cloud Foundry Lenley Hensarling

SVP Product Management EnterpriseDB


• The “why” and what we are trying to accomplish

• EDB Postgres Cloud Management – a DBaaS enablement console

• The Integration with Pivotal Cloud Foundry− Shared instance – setup and binding to an existing EDB Postgres DB− Private instance – provisioning of a Postgres cluster from within CF

using EDB Postgres Cloud Management

What we’ll cover


• For many enterprises there is a tension between IT and new efforts in app development for digital business initiatives

− Inventory of licenses− Control of resources versus enablement of agile development− The need for fluid and agile development motions

• We want to bridge that divide− Allow the IT side of the house visibility− Give them the ability to set up templates and control access− But not slow things down

• We want to give developers, quality engineers, and ops the ability to manage all of this in a hybrid cloud environment without lock-in

Why and what we are trying to accomplish


Support for Continuous Delivery in Dev/Ops


Cloud Management Architecture


EDB Cloud Management Architecture - Where we are going

PPCD Server

Cloud Controller

JCloud API

Cloud Cluster Manager

Configuration Manager

AWS Provider OpenStack Provider

OpenStack APIEC2 API

PPCD DB

Futures

VSphereProvider

VSphere API

pgPool

Master

PPAS 9.x

Node Manager

Replica 1

PPAS 9.x

Node Manager

Replica n

PPAS 9.x

Node Manager

GCPProvider

GCPAPI

AzureProvider

AzureAPI

Command Line I/FRESTful API

GUI Console


Quality Management

Production

Developers

TestDevelopment

Users

Operations / DBAs

One-click provisioning of database clusters with backup / restore and high availability

Define templates on size and capability, manage backups and restores, monitor databases, manage license inventory.

Cloud Management – enabling agility for development and deployment of apps while providing visibility and control for operations and DBAs


Console for use by DBAs & Operations

1

23

4

7

6

5

8

1 Scale replicas up / down2 Clone cluster3 Upgrade Postgres version4 Scale server class5 Server DNS / Port & Server statuses6 Cluster Healing or Failover modes7 User selectable scaling thresholds8 DB Monitoring & event log 9 Cluster statuses

9


• Any user with appropriate permissions can create a cluster.

• Specify name, type of DB version & server class.

• Choose which virtual network and IP pool to use.

• Configure number of nodes and storage size / encryption.

• Optionally choose to ‘yum update’ server after initial install.

• Specify Database username, password and notification info.

• Configure backup settings.

Self Service Create a New Cluster


• Snapshot based backups with Continuous WAL Archiving for PITR.

• View list of backups and their properties.

• Choose to restore a backup or remove backups no longer needed.

Manage and Use Backups


• View details about storage, connections and CPU load for the cluster

− Choose the time range you want to view

− Blue line shows utilization, red line shows threshold for adding more capacity

• View history of events that have occurred for the logged in user or the chosen cluster

Monitor System Usage and Event Log


• When you expect heavy read traffic or large reporting runs, you can manually add new read replicas in advance.

• Add new storage before a bulk load process.

• If your master database RAM or CPU requirements change, update to a larger or smaller virtual machine for your database.

Manual Scaling


• Load balancing is enabled by default on port 9999.

− We use pgPool and streaming replication.

• Read queries (most SELECT, COPY TO, DECLARE…) may be directed to any node.

− Will check log location on standbys to ensure they are up to date before routing queries.

• Transactions or Writes (SELECT that follows an INSERT, EXPLAIN, INSERT, UPDATE…) are directed to master node.

• Selective enforcement− /*IgnoreLB*/ SELECT * FROM emp;

− black_list & white_list

• Automatic monitoring and restart of load balancer upon failure.

Load Balancing


Cloud Management RESTful API

• A full set of management and provisioning primitives

− JSON compatible API with token based authentication

• Capabilities include:− Download Server Logs− View and Update Wall information− Get a list of clusters for the system

or a given user− Create and manage clusters− Manage users, server images and

db engines


• Pushbutton provisioning of read scalable database clusters− Streaming replication between master and replicas− Connection pooling handle more users over time− Load-balancing of READS over the whole cluster− SQL Routing for WRITES to master with white and black listing of SQL− Start, Connect and code in minutes instead of days or weeks!

• Automatic failover − Constant health and stats monitoring− Transaction preferred failover prevents loss of all committed transactions− Recovery Time preferred failover prioritizes recovery time with possible loss of last committed

transactions− Cluster configuration fully restored after failover− Unattended operation - No special DBA or Administrative skills required

Benefits of EDB Postgres Cloud Management


• What allows our integration with EPCM are the RESTful APIs of EPCM

• From the service broker a developer can gain access to Postgres DBs− provision data bases from within Cloud Foundry using templates set up by the data services

group− connect to existing databases that were set up by the data services group through EPCM or

were set up by another developer

• Developers and Quality Management engineers can provision populated databases by restoring a database backup to any instance provided they have rights to access them

• Supports a natural flow – allowing control and visibility for data services and operations and unimpeded development flow for apps developers

The Integration


• Shared instance− Connect to a running Postgres instance and provision a new database within it− Simple, available now

• In our labs and coming this fall− Integration with EPCM− Allows provisioning of a new database instance− Allows for cloning of database instance, a template with clone of the data. − Allows for attaching to different data sets for testing by restoring from various

“backups”− Allows for development on AWS and provisioning in a private cloud

− Or vice versa

The Integration


Cloud Foundry Service Broker


Marketplace


Integration with Cloud Foundry


Configuring the EDB Postgres Instance


Service plan instance


Binding to app


• Developer and Quality Management Engineers− Can provision instances of different sizes from within Cloud Foundry− Can restore a data set to use for testing and/or for initial schema set up− Can clone instances and the data volumes

• Ops can bind the application to an existing database for production

• EPCM bridges the gap of visibility and control of license instances and resource utilization for DBAs

• While allowing developers to set up instances and they are needed, all with a few clicks

• Test can access plans for functional test and for performance and scalability as required

With the addition of EPCM


• We are live with our initial shared instance integration on Pivotal Apps Manager Marketplace right now

• Integration with EPCM is there in our lab and can be demoed – just contact us for a deeper dive and demo

• We will release the integration to CF based on our EDB Postgres Cloud Manager this fall

− It will support both OpenStack and AWS− More targets to come in the first half of 2017

− Azure− On-prem virtualized environments− Google Compute Platform

Timeline\ for EDB / Cloud Foundry integration


• EPCM 2.0 Technical Overview on YouTube− https://www.youtube.com/watch?v=-1QiWocUY6U

• You can get to our Tile on Pivotal Marketplace− https://network.pivotal.io

Resources and additional info

https://www.youtube.com/watch?v=-1QiWocUY6U

https://www.youtube.com/watch?v=-1QiWocUY6U

https://network.pivotal.io/

integrating hybrid cloud database-as-a-service with cloud foundry’s service broker

Technology