best practices for a complete postgres enterprise architecture setup
TRANSCRIPT
© 2015 EnterpriseDB Corporation. All rights reserved. 1
Best Practices for a Complete Postgres Enterprise Architecture Setup
To watch to the presentation visit - www.EnterpriseDB.com > Resources > On-Demand Webcasts
© 2015 EnterpriseDB Corporation. All rights reserved. 2
• Introduction to EnterpriseDB • Enterprise Data Management Architecture
− OLTP Infrastructure − High Availability − Disaster Recovery − Data Integration − Monitoring and Management − Security
• Summary and Resources • Q&A
Agenda
We will take two quick polls during the webinar to gather group data on best practices of interest
© 2015 EnterpriseDB Corporation. All rights reserved. 3
POSTGRES innovation
ENTERPRISE reliability
24/7 support
Services & training
Enterprise-class features, tools &
compatibility
Indemnification
Product road-map
Control
Thousands of developers
Fast development
cycles
Low cost
No vendor lock-in
Advanced features
Enabling commercial adoption of Postgres
© 2015 EnterpriseDB Corporation. All rights reserved. 4
EDB Customers EDB currently has over 2,500 total customers including 50 of the Fortune 500 and 98 of the Forbes Global 2000
© 2015 EnterpriseDB Corporation. All rights reserved. 5
EDB is a Industry Leader Magic Quadrant for Operational DBMS, Q4 ‘14
• Gartner Comments − “EnterpriseDB is responsible
for many features of PostgreSQL, contributing to JSON, materialized views and partitioning.” − “Clients report that the
functionality of EnterpriseDB's Postgres Plus Oracle Compatibility Feature is now more than sufficient to run both mission-critical and non- mission-critical applications.” − “Customers commend the
compatibility with Oracle, the stability of the DBMS and the product support.”
© 2015 EnterpriseDB Corporation. All rights reserved. 6
EDB is an Open Source Community Leader
Amit Kapila
Ashesh Vashi
Bruce Momjian
Dave Page
Devrim Gunduz
Jan Wieck
Kevin Grittner
Korry Douglas
Muhammad Usama
Robert M Haas
Thom Brown
© 2015 EnterpriseDB Corporation. All rights reserved. 7
Where Do Users Need Help # Tickets Phase Category POC Dev Deployment Maintenance Grand Total Connectors 0% 2% 1% 0% 3%
Bug 0% 0% 0% 0% 1% How to 0% 1% 0% 0% 1% Product Awareness 0% 1% 0% 0% 1%
Database 10% 9% 17% 37% 73% Bug 1% 2% 1% 2% 6% Corruption 0% 0% 0% 2% 2% Enhancements 0% 0% 0% 0% 1%
How to 4% 3% 6% 10% 23% Product Awareness 4% 3% 8% 15% 31% Tuning 0% 0% 1% 7% 9%
Replication 1% 4% 4% 2% 12% Bug 0% 1% 0% 0% 2% How to 0% 0% 3% 1% 4% Product Awareness 1% 2% 1% 0% 4% Tuning 0% 0% 0% 1% 1%
Utilities 1% 3% 6% 3% 12% Bug 0% 1% 1% 0% 2% Enhancements 0% 0% 0% 0% 0% How to 1% 1% 3% 1% 6% Product Awareness 0% 1% 3% 1% 5%
Grand Total 12% 17% 28% 42% 100%
Questions focus on deploying, supporting
and maintaining Postgres in the
Enterprise
© 2015 EnterpriseDB Corporation. All rights reserved. 9
• Effective data center implementations use a reference architecture to: − Accelerate the implementation of a solution − Lower operational costs − Build-in flexibility − Enhance performance − Lower complexity − Reduce risk − Avoid the ‘OMG – MTP’ moment!
• EnterpriseDB Postgres reference architecture − modular and standards-based − supports a wide range of enterprise deployments
Why?
OMG! Tomorrow is
MTP!
© 2015 EnterpriseDB Corporation. All rights reserved. 10
• OLTP Infrastructure − ACID Compliant − High Performance − Scalable − Reliable
• High Availability − Protect against hardware
failure − Protect against software failure
• Disaster Recovery − Protect against site failure − Protect against operator error
• Data Integration − Integration with other Line of
Business systems − Heterogeneous integration with
Microsoft, Oracle, etc
• Monitoring and Management − Capacity Planning − Event management and
alerting
• Security − Authentication and
Authorization − Encryption and data protection
Key Components of a Enterprise Data Management Architecture
© 2015 EnterpriseDB Corporation. All rights reserved. 11
? Tool and Infrastructure Choices
Hig
h Av
aila
bilit
y Monitoring and Management
Disaster R
ecovery
Data Integration
Open PostgreSQL Monitoring
pgAdmin phpPgAdmin
repmgr HandyRep
PgHA
Slony Bucardo
Londiste BDR
BARMan pg_rman
TeamPostgreSQL OmniPITR
Circonus pgBadger
pgFouine
powa
pitrery wal-e
• Multiple license models (PostgreSQL, GPL, AGPL) • Multiple vendors with different SLAs • Overlap and gaps
check_postgres
PITRTools
Nagios XI wizard
© 2015 EnterpriseDB Corporation. All rights reserved. 12
EDB Integrated Postgres Portfolio
• OLTP Infrastructure − Postgres Plus Advanced
Server PPAS − Oracle Compliant − Enhanced partitioning, security
and tooling (Profiler, Index Advisor, Runtime Statistics)
• High Availability − EDB Failover Manager − Robust, cost effective,
commodity hardware
• Disaster Recovery − EDB BART (Backup and
Recovery Tool) − Native Streaming Replication
• Data Integration − xDB Replication (Single
Master and Multi Master) − Heterogeneous integration with
Oracle, MS SQL Server and Postgres
• Monitoring and Management − Postgres Enterprise Manager − Enterprise level management,
monitoring and alerting for database and infrastructure
• Security − PPAS Enhancements − Virtual Private Database (VPD) − Code protection − SQL Injection Attack Guard
© 2015 EnterpriseDB Corporation. All rights reserved. 13
PEM Client
Oracle/SQL Server/PG
Rep
licat
ion
DB Master Witness Offsite Replica
Virtual IP
DB Client
Backup & Recovery
PEM Server High Availability
Disaster Recovery HA Replica
Read/HA Replica
Robust, Resilient, Scalable Postgres
Enterprise Architecture
Mon
itorin
g
Dat
a In
tegr
atio
n
Simple, development
focused setup
© 2015 EnterpriseDB Corporation. All rights reserved. 15
Security
Tools
Performance
Compatibility
ADVANCED SERVER
POSTGRES
PLUS
© 2015 EnterpriseDB Corporation. All rights reserved. 16
Postgres Plus Advanced
Server
Resource Manager
(CPU & I/O)
Reporting
Transactions
80%
20%
PPAS 9.4 Resource Manager
• DBA assigns CPU & I/O to job groups
• Allocates and prioritizes consumption of resources
• Low priority jobs don’t hurt high priority jobs
© 2015 EnterpriseDB Corporation. All rights reserved. 17
SQL Performance Management
− SQL Profiler captures a SQL workload and locates the worst running SQL
− Both ad-hoc and scheduled operations supported
− Provides a number of SQL-based performance metrics along with EXPLAIN analysis
− Integrates with Index Advisor, which analyzes SQL statements and recommends new indexes to improve performance
− EDB SQL Profiler & Index advisor are components of EDB Postgres Enterprise Manager (PEM)
• Poorly optimized SQL code is the NUMBER ONE cause of database problems—EDB SQL Profiler & Index Advisor tools can help:
© 2015 EnterpriseDB Corporation. All rights reserved. 18
• Types of storage systems − Direct Attached Storage (DAS) − Storage Area Network (SAN) w. Fiber Channel − Storage Area Network (SAN) w. iSCSI − Network File System (NFS)
• DAS: fast and low latency • SAN: fast, expensive (w. Fiber Channel), scalable, can
include redundancy and smart file system operations • NFS: not an optimal solution for PostgreSQL data files
Caveat Emptor: File System and Storage Subsystem
© 2015 EnterpriseDB Corporation. All rights reserved. 20
• Shared Disk − Red Hat Cluster Server, Veritas Cluster, … − Mostly used for local failover − High $$$
• Native streaming replication-based − Available with Postgres 9.X − Log (WAL)-based, fast and reliable − Replicate the database locally or remote − Synchronous or asynchronous − Hub & spoke or cascading
• EDB Failover Manager (EFM) − Leverages native streaming replication − Integrates ‘Witness’ concept to create quorum and avoid false
alarms
High Availability Options for Postgres
© 2015 EnterpriseDB Corporation. All rights reserved. 21
EDB
EDB Failover Manager creates fault tolerant database clusters to minimize downtime when a master database fails by keeping data online in high availability configurations
FAILOVER MANAGER Master
Streaming Replica
Clients
Witness
Network / Internet
The MASTER is down!
I agree! Take control now!
© 2015 EnterpriseDB Corporation. All rights reserved. 23
Backup and Standby – Reality Check
hot 47%
warm 23%
None 18%
cold 12%
Standby Approach Physical
Only 36%
Logical Only 29%
Physical and
Logical 14%
FS Snapshot
7%
None 14%
Backup Approach 43% of customers could not execute PITR (prior to EDB Architecture Engagement) – some had NO backup in place
30% of customers would have been slow or unable to recover from failure
© 2015 EnterpriseDB Corporation. All rights reserved. 24
• Why do you need backup and recovery? Backup and recovery strategies protect you in case of: − Catastrophic device failure − Site failure − Maintenance − Operator error − Compliance − Data corruption
• Multiple components: − Logical backup provides granularity in objects (tables, table
spaces, databases) − Physical backups provide granularity in time for PITR − Cold/off-line backups − Hot/on-line backup, with WAL archiving
Backup and Recovery Strategies
© 2015 EnterpriseDB Corporation. All rights reserved. 25
• Backup Strategy Framework − Logical backup after structural changes or
major updates of reference/meta data − Physical backup (daily…) − WAL file archiving − Combine backup/recovery and streaming replication − Uses replica to offload hot backup and logical backup
• Consider − Allowable PITR timeframe (MTR – Mean Time to Recover) and
allowable data loss (RPO – Recovery Point Objective) − Data retention policy − Test, test, test − Periodic backup validation
Backup and Recovery Best Practices
© 2015 EnterpriseDB Corporation. All rights reserved. 26
Disaster Recovery for Postgres
• Two components − Offsite replication
− Streaming replication to Disaster Recovery site − Backup
− Logical backup − Physical backup − WAL archiving for PITR
© 2015 EnterpriseDB Corporation. All rights reserved. 27
• Simplifies and reduces errors with a system-wide catalog and command line tool
• Online backup and recovery across local and remote servers
• Local and remote online physical backups
• Auto-compression and MD5 checksum verification
• Continuous WAL Archiving and PITR
• Support for tablespace restoration on different paths
BART Architecture
© 2015 EnterpriseDB Corporation. All rights reserved. 29
• Monitor the health of your server and databases for proactive management and issue prevention
Monitoring and Management
• What should I monitor? • How do I reduce downtime? • How do I know if I need more space? • How to find bloat in tables & indexes?
© 2015 EnterpriseDB Corporation. All rights reserved. 30
Maintenance Highlights from 20+ AHCs
Unused Indexes
39%
Bloated Indexes
30%
Bloated Tables 13%
Missing Indexes
9% Missing
PK 9%
Bloat management (43%) is a key
production problem
specific to Postgres
© 2015 EnterpriseDB Corporation. All rights reserved. 31
• Utilize alerting and event management across statistics − Storage, user activity, connections − I/O, memory analysis − Database size, tablespace size − Session activity - workload, locks, waits
• Perform regular capacity planning − Analyze historical usage statistics of objects − Project the anticipated usage statistics for an object − Collect and analyze metrics for specific:
− Host/operating system − Postgres server − Database − Database object (table, index, function etc.)
Effective Monitoring and Capacity Planning
© 2015 EnterpriseDB Corporation. All rights reserved. 32
General Monitoring Recommendations Description Frequency Alerting Criteria / Parameters monitored Postgres instance is running and connection count
Every minute (i) When postgres instance is not running, servers down (ii) When number of connections reaches percentage of max_connections (iii) When number of connections crosses threshold
Load average, disk space
Every 15 minutes
(i) When load average is above threshold (ii) When disk consumption percentage above threshold (iii) When disk space available is below threshold
Long running and waiting queries in database
Every 10 minutes
(i) When any query is running for longer than threshold (ii) When any query is in state of "idle in transaction" for more than threshold (iii) When any query is waiting for more than threshold
Bloats for tables and indexes
Once a day (i) Total table bloat, highest table bloat (ii) When database tables and/or indexes individually
bloated more than critical threshold
Vacuum maintenance
Once a day
(i) Last vacuum (ii) Last auto-vacuum
Streaming replication sync with primary
Every 5 minutes
(i) When lag detected on streaming cluster exceeds threshold
Standby server lag behind the master by WAL segments Standby server lag behind the master by WAL pages
© 2015 EnterpriseDB Corporation. All rights reserved. 33
• Consumption projection: − At current consumption rates when will I drop below 100MB of disk? − At the consumption rate of the past 3 days when will I drop below 1 GB of disk
space?
• Point in Time activity − Users were complaining of performance problems from 3 to 4 PM today. What
was going on in the system, the database, how many users were connected? − We get a load spike every day at 11. What was happing at the time of the load
spike?
• Root cause for slower query performance − Queries that access the employ table seem to be taking longer and longer.
View the growth of the number of dead tuples in the table, see what kind of scans have been executing against the table (index or sequential)
• Activity monitoring − Which database on the server is the most active? What kind of growth is it
seeing?
Capacity Planning and Management – Use Cases
© 2015 EnterpriseDB Corporation. All rights reserved. 34
• Single management console allows easy visual control • Works with both PostgreSQL and Postgres Plus • Start/stop, configure, define and manage storage, security and database objects via
single graphical console
MONITOR MANAGE TUNE
EDB Postgres Enterprise Manager (PEM)
Mission critical OS and database statistics collection
Monitor real-time alerts
Predefined (200+) and custom alerts via SMTP or SNMP
Predefined & custom at-a-glance global dashboards
Replication monitoring
CRUD operations on all database objects
Bulk operations across multiple servers
Historical reports
Capacity Manager for planning & forecasting
Customizable GUI charts, tables & graphs
Identify poorly running SQL statements
SQL/Profiler to speed up large workloads
Index Advisor to suggest and create indexes
Postgres Expert for best practice enforcement
Tuning Wizard for machine utilization and load profiles
© 2015 EnterpriseDB Corporation. All rights reserved. 35
Unmanaged Hosts
An efficient distributed architecture perfectly suited for managing, monitoring and tuning large numbers of Postgres servers in multiple locations
Monitoring Data
Enterprise Management Connections
PEM Clients or Web Clients
Managed Hosts
with PEM Agents
Enterprise Manager Agent Installed on each managed machine; collects data on OS and database health and operations PEM
Server
Enterprise Manager Server Centralized storage for agent-collected data and client dashboards
Host Database Management
Connection
Enterprise Manager Clients User GUI console with global at-a-glance monitoring dashboards; used to carry out centralized database administration and tuning
Platform-specific and web clients
POSTGRES ENTERPRISE MANAGER (PEM) Architecture
EDB
© 2015 EnterpriseDB Corporation. All rights reserved. 36
Monitor All Your Postgres Databases From One Screen
• Customized global dashboard
• View up/down status of all agents
• Monitor alerts from many servers in one place
• Navigate to Dashboards for further analysis
© 2015 EnterpriseDB Corporation. All rights reserved. 38
• Postgres solutions exist in context with other Line of Business solutions
• Options − Physical data replication – xDB Replication Engine − Virtual data replication – Foreign Data Wrappers − ETL (Extract, Transfer, Load)
Data Integration with Postgres
© 2015 EnterpriseDB Corporation. All rights reserved. 39
Oracle SQL Server PostgreSQL Advanced Server
READ/WRITE
PostgreSQL Advanced Server
Data filtering
Scheduling
READ
MASTER
REPLICA
Flexible § Heterogeneous § Filter only data you need replicated § Continuous or scheduled § Cascading
Multiple Applications § Performance—shift read operations
from master to hot replica Backup § Testing/new development copies § Migrate data to new systems
EDB xDB REPLICATION SINGLE MASTER
Near real-time replication
© 2015 EnterpriseDB Corporation. All rights reserved. 40
xDB Graphical UI: Making Replication Easy
Browsing publications and subscriptions
Setting up new publications
Configuring new publications
Adding criteria
Adding subscriptions
Reviewing results
© 2015 EnterpriseDB Corporation. All rights reserved. 42
Security
• Conceptual Layers − Authentication − Authorization − Auditing − Data Security − SQL Injection Attacks
• Approach − Secure physical access − Network access limitation − Host access limitation − Database access
limitation − Data access limitation
© 2015 EnterpriseDB Corporation. All rights reserved. 43
“By default, PostgreSQL is probably the most security-aware database available ...”
--Database Hacker's Handbook
Postgres Plus Advanced Server Security • Object level privileges assigned to roles and users • Row Level Security (Virtual Private Database) • EAL2 Certification (augmented with ALC_FLR.2), NIPRNet, SIPRNet, JWICS • Stored procedure obfuscation • Protection against SQL injection attacks • Kerberos and LDAP authentication • SQL USAGE privilege and VIEW Security Barriers • SSL communication • Data Level Encryption (AES, 3DES, etc.) • Ability to utilize 3rd party Key Stores in a full PKI Infrastructure • Foundation for full compliance with the strictest of security standards (PCI Data
Security Standard)
© 2015 EnterpriseDB Corporation. All rights reserved. 44
Security Best Practices for PostgreSQL Security Best Practices for PostgreSQL and Postgres Plus Advanced Server
Executive Summary ................................................................3
Introduction ….........................................................................4
Postgres Security Features within the Above Framework ......6
Authentication .........................................................6
Authorization ….......................................................7
Accounting/Auditing ................................................8
Data Security ….......................................................9
SQL Injection Attacks ............................................10
Postgres Plus Advanced Server Security Features ..............11
Further Reading and Useful Links ........................................12
About EnterpriseDB ..............................................................12
© 2013 EnterpriseDB Corporation. All rights reserved. EnterpriseDB and Postgres Plus are trademarks of EnterpriseDB Corporation. Other names may be trademarks of their respective owners. http://www.enterprisedb.com
Page: 2
Table of Contents
Disclaimer The following is intended as an outline of EnterpriseDB’s general product direction. It is intended for informational purposes only, and it should not be relied upon in making purchasing decisions. This information may not be incorporated into any contract. It is not a commitment or obligation on the part of EnterpriseDB to release, launch or deliver any updates, modifications, material, code or functional improvements and may change at EnterpriseDB’s sole discretion.
http://info.enterprisedb.com/rs/enterprisedb/images/Whitepapers_Security_BP_PostgreSQL_and_Postgres_Plus_AS.pdf
© 2015 EnterpriseDB Corporation. All rights reserved. 45
• A successful Postgres implementation is about more than using Postgres
• It includes − High Availability − Disaster Recovery − Backup Strategy − Integration Strategy − Monitoring and Management − Security
Summary
© 2015 EnterpriseDB Corporation. All rights reserved. 46
PEM Client
Oracle/SQL Server/PG
Rep
licat
ion
DB Master Witness Offsite Replica
Virtual IP
DB Client
Backup & Recovery
PEM Server High Availability
Disaster Recovery HA Replica
Read/HA Replica
Mon
itorin
g
Dat
a In
tegr
atio
n
EDB Failover Manager
EDB Backup
and Recovery
xDB Replication
Postgres Enterprise Manager
Postgres Streaming Replication
© 2015 EnterpriseDB Corporation. All rights reserved. 47
PEM Client
Oracle/SQL Server/PG
Rep
licat
ion
DB Master Witness Offsite Replica
Virtual IP
DB Client
Backup & Recovery
PEM Server High Availability
Disaster Recovery HA Replica
Read/HA Replica
Mon
itorin
g
Dat
a In
tegr
atio
n
EDB Failover Manager
EDB Backup
and Recovery
xDB Replication
Postgres Enterprise Manager
Postgres Streaming Replication