risk-free administration for expert database · pdf filecapacity planning using database...

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Risk-Free Administration for Expert Database Administrators

Kurt Engeleiter, Oracle Jason Lentz, Cerner Bart Gaddis, Cerner

Oracle Confidential – Internal


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

3 Oracle Confidential – Internal


Program Agenda

• DBA Challenges

• Risk Free Administration Tools

• New Performance Management Tools



Program Agenda

• DBA Challenges





Top Challenges

Database Management

Accurate capacity planning with 20-40% data store growth

40%

Source: From Database Clouds to Big Data: 2013 IOUG Survey On Database Manageability

?

78% Downtime resulting from unverified changes

48% Unpredictable application (SQL) performance

Key Takeaway:

Improve Quality of Service



DBA Actions That Affect Performance

• Adding or dropping indexes, partitioning tables, adding materialized views

• Gathering optimizer statistics

• Applying patches – one offs, PSUs, CPUs

• Modifying init.ora parameters

• Implementing new features such as Compression, Database In-Memory

• Version or patchset upgrades, e.g.

– 11g -> 12c

– 12.1.0.1 -> 12.1.0.2



Environment Changes That Affect Performance

• Data growth

• User population growth

• Infrastructure changes

– Server upgrades

– Changes in size or type of disk storage

– Migration to new platforms including migration to Cloud

• Database Consolidation



Program Agenda

• DBA Challenges





Database Replay

Concurrent Replay SPA

SPA Quick Check

• Proactively identifies impact of routine DBA tasks on database performance

Load and throughput testing solution

Designed for detecting and remediating throughout problems

• SQL response time (unit) testing solution

• Designed for identifying and tuning regressed SQLs

• Consolidation and capacity planning solution

Risk Free Performance Management Tools

Lower Risk

Automated Cut Costs



Program Agenda

• DBA Challenges


• SPA Quick Check

• SQL Performance Analyzer

• Database Replay

• Concurrent Database Replay




SPA Quick Check Overview • Helps users predict the impact of

specific system changes on SQL workload

• Input is SQL Tuning Set (STS) and a database configuration change

• Automatically executes two trials – pre and post change - and generates report

• Integrated with STS, SQL Plan Baselines, & SQL Tuning Advisor to form an end-to-end solution

12

SQL Plans + Stats

Analysis Report

Compare SQL Performance

Pre-change Trial Post-change Trial

SQL Workload (STS)

SQL Plans + Stats


Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 13

SPA Quick Check

• Optimized for use on prod systems

• Optimal Trial or Explain Plan mode

• Disable multi-executions, full DML execute disabled

Optimized

• Per SQL time limits

• Testing scoped to private session

• Associate with Resource Consumer Group

Controlled

• Context-aware change testing workflows, such as,

• Optimizer gather statistics

• Init.ora parameter changes

Change-Aware

SPA Quick Check

Production Database

Optimized Controlled/S

coped

Change-Aware

Optimal Trial

Mode, no DML

execute

Pre-selected

STS and

default SPA

settings

Per SQL Time Limits,

Limits testing scope to

private session

Context-aware

change testing

DBA



SPA Quick Check Report

1

2

3 4

5



Risk Free Changes with SPA Quick Check

• Statistics management

– Gather statistics in pending mode

– Use SPA Quick Check to determine whether to publish

• Init.ora parameter changes – Current parameter value is used for pre-change

– Select new value

– SPA Quick Check validates new parameter setting

• SQL Tuning Validation

– Validates SQL Profiles



SPA Quick Check Demo



Program Agenda

• DBA Challenges


• SPA Quick Check


• Database Replay





SQL Performance Analyzer (SPA) Overview • Helps users predict the impact of any

system change on SQL workload

• User selects SQL Tuning Set

• User executes different SQL trials (experiments) of SQL statements performance by test execution or explain plan

• Analysis shows per SQL execution differences

• Integrated with STS, SQL Plan Baselines, & SQL Tuning Advisor to form an end-to-end solution



Risk Free Changes with SPA

• Access structure changes:

– Indexes

– Materialized views

– Parallelism

• Patching

• Upgrades

• Implementing new features such as Compression or Database In-Memory

• Validating new hardware – servers or storage



Program Agenda

• DBA Challenges


• SPA Quick Check


• Database Replay





• Database load and performance testing with real production workloads

– Production workload characteristics such as timing, transaction dependency, think time, etc., fully maintained

• Test and measure transaction throughput improvements

• Identify application scalability and concurrency problems on test system before production deployment

• Usage scenarios: platform migration (e.g., Exadata), upgrades, patching, new database features or options, etc.

• 21

Database Replay

Test

Capture Process Replay Analysis & Reporting

Production Replay Driver Clients

Storage Storage



Risk Free Changes with Database Replay

• Patching

• Upgrades

• Implementing new features

• Validating new hardware – servers or storage

• Stress testing for capacity planning



Program Agenda

• DBA Challenges


• SPA Quick Check


• Database Replay





Capacity Planning: Motivation • User populations and data are continually growing

– How long can my environment handle the workload

• Organizations are pursuing database consolidation to deliver cost savings while increasing business agility

– How can we guarantee service levels between different workloads

– How can we test if one errant application is not affecting the others

• Consolidating workloads and databases has significant challenges

– Applications have different workload profiles – CPU – Memory – Storage – Network

– Will my Multitenant database handle peak workloads and co-exist?

– Is there enough headroom?



• Capture workload on different databases that needs to co-exists concurrently

• Works for schema consolidation and Pluggable Databases

• Use for schema and CDB consolidation

• Identify and remediate inter-application scalability and concurrency problems

• Perform capacity planning through scale up, subsetting, scheduling (time-shifting) of multiple workloads

25

Concurrent DB Replay For Risk Free Consolidation and Capacity Planning



Capacity Planning Using Database Replay and Concurrent Database Replay

• Comprehensive scale-up support and what-if scenarios testing

• Scale-up techniques superior to traditional methods

– Zero-scripting approach extended for scale-up

– Scales data and user population

– Realistic data and bindsets

– Flexible, supports custom workload creation

• Capacity Planning Strategies

– Scale-up by scheduling concurrent replays

– Scale-up with multiple PDBs

– Scale-up by workload folding


ERP

DW

Replay Captured Workload Align Workload Peaks

CRM

Application Workloads

Capacity Planning Strategies: Peak Workload Testing – Time-Shifting

Peak Workload Testing Through Time-Shifting Exercise worst case scenario where workload peaks line up Evaluate and experience fallout from the safety of a test system



Scale-up with Multiple Duplicate PDBs

Perform scale-up testing to identify possible host bottlenecks when deploying multiple instances of an application

Deploying a multitenant application—or adding a new geographical area to an existing application or line of business

Capacity Planning Strategies: Multiple PDBs

Replay Concurrent Workload Mapping

Test Database

PDB Clones

Sales Workload



Capacity Planning Strategies: Workload Folding

Replay Concurrent Workload Subset

Subset 2

Subset 1

Workload Subsets Sales Workload

Scale-up Through Workload Folding

Perform scale-up testing by subsetting an existing workload capture into 2 & then perform concurrent replay of workload subsets

Suitable for workloads where transactions in subsets are mostly independent 29 Oracle Confidential – Internal


Concurrent Replay Capacity Planning Strategies Summary

Scale-up Strategy Workload Suitability

Concurrent Database Replay Enables consolidation validation

Can be used for all workloads

Multiple identical PDBs Simultaneous workload and data scale-up

Multitenant what-if scenarios

Workload Folding N * Workload scale-up

Suitable for relatively stateless workloads


Program Agenda

• DBA Challenges


• SPA Quick Check


• Database Replay





Database Performance Diagnostics Tools

• Automatic Workload Repository – AWR

– Reports about performance and workload data from AWR

• Active Session History – ASH

– Gathers fine-grain data about every active database session every second

• Automatic Database Diagnostics Monitor - ADDM

– Data Analysis and Problem Identification

– Findings and Advise on how best to resolve bottlenecks

• Real-time SQL and Database Operations Monitoring

– Provides in-depth diagnostics about SQL execution at row source level

–

32

Database Performance Hub Provides Unified Monitoring Solution!



Database Performance Hub

33

Unified Performance Monitoring

• Single view of DB performance

ADDM, ASH analytics, Real-Time SQL Monitoring, SQL Tuning…

• Switch between ASH analytics, workload view, ADDM findings and SQL monitoring seamlessly

• Supports both real-time & historical mode

• Historical view of SQL Monitoring reports

• Dedicated tab for RAC



Performance Hub Report

• New interactive report for analyzing AWR data

• Performance Hub report generated from SQL*Plus

– @$ORACLE_HOME/rdbms/admin/perfhubrpt.sql

OR calling dbms_perf.report_perfhub(….) function

– Single view of DB performance

– ADDM, SQL Tuning, Real-Time SQL Monitoring, ASH Analytics

• Switch between ASH analytics, workload view, ADDM findings and SQL monitoring seamlessly

• Supports both real-time & historical mode

• Historical view of SQL Monitoring reports



Perfhub Demo



New AWR “Active-HTML” Report

• New AWR report type “active-html” introduced in Oracle Database 12.1.0.2

• Provides best of HTML and Performance Hub Reports

– HTML report contains embedded Performance Hub Report as the last section

– ADDM task finds and recommendations are also presented

– Exadata-aware

• Highly recommended to use AWR “active-html” reports instead of HTML reports

– Combines power for EM navigation and drill down for offline analysis

– Can be saved and mailed like other Active Reports and does not need EM connectivity for viewing


Jason Lentz – Cerner Abilities Center

Bart Gaddis – Cerner Database Architecture

October 26, 2015

Customer Use Case Oracle Real Application Testing

Health care is too important to stay the same.TM

BRNDEXP 2.1 0714 © 2014 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

40

Cerner today


41

Abilities Center – Cerner’s Wind Tunnel

Test Partners / Upgrade Center

CernerWorks

• Release + Tech Production Validation

• Controlled / Monitored Implementation

• Experienced IP Resources Available

IP Development Ability-Proven

Controlled Testing Environment

Hardware Technologies

iBus Platform

Java Services, RCP, Thin Client

Layered Software Technologies

Country Specific Solutions

Millennium Platform

Test Design

• Leverage clinical and domain expertise

• Automate client representative workflows

• Test with role and venue data complexity

Lights On Network

• Collects 7 billion client timers per month

• Built-in instrumentation allows lab/client comparison

• Enables focused enhancements in workflow coverage

Perform-ability 2 Seconds is Too Slow

High Availability 99.99% Uptime

Scalability Regional – National – Global

Affordability Reduce TCO

Reliability 1 Crash is Too Many

Test Measure Analyze

Correct

Quality Dedicated Testing Resources

• Isolated Network

• Secured Environment

• Experienced Associates

• IP Engineering Teams

• 30+ Millennium Domains

• Real Client Databases

• Automated Testing Tools

• SAN Storage

• AIX

• Citrix

• HPUX

• Linux

• MQ

• Oracle

• VMWare

• WebSphere

CAMM Platform


42

Why RAT?

• Capture and Replay client-like production workloads • Volume

• Data diversity

• Minimal effort to utilize

• Workload Scale-up and Folding • Test hardware at 10x current volume

• Stress storage and compute nodes

• Integrate/augment with existing workflow automation • Consume replay activity data in real-time

• Measure impact of replay load on automated workflows


43

Test Methodology

Controlled Approach

• Measure one change

• Same SGA

• Same replay parameters

Repeatable Replays

• Same number of user calls

• Low divergence

• Consistent duration

Actionable Data

• Confidence in metrics

• Process/parameter changes

• Purchase decisions


44

Workload Capture and Replay Procedure

• Two hour capture during peak volume on live client production system

• Database duplicate and recovery to capture SCN (Full copy)

• Database upgrade to 11.2.0.4

• SQL regression tuning

• Preprocess and tune replay clients

• Repeatable Workload Replays • Flashback to guaranteed restore point

• Cache warming - Consolidated replay of entire workload (read only)

• Read-write workload replay

• Standardized data and report collection (Replay, AWR, OS)


45

Test Methodology

Controlled Approach


• Same SGA


Repeatable Replays


• Low divergence


Actionable Data





46

Database Replay Analysis – SQL Perspective

SQL Tuning Sets, primary tool

• Capture SQL during baseline replay and subsequent replays

• SPA (SQL Performance Analyzer) has built in reporting using SQL Tuning Sets

• Find SQL which has regressed or improved

• Find SQL with plan changes

• Relate the change to the impact on workload

• STS comparison - Example Report


47

SQL Tuning Sets – Use Cases

Optimizer Related Changes • DBMS_STATS sample size, impact on execution plan changes

• DBMS_STATS histogram methodology (frequency histograms with autosample)

• Optimizer patches, impact on execution plan changes

• Optimizer parameters, impact on execution plan changes

• Discovery of bug related to ORA-7445 on hard parse due to corrupt histogram.

Structural Changes • Validation of Cluster Tables, impact on SQL execution statistics

• Investigation/evaluation of plan changes associated to Cluster Tables

Key Views • DBA_SQLSET_STATEMENTS, DBA_SQLSET_PLANS, DBA_SQLSET_BINDS


48

Test Methodology

Controlled Approach


• Same SGA


Repeatable Replays


• Low divergence


Actionable Data





49

Major Replay Use Cases

Vertical scaling (Server hardware upgrade)

Horizontal scaling (Add RAC instances)

OS tuning (OS version, parameter change)

Notable Results

• IP Fragmentation Thresholds (IPFRAG)

• RESULTS: Increasing all thresholds

decreased packet reassembles and failures

and improved global cache metrics while

remaining within hardware capabilities.

• DETERMINATION: Safe and effective to roll

out to clients.


50





Notable Results

• Migrate to new hosts with more CPU and RAM

• RESULTS: More efficient reads and gets

increased user/application activity while

decreasing waits (concurrency, cluster,

commit).

• DETERMINATION: Improved performance

and scaling point. Benefits outweigh cost.


51





Notable Results

• Add RAC instances, up to 6 wide

• RESULTS: Resource utilization gradually

decreased with every instance added to

cluster.

• DETERMINATION: Adding additional

hardware is an effective solution for

increasing capacity.

Contact

Jason Lentz [email protected]

Bart Gaddis [email protected]

risk-free administration for expert database · pdf filecapacity planning using database...

Documents