high performance solution for oracle 11g rac with xtremcache
TRANSCRIPT
White Paper
EMC Solutions
Abstract
This solution demonstrates the benefits of deploying EMC® XtremCache™ and VMAX® to increase IOPS and decrease latency for OLTP databases. It provides scalability, high performance, and ease of use for mission-critical business demands.
December 2013
HIGH PERFORMANCE SOLUTION FOR ORACLE 11g RAC WITH XTREMCACHE 2.0.1 ON VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition
Optimum IOPS for OLTP workloads
Sustained Average Response Time Less Than One Millisecond
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
2
Copyright © 2013 EMC Corporation. All Rights Reserved.
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.
All trademarks used herein are the property of their respective owners.
Part Number H11828.2
3 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
Table of contents
Executive summary ............................................................................................................................... 5
The business challenge .................................................................................................................... 5
The technology solution: open, best-in-class components ............................................................... 5
Open standards benefits ............................................................................................................. 6
Operational advantages .............................................................................................................. 6
Solution overview ............................................................................................................................ 6
Key results and recommendations ................................................................................................... 6
Introduction.......................................................................................................................................... 8
Purpose ........................................................................................................................................... 8
Scope .............................................................................................................................................. 8
Audience ......................................................................................................................................... 8
Terminology ..................................................................................................................................... 9
Technology overview .......................................................................................................................... 10
Architecture layers ......................................................................................................................... 10
Architecture diagram ...................................................................................................................... 10
EMC Oracle Performance Solution: Server layer ................................................................................. 11
Server hardware ............................................................................................................................. 12
Server software .............................................................................................................................. 13
Red Hat Enterprise Linux ................................................................................................................ 13
Oracle Grid Infrastructure and Database 11g Release 2 .................................................................. 14
XtremSF and XtremCache deployment ............................................................................................ 14
XtremSF and XtremCache overview ................................................................................................ 14
EMC XtremCache configuration .................................................................................................. 15
EMC Oracle Performance Solution: Network layer .............................................................................. 16
EMC Oracle Performance Solution: Storage layer ............................................................................... 18
Introduction ................................................................................................................................... 18
Storage components ...................................................................................................................... 18
Storage hardware ........................................................................................................................... 18
EMC Symmetrix VMAX 40K eight-engine configuration .................................................................. 19
Storage software ............................................................................................................................ 20
EMC Oracle Performance Solution: Oracle Database layer .................................................................. 21
Introduction ................................................................................................................................... 21
Storage virtual provisioning design ................................................................................................ 21
Drive type....................................................................................................................................... 21
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
4
ASM disk group configuration for OLTP database ........................................................................... 21
OLTP database and workload profile .............................................................................................. 22
EMC Oracle Performance Solution: Performance testing .................................................................... 23
Overview ........................................................................................................................................ 23
Test objective ................................................................................................................................. 24
Query only test scenarios and methodology ................................................................................... 24
Read test results ............................................................................................................................ 24
Update only test scenarios and methodology ................................................................................. 26
Peak write test results .................................................................................................................... 27
Conclusion ......................................................................................................................................... 29
Summary ....................................................................................................................................... 29
Findings ......................................................................................................................................... 29
References.......................................................................................................................................... 30
EMC documentation ....................................................................................................................... 30
Oracle documentation.................................................................................................................... 30
5 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
Executive summary
Customers require an open, scalable, tiered, highly available, and high performance infrastructure to run their critical Oracle systems. IT organizations must strive for better performance and increased efficiency in their Oracle infrastructure and Oracle database and storage administration operations, including the following requirements:
Reduce capital expenditures and operational expenditures by deploying an open, non lock-in technology
Consolidate many Oracle databases (Oracle database versions 10gR1 to 11gR2 and Oracle 12c) and database workloads, including OLTP and Data Warehouse, to maximize the efficiency of the data center infrastructure
Deliver maximum performance while effectively utilizing the existing arrays and Oracle infrastructure
Maintain the highest performance levels and provide predictable performance to deliver the quality of service required in these Oracle mixed workload environments
The EMC Proven High Performance Solution for Oracle Real Application Clusters (RAC) on EMC Symmetrix® VMAX® is an open architecture that incorporates open, best-in-class Intel servers with EMC server-side flash storage (XtremSF™) and EMC VMAX storage arrays.
The solution uses optimal servers to balance performance, scalability, and Oracle license costs. The use of Xtrem™ technologies XtremSF and XtremCache™ software in the servers provides distinct performance and operational advantages over equivalent systems that do not contain server-side flash technologies.
XtremCache does the following to accelerate an Oracle RAC environment:
Features an ultra-performance tier: Accelerates any application that benefits from low-latency, high bandwidth physical read I/O
Hottest data resides on database server flash
Data is as close to the Oracle Database server CPU as any storage model will allow
Cooperates with Oracle Clusterware: Oracle Clusterware is the final authority on all node membership information in an Oracle RAC deployment
Has no awareness of database instances:
Has no concern for content of blocks of cached LUNs
Only XtremCache nodes can access LUNs cached by XtremCache
Does not impose a performance penalty on active transactions for cache insertions or cache coherency
The business challenge
The technology solution: open, best-in-class components
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
6
Offers optimized performance: VMAX arrays are relieved of read IOPS, leaving more bandwidth for handling writes. VMAX FAST enables automatic data placement as data goes from hot to cool usage.
Delivers the highest performance levels in the industry: This solution delivers the highest performance for mixed workload Oracle environments. EMC Proven Solutions for Oracle have demonstrated sustained metrics over 3.7 million IOPS with latency of less than half a millisecond for OLTP workload, and Data Warehouse workloads with sustained throughput of 32 GBs with a data load rate of 21 TB/hour. Key results and recommendations on page 7 provides details.
Open standards benefits
This solution is based on open standards. Advantages resulting from the open standards commitment include the following:
Flexible adaptation to existing and future customer needs and open industry standards
Lower capital investment and operational expense without vendor lock-in
Operational advantages
EMC open architecture not only supports different releases of Oracle Database software (10g, 11g, and 12c), but it also provides the capability of running the databases concurrently; that is, it supports database consolidation.
Open architecture and flexible adaptation means that application modification is not required for database deployment for this solution, lessening the potential for unforeseen impact to business operations and systemic data flow throughout the enterprise.
The purpose of the solution is to build an EMC High Performance Solution for Oracle on VMAX infrastructure, based on an open architecture, and demonstrate the following capabilities of the infrastructure:
High performance and flexibility
Low operational costs
Reduced risk
This white paper validates the performance of the solution and provides guidelines to build similar solutions. This document is not a comprehensive guide to every aspect of this solution.
The core advantages of the EMC Proven High Performance Solution for Oracle RAC on VMAX are as follows:
Delivers the highest performance for Oracle RAC workload environments. EMC Proven High Performance Solution for Oracle RAC on VMAX has demonstrated sustained metrics over 3.7 million IOPS with latency of less than ½ millisecond for OLTP workload. This impressive performance is achieved by utilizing open
Solution overview
Key results and recommendations
7 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
best-in-class components at the computer, network and storage layers. Details are listed in Table 1:
Table 1. IOPS test results with workload when XtremCache is enabled
Workload type
Performance statistics
One node
Two nodes
Four nodes
Eight nodes
Read only workload
IOPS 457,136 962,155 1,914,963 3,765,176
Response time (ms) 0.74 0.69 0.68 0.75
UPDATE transaction workload
Aggregate IOPS 53,492 99,649 190,809 303,330
Write IOPS 26,950 50,110 95,932 153,276
Read IOPS 26,542 49,539 94,877 150,054
Redo throughput (MB/s)
20 37 71 115
Note: Because EMC XtremCache is write through cache, for the UPDATE workload, the data blocks that have been read into buffer cache are accelerated by EMC XtremCache. Meanwhile, the dirty blocks that have been flushed by DBWR are passing through EMC XtremCache and are directly written to the back-end VMAX array.
Uses the following EMC technology enablers in the reference architecture:
EMC VMAX 40K
XtremSF PCIe flash card
XtremCache caching software
This solution provides a foundation that can be scaled in a flexible, predictable, and nearly linear way using additional server resources, including CPUs and memory, HBA ports, and front-end ports, to provide higher IOPS and throughput based on the configuration in this solution.
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
8
Introduction
The purpose of this white paper is to describe an EMC High Performance Solution for Oracle on VMAX infrastructure based on an open architecture, and to demonstrate the following capabilities of the infrastructure:
High performance and flexibility
Low operational costs
Reduced risk
This solution validates the performance of the solution and provides guidelines to build similar solutions. This document is not a comprehensive guide to every aspect of this solution.
This white paper serves the following purposes:
Introduces the key solution technologies
Describes the solution architecture and design
Describes the solution test scenarios and present the results of performance testing
Identifies the key business benefits of the solution
This white paper is intended for chief information officers (CIOs), data center directors, Oracle DBAs, storage administrators, system administrators, technical managers, and any others involved in evaluating, acquiring, managing, operating, or designing Oracle database environments.
Purpose
Scope
Audience
9 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
Table 2 lists the terminology used in this solution paper.
Table 2. Terminology
Term Definition
AWR Automatic Workload Repository
ASM Automatic Storage Management
DML Data Manipulation Language
IOPS I/Os Per Second
LUN Logical Unit Number
OLTP Online Transaction Processing
PCIe Peripheral Component Interconnect Express
PGA Program Global Area
RAID Redundant Array of Independent Disks
RAC Real Application Clusters
SATA Serial Advanced Technology Attachment
SLOB Silly Little Oracle Benchmark
SGA System Global Area
Terminology
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
10
Technology overview This section describes the architecture of the solution.
The solution provides an optimal cost-to-performance ratio for Oracle mission-critical application environments. The architecture is composed of the following layers:
Server–Eight Cisco UCS C240 M3 servers each with:
Two 700 GB EMC XtremSF SLC flash card (1400 GB total)
EMC XtremSF driver and firmware
Network–Two Cisco Director - MDS 9506 switches
Storage–EMC Symmetrix VMAX 40K storage system
Database–One eight-node Oracle RAC database is installed on the eight servers
Figure 1 depicts the EMC Proven High Performance Solution for Oracle RAC on VMAX.
Figure 1. Solution architecture
Architecture layers
Architecture diagram
11 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
EMC Oracle Performance Solution: Server layer Comprising the server layer of the solution, eight Cisco UCS C240 M3 servers utilize a total of 128 cores with 2.90 GHz E5-2690 processors, 2.56 TB RAM, and 11 TB XtremSF flash PCIe cards. The Cisco UCS C240 M3 is an enterprise-class rack server designed for performance and expandability. As part of the EMC Proven High Performance Solution for Oracle RAC on VMAX, the server layer enables a high-performing, consolidated approach to an Oracle infrastructure, resulting in deployment flexibility without the need for application modification. Features and benefits include the following:
Intel Xeon processor E5-2600 product family for hardened protection for virtual and cloud environments
Fully integrated quad-port gigabit Ethernet
Figure 2 shows one of the eight Cisco UCS C240 M3 rack servers used in the EMC Proven High Performance Solution for Oracle RAC on VMAX solution.
Figure 2. Cisco UCS C240 M3 rack server (1 of 8)
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
12
Table 3 describes the various hardware components of the EMC Proven High Performance Solution for Oracle on the VMAX server layer.
Table 3. Server hardware
Server
hardware Quantity Configuration Description
Cisco UCS C240 M3
8 2 x 8-Core Sandy-Bridge E5-2690 Processors – 512 GB RAM - 4x 200 GB SSD server
Servers
Each server consists of the following components:
PCIEHHS-7XXM
2 700 GB SLC PCIe card EMC XtremSF
UCSC-C240-M3S
1 UCS C240 M3 SFF w/o CPU mem HD PCIe with rail kit expdr
Server housing
UCS-CPU-E5-2690
16 2.90 GHz E5-2690/130W 4C/10 MB Cache/DDR3 1600 MHz
CPUs
UCS-ML-1X324RY-A
16 32 GB DDR3-1600 MHz LR DIMM/PC3-12800/quad rank/x4/1.35v
DRAM
UCS-SD200G0KA2-E
4 200 GB std height 15 mm SATA SSD hot plug/drive sled mounted
Internal hard drives
UCSC-SD-16G-C240
1 16 GB SD card module for C240 servers
SD card
UCSC-RAIL-2U
1 2U rail kit for UCS C-Series servers
Rail kit
N20-BBLKD 20 UCS 2.5-inch HDD blanking panel
HDD panels
UCSC-HS-C240M3
2 Heat sink for UCS C240 M3 rack server
Heat sinks
UCSC-PCIF-01F
4 Full-height PCIe filler for C-Series
PCI slot fillers
UCSC-PCIF-01H
1 Half-height PCIe filler for UCS PCI slot filler
UCSC-RAID-11-C240
1 LSI 2008 SAS RAID Mezzanine Card for UCS C240 server
RAID card
CAB-C13-C14-AC
2 Power cord C13 to C14 (recessed receptacle) 10A
Power cables
UCSC-PSU-650W
2 650 W power supply for C-Series rack servers
Power supplies
Server hardware
13 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
Server
hardware Quantity Configuration Description
LPE12004-M8
2 Emulex Quad Channel 8Gb FC PCIe HBA
Fibre Channel cards
E10G42BTDA 1 Intel X520-DA2- Network adapter - PCI Express 2.0 x8 low profile - 10 gigabit Ethernet - 2 ports
10 GbE network card
E10GSFPSR 2 Intel Ethernet SFP+ SR Optics - SFP+ transceiver module - 1000Base-SX, 10GBase-SR - 850 nm
Optical ports for FC
Table 4 describes the various software components of the solution server layer.
Table 4. Server software
Server software Configuration Description
Red Hat Enterprise Linux 6.3 Operating system for database servers
Oracle Grid Infrastructure 11g Release 2
Enterprise Edition 11.2.0.3
Software provides Clusterware and ASM storage volume management
Oracle Database 11g Release 2
Enterprise Edition 11.2.0.3
Oracle Database software
EMC XtremCache software
2.0.1 EMC XtremCache software for server-side flash cache
Red Hat Enterprise Linux includes enhancements and new capabilities, including developer tools, virtualization features, security, scalability, file systems, and storage. Red Hat Enterprise Linux is a versatile platform that can be deployed on physical systems, as a guest on the major hypervisors, or in the cloud. It supports all leading hardware architectures with compatibility across releases.
Server software
Red Hat Enterprise Linux
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
14
Oracle Database 11gR2 is available in a variety of editions tailored to meet the business and IT needs of an organization. This solution utilizes Oracle Database 11gR2 Enterprise Edition (EE). Oracle Database 11g R2 EE delivers industry-leading performance, scalability, security, and reliability on a choice of clustered or single servers running Windows, Linux, or UNIX. The database supports advanced features such as Virtual Private Database, and data warehousing options such as partitioning and Advanced Analytics, that are either included or available as extra-cost options.
XtremSF and XtremCache overview In this solution, two 700 GB EMC XtremSF flash cards are used in each RAC node, and one XtremCache cache device is created from one XtremSF card.
EMC XtremSF is an advanced flash storage technology deployed in a server and designed to deliver unprecedented performance acceleration by reducing latency and increasing I/O throughput. EMC XtremSF allows applications to access data in the most efficient manner possible. Residing on the server PCIe interconnect bus, XtremSF reduces application response time from milliseconds to microseconds by caching the frequently accessed data at the server side, which reduces the number of I/O requests sent to backend disk array.
EMC XtremCache is server flash-caching software that reduces latency and accelerates throughput to dramatically improve application performance when coupled with EMC XtremSF PCIe flash technology. XtremCache accelerates reads and protects data by using a write-through cache policy to the networked storage to deliver persistent high availability, integrity, and disaster recovery.
XtremCache coupled with array-based EMC FAST™ software provides the most efficient and intelligent I/O path from the application to the underlying disk array. The result is a networked infrastructure that is dynamically optimized for performance, intelligence, and protection for both physical and virtual environments.
Benefits of XtremCache include the following:
Provides performance acceleration for read-intensive workloads with the protection of a back-end, networked storage array
Provides an intelligent path for the I/O and ensures that the right data is in the XtremCache of the servers at the right time
By offloading flash and wear-level management onto the XtremSF PCIe flash card, uses minimal CPU and memory resources from the server
Works in both physical and virtual environments
Provides better data protection: since XtremCache is a write-through cache, it does not comprise data consistency in the storage array, even if the cards fail in the middle of I/O processing
No need to warm up the XtremCache across database instances reboot
Works for any kind of I/O; for example, any applications and any database platforms
Is supported on various operating systems and server platforms
Oracle Grid Infrastructure and Database 11g Release 2
XtremSF and XtremCache deployment
15 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
Allows customers flexibility in choice of cache capacity on the cards
Supports the Oracle RAC database, and even these RAC databases are “stretched” with EMC VPLEX
As XtremCache is installed in a greater number of servers in the environment, more I/O processing is offloaded from the storage array to XtremCache configured on the servers. This provides a highly scalable performance model in the storage environment. For more detail, refer to:
Introduction to EMC XtremCache for Oracle Real Application Clusters listed in References
Introduction to EMC XtremCache for Oracle Real Application Clusters video listed in References
EMC XtremCache configuration
XtremCache supports Oracle RAC using a distributed cache coherency algorithm. XtremCache automatically recognizes the presence of Oracle RAC, and switches operation to clustering mode.
All working Oracle RAC nodes must have XtremCache installed for the distributed Cache feature to come online. EMC recommends using XtremCache with Oracle RAC to cache LUNs that holding data files and TEMP files. Do not use XtremCache to cache redo logs, archives, or clusterware files.
Steps for configuring XtremCache devices are shown in the XtremCache manual.
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
16
EMC Oracle Performance Solution: Network layer The switch component level is made up of two Cisco MDS 9506 director-class SAN switches (like the one shown in Figure 3). The Cisco MDS 9506 is designed for deployment in storage networks supporting virtualized data centers and enterprise clouds. The switch combines high performance and low total cost of ownership, a core architectural requirement at all levels of the VMAX Performance Block.
Figure 3. Cisco MDS 9506 Multilayer Director
The Cisco MDS 9506 also offers these benefits:
Highly available scalability through a combination of nondisruptive software upgrades, stateful process failover, and full redundancy of all core components
Superior platform for accelerated, intelligent storage applications such as EMC replication and backup, data migration, and storage media encryption
Virtual machine transparency and end-to-end visibility all the way from the virtual machine down to the EMC storage, enabling scalable, mobile virtual machines
Nexus 5.2(8) software is used in the EMC High Performance Solution for Oracle on VMAX.
Table 5 lists the hardware components of the network layer of this solution.
17 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
Table 5. Hardware components of the network layer
Network
hardware Quantity Configuration Description
MDS-PBF-ADV32 8 Cisco 32-port 8-Gbps FC Port Module
Line cards
MDS-9506-V2 2 Chassis SUP2 no ports Director Director chassis
FC10M-50MLC 128 FCHNL 10M 50/125 LC-LC LC adapters
MDS-PW19-TWST 2 Cisco 9506 Twist Lock Power Cord US
Power cord
FC1M-50MLC 96 FCHNL 1M 50/125 LC-LC
MDS-8G-SW 192 MDS 2/4/8-Gbps FC shortwave SFP LC
MDS-ENT-9500 2 Enterprise license key 9500
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
18
EMC Oracle Performance Solution: Storage layer
EMC is not only the world’s largest storage provider, but also the leader in storage infrastructure in Oracle Database environments. Storage User Demand Study, 2012 – Fall Edition: Many Uses of Internal Storage provides more details.
The storage components comprising the EMC Proven High Performance Solution for Oracle on VMAX include the following:
EMC Symmetrix VMAX 40K with eight engines (the specification of the engine is shown in Figure 4)
EMC PowerPath®
Dynamic cache partitioning (VMAX)
Figure 4. EMC Symmetrix VMAX 40K
Table 6. EMC Symmetrix VMAX storage hardware
Storage
hardware Quantity Configuration Description
SD-DBV-DIR-1P 4 VMAX 40K DBV DIR 1 Phase
SD-DBV-SPS 16 VMAX 40K SPSV
SD-1P 1 VMAX 40K 1P INFRAST
SVDBSOLDOR1P 4 VMAX 40K Drive Bay Solid Door 1P
SYMV2-MIGRBAS 1 Symmetrix 40K Migration Bundle
PP-SE-SYM 1 PPATH SE SYM
Introduction
Storage components
Storage hardware
19 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
Storage
hardware Quantity Configuration Description
SD-VCONFIG32 1 VMAX 40K VCONFIG 32
SD-FE80000E 16 VMAX 40K 8 MM 8 G Fibre Fibre Ports
SD-INTBKVKIT 8 VMAX 40K Internal Cable Bracket Titan
SD-PW40U-US 10 30A 1Phase Namer Japan l6-30P Power
VL4103001B 376 VMAX 40K 4 G 10K 300 GB SAS Drive
Disk drives
SD-ADD192C 7 VMAX 40K Add Engine-192GB-C VMAX engine
SD-192-BASEC 1 VMAX 40K Base-192GB-C VMAX cache
SD-DE25-DIR 64 VMAX 40K 25SLT DR ENCL
The EMC Symmetrix VMAX 40K is designed for high efficiency, scalability, and secure data persistence. Built on the strategy of powerful, trusted, and smart storage, and founded in EMC Virtual Matrix Architecture that allows for seamless, cost-effective growth, the EMC Symmetrix VMAX 40K offers the following:
Zero downtime migration technology and lower cost and greater efficiency through automated tiering
More scalability for less management complexity and operational expense
The configuration supporting this solution is described in Table 7:
Table 7. EMC Symmetrix VMAX 40K configurations
Component Quantity Configuration
Engines 8 192 GB cache each, total of 1536 GB raw cache
Directors 16 Eight ports on each director with 8 GB Fibre Channel
Bays 5 1 system, 4 disk
10K SAS drives 376 100 TB Raw, 45 TB Usable (RAID1 Configured)
EMC Symmetrix VMAX 40K eight-engine configuration
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
20
Table 8 lists the software used in the solution storage layer.
Table 8. Storage software
Storage software Configuration Description
VMAX Enginuity™ code 5876 VMAX micro code
EMC Solutions Enabler 7.6 Host CLI storage management software
EMC PowerPath 5.7 SP1 Multipathing and load balancing software
Storage software
21 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
EMC Oracle Performance Solution: Oracle Database layer
In Oracle 11g R2, Oracle ASM and Oracle Clusterware have been integrated into the Oracle Grid Infrastructure. In the solution, we use ASM to store the database files.
EMC Virtual Provisioning™ automatically stripes data across all data devices in a virtual pool and balances the workload across storage devices. To ensure even striping of data, all data devices in a virtual pool should be the same size.
Table 9 shows the RAID selections and number of spindles for each virtual pool. In this solution, Oracle data files and redo log files are located on thin devices using RAID 1 protection and all physical spindles for the best performance and capacity.
Table 9. Virtual pool design on VMAX 40K
Virtual pool
RAID protection
Drive type
Physical spindles size
Number of active spindles Item
FC_R1_1 RAID 1 (2-way, mirror)
FC 10K 300 GB 376+ 5 (spare disk)
CRS, DATA, REDO
Table 10 details the RAC database Oracle ASM (automatic storage management) disk group design. On each database, we used three ASM disk groups to store the relevant database files, including data files, control files, online redo log files, and temporary files. Default settings are used for ASM disk groups.
Table 10. ASM disk group design for RAC database
Item LUN size (GB) Number of LUNs ASM disk group name
CRS 10 2 +CRS
DATA 1024 18 +DATA
REDO 60 4 +REDO
Introduction
Storage virtual provisioning design
ASM disk group configuration for OLTP database
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
22
Table 11 details each OLTP database workload profile for the solution. We used the SLOB toolkit to generate an OLTP database and deliver the OLTP-like workloads, including the query-only and UPDATE-only workloads required for the solution.
Table 11. Database workload profile for each OLTP database
Profile characteristic Details
Database type OLTP
Database size 16 TB
Oracle Database 11gR2 eight-node RAC database on ASM
Instance configuration for read only workload
SGA size: 16 GB
Note: Because more SGA causes less I/O activity, we reduced SGA to this small number to generate a stable and high I/O workload
Workload profile OLTP-like workload simulated by SLOB
Network connectivity 8 Gb Fibre Channel for SAN; 10 GbE for IP
OLTP database and workload profile
23 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
EMC Oracle Performance Solution: Performance testing
The solution characterizes the Oracle OLTP system performance on VMAX array with EMC XtremSF cards installed on database servers. We created an eight-node Oracle RAC database on eight physical servers. Silly Little Oracle Benchmark (SLOB) was used to generate workload because it is the premiere SQL workload for driving maximum physical random I/O from a database platform.
SLOB is a SQL-driven Oracle database I/O generator, instead of a synthetic I/O generator. SLOB uniquely drives massive physical I/O using minimal host CPU resources, and it specifically targets the Oracle I/O subsystem. SLOB performs all of its physical I/O buffered in the Oracle SGA. No physical I/O buffered in the Oracle PGA is performed. SLOB possesses the following characteristics:
Supports testing Oracle logical read (SGA buffer gets) scaling
Supports testing physical, random single-block reads (db file sequential read/db file parallel read)
Supports testing random single block writes (db file parallel write)
Supports testing extreme REDO logging I/O
Consists of simple PL/SQL
Is entirely free of all application contention
We used SLOB to generate an OLTP-like workload on the eight-node Oracle RAC database to demonstrate sustained flash and storage array IOPS. The database performance metrics including IOPS and latency were gathered primarily from the AWR report. In addition, we gathered metrics for I/O throughput at the server/database and storage levels.
Notes: Benchmark results are highly dependent on workload, specific application requirements, and system design and implementation. Relative system performance will vary because of these and other factors. Do not use solution test workloads as a substitute for a specific customer application benchmark when making critical capacity planning or product evaluation decisions All performance data contained in this report was obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly. EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute.
Overview
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
24
This test measures the physical I/O scalability along with the scaling of the number of concurrent SLOB zero-think-time sessions (simulated concurrent users) and the number of RAC nodes. Multiple concurrent sessions (reader sessions) executing similar query SQL statements were run to validate read only workload, and multiple concurrent sessions (writer sessions) executing similar UPDATE SQL statements were run to validate the physical read/write workload.
We gradually increased the number of instances of the RAC database and the number of concurrent users with each user running similar OLTP queries simultaneously.
When we added a RAC node, additional resources were added, including CPU power and XtremSF cards. With the addition of new RAC nodes, we tested the system again by running the similar SLOB workload. For this test, workloads were running simultaneously from all the RAC nodes added. We then increased the number of concurrent users and measured the performance scalability.
The test process included the following steps:
1. Run the OLTP query only workload with 64 concurrent simulated users (zero-think-time sessions) on the first node of an eight-node RAC database using SLOB.
2. Add the second node into the system, then run the workload with 64 concurrent users on each node; that is, with 128 concurrent users in total running simultaneously on the two-node RAC database.
3. Add two additional nodes into the system, then run the workload with 64 concurrent users on each node separately; that is, with a total of 256 concurrent users running simultaneously on the four-node RAC database.
4. Add four additional nodes into the system, then run the workload with 64 concurrent users on each node separately; that is, with a total of 512 concurrent users running simultaneously on the eight node RAC database.
Performance statistics were captured using Oracle Automatic Workload Repository (AWR) RAC reports. We observed the “physical reads” value in the AWR report to assess read IOPS statistics. Query average response time was calculated from the “db file parallel read” and “db file sequential read” record in the “Top Timed Events” section of the AWR report, as shown in Figure 5.
Figure 5. AWR RAC report snippet for read IO response time calculation
Test objective
Query only test scenarios and methodology
Read test results
25 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
We used the following formula to calculate the I/O latency:
For “db file sequential read”
The total wait time is T1 which is 19,485.48 seconds as shown in Figure 5.
The total number of waits is N1 which is 30,614,070 as shown in Figure 5.
For “db file parallel read”
The total wait time is T2 which is 9,803.47 seconds as shown in Figure 5.
The total number of waits is N2 which is 8,610,615 as shown in Figure 5.
The average read response time is: (T1+T2) / (N1+N2), which is (19,485.48 + 9,803.47) * 1,000/ (30,614,070 + 8,610,615) as shown in Figure 5, and 0.75ms is the average response time.
Table 12 depicts how IOPS increase when RAC nodes scale up.
Table 12. Scaling of nodes and resulting increases in IOPS
Figure 6 shows that IOPS increase when the number of RAC nodes scales, while average read I/O response time remains under one millisecond.
Figure 6. Query only IOPS scaling and average physical read I/O response time
As Figure 6 shows, we achieved a total of 3,765,176 read IOPS and an average latency of 0.75 milliseconds with eight-node RAC database when running 64 concurrent sessions executing similar queries SQL statements on each of the RAC nodes.
Metrics 1 node 2 nodes 4 nodes 8 nodes
IOPS 457,136 962,155 1,914,963 3,765,176
Average response time (ms) 0.74 0.69 0.68 0.75
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
26
The IOPS increased nearly linearly with each additional node that was added into the Oracle RAC database. For example, the total IOPS of four RAC nodes reached 1,914,963. After another four RAC nodes are added for a total of eight database nodes, the IOPS are almost doubled to 3,765,176.
The read hit ratio for XtremCache is about 98 percent for each cache device during the test. Two percent of the I/Os are served from the storage array. The statistics can be monitored by running the following command line:
vfcmt display -cache_dev <device>
We gradually increased the number of RAC nodes and ran a total of 56 concurrent sessions on the database to generate stable and high read and write IOPS, with each session running similar SLOB UPDATE SQL statements
We decreased buffer cache for each database instance to push consistent write I/O workload to the back-end storage. The write workload was driven by the UPDATE SQL statement. Generally, it incurs the following operations sequentially:
1. Read the data blocks that need to be updated into the buffer cache.
2. Update the rows in the data blocks.
3. Commit the updated rows and trigger LGWR flushing redo entries to online log files.
During the SQL UPDATE only workload running, the background DBWR process flushes the dirty blocks out of the buffer cache into data files. Because we used a very small buffer cache, the data blocks were read into the buffer cache and written out of the buffer cache soon after the rows were updated. Thus the execution of each UPDATE operation caused physical reads, which were accelerated by the EMC XtremCache (when cache hit) or the back-end VMAX array (cache miss). When the updated data blocks were written out of the buffer cache by the DBWR process, as being a write through cache for EMC XtremCache , the data blocks were written to the back-end VMAX array. Once the application got the acknowledgement from the back-end array, the application I/O request was complete, and the updated data was asynchronously written to XtremCache.
The test process included the following steps:
1. Run the Update only workload with 56 concurrent users on one RAC Database with only one node using the scripts in SLOB.
2. Add one additional node into the RAC database, then run the workload with 56 concurrent users; that is, with 28 concurrent users running simultaneously on each node of this two-node RAC.
3. Add two additional nodes into the RAC database, then run the workload with 56 concurrent users; that is, with 14 concurrent users running simultaneously on each node of this four-node RAC.
4. Add four additional nodes into the RAC database, then run the workload with 56 concurrent users; that is, with seven concurrent users running simultaneously on each node of this eight-node RAC
Update only test scenarios and methodology
27 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
Performance statistics are captured using AWR reports. We read the “physical writes” row in the AWR report for the peak write IOPS statistics. Because the write workload is generated by UPDATE statements as described previously, we also collected “physical reads” from the AWR report for the peak read IOPS statistics that were part of the write transaction.
We calculated updated write average response time by dividing the “Total Wait Time(s)” by the “Waits” of the “db file parallel write” record in the “Top Timed Events” section of the AWR report, as shown in Figure 7. We also calculated LGWR latency by dividing the “Total Wait Time(s)” by the “Waits” of the “log file parallel write” record in the “Top Timed Events” section of the AWR report, as shown in Figure 7.
For example, in following AWR snippet, the total wait time of the “db file parallel write” wait event is 287.00 seconds, which is 287,000 ms, and the number of waits is 319,941; thus, the write average response time can be calculated as 287,000 / 319,941= 0.90 ms. The LGWR latency is: 1,087.81*1,000/1,323,505 = 0.82.
Figure 7. Update only write average response time measurement from the RAC AWR report
Table 13 and Figure 8 depict the peak read/write IOPS increases as the RAC node scaling.
Table 13. Scaling of users and resulting increases in peak disk array IOPS
IOPS 1 node 2 nodes 4 nodes 8 nodes
Write 26,950 50,110 95,932 153,276
Read 26,542 49,539 94,877 150,054
Aggregate 53,492 99,649 190,809 303,330
Write response time
0.30 0.40 0.30 0.90
Read response time(ms)
0.90 0.80 0.20 0.30
Redo size (MB/s) 20 37 71 115
LGWR latency (ms)
0.55 0.54 0.66 0.82
Peak write test results
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
28
Figure 8 shows the peak IOPS, and update-only IOPS for RAC node scaling, while average response time remains under one millisecond.
Figure 8. Peak writes – update-only IOPS scaling along with RAC nodes scaling
During an UPDATE transaction, the backend VMX only needs to handle the writing I/O activities, as the reading I/O activities have been cached and accelerated by XtremCache. Because of this, the solution can scale to accommodate a very heavy transaction workload, as confirmed in testing.
As shown in Table 13 and Figure 8, when running 56 concurrent sessions executing similar UPDATE SQL statements on eight-node RAC database, we achieved 303,330 aggregated IOPS including 153,276 write IOPS and 150,054 read IOPS, which were part of the write transaction. The average latency of write was 0.9 milliseconds, because we used very small buffer cache (only 128 MB, almost no data cached in the server) to generate a high physical write I/O workload.
The IOPS increased nearly linearly when additional RAC nodes were added into the workload. For example, the aggregate IOPS were 53,492 when running write workload on one RAC node, and this increased to 99,649 when running workload on two RAC nodes.
Redo size is also a key metric used to measure the transaction capability. As demonstrated through testing, the workload on one node generated 20 MB/second redo entries, and almost doubled to 37 MB/second with the workload running on two nodes. When we ran workload on four nodes the redo throughput is almost doubled again to 71 MB/second. The transaction capability is scaling along with the node scaling.
29 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
Conclusion
Implementing the EMC Proven High Performance Solution for Oracle RAC on VMAX with innovative and proven products like EMC’s VMAX gives customers choices within an open infrastructure, enabling them to easily integrate into existing data center operations and take advantage of new technologies such as XtremCache, which now also fully supports Oracle RAC. Customers can efficiently use people, processes, and technologies through virtualization, database, and applications consolidation. Customers can also independently scale capacity and processing capability without the limitations imposed by a single-purpose appliance
As the infrastructure undergoes changes to applications, databases, and non-database software, this stack can align with the shifting technical demands imposed by the business needs. This white paper explains how to balance OLTP workloads while maintaining the protection and resiliency of the data. That adaptability to change and the ability to apply the technology where it is needed protects the capital investment, and can be fluid as the requirements change without sacrificing any of the other datacenter operations.
The core advantages of this solution are as follows:
With its open architecture, supports Oracle OLTP workload built on superior functional integration throughout the solution
Leverages the following EMC technology enablers in the solution:
EMC VMAX 40K
XtremSF flash cards
XtremCache
Provides full support for EMC Performance Boost, HA, and replication technologies
Increases IOPS for OLTP workloads when servers are added
The aggregate read IOPS increased from 457,136 to 1,914,963 when RAC nodes were scaled out from one to four, and it increased to 3,765,176 when eight RAC nodes ran the workload together.
Average response time less than one millisecond (0.75ms)
The aggregate IOPS for UPDATE transaction workload was 53,492 on one RAC node and caused 20 MB/s redo throughput, and it increased to 190,809 and caused 71 MB/s redo throughput when four RAC nodes were added. The IOPS increased to 303,330 and generated 115 MB/s redo throughput when running on eight RAC nodes.
Average response time for DBWR background process less than one millisecond (0.90 ms)
This solution provides a foundation that can be scaled in a flexible, predictable, and near-linear way, by adding additional server resources including CPUs and memory, HBA ports, and front-end ports, to provide higher IOPS and throughput based on the described configuration.
Summary
Findings
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
30
References
The following documents provide additional and relevant information. Access to these documents depends on your login credentials. If you do not have access to a document, contact your EMC representative.
The following reference materials are available on EMC Powerlink:
EMC Infrastructure for High Performance Microsoft and Oracle Database Systems
Introduction to EMC XtremCache
EMC XtremCache Data Sheet
In addition, XtremCache documentation is available at EMC Online Support: https://support.emc.com/products/25208_XtremCache-Cache/Documentation/
A video entitled Introduction to EMC XtremCache for Oracle Real Application Clusters is visible via this link: https://community.emc.com/videos/6740
For additional information, see the following documents:
Oracle Grid Infrastructure Installation Guide 11g Release 2(11.2) for Linux
Oracle Database Installation Guide 11g Release 2 (11.2) for Linux.
EMC documentation
Oracle documentation