emc vspex for virtualized oracle database 11g oltp … · contents 4 emc vspex for virtualized...

IMPLEMENTATION GUIDE

EMC VSPEX FOR VIRTUALIZED ORACLE DATABASE 11g OLTP Enabled by EMC Next-Generation VNX and EMC Backup

EMC VSPEX

Abstract

This Implementation Guide describes the high-level steps and best practices required to implement the EMC® VSPEX® Proven Infrastructure for Virtualized Oracle Databases on a VSPEX Private Cloud with VMware vSphere enabled by EMC Next-Generation VNX® and E MC Backup. It describes implementation for the FAST™ Cache and FAST VP® features and explains their usage in VNX storage systems. Usage guidelines and major customer benefits are also included.

October 2013

2 EMC VSPEX For Virtualized Oracle Database 11g OLTP Enabled by EMC Next-Generation VNX and EMC Backup Implementation Guide

Copyright © 2013 EMC Corporation. All rights reserved. Published in the USA.

Published October 2013

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

The information in this publication is provided “as is”. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

EMC VSPEX for Virtualized Oracle Database 11g OLTP Enabled by EMC Next-Generation VNX Implementation Guide

Part Number H12064.1

http://www.emc.com/legal/emc-corporation-trademarks.htm

http://www.emc.com/legal/emc-corporation-trademarks.htm

Contents

3 EMC VSPEX For Virtualized Oracle Database 11g OLTP Enabled by EMC Next-Generation VNX and EMC Backup

Implementation Guide

Contents

Chapter 1 Introduction 9

Purpose of this guide ................................................................................................ 10

Business value ......................................................................................................... 10

Scope ....................................................................................................................... 11

Audience .................................................................................................................. 11

Terminology.............................................................................................................. 12

Chapter 2 Before You Start 13

Overview .................................................................................................................. 14

Pre-deployment tasks ............................................................................................... 14

Deployment workflow ............................................................................................... 14

Deployment prerequisites ........................................................................................ 15

Planning and sizing Oracle Database 11g R2 ............................................................ 17

Essential Reading ..................................................................................................... 17

Design guide........................................................................................................ 17

Solution overviews .............................................................................................. 17

VSPEX Proven Infrastructure ................................................................................ 17

VMware documentation ....................................................................................... 17

Backup and recovery ........................................................................................... 17

Chapter 3 Solution Overview 19

Overview .................................................................................................................. 20

Solution architecture ................................................................................................ 20

Key components ....................................................................................................... 21

Introduction ......................................................................................................... 21

EMC VSPEX .......................................................................................................... 22

Oracle Database 11g ........................................................................................... 24

VMware vSphere 5.1 ............................................................................................ 25

VMware vSphere HA ............................................................................................ 25

VMware vSphere Distributed Resource Scheduler ................................................ 25

VMware vSphere PowerCLI ................................................................................... 25

EMC Next-Generation VNX .................................................................................... 25

VNX performance ................................................................................................. 27

Virtualization management .................................................................................. 28

Red Hat Enterprise Linux 6.3 ................................................................................ 30

Contents


EMC backup and recovery solutions ..................................................................... 30

Chapter 4 Solution Implementation 31

Overview .................................................................................................................. 32

Setting up the physical components ......................................................................... 32

Implementing the network ........................................................................................ 33

Implementing the storage array ................................................................................ 34

Overview .............................................................................................................. 34

Setting up initial VNX configuration ..................................................................... 35

Provisioning storage for VMFS example ............................................................... 35

Provisioning storage for NFS example .................................................................. 36

Configuring FAST Cache (optional) ....................................................................... 40

Configuring FAST VP (optional) ............................................................................. 41

Example storage layout ........................................................................................ 43

Implementing ESXi and vCenter ................................................................................ 44

Preparing the Oracle Database 11g R2 virtual machine template .............................. 45

Overview .............................................................................................................. 45

Creating the Oracle Database 11g R2 virtual machine .......................................... 45

Completing the Oracle install pre-requisites ........................................................ 46

Installing the Oracle Database 11g R2 binaries and patches ................................ 48

Preparing the Oracle Database 11g R2 virtual machine template ......................... 50

Customizing the VMware template ....................................................................... 50

Deploying the Oracle Database 11g R2 virtual machine ............................................ 51

Overview .............................................................................................................. 51

Deploying new virtual machines .......................................................................... 51

Cloning the Oracle binaries ona new virtual machine ........................................... 52

Determining the database prerequisites .............................................................. 53

Configuring the database file layout for NFS ......................................................... 54

Configuring the database file layout for Block ...................................................... 56

Configuring the initialization parameters ............................................................. 57

Enabling the HugePages setting ........................................................................... 58

Creating the database using DBCA ....................................................................... 58

Performing the database post creation activities ................................................. 59

Implementing backup and recovery .......................................................................... 61

Overview .............................................................................................................. 61

Chapter 5 Solution Verification 63

Verifying the baseline infrastructure ......................................................................... 64

Overview .............................................................................................................. 64

Verifying the ESXi functionality ............................................................................ 64

Contents



Verifying the solution component redundancy ..................................................... 64

Verifying the Oracle Database configuration ........................................................ 65

Using the health monitoring tools ........................................................................ 65

Verifying the Oracle Database................................................................................... 66

Overview .............................................................................................................. 66

Understanding the key metrics ............................................................................ 66

Reviewing the Oracle dataset ............................................................................... 68

Backup and recovery verification methodology ......................................................... 68

Chapter 6 Reference Documentation 69

EMC documentation ................................................................................................. 70

EMC ..................................................................................................................... 70

Oracle .................................................................................................................. 70

VMware ............................................................................................................... 70

Other documentation ............................................................................................... 71

Links ........................................................................................................................ 71

Appendix A Configuration Worksheet 72

Pre-configuration worksheet for VSPEX ..................................................................... 73

Appendix B: Example Virtual Disk Layout 77

Example Virtual Disk Layout of Oracle Database on the VMFS Datastore ................... 78

Figures

6 EMC VSPEX For Virtualized Oracle Database 11g OLTP Enabled By EMC Next-Generation VNX and EMC Backup Implementation Guide

Figures Figure 1. Architecture of the validated infrastructure .......................................... 21

Figure 2. VSPEX Proven Infrastructure ................................................................ 22

Figure 3. Next-Generation VNX with multicore optimization................................ 27

Figure 4. Active/active processors increase performance, resiliency, and efficiency ............................................................................................. 28

Figure 5. New Unisphere Management Suite ...................................................... 29

Figure 6. Redundant 10 GbE network configuration ............................................ 34

Figure 7. Redundant Fibre Channel storage network........................................... 34

Figure 8. Oracle Database 11g R2 storage elements .......................................... 35

Figure 9. Example storage layout of Oracle database on VMware ....................... 36

Figure 10. Configuring EMC Unified Storage Management in vSphere Client ......... 37

Figure 11. Configuring NFS Storage on VNX for Oracle VMs .................................. 37

Figure 12. Configuring VNX storage pools ............................................................ 38

Figure 13. Unisphere Create File System Wizard ................................................... 38

Figure 14. EMC Unisphere Create NFS Export Wizard ............................................ 39

Figure 15. Enabling FAST Cache in the Storage Pool Properties dialog .................. 40

Figure 16. FAST VP creating storage pool.............................................................. 41

Figure 17. Tier status before data movement ....................................................... 42

Figure 18. FAST VP in a steady state ..................................................................... 42

Figure 19. Example virtualized oracle storage layout for VSPEX ............................ 43

Figure 20. Oracle Universal Installer – Install choose database software only ...... 48

Figure 21. Enable Oracle Direct NFS client ............................................................ 49

Figure 22. Example of script root_clone.sh ........................................................... 49

Figure 23. Example of script cloneHome.sh .......................................................... 49

Figure 24. Virtual machine template example ...................................................... 50

Figure 25. vSphere custom specification example ............................................... 50

Figure 26. Example PowerCLI script to deploy a virtual machine from template .... 52

Figure 27. Example PowerCLI script to clone and reattach the ORACLE_HOME ...... 53

Figure 28. Oracle Net Configuration Assistant utility............................................. 53

Figure 29. Example /etc/mtab NFS mount options ............................................... 54

Figure 30. Oranfstab example .............................................................................. 55

Figure 31. Example of /etc/fstab .......................................................................... 57

Figure 32. Oracle Database Configuration Assistant setup screen ........................ 59

Figure 33. Oracle startup/shutdown script /etc/init.d/dbora ............................... 60

Figure 34. Swingbench interface load panel screenshot ....................................... 67

Figure 35. Swingbench order entry screen for workload simulation ...................... 67

Figures



Tables


Tables Table 1. Terminology......................................................................................... 12

Table 2. Tasks for pre-deployment .................................................................... 14

Table 3. Virtualized oracle deployment workflow for VSPEX .............................. 14

Table 4. Deployment prerequisites checklist ..................................................... 16

Table 5. Reference virtual machine characteristics ............................................ 23

Table 6. Mapping the Oracle sizing model to the VSPEX Reference Virtual Machine .............................................................................................. 23

Table 7. Tasks for physical setup ...................................................................... 32

Table 8. Tasks for switch and network configuration ......................................... 33

Table 9. Tasks for storage configuration ............................................................ 34

Table 10. Steps for installing and configuring ESXi and vCenter .......................... 44

Table 11. Virtual machine template specification ................................................ 45

Table 12. Oracle install prerequisites .................................................................. 46

Table 13. RHEL 6 RPM packages for Oracle Database 11g R2 .............................. 47

Table 14. Installing the Oracle Database binaries and patches ........................... 48

Table 15. Oracle Database 11g R2 reference virtual machine model ................... 51

Table 16. Database file layout for NFS ................................................................. 54

Table 17. Tasks for verifying the VSPEX Proven Infrastructure installation .......... 64

Table 18. Tools for monitoring the solution ......................................................... 65

Table 19. General operations .............................................................................. 66

Table 20. Schema tables and indexes ................................................................. 68

Table 21. Common server information ................................................................ 73

Table 22. ESXi server information ....................................................................... 73

Table 23. Array information ................................................................................. 74

Table 24. Network infrastructure information ...................................................... 74

Table 25. VLAN information ................................................................................ 74

Table 26. Service accounts ................................................................................. 75

Table 27. Example Virtual Disk Layout ................................................................ 78

Chapter 1: Introduction



Chapter 1 Introduction

This chapter presents the following topics:

Purpose of this guide ............................................................................................... 10

Business value ......................................................................................................... 10

Scope ....................................................................................................................... 11

Audience .................................................................................................................. 11

Terminology ............................................................................................................. 12



Purpose of this guide

EMC® VSPEX® Proven Infrastructures are optimized for virtualizing business-critical applications. VSPEX provides modular solutions built with technologies that enable faster deployment, more simplicity, greater choice, higher efficiency, and lower risk.

VSPEX provides partners with the ability to design and implement the virtual assets required to support a fully integrated virtualized solution for Oracle relational database management systems on a VSPEX private cloud infrastructure.

The VSPEX for virtualized Oracle infrastructure provides customers with a modern system, capable of hosting a virtualized database solution that is scalable and delivers a constant performance level. This solution uses VMware vSphere to provide virtualization, with EMC Next-Generation VNX® providing storage, and EMC Avamar® and EMC Data Domain® for backup. The compute and network components, while vendor-definable, are designed to provide redundancy and sufficient power to handle the processing and data needs of the virtual machine environment.

This Implementation Guide describes how to implement, using best practices, the VSPEX Proven Infrastructure for virtualized Oracle Databases on a VSPEX Private Cloud with VMware vSphere.

Business value

Database management system software continues to be the dominant software used to manage data in nearly all commercial segments. This growth is expected to continue despite the increasing market share of other data management tools. This growth is expected to accelerate as customers continue to diversify their infrastructures and supporting technologies and drive towards more hardware and software appliances and configurations.

This VSPEX Proven Infrastructure is focused on helping EMC partners understand the value that the VNX series, EMC backup and recovery systems, and Oracle bring to customers who often have growing, isolated IT environments running server-centric applications and who face increasing Oracle backup and recovery issues.

This VSPEX solution is designed to meet the customer’s Oracle database challenges while enabling customers to grow in performance, scalability, reliability, and automation. By consolidating their database applications on VNX, they can consolidate onto a single centralized storage platform that enables them to more effectively manage the exploding growth in data that is challenging businesses today. This solution has been sized and proven to:

Deploy faster, saving time and effort with Proven solutions

Increase performance and scalability out of the box

Reduce the customer’s backup storage requirements and costs

Accommodate backup windows

Enable fast disk-based recovery




Scope

This Implementation Guide describes the high-level steps required to deploy the VSPEX solution for Oracle RDBMS and Online Transaction Processing (OLTP) systems. This VSPEX application solution is based on the current VSPEX private cloud with VMware solution and has been modified to support Oracle Database requirements. This solution uses the VSPEX standard or “reference virtual machine” building blocks of vCPU, memory, and storage and I/O configurations, and describes the implementation based on three (small, medium, and large) different database workload thresholds.

This Implementation Guide assumes that a VSPEX Proven Infrastructure already exists in the customer’s environment, which includes VMware vSphere,VNX, and EMC backup and recovery systems.

Audience

This guide is intended for internal EMC personnel and qualified EMC VSPEX partners. This guide assumes that the VSPEX partners who intend to deploy this solution are:

Qualified by EMC to sell, install, and configure the EMC VNX family of storage systems

Qualified to sell, install, and configure the network and server products required for VSPEX Proven Infrastructures

Certified for selling VSPEX Proven Infrastructures

Partners implementing this solution should also have the necessary technical training and background to install and configure:

VMware vSphere 5.1

Red Hat Enterprises Linux 6.3

Oracle Database 11g R2 or above

EMC next-generation backup, which includes EMC Avamar and EMC Data Domain

This document provides external references where applicable. EMC recommends that readers are familiar with these documents. For details, refer to the section entitled EMC documentation.



Terminology

Table 1 lists the terminology used in the guide.

Table 1. Terminology

Term Definition

AWR Automatic Workload Repository

DNFS Oracle Direct NFS client

DNS Domain name system

eMLC Enterprise multilevel cell

FAST VP Fully Automated Storage Tiering for Virtual Pools

FQDN Fully Qualified Domain Name

IOPS Input/output operations per second

FRA Fast Recovery Area (Oracle)

NFS Network File System

NL-SAS Near-line serial-attached SCSI

OLTP Online transaction processing

Oracle EE Oracle Enterprise Edition

Oracle SE Oracle Standard Edition

PowerCLI A Windows PowerShell interface to the VMware vSphere and vCloud APIs

Reference virtual machine

Represents a unit of measure for a single virtual machine to qualify the compute resources in a VSPEX Proven Infrastructure

SGA System global area

Statspack Oracle database monitoring and reporting utilities

TPS Transactions per second

VDM Virtual Data Mover

VMDK VMware virtual machine disk

VMFS VMware virtual machine file system

Chapter 2: Before You Start



Chapter 2 Before You Start


Overview .................................................................................................................. 14

Pre-deployment tasks .............................................................................................. 14

Deployment workflow .............................................................................................. 14

Deployment prerequisites ........................................................................................ 15

Planning and sizing Oracle Database 11g R2 ........................................................... 17

Essential Reading .................................................................................................... 17



Overview

Before you layer the Oracle OLTP on a VSPEX private cloud, EMC recommends that you complete the pre-deployment tasks listed in Table 2.

Pre-deployment tasks

Pre-deployment tasks include procedures that do not directly relate to environment installation and configuration, but whose results are needed during installation. Examples of pre-deployment tasks include the collection of hostnames, IP addresses, VLAN IDs, license keys, installation media, and so on. Perform these tasks before visiting the customer to decrease the time required on site.

Table 2. Tasks for pre-deployment

Task Description Reference

Gather documents Gather the related documents listed in Essential Reading. These are used throughout the text of this document to provide details on setup procedures and deployment best practices for the various components of the solution.

Essential Reading

Gather tools Gather the required and optional tools for the deployment. Use Table 4 to confirm that all equipment, software, and appropriate licenses are available before the deployment process.

Deployment prerequisites

Gather data Collect the customer-specific configuration data for networking, naming, and required accounts. Enter this information into the Customer Oracle Database 11g R2 configuration worksheet for reference during the deployment process.

Appendix A Configuration Worksheet

Deployment workflow

EMC recommends that you follow the process flow in Table 3 to design and implement your VSPEX Proven Infrastructure for virtualized Oracle Database 11g R2 solution.

Table 3. Virtualized oracle deployment workflow for VSPEX

Step Action

1 Use the VSPEX qualification worksheet to collect user requirements. The qualification worksheet is in Appendix A of the companion ”EMC VSPEX FOR VIRTUALIZED ORACLE DATABASE 11g OLTP Design Guide”




Step Action

2 Use the VSPEX Sizing Tool to determine the recommended VSPEX Proven Infrastructure for a virtualized Oracle Database 11g R2 solution. For more information about the VSPEX Sizing Tool, refer to the VSPEX Sizing Tool on the EMC Business Value Portal.

Note You need to register the first time you access the tool. If the VSPEX Sizing Tool is not available, you can manually size the application using the sizing guidelines in Appendix B of the companion Design Guide.

3 To determine the final design for the VSPEX Proven Infrastructure for virtualized Oracle Database 11g, refer to the companion Design Guide.

Note Ensure that all application requirements are considered, not just this application.

4 To select and order the right solution, refer to the appropriate VSPEX Proven Infrastructure document in Essential Reading.

5 To deploy and test your VSPEX solution, refer to this guide.

Deployment prerequisites

Table 4 itemizes the hardware and software requirements to configure the solution.

For additional information, refer to the hardware and software tables in the relevant document in the Essential Reading section.

http://express.salire.com/go/emc



Table 4. Deployment prerequisites checklist

Require-ment Description Version Reference notes

Hardware

Physical servers: sufficient physical server capacity to host the required number of virtual machines as recommended by the companion Design Guide and VSPEX Sizing Tool.

Reference Architecture:

EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 1000 Virtual Machines

EMC Avamar 6.1 hardware and licenses

EMC Data Domain hardware and licenses

VMware vSphere 5.1 servers to the host virtual infrastructure servers. This requirement may be covered in the existing infrastructure.

Networking: switch port capacity and capabilities as required by the virtual server infrastructure.

EMC VNX: multiprotocol storage array with the required disk layout.

EMC Backup: Avamar 6.1, Data Domain

Software

VNX OE for file Release 8.0.10000.34653

EMC online support

VNX OE for block Release 05.33.000.3.653

EMC VSI for VMware vSphere: Unified Storage Management

5.4

EMC VSI for VMware vSphere: Storage Viewer 5.4

VMware ESXi installation media 5.1

VMware vCenter Server installation media 5.1

EMC Avamar 6.1 in server and client versions

Red Hat Enterprise Linux 6.3 Red Hat Enterprise Linux 6.3 media

www.redhat.com

Licenses

VMware vSphere license keys. This may be covered in the VSPEX Proven Infrastructure.

5.1 www.vmware.com

Red Hat Enterprise Linux 6.3 www.redhat.com

Oracle Database 11g R2 11.2.0.3.6 www.oracle.com

http://www.redhat.com/

http://www.redhat.com/

http://www.oracle.com/




Planning and sizing Oracle Database 11g R2

To plan and size your Oracle Database 11g R2 deployment on the VSPEX infrastructure, you should follow the recommendations in the companion Design Guide. Size the storage accordingly to information obtained from the VSPEX for Virtualized Oracle 11g Qualification worksheet as detailed in Appendix A and B of the companion Design Guide.

Essential Reading

Before implementing the solution described in this document, EMC recommends that you read the following documents, available from the VSPEX space in the EMC Community Network or from EMC.com or the VSPEX Proven Infrastructure partner portal.

Design Guide: EMC VSPEX for Virtualized Oracle Database 11g OLTP

EMC VSPEX Server Virtualization for Midmarket Businesses

EMC VSPEX Server Virtualization for Small and Medium Businesses

Reference Architecture: EMC VSPEX Private Cloud: VMware vSphere for up to 1,000 Virtual Machines

White Paper: EMC Avamar Backup for Oracle Environments

White Paper: EMC Avamar Backup with Data Domain VMware VSphere Product Documentation

Refer to the following backup and recovery papers:

White Paper: EMC Avamar Backup for Oracle Environments

White Paper: EMC Avamar Backup with Data Domain

White Paper: EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11gR2 Design and Implementation Guide i

Design guide

Solution overviews

VSPEX Proven Infrastructure

VMware documentation

Backup and recovery

https://community.emc.com/community/partner/vspex?view=overview

https://community.emc.com/community/partner/vspex?view=overview

http://www.emc.com/

http://www.emc.com/platform/virtualizing-information-infrastructure/vspex.htm


Chapter 3: Solution Overview



Chapter 3 Solution Overview


Overview .................................................................................................................. 20

Solution architecture ............................................................................................... 20

Key components ...................................................................................................... 21



Overview

This chapter provides an overview of the VSPEX Proven Infrastructure for Oracle Database 11g and the key technologies used in this solution. The solution described in this Implementation Guide includes servers, storage, network components, and Oracle Database 11g components.

The solution enables customers to quickly and consistently deploy a virtualized Oracle Database 11g in the VSPEX Proven Infrastructure. The reference architecture will consume the reference virtual machine resources, based on the sizing guidance in the VSPEX Proven Infrastructure, and combine with additional storage for Oracle Database 11g application data.

This Implementation Guide can help EMC personnel and qualified EMC VSPEX Partners to deploy a simple, effective, and flexible Oracle Database 11g solution on a VSPEX Proven Infrastructure for their customers.

Solution architecture

Figure 1 shows the architecture that characterizes the infrastructure validated for an Oracle Database 11g overlay on a VSPEX infrastructure. To validate this solution, we1:

Deployed all Oracle Database 11g servers as virtual machines on VMware vSphere 5.1.

Used the VSPEX sizing tool for Oracle Database 11g to determine the number of, and the detailed compute resources for, each Oracle Database 11g database. Figure 1 displays an example with three Oracle sizing options (small, medium, and large). Use the sizing tools provided with this solution to size your customer’s environment and choose the options that best suit your customer.

Determined the recommended storage layout for Oracle Database 11g and the virtual infrastructure pool in the VNX series storage arrays (using the VSPEX sizing tool).

Note: The minimum Oracle version for this solution is 11.2.0.3. We refer to this as 11gR2 throughout this document.

1 In this paper, "we" refers to the EMC Solutions engineering team that validated the solution.




Figure 1. Architecture of the validated infrastructure

Key components

This section provides an overview of the key technologies used in this solution:

EMC VSPEX

Oracle Database 11g

VMware vSphere 5.1

VMware vSphere HA

vSphere Distributed Resources Scheduler

VMware vSphere PowerCLI

EMC Next-Generation VNX®

Red Hat Enterprise Linux 6.3

EMC Avamar

EMC Data Domain

Introduction



EMC has joined forces with the industry’s leading providers of IT infrastructure to create a complete virtualization solution that accelerates the deployment of private cloud technologies. Built with best-of-breed technologies, VSPEX enables faster deployment, more simplicity, greater choice, higher efficiency, and lower risk.

VSPEX Proven Infrastructure, as shown in Figure 2, is a modular, virtualized system validated by EMC and delivered by EMC partners. VSPEX includes a virtualization layer, server, network, and storage, designed by EMC to deliver reliable and predictable performance.

Figure 2. VSPEX Proven Infrastructure

VSPEX provides the flexibility to choose the best network, server, and virtualization technologies that fit a customer’s environment to create a complete virtualization solution.

VSPEX provides a virtual infrastructure for customers looking to gain the simplicity of a truly converged infrastructure, while gaining flexibility in individual components of the stack. VSPEX solutions, proven by EMC, are packaged and sold exclusively by EMC channel partners. VSPEX provides channel partners with more opportunity, a faster sales cycle, and end-to-end enablement. By working even more closely together, EMC and its channel partners can now deliver an infrastructure that accelerates the journey to the cloud for even more customers.

EMC VSPEX




Reference virtual machine

To simplify the virtual infrastructure discussion, the VSPEX solution has defined a typical customer workload (described in this section) as a reference virtual machine. For VSPEX solutions, we define the reference virtual machine as a measure unit of a single virtual machine to qualify the compute resources in the VSPEX virtual infrastructure. Table 5 lists the characteristics of this virtual machine.

Table 5. Reference virtual machine characteristics

Characteristic Value

Virtual processors per virtual machine 1

RAM per virtual machine 2 GB

Available storage capacity per virtual machine 100 GB

I/O operations per second (IOPS) per virtual machine

25

I/O pattern Random

I/O read/write ratio 2:1

VSPEX for virtualized Oracle sizing model

Scale-up testing formed part of the validation process. We used a standard compute-sizing model for Oracle, which simplified and standardized the validation testing. It also enabled us to identify the configuration required to run a TCP-C like OLTP database workload with a 60:40 read/write ratio, yielding acceptable response times.

Table 6 shows how we mapped the Oracle sizing model to the VSPEX reference virtual machine.

Table 6. Mapping the Oracle sizing model to the VSPEX Reference Virtual Machine

Oracle model Resources Equivalent reference virtual machine

Small—virtual machine for up to 150 users

Compute requirements:

2 vCPU

8 GB memory

Storage requirements (OS & Oracle binaries):

100 GB

25 IOPS

4

Medium—virtual machine for up to 250 users


4 vCPU

16 GB of memory


100 GB

25 IOPS

8



Oracle model Resources Equivalent reference virtual machine

Large—virtual machine for more than 250 Users


8 vCPU

32 GB of memory


100 GB

25 IOPS

16

We calculated the database storage I/O thresholds and capacity separately from those required by the VSPEX reference virtual machine.

Oracle Database 11g is available in a variety of editions tailored to meet the business and IT needs of an organization. In this solution we will be considering

Oracle Database 11g Release 2 Standard Edition (SE)

Oracle Database 11g Release 2 Enterprise Edition (EE)

Oracle Database 11g SE is an affordable, full-featured data management solution that is ideal for all companies. It is available on single or clustered servers and can be licensed on a maximum capacity of four processor sockets, regardless of core count. The SE license includes Oracle Real Application Clusters (RAC) as a standard feature with no additional cost.

Oracle Database 11g EE delivers industry-leading performance, scalability, security, and reliability on a choice of clustered or single servers running Windows, Linux, or UNIX. It supports advanced features, either included or as extra-cost options, that are not available with Oracle Database 11g SE. These include security features such as Virtual Private Database and data warehousing options such as partitioning and advanced analytics. Oracle Database 11g Release 2 EE extends the processor-licensing model for multi-core processors and is priced using the following formula:

(number of Processors) x (number of cores) x (Oracle Processor Core Factor)

For example, two 10-core Intel Xeon Processor E7-2870s (with an Oracle Processor Core Factor of 0.5) are licensed as follows

Oracle Database 11g Release 2 SE: 2 processor socket SE licenses

Oracle Database 11g Release 2 EE: 2 x 10 x 0.5 = 10 EE licenses

The Oracle Database 11g R2 edition can affect the licensing cost and the size and number of VMware ESXi clusters you can configure. This affects how you place and manage the virtual machines. Refer to the document entitled Design Guide for EMC VSPEX for Virtualized Oracle Database 11g OLTP, in the DRS Host Affinity and Oracle processor licensing section, for more information on virtualization and Oracle processor licensing.

Oracle Database 11g




VMware vSphere 5.1 abstracts applications and information from the complexity of underlying infrastructure through comprehensive virtualization of server, storage, and networking hardware. This transformation creates fully functional virtual machines that run isolated and encapsulated operating systems and applications just like physical computers. This virtualization of hardware resources enables efficiencies through consolidation of multiple applications on fewer physical servers.

VMware vSphere High Availability (HA) provides easy-to-use, cost effective high availability for applications running in virtual machines. In the event of a physical server failure, affected virtual machines automatically restart on other production servers with spare capacity.

HA enables you to create a cluster out of multiple ESXi servers, enabling you to protect virtual machines. If one of the hosts in the cluster fails, the impacted virtual machines automatically restart on other ESXi hosts within that same VMware vSphere cluster.

VMware vSphere Distributed Resource Scheduler (DRS) is an infrastructure service run by VMware vCenter Server (vCenter). DRS aggregates ESXi host resources into clusters and automatically distributes these resources to virtual machines by monitoring utilization and continuously optimizing virtual machine distribution across ESXi hosts. DRS can also use vMotion® and Storage vMotion® to ensure that the virtual machines have access by rebalancing resource capacity to make room for larger virtual machines. VMware recommends enabling DRS to achieve higher consolidation ratios.

VMware vSphere PowerCLI provides a Windows PowerShell interface for the users of vSphere 5.1 and above and VMware Infrastructure 4.x and above. VMware vSphere PowerCLI is a powerful command-line tool that lets you automate all aspects of vSphere management, including network, storage, VM, guest OS and more. PowerCLI is distributed as a Windows PowerShell snap-in, and includes 330 PowerShell cmdlets for managing and automating vSphere and vCloud, along with documentation and samples.

The EMC VNX flash-optimized unified storage platform delivers innovation and enterprise capabilities for file, block, and object storage in a single, scalable, and easy-to-use solution. Ideal for mixed workloads in physical or virtual environments, VNX combines powerful and flexible hardware with advanced efficiency, management, and protection software to meet the demanding needs of today’s virtualized application environments.

VNX includes many features and enhancements designed and built upon the first generation’s success. These features and enhancements include:

More capacity with multicore optimization with Multicore Cache, Multicore RAID, and Multicore FAST Cache (MCx)

Greater efficiency with a Flash-optimized hybrid array

Better protection by increasing application availability with active/active

VMware vSphere 5.1

VMware vSphere HA

VMware vSphere Distributed Resource Scheduler

VMware vSphere PowerCLI

EMC Next-Generation VNX



Easier administration and deployment by increasing productivity with new Unisphere Management Suite

VSPEX is built with the next generation of VNX to deliver even greater efficiency, performance, and scale than ever before.

Flash-optimized hybrid array

VNX is a flash-optimized hybrid array that provides automated tiering to deliver the best performance to your critical data, while intelligently moving less frequently accessed data to lower-cost disks.

In this hybrid approach, a small percentage of flash drives in the overall system can provide a high percentage of the overall IOPS. Flash-optimized VNX takes full advantage of the low latency of flash to deliver cost-saving optimization and high performance scalability. The EMC Fully Automated Storage Tiering Suite (FAST Cache and FAST VP) tiers both block and file data across heterogeneous drives and boosts the most active data to the flash, ensuring that customers never have to make concessions for cost or performance.

Data is generally accessed most frequently at the time it is created, therefore new data is first stored on flash drives to provide the best performance and latency. As that data ages and becomes less active over time, FAST VP tiers the data from high-performance to high-capacity drives automatically, based on customer-defined policies. This functionality has been enhanced with four times better granularity and with new FAST VP solid-state disks (SSDs) based on enterprise multi-level cell (eMLC) technology to lower the cost per gigabyte. FAST Cache dynamically absorbs unpredicted spikes in system workloads. All VSPEX use cases benefit from the increased efficiency.

VSPEX Proven Infrastructures deliver private cloud, end-user computing, and virtualized application solutions. With VNX, customers can realize an even greater return on their investment. VNX provides out-of-band, block-based deduplication that can dramatically lower the costs of the Flash tier.

VNX Intel MCx Code Path Optimization

The advent of Flash technology has been a catalyst in totally changing the requirements of midrange storage systems. EMC redesigned the midrange storage platform to efficiently optimize multicore CPUs to provide the highest performing storage system at the lowest cost in the market.

MCx distributes all VNX data services across all cores—up to 32, as shown in Figure 3. The VNX series with MCx has dramatically improved the file performance for transactional applications like databases or virtual machines over network-attached storage (NAS).




Figure 3. Next-Generation VNX with multicore optimization

Multicore Cache The cache is the most valuable asset in the storage subsystem; its efficient use is key to the overall efficiency of the platform in handling variable and changing workloads. The cache engine has been modularized to take advantage of all the cores available in the system.

Multicore RAID Another important part of the MCx redesign is the handling of I/O to the permanent back-end storage—hard disk drives (HDDs) and SSDs. Greatly increased performance improvements in VNX come from the modularization of the back-end data management processing, which enables MCx to seamlessly scale across all processors.

VNX storage, enabled with the MCx architecture, is optimized for FLASH 1st and provides unprecedented overall performance, optimizing for transaction performance (cost per IOPS), bandwidth performance (cost per GB/s) with low latency, and providing optimal capacity efficiency (cost per GB).

VNX provides the following performance improvements:

Up to four times more file transactions when compared with dual controller arrays

Increased file performance for transactional applications (for example, Microsoft Exchange on VMware over NFS) by up to three times with a 60 percent better response time

Up to four times more Oracle and Microsoft SQL Server OLTP transactions

Up to six times more virtual machines

VNX performance



Active/active array service processors

The new VNX architecture provides active/active array service processors, as shown in Figure 4, which eliminate application timeouts during path failover since both paths are actively serving I/O.

Note: The active/active processors are only available for classic LUNs not for pool LUNs.

Load balancing is also improved and applications can achieve an up to two times improvement in performance. Active/active for block is ideal for applications that require the highest levels of availability and performance, but do not require tiering or efficiency services like compression, deduplication, or snapshot.

With this VNX release, VSPEX customers can use virtual Data Movers (VDMs) and VNX Replicator to perform automated and high-speed file system migrations between systems. This process migrates all snaps and settings automatically, and enables the clients to continue operation during the migration.

Figure 4. Active/active processors increase performance, resiliency, and efficiency

VMware Virtual Storage Integrator

Virtual Storage Integrator (VSI) is a no-charge VMware vCenter plug-in available to all VMware users with EMC storage. VSPEX customers can use VSI to simplify management of virtualized storage. VMware administrators can gain visibility into their VNX storage using the same familiar vCenter interface to which they are accustomed.

With VSI, IT administrators can do more work in less time. VSI offers unmatched access control that enables you to efficiently manage and delegate storage tasks with confidence. Perform daily management tasks with up 90 percent fewer clicks and up to 10 times higher productivity.

VMware vStorage APIs for Array Integration

VMware vStorage APIs for Array Integration (VAAI) offloads VMware storage-related functions from the server to the storage system, enabling more efficient use of server and network resources for increased performance and consolidation.

Virtualization management




VMware vStorage APIs for Storage Awareness

VMware vStorage APIs for Storage Awareness (VASA) is a VMware-defined API that displays storage information through vCenter. Integration between VASA technology and VNX makes storage management in a virtualized environment a seamless experience.

EMC Storage Integrator

EMC Storage Integrator (ESI) is targeted towards the Windows and Application administrator. ESI is easy to use, delivers end-to end monitoring, and is hypervisor agnostic. Administrators can provision in both virtual and physical environments for a Windows platform, and troubleshoot by viewing the topology of an application from the underlying hypervisor to the storage.

Unisphere Management Suite

EMC Unisphere is the central management platform for the VNX series, providing a single, combined view of file and block systems, with all features and functions available through a common interface. Unisphere is optimized for virtual applications and provides industry-leading VMware integration, automatically discovering virtual machines and ESX servers and providing end-to-end, virtual-to-physical mapping. Unisphere also simplifies configuration of FAST Cache and FAST VP on VNX platforms.

The new Unisphere Management Suite extends Unisphere’s easy-to-use, interface to include VNX Monitoring and Reporting for validating performance and anticipating capacity requirements. As shown in Figure 5, the suite also includes Unisphere Remote for centrally managing up to thousands of VNX and VNXe systems with new support for XtremSW Cache.

Figure 5. New Unisphere Management Suite



Red Hat Enterprise Linux is a versatile platform for x86 and x86-64 that can be deployed on physical systems, as a guest on the major hypervisors, or in the cloud. It supports all leading hardware architectures with compatibility across releases. Red Hat Enterprise Linux 6.3 includes enhancements and new capabilities that provide rich functionality, especially the developer tools, virtualization features, security, scalability, file systems, and storage.

EMC Avamar and EMC Data Domain deliver the protection confidence needed to accelerate deployment of virtualized Oracle. Optimized for virtualized application environments, EMC backup and recovery reduces backup times by 90 percent and increases recovery speeds by 30 times,even offering instant virtual machine access,for worry-free protection.

EMC backup also delivers big savings. Our deduplication solutions reduce backup storage by 10 to 30 times, backup management time by 81 percent, and bandwidth by 99 percent for efficient offsite replication, delivering a seven month payback on average.

Furthermore, EMC backup offers a solution with Data Domain systems and DD Boost software that allows for full DBA control of Oracle backup, recovery, and replication while the backup team maintains control of the infrastructure. This eliminates the occurrence of protection silos, increasing efficiency and lowering risk.

Red Hat Enterprise Linux 6.3

EMC backup and recovery solutions

Chapter 4: Solution Implementation



Chapter 4 Solution Implementation


Overview .................................................................................................................. 32

Setting up the physical components ........................................................................ 32

Implementing the network ....................................................................................... 33

Implementing the storage array ............................................................................... 34

Implementing ESXi and vCenter ............................................................................... 44

Preparing the Oracle Database 11g R2 virtual machine template ............................ 45

Deploying the Oracle Database 11g R2 virtual machine ........................................... 51

Implementing backup and recovery ......................................................................... 61



Overview

This chapter discusses the steps required to install, configure, and implement the VSPEX Proven Infrastructure environment, which involves:

Preparing the components and their physical setup.

Implementing the network, including the configuration steps for physical and virtual switches in line with vendor guidelines and The vSphere Networking Guide.

Configuring the VNX storage array and optional FAST Cache or FAST VP feature.

Building the VMware vSphere environment with ESXi server and vCenter implementation.

Preparing a virtual machine template for rapid deployment of Oracle Database 11g R2.

Quickly and simply deploying an Oracle Database 11g R2 virtual machine using PowerCLI.

Implementing backup and recovery.

Setting up the physical components

This section includes information about how to prepare the solution’s physical components. After you complete the steps listed in Table 7, the new hardware components will be racked, cabled, powered, and ready for network connection.

Note If you already have a VSPEX Proven Infrastructure environment, you can skip this section.

Table 7. Tasks for physical setup


Prepare network switches

Install switches in the rack and connect them to power.

Your vendor’s installation guide

Prepare servers Install the servers in the rack and connect them to power.

Your vendor’s installation guide

Prepare VNX Install the VNX in the rack and connect it to power.

EMC VNX System Installation Guide

For details of the physical setup, refer to Reference Architecture: EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 1,000 Virtual Machines




Implementing the network

This section documents requirements for the network infrastructure needed to support this architecture. Table 8 provides a summary of the tasks for switch and network configuration and references for further information.

Note: If you already have a VSPEX Proven Infrastructure environment, you can skip this section.

Table 8. Tasks for switch and network configuration


Configure network

Configure storage network and host infrastructure networking as specified in the solution reference architecture.

Refer to the appropriate document in Essential Reading.

Complete network cabling

Connect:

Switch interconnect ports

VNX ports

ESXi NIC/FC ports

N/A

Configure VLAN Configure private and public VLANs as required.

Vendor’s switch configuration guide

Configure VMare vSphere networking

Create virtual switches for separation of application, management, and storage networks.

vSphere Networking Guide

For details of network implementation, refer to the appropriate document in Essential Reading.

Figure 6 and Figure 7 show the configured vSphere network for this implementation. As highlighted, redundant 10GbE network adapters or FC Host adapters have been dedicated for storage network traffic.



Figure 6. Redundant 10 GbE network configuration

Figure 7. Redundant Fibre Channel storage network

Implementing the storage array

This section describes how to configure the VNX storage array. In this solution, the VNX provides Network File System (NFS) or Virtual Machine File System (VMFS) data storage for the VMware host.

Table 9. Tasks for storage configuration


Set up initial VNX configuration

Configure the IP address information and other key parameters on the VNX.

VNX System Installation Guide

VNX Series Configuration Worksheet

Provision storage Create VMFS datastores or create NFS file systems that will be mounted on hosts

Overview




Figure 8 depicts the high-level architecture of this solution.

Figure 8. Oracle Database 11g R2 storage elements

Ensure that network interfaces, IP address information, and other key parameters such as DNS and NTP are configured on the VNX before provisioning the storage.

For more information on how to configure the VNX platform, refer to the appropriate document in Essential Reading.

Before you provision the storage for a VMFS datastore, follow the recommendations and VSPEX Sizing Tool proposals introduced in the companion Design Guide.

Before you start, refer to the Reference Architecture titled EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 1000 Virtual Machines to provision storage for virtual machine operating system on VNX.

Figure 9 shows an example of how to provision a VMFS datastore for the Oracle Database 11g R2 data in VNX. For more information about the storage layout recommendations and design, refer to the Design Guide.

Setting up initial VNX configuration

Provisioning storage for VMFS example



Figure 9. Example storage layout of Oracle database on VMware

For more information on how to lay out virtual disks for the Oracle Database 11g R2 data on the VMFS datastore, refer to the example in Table 27 on page 78.

Present the NFS filesystem for the virtual machine datastore to all ESXi hosts, using either EMC Unisphere or the VSI plugin for VMware vSphere client. Figure 11 displays how how to mount the NFS file systems to ESXi hosts that are used to store virtual servers on the VNX array using VSI.

The EMC Unified Storage Management plugin for VMware vSphere provides an interface for creating and managing NFS file systems for VMware ESXi host directly from the vSphere Client interface. Figure 10 shows the configuration screen accessed through Home >Solutions and applications >EMC from the vSphere Client, which is accessible after running the installer on your management host.

Provisioning storage for NFS example




Figure 10. Configuring EMC Unified Storage Management in vSphere Client

Unified Storage Management enables the vSphere administrator to mount NFS file systems directly to the ESXi cluster nodes from vSphere Client, as shown in Figure 11.

Figure 11. Configuring NFS Storage on VNX for Oracle VMs

To configure NFS network settings, storage pools for file, file systems, and NFS exports on the VNX array in EMC Unisphere, perform the following steps:

1. In Unisphere, select the VNX array for this solution.

2. Select Settings Network Settings for File

3. Configure the IP address for network ports used for NFS. For detailed steps refer to EMC Procedure Generator for VNX.

4. Select Storage Storage Configuration



Figure 12. Configuring VNX storage pools

5. Select the Storage Pools for File tab and create the additional storage pools in the VNX for Oracle Database files, as shown in Figure 12. Refer to Table 27 on page 78 for detailed information.

6. Create the required file systems. Figure 13 shows the Create File System Wizard from Unisphere.

Figure 13. Unisphere Create File System Wizard




7. Create the NFS exports for the database components that the Oracle NFS client will mount on the database virtual machines. Figure 14 shows the Unisphere Create NFS Export Wizard. Root and Access permissions are granted to the NIC.

Figure 14. EMC Unisphere Create NFS Export Wizard



Enabling FAST Cache is a transparent operation to Oracle Database 11g R2 and no reconfiguration or downtime is necessary. To make the best use of either of the FAST technologies, EMC recommends that you first enable FAST Cache on the Oracle Database 11g R2 storage pool. For more details, refer to the EMC VSPEX for Virtualized Oracle Database 11g OLTP Design Guide.

To create and configure FAST Cache, use the following steps:

1. Refer to the Reference Architecture paper entitled EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 1,000 Virtual Machines for detailed steps about how to create FAST Cache.

2. In Unisphere, after creating the FAST Cache, click the Storage tab and select Storage Pool. Select Data Pool, and click Properties.

3. Select the Advanced tab in Storage Pool Properties and click Enabled to enable FAST Cache, as shown in Figure 15.

4. Click OK to complete the configuration.

Note The FAST Cache feature on the VNX series array does not cause an instant performance improvement. The system must collect data about access patterns and promote frequently used information into the cache. This process can take a few hours during which the performance of the array steadily improves.

Figure 15. Enabling FAST Cache in the Storage Pool Properties dialog

Configuring FAST Cache (optional)




We created a three-tier FAST VP with a mixed storage pool, consisting of 5 flash drives, 25 SAS drives and 8 NL-SAS drives on VNX8000. FAST VP automatically relocates the LUN data from one tier to another within a pool.

In this solution, we set the Auto-Tiering policy to Scheduled.For demonstration purpose, we configured the Data Relocation Schedule setting as Monday to Sunday, starting from 00:00 to 23:45, which determines the time window when FAST VP moves data between tiers.

Note: The Data Relocation Rate and Data Relocation Schedule are highly dependent on the real workload in a customer environment. Usually, setting the Data Relocation Rate to Low has less impact on the current running workload.

Set the Tiering Policy for all LUNs containing datafiles to Auto-Tier, so that FAST VP can automatically move the most active data to flash drive devices.

Figure 16. FAST VP creating storage pool

Configuring FAST VP (optional)



Initially, all datafiles were placed on SAS devices, as shown in Figure 17.

Figure 17. Tier status before data movement

With the workload running against the database for a few hours, FAST VP monitored and provided ongoing load balancing of LUNs across available drives until it reached a steady state, as shown in Figure 18.

Figure 18. FAST VP in a steady state




Figure 19 shows an example storage layout for VNX with FAST Cache or FAST VP enabled on the Oracle Database 11g R2 pool.

Figure 19. Example virtualized oracle storage layout for VSPEX

Example storage layout



Implementing ESXi and vCenter

This section documents the requirements for the installation and configuration of the ESXi hosts, vCenter configuration, and infrastructure servers required to support the architecture. Table 10 describes the tasks that must be completed.

Table 10. Steps for installing and configuring ESXi and vCenter

Task Description Reference documents

Install ESXi Install the ESXi 5.1 hypervisor on the physical servers that are being deployed for the solution.

vSphere Installation and Setup

Configure and deploy vCenter Server

Install VMware vCenter Server 5.1 vCenter Server and Host Management Guide

Installing vCenter Server 5.1 best practices

Configure ESXi networking

Configure ESXi networking including NIC trunking, VMkernel port, virtual machine port groups, and Jumbo Frames.

vSphere Networking

Connect VMware datastores

Connect the VMware datastore to the ESXi hosts deployed for the solution.

vSphere Storage

Create a virtual data center

Create a virtual data center. vCenter Server and Host Management

Apply vSphere license keys

Type the vSphere license keys in the vCenter licensing menu.


Add ESXi hosts Connect vCenter to ESXi hosts. vCenter Server and Host Management

Configure vSphere clustering

Create a vSphere cluster vSphere Resource Management

Perform array ESXi host discovery

Perform ESXi host discovery within the Unisphere console.

Using EMC VNX Storage with VMware vSphere–TechBook

Enable VMware High Availability (HA), DRS, and vMotion functionality

Once you enable DRS, you may need to consider the use of VMware vSphere DRS Affinity and Anti-Affinity must rules for specific groups of virtual machines, for example, Oracle VMs may be restricted to a particular set of VMs due to licensing

Automating High Availability (HA) Services with VMware HA –VMware Technical note

Install EMC VNX USM Install the EMC VNX USM on the administration console.

EMC VSI for VMware vSphere: Unified Storage Management—Product Guide

Install the EMC VSI plug-in

Install the EMC VSI plug-in on the administration console.


For details of ESXi and vCenter, refer to the Reference Architecture titled EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 1,000 Virtual Machines.




Preparing the Oracle Database 11g R2 virtual machine template

VMware provides the ability to create a template that you can use as a master copy to quickly create and provision virtual machines. By using a template, you can install a guest OS and apply it to a virtual machine with application users and software configured and ready for use with minimal administrative intervention. This minimizes deployment time and avoids repetitive installation and configuration tasks for each virtual machine that is required.

Customization specifications, maintained in vCenter, further simplify the rollout of virtual machines. A deployment wizard, automation tool, or script can use these templates to automatically pre-create or amend server settings (such as server name, time zone, and network configuration) prior to building the new virtual machine.

This section documents the preparation of a VMware virtual machine template to enable rapid deployment of new Oracle Database 11g R2 environments.

To configure the template with the requirements and prerequisites for the Oracle software install, complete the following steps:

1. Create a virtual machine with the following resources:

2 vCPUs

8 GB of vRAM

100 GB virtual disk

2. Install the operating system and RPM packages

3. Configure the system and kernel parameters

4. Create the OS groups and users

5. Install and detach the Oracle binaries ready for clone

6. Convert the virtual machine to a VMware template

Create a virtual machine on the ESXi server with the customer’s guest OS configuration. Table 11 lists an example specification used when building the virtual machine template.

Table 11. Virtual machine template specification

Part Description

CPU 2 vCPUs

Memory 8 GB

Virtual disk 100GB

8GB swap partition

92GB bootable root partition

Operating system Red Hat Enterprise Linux Server release 6.3

Kernel 2.6.32

Overview

Creating the Oracle Database 11g R2 virtual machine



Part Description

Network interfaces Eth0: public/management IP network

Eth1: Storage Network

OS users Username: oracle

OS groups Group: oinstall, dba

Oracle 11g R2 Binaries Oracle 11.2.0.3 installed and patched

RPM packages installed Refer to the Oracle Installation Guide for prerequisites

System configuration See the Oracle Installation Guide for prerequisites.

This section describes how to install and configure the operating system and Oracle prerequisites in the virtual machine prior to converting it to a template.

Table 12. Oracle install prerequisites


Install the guest OS Install the Red Hat Enterprise Linux 6.3 64-bit operating system and configure the network interfaces.

Red Hat Enterprise Linux 6 Installation Guide

Install VMware tools Install VMware tools using Red Hat Package Manager (RPM)

VMware knowledge Base article ID(1018392)

Install the required RPM packages

Install or update required RPM packages as listed in Table 13.

Oracle Database Installation Guide 11g Release 2 (11.2) for Linux

Set the kernel parameters

Configure the kernel with Oracle-specific settings for semaphores, virtual and shared memory, network, and asynchronous I/O.


Set the resource limits for the Oracle user

Check and adjust the resource limits for the Oracle software installation users


Create the required software directories

Create the required directories for the Oracle software


Configure HugePages

HugePages reduce the number of memory pages and reduces virtual memory management overhead.

Oracle Database 11g Release 2 on Red Hat Enterprise Linux 6 Deployment Recommendations

Table 13 lists the Red Hat Enterprise Linux 6 RPM packages required to install Oracle database 11g R2.

Completing the Oracle install pre-requisites




Table 13. RHEL 6 RPM packages for Oracle Database 11g R2

binutils-2.20.51.0.2-5.11.el6 (x86_64) libstdc++-4.4.4-13.el6.i686 compat-libcap1-1.10-1 (x86_64) libstdc++-devel-4.4.4-13.el6 (x86_64) compat-libstdc++-33-3.2.3-69.el6 (x86_64) libstdc++-devel-4.4.4-13.el6.i686 compat-libstdc++-33-3.2.3-69.el6.i686 libaio-0.3.107-10.el6 (x86_64) gcc-4.4.4-13.el6 (x86_64) libaio-0.3.107-10.el6.i686 gcc-c++-4.4.4-13.el6 (x86_64) libaio-devel-0.3.107-10.el6 (x86_64) glibc-2.12-1.7.el6 (i686) libaio-devel-0.3.107-10.el6.i686 glibc-2.12-1.7.el6 (x86_64) make-3.81-19.el6 glibc-devel-2.12-1.7.el6 (x86_64)10 sysstat-9.0.4-11.el6 (x86_64)

glibc-devel-2.12-1.7.el6.i686 On Oracle Linux 6, and Red Hat Enterprise Linux 6:

ksh unixODBC-2.2.14-11.el6 (x86_64) or later libgcc-4.4.4-13.el6 (i686) unixODBC-2.2.14-11.el6.i686 or later libgcc-4.4.4-13.el6 (x86_64) unixODBC-devel-2.2.14-11.el6 (x86_64) or later libstdc++-4.4.4-13.el6 (x86_64) unixODBC-devel-2.2.14-11.el6.i686 or later



As advised in the Design Guide, decide whether to install Oracle SE or Oracle EE Database 11g R2 software. Both editions use the same installation files.

Table 14 includes the steps required to install the Oracle Database binaries and patches.

Table 14. Installing the Oracle Database binaries and patches


Install the Oracle binaries

Run the Oracle Universal installer and choose to “Install choose database software only” as shown in Figure 20.


Install the latest PSU and CPU

Install the latest database Patch Set Update and instructions available from the My Oracle Support website. A login ID and password as well as a support contract are required.

See Critical Patch Updates, Security Alerts and Third Party Bulletin

Enable DNFS Oracle Database is not shipped with Direct NFS enabled by default. Enable Direct NFS as shown in Figure 21.


Detach the Oracle Home directory?

To remove the Oracle Database 11g R2 software from the Oracle Universal Installer inventory run the script $ORACLE_HOME/oui/bin/detachhome.sh

Oracle Universal Installer and OPatch User's Guide 11g Release 2 (11.2) for Windows and UNIX Chapter 2

Create the ORACLE_HOME cloning scripts

To automate the cloning Oracle software process, create the scripts /root/root_clone.sh shown in Figure 23

Oracle Universal Installer and OPatch User's Guide 11g Release 2 (11.2) for Windows and UNIX Chapter 6

As well as the official installation guide, Oracle Database Installation Guide 11g Release 2 (11.2) for Linux, Red Hat has produced a helpful whitepaper titled Oracle Database 11g Release 2 on Red Hat Enterprise Linux 6 Deployment Recommendations which offers guidance specific to the Red Hat operating system.

Figure 20. Oracle Universal Installer – Install choose database software only

Installing the Oracle Database 11g R2 binaries and patches

http://www.oracle.com/technetwork/topics/security/alerts-086861.html






Oracle Database is not shipped with Direct NFS enabled by default. To enable Direct NFS, complete the following steps:

Change the directory to $ORACLE_HOME/rdbms/lib.

cp libodm11.so libodm11.so_stub

ln -s libnfsodm11.so libodm11.so

Enter the following command:

make -f ins_rdbms.mk dnfs_on

Figure 21. Enable Oracle Direct NFS client

[root@vspex-ora02 ~]# cat /root/root_clone.sh

#!/bin/bash

OHOME=/u01/app/oracle/11.2.0.3

echo "Cloning Oracle Binaries as Oracle at $(date +%Y%m%d%H%M)"

su - oracle -c "$OHOME/oui/bin/cloneHome.sh"

echo "Clone Finished at $(date +%Y%m%d%H%M)"

echo "Running root.sh at $(date +%Y%m%d%H%M)"

$OHOME/root.sh

echo "Install Finished at $(date +%Y%m%d%H%M)"

Figure 22. Example of script root_clone.sh

[root@vspex-ora02 ~]# cat $OHOME/oui/bin/cloneHome.sh

#!/bin/sh

OHOME=/u01/app/oracle/11.2.0.3

OHOMENAME=OraDb11g_home1

OBASE=/u01/app/oracle

CUR_DIR=`pwd`

cd $OHOME/oui/bin

${OHOME}/clone/bin/clone.pl ORACLE_HOME="${OHOME}" ORACLE_HOME_NAME="${OHOMENAME}" ORACLE_BASE="${OBASE}" -ignoreSysPrereqs

cd $CUR_DIR

Figure 23. Example of script cloneHome.sh



Convert the virtual machine to a template called vspx-Orac11203EE-2vcpu-8GB-tmplt as shown in Error! Reference source not found..

Figure 24. Virtual machine template example

Create a custom specification, Oracle_2vCPU_8GB_11203_EE_spec, using the vCenter client. This requests the new virtual machine name and sets place holder IP addresses for the network interfaces as shown in Figure 25.

Figure 25. vSphere custom specification example

Preparing the Oracle Database 11g R2 virtual machine template

Customizing the VMware template




Deploying the Oracle Database 11g R2 virtual machine

Table 15 shows the mapping of the compute requirements for the Oracle Database 11g R2 virtual machines.

Table 15. Oracle Database 11g R2 reference virtual machine model

Virtual Machine Compute Requirement for Oracle Database 11g R2

vCPU

Memory in GB Total reference virtual machines

Small 2 8 4

Medium 4 16 8

Large 8 32 16

There are serveral methods available to deploy a virtual machine from a template using custom specifications:

By manually choosing Deploy Virtual Machine from this Template in vCenter

Choosing Customize using an existing customization specification

By creating a workflow with an orchestration tool such as vCenter Orchestrator

By creating a custom PowerCLI script that can be edited or parameterized

The example PowerCLI script in Figure 26:

Assigns variables for: template name, custom specification name, virtual machine name, ESXi host name, VMFS datastore name, virtual machine public IP address, virtual machine storage IP address, and the number of vCPUs, and vRAM size in MB.

Assigns the virtual machine public and storage IP addresses to the two vNICs included in the custom specification in Figure 26.

Creates the new virtual machine using the template and custom specification.

Changes the CPU and VRAM assignment for the virtual machine.

Starts the virtual machine.

Overview

Deploying new virtual machines



# Connect to Vcenter

Connect-VIServer -server 192.168.28.33 -user Administrator -password <Password>

{

#Assign Variables for template, specification, VM, ESXi Host , Datastore

# IP addresses for the Public and Storage network interfaces

# and the vCPU and vRAM assigned to the Virtual Machine

$Template="vspx-Orac11203EE-2vcpu-8GB-tmplt"

$Customization="Oracle_2vCPU_8GB_11203_EE_spec"

$VMName="<Virtual Machine Name>"

$VMHost=" <ESXi Host Name>"

$Datastore="<Datastore Name>"

$IPAdressPublic=”< Public IP Address>”

$IPAdressStorage=”<Storage IP Address>”

$NumVCPU=”4 ”

$vRAMMB=”16384”

# Assign IP addresses to the two vNICs included in the custom specification

Get-OSCustomizationSpec $Customization | Get-OSCustomizationNicMapping | where { $_.Position -eq '1'} | Set-OSCustomizationNicMapping -IpMode UseStaticIP -IpAddress $IPAdressPublic -Subnetmask 255.255.255.0 -DefaultGateway 192.168.28.1

Get-OSCustomizationSpec $Customization | Get-OSCustomizationNicMapping | where { $_.Position -eq '2'} | Set-OSCustomizationNicMapping -IpMode UseStaticIP -IpAddress $IPAdressStorage -Subnetmask 255.255.255.0 -DefaultGateway 192.168.40.1

# Create the new VM using the Template and custom spec

New-VM -Name $VMName -OSCustomizationSpec $Customization -Template $Template -VMHost $VMHost -Datastore $Datastore

# Change the CPU and memory assignment for the VM

Set-VM $VMName -NumCPU $NumVCPU -MemoryMB $vRAMMB -Confirm:$false

# Start up the VM

Start-VM -VM $VMName -Confirm:$false

}

Figure 26. Example PowerCLI script to deploy a virtual machine from template

Edit and run the example PowerCLI script, or edit it to read a parameter file and deploy multiple virtual machines.

Run the clone process and reattach the Oracle home directory to edit and run a second PowerCLI to complete the virtual machine deployment. This script, shown in Figure 27, calls the root_clone.sh from Figure 22 which does this.

Cloning the Oracle binaries ona new virtual machine




{

$script="/root/root_clone.sh"

$huser="root"

$hpwd="<ROOT Password>"

$VMName="<Virtual Machine Name>"

Invoke-VMScript -ScriptText $script -VM $VMName -GuestUser $huser -GuestPassword $hpwd -ScriptType bash

}

Figure 27. Example PowerCLI script to clone and reattach the ORACLE_HOME

Use the VSPEX for Virtualized Oracle Qualification Worksheet in the Design Guide to size the virtualization infrastructure and the overall NFS filesystems in which to store the Oracle database. These documents also provide guidance on the Oracle initialization parameters required by the database.

Table 15 on page 51 shows the sizing of the virtual machine.

Determine the size of the individual datafiles based on the schema design and tablespace placement of tables and indexes. Datafile placement is covered in the section Configuring the database file layout for NFS.

For guidance on configuring the initialization parameters, see the Design Guide and the Configuring the initialization parameters section of this document.

Before creating the database, you must create a listener process to receive the incoming client connection requests using the Oracle Net Configuration Assistant (NETCA) utility, as shown in Figure 28.

Figure 28. Oracle Net Configuration Assistant utility

Determining the database prerequisites



Consider these items before creating an Oracle 11g R2 database:

Select the required database character set to meet your localization and globalization needs.

Consider the time zones your database must support.

Ensure the SYSTEM and SYSAUX tablespace are created as Locally Managed Tablespaces, appropriately sized, and have AUTOEXTEND enabled.

Plan to use an undo tablespace to manage your undo data.

In this example, the components of the Oracle Database: datafiles, online redo log files, temp files, control files, and FRA files reside on NFS file systems. These file systems are designed (in terms of the RAID level and number of disks used) to be appropriate for each type of file.

Table 16 lists each file type and indicates where they reside.

Table 16. Database file layout for NFS

Content Location

Datafiles /u02/app/oracle/oradata/${ORACLE_SID}

Temp files /u04/app/oracle/oradata/${ORACLE_SID}

Online redo logs, control files /u03/app/oracle/oradata/${ORACLE_SID}

FRA files /u05/app/oracle/oradata/${ORACLE_SID}

This solution uses the Oracle Direct NFS client, which offers improved performance over the standard NFS client supplied with the operating system.

By default, Direct NFS serves the mount entries found in the file /etc/mtab, which lists all currently mounted filesystems along with their initialization options. This should match the system configuration file /etc/fstab where the mount options are defined. Figure 29 provides a sample /etc/mtab with NFS mount options:

NFSServer1:/vspex_ora2_data /u02/app/oracle/oradata/VSPEX1 nfs

rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,t

imeo=600,addr=192.168.40.11 0 0

NFSServer2:/vspex_ora2_redo /u03/app/oracle/oradata/VSPEX1 nfs


imeo=600,addr=192.168.40.10 0 0

NFSServer1:/vspex_ora2_temp /u04/app/oracle/oradata/VSPEX1 nfs


imeo=600,addr=192.168.40.11 0 0

NFSServer1:/vspex_ora2_fra /u05/app/oracle/oradata/VSPEX1 nfs


imeo=600,addr=192.168.40.11 0 0

Figure 29. Example /etc/mtab NFS mount options

Oracle recommends that you set the NFS buffer size parameters rsize and wsize to 32768 in the system configuration file /etc/fstab. However, you can use the file oranfstab to specify additional Oracle Database specific options for Direct NFS.

Configuring the database file layout for NFS




The oranfstab is located in one of the two following locations:

/etc/oranfstab where the options are global to all Oracle databases on the server

$ORACLE_HOME/dbs/oranfstab where the options are specific to a single database

The Oracle Database requires that mount points be mounted by the kernel NFS system even when served through Direct NFS.

To enable Direct NFS, you can optionally create an oranfstab file with the following attributes for each NFS server to be accessed using Direct NFS:

Server: The NFS server name.

Path: Up to four network paths to the NFS server, specified either by IP address, or by name, as displayed using the ifconfig command on the filer.

Local: Up to four local paths on the database host, specified by IP address or by name, as displayed using the ifconfig command run on the database host.

Export: The exported path from the NFS server.

Mount: The corresponding local mount point for the exported volume.

Dontroute: Specifies that outgoing messages should not be routed by the operating system, but instead sent using the IP address they are bound to. Please note that this attribute does not work on Linux with multiple paths in the same subnet.

mnt_timeout: Specifies (in seconds) how long Direct NFS client should wait for a successful mount before timing out. This parameter is optional and the default timeout is 10 minutes.

Figure 30 provides an example of an oranfstab file with two NFS server entries:

server: NFSServer1

local: vspex_ora2

path: NFSServer1

export: /vspex_ora2_data mount: /u02/app/oracle/oradata/vspex1

export: /vspex_ora2_temp mount: /u04/app/oracle/oradata/vspex1

export: /vspex_ora2_fra mount: /u05/app/oracle/oradata/vspex1

server: NFSServer2

local: vspex_ora2

path: NFSServer2

export: /vspex_ora2_redo mount: /u03/app/oracle/oradata/vspex1

Figure 30. Oranfstab example



Red Hat 6.3 supports multiple and various file systems, like VFAT, ext2,ext3, ext4 and Reiser file system. Oracle generally does not certify file systems, but Linux is a specific case.The current support includes ext2, ext3, ext4 (Oracle Linux 5.6 and later),etc. The ext4 is the default file system in this solution.

Logical Volume Management (LVM) can be used to allocate disk space and provide disk striping.

Formatting Physical Volumes (PVs)

To initialize a disk or disk partition as a physical volume we just run the "pvcreate" command on the whole disk or the partition. For example:

pvcreate /dev/sdb

or pvcreate /dev/sdb1

This creates a volume group descriptor at the start of the second SCSI disk or the /dev/sdb1 partition.

Creating Volume Groups (VGs)

Use the "vgcreate" command to group selected PVs into VGs. The following command creates a volume group named " data_volume_group" from five disk partitions from different disks:

vgcreate data_volume_group /dev/sdb1 /dev/sdc1 /dev/sdf1 /dev/sdd1

/dev/sde1

To activate a single volume group ”data_volume_group”, use the command:

vgchange -a y /dev/data_volume_group

Creating Logical Volumes (LVs)

The following is an example of creating logical volume data_vg-data_lv from the volume group data_volume_group. This command creates a 2048G LV named “data-vg-data_lv” for the vg named “data_volume_group” and its block device special file “/dev/ data_volume_group/ data_vg-data_lv”

lvcreate -i5 -I4 -L2048G -n data_vg-data_lv

data_volume_group

Where size is either "-l num_extents" or "-L num_bytes", where num_bytesis a number followed by one of k, m, g, or t. One of the most common options is "-n name" (you can use "--name" for "-n") to specify a name for the logical volume.

Another option includes "-i num_stripes -I stripe_size" to create an LV with striped mapping (stripe_size is a number between 2 and 9)

Once the LVs have been created you can format them with filesystems (or as swap space) using standard tools such as "mkfs". One example of formatting LV with ext4 filesystem:

mkfs.ext4 -E stride=16 /dev/data_volume_group/data_vg-data_lv

Configuring the database file layout for Block




If the new filesystem can be successfully mounted, a final step is to edit the /etc/fstab file and possibly the rc.sysinit file, so that the volumes are mounted automatically at boot time. Figure 31 shows an example of /etc/fstab.

Figure 31. Example of /etc/fstab

For more information, see My Oracle Support Note ID 1476869.1 (Available Mount Options to Improve ext4 Filesystem Performance) and 236826.1 (Supported and Recommended File Systems on Linux).

To configure the initialization parameters, you can use one of the following methods.

Automatic Shared Memory Management

To automate the management of shared memory structures, implement Automatic Shared Memory Management (ASMM). The following shared memory structures will be automatically managed:

DB_CACHE_SIZE

SHARED_POOL_SIZE

LARGE_POOL_SIZE

JAVA_POOL_SIZE

STREAMS_POOL_SIZE

When implemented, ASMM enables the Oracle database to distribute memory between these components based on workload requirements. Hence, these components are considered autotuned parameters. The background process, Memory Manager (MMAN), coordinates the sizing of these components and moves memory to where it is needed most.

To implement this feature, set the following initialization parameters:

SGA_TARGET set to a nonzero value

STATISTICS_LEVEL=TYPICAL (or ALL)

Automatic Memory Management

Do not use Oracle Automatic Memory Management (AMM) since AMM is incompatible with HugePages. If you want to use HugePages make sure that both MEMORY_TARGET and MEMORY_MAX_TARGET initialization parameters are not set.

For more information, see My Oracle Support Note ID 749851.1.

Configuring the initialization parameters



I/O operations for file system files

To enable direct I/O and async I/O for file system files, set FILESYSTEMIO_OPTIONS=SETALL

With async I/O, processing continues while the I/O request is submitted and processed. This allows async I/O to bypass some of the performance bottlenecks associated with I/O operations. Direct I/O is a feature available in the filesystem that delivers data directly to the application without caching in the file system buffer cache.

Direct NFS does not depend on the value of FILESYSTEMIO_OPTIONS. Direct NFS always issues async and direct I/O as it does not depend on OS support. However, so that there is a backup to the OS NFS client in case of mis-configuration, set filesystemio_options to SETALL if the OS supports it.

HugePages was enabled in the Linux Kernel as part of the Template creation process described in Table 12 on page 46.

The database initialization parameter, USE_LARGE_PAGES, was involved from 11.2.0.2 to manage HugePages for use by the database. From version 11.2.0.3, setting USE_LARGE_PAGES to AUTO causes the oradism process to try to reconfigure the Linux kernel to increase the number of HugePages to match the database requirements.

See My Oracle Support article USE_LARGE_PAGES To Enable HugePages In 11.2 [ID 1392497.1] for more details.

The Database Configuration Assistant (DBCA) is the preferred way to create a database because it is a more automated approach and the database is ready to use when DBCA completes. Launch DBCA as a standalone tool to create an Oracle Database.

To start DBCA on UNIX, Linux, or at a Windows command-line prompt, enter the following command:

dbca

Note: The dbca utility is typically located in the ORACLE_HOME/bin directory.

Enabling the HugePages setting

Creating the database using DBCA




Figure 32. Oracle Database Configuration Assistant setup screen

Oracle DBCA runs in interactive mode. Using the screens in Oracle DBCA, you can either modify one of the preconfigured database types or customize the database.

General Purpose/Transaction Processing is a preconfigured database template that you can create and customize during the installation. This involves amending the

Initialization parameters

Size and placement of the database files to use the NFS mount points

After you create a database as described in Creating the database using DBCA, the instance is open and available for normal database use. However, the Oracle database will not start up or shut down automatically with the virtual machine.

With Oracle Database 11g R2, there are two methods to automate the Oracle Database startup and shutdown process:

Use an initialization script and the /etc/oratab file as defined by My Oracle Support Document How to Automate Startup/Shutdown of Oracle Database on Linux [ID 222813.1]

Install the additional 11.2 Grid Infrastructure software using the single instance option. This will use a modified Cluster Ready Services (CRS) component to automate the startup of the database, listener, and Oracle Automatic Storage Management (a grid Infrastructure component) if required.

This solution uses the initialization script shown in Figure 33. Follow these steps to set up automated startup and shutdown of the database and listener.

1. Update the /etc/oratab file with an entry in the following format:

<ORACLE_SID>:<ORACLE_HOME>:Y

Where:

ORACLE_SID is the name of the database

Performing the database post creation activities



ORACLE_HOME is the directory where the Oracle 11g R2 database binaries are installed

Y states that the database can be started up and shutdown automatically

For example: VPLEXM1: /u01/app/oracle/11.2.0.3:Y

2. Figure 33 shows an example of how to create the service script /etc/init.d/dbora as root user.

#!/bin/bash

#

# chkconfig run levels and script numbering

# chkconfig: 35 99 10

# description: Starts and stops Oracle processes

#

# Set ORA_HOME to be equivalent to the $ORACLE_HOME

# from which you wish to execute dbstart and dbshut;

# Set ORA_OWNER to the user id of the owner of the

# Oracle database in ORA_HOME.

#

ORA_HOME=/u01/app/oracle/11.2.0.3

ORA_OWNER=oracle

case "$1" in

'start')

# Start the TNS Listener

su - $ORA_OWNER -c "$ORA_HOME/bin/lsnrctl start"

# Start the Oracle databases:

su - $ORA_OWNER -c "$ORA_HOME/bin/dbstart"&

touch /var/lock/subsys/dbora

;;

'stop')

# Stop the TNS Listener

su - $ORA_OWNER -c "$ORA_HOME/bin/lsnrctl stop"

# Stop the Oracle databases:

su - $ORA_OWNER -c "$ORA_HOME/bin/dbshut"

rm -f /var/lock/subsys/dbora

;;

esac

# End of script dbora

Figure 33. Oracle startup/shutdown script /etc/init.d/dbora

3. Execute the script:

chmod 755 /etc/init.d/dbora




4. Register the service to the Linux service mechanism for automatic startup at run level 3 and 5 by issuing the following command.

/sbin/chkconfig --add dbora

With the infrastructure build tasks complete, the database is now ready for use.

Implementing backup and recovery

For information about implementing backup and recovery for an Oracle 11gR2 Database, refer to the document entitled EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11gR2 Design and Implementation Guides available on EMC’s online support.

Overview

Chapter 5: Solution Verification



Chapter 5 Solution Verification


Verifying the baseline infrastructure ........................................................................ 64

Verifying the Oracle Database .................................................................................. 66

Backup and recovery verification methodology ........................................................ 68



Verifying the baseline infrastructure

This section provides a list of items that you need to review after configuring the solution. The goal of this section is to verify the configuration and functionality of the solution, and to ensure that the configuration supports the core availability requirements. Table 17 describes the tasks that must be completed.

Table 17. Tasks for verifying the VSPEX Proven Infrastructure installation


Verify ESXi functionality Verify the basic ESXi functionality of the solution with a post-installation checklist.

Using EMC VNX storage with VMware vSphere

http://www.vmware.com

Verify redundancy of the solution components

Verify the redundancy of the solution of the components:

Storage

ESXi host

Network switch

Vendor documentation

Verify the Oracle Database configuration in the VSPEX Proven Infrastructure

Verify the Oracle Database configuration.

http://support.oracle.com

Monitor the health of the VSPEX Proven Infrastructure for virtualized Oracle Database

Use the tool to monitor the VSPEX Proven Infrastructure’s health

View alert logs (Oracle Database 11g R2)

View storage status from Unisphere management console

View VM logs from vSphere 5.1 client

EMC recommends that you verify the ESXi configurations before deploying the system to production on each ESXi server.

For more detailed information, refer to appropriate documents accessible from the VSPEX Proven Infrastructure Launch Page.

To ensure that the various components of the solution maintain availability requirements, it is important to test specific scenarios related to maintenance or hardware failure. EMC recommends that you verify redundancy of the solution components including storage, ESXi hosts, and network switches.

For detailed steps, refer to the appropriate documents accessible from the VSPEX Proven Infrastructure Launch Page.

Overview

Verifying the ESXi functionality

Verifying the solution component redundancy

http://www.vmware.com/

http://support.oracle.com/

http://www.emc.com/platform/virtualizing-information-infrastructure/vspex.htm#!resources






To ensure that Oracle Database 11g R2 is running smoothly in the VSPEX infrastructure, use these steps to verify and monitor the Oracle Database configuration:

1. Check the installed Oracle Database contents and directory location. Make sure Oracle Database has been installed successfully.

2. Verify that the Oracle Database that you plan to use is accessible. Use the TNSPING utility to test connectivity from the client with the following command: tnsping net_service_name count.

Health monitoring is a simplified measurement that reflects the reliability, stability, and performance of the entire VSPEX Proven Infrastructure for virtualized Oracle Database. Table 18 lists some tools for you to monitor and troubleshoot your VSPEX solution.

Table 18. Tools for monitoring the solution

Tools Description

esxtop The esxtop tool provides a real-time view (updated every five seconds, by default) of ESXi Hypervisor performance metrics, such as CPU. It is important that you understand the performance for hypervisor in order measure or troubleshoot the Oracle Database performance-related issues.

Unisphere Analyzer Unisphere Analyzer software is preloaded on all VNX storage systems with the FLARE/block bundles (or later). It is an easy to monitor, collect, and analyze the all aspects of storage related data from it.

Oracle Enterprise Manager Database Control

Oracle Enterprise Manager Database Control (Database Control) is the primary tool for managing your Oracle database. It is installed with the database.

Using Database Control, you can perform administrative tasks such as creating schema objects (tables, views, indexes, and so on), managing user security, managing database memory and storage, backing up and recovering your database, and importing and exporting data. You can also view performance and status information about your database.

Automatic Workload Repository (AWR)

The Automatic Workload Repository (AWR) collects, processes, and maintains performance statistics for problem detection and self-tuning purposes. This data is both in memory and stored in the database. The gathered data can be displayed in both reports and views.

EMC recommends that you use the above tools to monitor the performance of the entire Oracle Database environment. For details about performance verification and test methodology, refer to the companion Design Guide.

Verifying the Oracle Database configuration

Using the health monitoring tools



Verifying the Oracle Database

This section provides a verification example of an Oracle OLTP Database in the VSPEX Proven Infrastructure for virtualized Oracle Database solution. The purpose is to measure the performance of the Oracle Database to ensure that the VSPEX Proven Infrastructure meets the customer’s business requirements.

Before you verify your own VSPEX Proven Infrastructure, EMC recommends that you read the Application validation methodology section in the companion Design Guide to understand the test methodology.

After you identify the business requirements, you need to know the key metrics for the Oracle Database testing when running the Oracle Database validation tests. The key metrics are identified in the Design Guide in the Application validation methodology section. My Oracle Support Note 1275596.1 provides a list of typical acceptable I/O response times.

Loading the test configuration

The Swingbench Order Entry – PL/SQL (SOE) schema delivers the OLTP workloads required by the solution. The Order Entry schema introduces heavy contention on a small number of tables and is designed to stress CPU and memory. To establish an optimal configuration, we tested different users accessing the order entry schema.

Before running an Oracle Database 11g R2 test, it is important to understand the test scenarios according to the business requirements. You can easily create your own test scenarios using Swingbench. By editing the transaction list, you can define the different operations that end users perform in the Oracle Database. Table 19 lists some of the general operations we considered in the Oracle Database 11g R2 VSPEX solution.

Table 19. General operations

Workload Operation Weight

Swingbench Order Entry

Customer Registration 30

Browse Products 60

Order Products 50

Process Orders 30

Browse Orders 10

Swingbench allows you to select a variable “think time” (delay) between transactions. Figure 34 shows the load panel on the Swingbench interface, with the appropriate think time parameters set.

Overview

Understanding the key metrics




Figure 34. Swingbench interface load panel screenshot

We simulated the workload using Swingbench against the database as shown in Figure 35.

Figure 35. Swingbench order entry screen for workload simulation

Once the test is complete, you can use the Automatic Workload Repository (AWR) report to check the results and verify that you have achieved the key metrics in your test environment. Since the metrics can vary with different kinds of Oracle Database configuration and VSPEX offerings for VMware, refer to the VSPEX Sizing Tool for the detailed metrics.

If the test results are not ideal, it is easy to identify the bottleneck from the AWR report. For detailed information about the AWR report, refer to the My Oracle Support Note ID 1359094.1.

After identifying potential bottlenecks, refer to the Implementing the storage section to make sure that you have the correct configuration of VNX, switches, hypervisor, or load balancer for the VSPEX for virtualized Oracle solution. System bottlenecks represent a point of contention where there are insufficient resources to service user transaction requests. Consider adding more resources into the solution to meet the requirement. Often, the reason for the bottleneck is inefficient custom code or third-party solutions due to customization.


https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?_afrLoop=171689848024001&recommended=true&type=DOCUMENT&id=1359094.1&_afrWindowMode=0&_adf.ctrl-state=17ns2uvjcp_97

https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?_afrLoop=171689848024001&recommended=true&type=DOCUMENT&id=1359094.1&_afrWindowMode=0&_adf.ctrl-state=17ns2uvjcp_97



In this VSPEX validation test, we used the Swingbench Order Entry - PL/SQL (SOE) schema to deliver the OLTP workloads. This schema models a traditional OLTP database. Tables and indexes reside in separate tablespaces. The SOE schema used in this solution contains the tables and indexes shown in Table 20.

Table 20. Schema tables and indexes

Table name Index

CUSTOMERS CUSTOMERS_PK (UNIQUE), CUST_ACCOUNT_MANAGER_IX, CUST_EMAIL_IX, CUST_LNAME_IX, CUST_UPPER_NAME_IX

INVENTORIES INVENTORY_PK (UNIQUE), INV_PRODUCT_IX, INV_WAREHOUSE_IX

ORDERS ORDER_PK (UNIQUE), ORD_CUSTOMER_IX, ORD_ORDER_DATE_IX, ORD_SALES_REP_IX, ORD_STATUS_IX

ORDER_ITEMS ORDER_ITEMS_PK (UNIQUE), ITEM_ORDER_IX, ITEM_PRODUCT_IX

PRODUCT_DESCRIPTIONS PRD_DESC_PK (UNIQUE), PROD_NAME_IX

PRODUCT_INFORMATION PRODUCT_INFORMATION_PK (UNIQUE), PROD_SUPPLIER_IX

WAREHOUSES WAREHOUSES_PK (UNIQUE)

LOGON –

Backup and recovery verification methodology

To verify your backup and recovery methodology, refer to the EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11gR2 Design and Implementation Guide s

.

Reviewing the Oracle dataset

Chapter 6: Reference Documentation



Chapter 6 Reference Documentation


EMC documentation ................................................................................................. 70

Other documentation ............................................................................................... 71

Links ........................................................................................................................ 71



EMC documentation

EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 1000 Virtual Machines – Reference Architecture

White Paper: EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11gR2 Design and Implementation Guide

VNX System Installation Guide

VNX Series Configuration Worksheet

Using EMC VNX Storage with VMware vSphere–TechBook



VNX family

http://www.emc.com/storage/vnx/vnx-family.htm

VNX series documentation on EMC online support site

https://support.emc.com/products/12781_VNX-Series/Documentation/

Oracle Database 11g R2 Documentation Library 11g Release 2 (11.2)

http://www.oracle.com/pls/db112/homepage

http://support.oracle.com

Oracle Database 11g Release 2 documentation: http://docs.oracle.com/cd/E11882_01/install.112/e24321.pdf

http://www.oracle.com/technetwork/indexes/documentation/index.html Oracle Database Release Notes for Linux

My Oracle Support https://support.oracle.com/ latest OPatch and latest PSU

HugePages and Oracle Database 11g Automatic Memory Management (AMM) on Linux [Article ID 749851.1]

USE_LARGE_PAGES To Enable HugePages [Article ID 1392497.1]

HugePages on Oracle Linux 64-bit [Article ID 361468.1]

FAQ: How to Use AWR Reports to Diagnose Database Performance Issues [Article ID 1359094.1]


vCenter Server and Host Management Guide

Installing vCenter Server 5.1 best practices

vSphere Networking

vSphere Storage

vCenter Server and Host Management

EMC

Oracle

VMware

http://www.emc.com/storage/vnx/vnx-family.htm

https://support.emc.com/products/12781_VNX-Series/Documentation/

http://www.oracle.com/pls/db112/homepage

http://www.oracle.com/technetwork/indexes/documentation/index.html

http://www.google.com/url?sa=t&rct=j&q=oracle%20support&source=web&cd=1&cad=rja&sqi=2&ved=0CDAQFjAA&url=https%3A%2F%2Fsupport.oracle.com%2F&ei=mqQGUcCpJJC2hAf204GADw&usg=AFQjCNG0526XG7uf7fF9Za56TPf9auhPTw&bvm=bv.41524429,d.ZG4

https://support.oracle.com/




vSphere Installation and Setup vCenter Server and Host Management

vSphere Resource Management

Automating High Availability (HA) Services with VMware HA –VMware Technical note

General documentation: http://www.vmware.com

VMware vSphere Documentation: http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html

vSphere PowerCLI Documentation:

http://www.vmware.com/support/developer/PowerCLI/

Other documentation

Oracle Database 11g Release 2 on Red Hat Enterprise Linux 6 Deployment Recommendations

http://www.redhat.com/resourcelibrary/reference-architectures/deploying-oracle-11g R2-on-rhel-6

Links

Note: The links provided were working correctly at the time of publication.

Everything Oracle at EMC

https://community.emc.com/community/connect/everything_oracle

My Oracle Support: Oracle Support Website

https://support.oracle.com (user account required)

FAQ: How to Use AWR reports to Diagnose Database Performance Issues [ID 1359094.1]

Requirements for Installing Oracle 11g R2 RDBMS on RHEL6 or OL6 64-bit (x86-64) [ID 1441282.1]

How to Tell if the IO of the Database is Slow [Article ID 1275596.1]

HugePages on Linux: What It Is... and What It Is Not... [ID 361323.1]

USE_LARGE_PAGES To Enable HugePages In 11.2 [ID 1392497.1]

HugePages on Oracle Linux 64-bit [ID 361468.1]

Installing and Using Standby Statspack in 11g [ID 454848.1]

Installing and Configuring StatsPack Package[Article ID 149113.1]

http://www.vmware.com/

http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html

http://www.vmware.com/support/developer/PowerCLI/

http://www.redhat.com/resourcelibrary/reference-architectures/deploying-oracle-11gr2-on-rhel-6

http://www.redhat.com/resourcelibrary/reference-architectures/deploying-oracle-11gr2-on-rhel-6

https://community.emc.com/community/connect/everything_oracle

https://support.oracle.com/

Appendix A: Configuration Worksheet


Appendix A Configuration Worksheet

This appendix presents the following topic:

Pre-configuration worksheet for VSPEX .................................................................... 73




Pre-configuration worksheet for VSPEX

Before configuring Oracle Database 11g R2 for this solution, you need to gather some customer-specific configuration information such as IP addresses, hostnames, and so on. The following tables provide a worksheet that you can use to record the information. You can also print and use the worksheet as a customer “leave behind” document for future reference.

To confirm the customer information, cross-reference with the relevant array configuration worksheet: EMC VNX File and Unified Worksheets.

Table 21. Common server information

Server Name Purpose Primary IP

Domain Controller

DNS Primary

DNS Secondary

DHCP

NTP

SMTP

SNMP

vCenter Server

Swingbench server

Database server 01

Database server 02

Database server 03

Table 22. ESXi server information

Server Name Purpose Primary IP

Private Net (storage) addresses

ESXi

Host 1

ESXi

Host 2

…



Table 23. Array information

Array name

Root password

NAS admin password

CS0 Primary hostname

CS1 Secondary hostname (if applicable)

CS0 Primary IP address

CS1 Secondary IP address (if applicable)

SPA IP address

SPB IP address

Virtual machine infrastructure pool name

Virtual machine infrastructure file system name

Datafile storage pool name

Datafile file system name

Redo log storage pool name

Redo log file system name

Fra storage pool name

Fra file system name

Table 24. Network infrastructure information

Name Purpose IP Subnet Mask Default Gateway

Ethernet Switch 1

Ethernet Switch 2

…

Table 25. VLAN information

Name Network Purpose VLAN ID Allowed Subnets

Virtual Machine Networking Management

NFS networking

vMotion




Table 26. Service accounts

Account Purpose Password (optional, secure appropriately)

vCenter Server administrator

Array administrator

Oracle Database administrator

Appendix B: Example Virtual Disk Layout




This chapter presents the following topic:

Appendix B: Example Virtual Disk Layout 77

Example Virtual Disk Layout of Oracle Database on the VMFS Datastore .................. 78



Example Virtual Disk Layout of Oracle Database on the VMFS Datastore

For customers who plan to virtualize Oracle database on the VMware platform, storage layout guidelines for Oracle database on VMware technology, with respect to the number of LUNs and maximizing I/O performance, are the same as with a physical installation.

Table 27 shows an example of virtual disk layout.

Table 27. Example Virtual Disk Layout

Datastore Name

Virtual Disk on ESX

Guest OS Device Name

Virtual device

Virtual SCSI Driver

VMDK Size (GB)

Purpose

OSDATASTORE VMDK-

hard disk 1

/dev/sda SCSI 0:0 Paravirtual 100 Redhat Linux 6.3 OS and Oracle binaries

VMFSDATA01 VMDK-

hard disk 2

/dev/sdb1 SCSI 1:0 Paravirtual 500 Database datafile

VMFSDATA02 VMDK-

hard disk 3

/dev/sdc1 SCSI 1:1 Paravirtual 500 Database datafile

VMFSDATA03 VMDK-

hard disk 4

/dev/sdd1 SCSI 1:2 Paravirtual 500 Database datafile

VMFSDATA04 VMDK-

hard disk 5

/dev/sde1 SCSI 1:3 Paravirtual 500 Database datafile

VMFSDATA05 VMDK-

hard disk 6

/dev/sdf1 SCSI 1:4 Paravirtual 500 Database datafile

VMFSREDO01 VMDK-

hard disk 7

/dev/sdg1 SCSI 2:0 Paravirtual 100 Database redo file

VMFSREDO02 VMDK-

hard disk 8

/dev/sdh1 SCSI 2:1 Paravirtual 100 Database redo file

VMFSREDO03 VMDK-

hard disk 9

/dev/sdi1 SCSI 2:2 Paravirtual 100 Database redo file

VMFSREDO04 VMDK-

hard disk 10

/dev/sdj1 SCSI 2:3 Paravirtual 100 Database redo file

VMFSREDO05 VMDK-

hard disk 11

/dev/sdk1 SCSI 2:4 Paravirtual 100 Database redo file

VMFSFRA01 VMDK-

hard disk 12

/dev/sdl1 SCSI 3:0 Paravirtual 500 Database FRA file




Datastore Name

Virtual Disk on ESX

Guest OS Device Name

Virtual device

Virtual SCSI Driver

VMDK Size (GB)

Purpose

VMFSFRA02 VMDK-

hard disk 13

/dev/sdm1 SCSI 3:1 Paravirtual 500 Database FRA file

VMFSFRA03 VMDK-

hard disk 14

/dev/sdn1 SCSI 3:2 Paravirtual 500 Database FRA file

VMFSFRA04 VMDK-

hard disk 15

/dev/sdo1 SCSI 3:3 Paravirtual 500 Database FRA file

VMFSFRA05 VMDK-

hard disk 16

/dev/sdp1 SCSI 3:4 Paravirtual 500 Database FRA file

emc vspex for virtualized oracle database 11g oltp … · contents 4 emc vspex for virtualized...

Documents