oracle rac on system z storage configurations for rac assignment of disk from z/vm oracle asm,...

1

© 2007 IBM Corporation

Oracle RAC on System z

David J SimpsonIBM - Certified Oracle [email protected]

Chart 2© 2008 IBM Corporation

TrademarksTrademarks

The following are trademarks of the International Business Machines Corporation in the United States and/or other countries. For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml: AS/400, DBE, e-business logo, ESCO, eServer, FICON, IBM, IBM Logo, iSeries, MVS, OS/390, pSeries, RS/6000, S/30, VM/ESA, VSE/ESA, Websphere, xSeries, z/OS, zSeries, z/VM

The following are trademarks or registered trademarks of other companies

Lotus, Notes, and Domino are trademarks or registered trademarks of Lotus Development CorporationJava and all Java-related trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countriesLINUX is a registered trademark of Linux TorvaldsUNIX is a registered trademark of The Open Group in the United States and other countries.Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation.SET and Secure Electronic Transaction are trademarks owned by SET Secure Electronic Transaction LLC.Intel is a registered trademark of Intel Corporation* All other products may be trademarks or registered trademarks of their respective companies.

NOTES:

Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here.

IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.

All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.

This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.

All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

References in this document to IBM products or services do not imply that IBM intends to make them available in every country.

Any proposed use of claims in this presentation outside of the United States must be reviewed by local IBM country counsel prior to such use.

The information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

2


Oracle Real Application Clusters IntroductionNetworking Configuration Options for Oracle RACOracle Storage Configurations for RACAssignment of Disk from z/VMOracle ASM, OCFS2, Raw and Block storageECKD vs. SCSI storage Raid 5 vs. Raid 10FCP MultipathBackup considerationsOracle ASM features for adding/modifying storage

Agenda


Oracle Real Application Clusters - RAC

What is Oracle RAC?Oracle Real Application Clusters (RAC) is Oracle’s clustering

solution for highly available systems running the Oracle database.

Why Use Oracle RAC?Provides High Availability to survive node and instance

failures.Scalability with the ability to add more nodes as demand and

load increases.Provides the capability to patch one instance while other is

being used. With Oracle 11g this has been expanded.Some downtime is needed for database upgrades.

3


Oracle RAC – points of failureValue Proposition:

Very LowHighApplication

Medium Low Disk Subsystem microcode

Very LowLowLinux

LowLowz/VM

LowVery LowLPAR

HighVery LowSystem z hardware

Cost to fixProbability of FailureSingle Point of Failure

Migrating an instance to RAC:- 15% more shared Pool

- 10% more buffer Cache

- Memory requirements per instance are reduced by multiple nodes


Typical Hardware Component Performance

Run at Gbyte/smemory speed

Hipersockets (CPU)

80 Mbytes/s1 Gbits/sGigE Nic

200 Mbytes/s2 Gbits/sDisk Controller

200 Mbytes/s2 Gbits/sFibre Channel

1600 Mbytes/s8 x 2 Gbits16 Port Switch

100/200 Mbytes/s½ Gbit/sHBA

Maximal Bytes/sTheoretical (Bit/s)Component

4


Multiple LPAR ConfigurationHardware Hipersocket

LPAR 1

Linux Guest2

Gue

st LAN

Linux Guest1

z/VM

OSA Card OSA Card

Gue

st LAN

z/VM

Linux Guest3

Linux Guest4

LPAR 2

VSWITCHVSWITCH


1 LPAR Environment

Oracle Installation codes/mnt//Oracle10gRAC

Server1(Oracle)

LINUXCode Server

VMTCP/IP

Hipersockets (Guest LAN)

VSWITCH

z/VM V5.3

Server2(Oracle)

Oracle 10gR2 RAC Oracle 10gR2 RAC

inter connect

5


Multiple System z Configuration

OSA Card OSA Card

System z System z

Switch


Oracle RAC Network Configuration Summary

6


Network Configuration Tips

Three IP addresses (assign 2) are needed per Oracle RAC node, Oracle Clusterware will bind to the first public interface e.g. eth0 to make eth0:1, so do not use eth0:1 its needed for Virtual IP address.Real Application Clusters uses the interconnect to transfer blocks & messages between instances check average cr block receive time in the AWR report or the GV$SESSION_WAIT view.Check the following parameters which can be set to up to 1MB/proc/sys/net/core/rmem_default - default receive window /proc/sys/net/core/rmem_max - maximum receive window /proc/sys/net/core/wmem_default - default send window /proc/sys/net/core/wmem_max - maximum send window

Something else you can do is use jumbo frame sizes of 9000 MTU instead of the default of 1500 MTU for private interconnect, you must set the private interface the same on both nodes or RAC will not start.


Oracle RAC Storage Considerations

Many things to consider when setting up storage:

Assignment of Disk from z/VMOracle ASM, OCFS2, Raw and Block storageECKD vs SCSI storage Raid 5 vs Raid 10FCP MultipathBackup considerationsOracle ASM features for adding/modifying storage

7


Oracle RAC and ASM Instances


Assigning Shared Disk from z/VM between RAC nodes

It is best to dedicate the devices that will be shared to the individual Linux guests when using multiple LPARs.

Use an Rdevice statements for the shared DASD devices between LPARs, such as the following:

Rdevice 2E13-2E1C Type DASD Shared Yes MDC OFF

Oracle is managing the “locking” of writes to the disk. zVM and Linux do not manage which guests can write to a device, so its important to limit access just to the nodes that are in the Oracle cluster.

Ensure that Mini Disk Cache is off at the zVM and the Disk level, and when installing tools like Velocity, ensure that these tools do not inadvertently turn on mini disk caching.

8


Oracle ASM, OCFS2, Raw and Block Storage

Oracle Automated Storage Management is Oracle’s direction for storage of databases going forward (per Oracle training material)ASM is a software striping aggregate algorithm.LVM utilizes a hashing algorithm to determine the stripe Oracle ASM utilizes a separate instance to maintain stripping information which can makes things quicker for writes.Use either Oracle ASM or LVM for database files never both.For Oracle RAC use ASM.You cannot use LVM with OCFS2 (Oracle Cluster File System) which can effect performance by not being able to utilize stripping.Oracle Cluster Registry (OCR) and Voting Disk can be on Raw or Block devices (Note - only on zSeries for Block Devices)


Benefit of striping

If you set no striping, all I/O requests to a LV are concentrated on one PV. If you setup striping, all I/O requests to a LV are balanced to multiple PV. It may remove a bottleneck and get the best performance.

Application Application

123

4

5

1 23

4

5Logical Volume Logical Volume

Striping with 2 PVsNo striping

Application issues I/O request to LVs

This disk may be bottlenecks

I/O requests are balanced to multiple disks.

9


Oracle ECKD Disk vs SCSI Disk

Two types of disk storage typically used. Disk applications can use either SCSI storage devices utilizing fixed (512-byte) blocks or Extended Count Key Data (ECKD) disk devices. These disks can then be connected by the following connection technologies, Enterprise System Connection (ESCON), Fiber Connector (FICON) or Fiber Channel Protocol (FCP). FICON and FCP: the number of stripes should be equal to the number of disks ESCON the number of stripes should be at least equal to the number of available channels. Good stripe sizes are 32K and 64K for data warehouse databases, and 16K or 8K for high online transaction processing databases. (OLTP) for LVM file systems.


Oracle ASMSince the I/O’s are per channel with one active at a time, then by having 4 ASM devices we can utilize 4 separate channel writes at a time and hence more I/O's in parallel instead of oneat a time through a single device (even though the underlying device may be setup across multiple disk devices)For performance, use disks from several array groups that are the same size and from all clusters in the storage array. This will provide better I/O concurrency and load balancing. Oracle ASM database uses 1MB stripe sizes for datafiles, and a 128KB stripe sizes for redo files as a default.Try and make the LUN stripe size is as close to 1MB as possible for the data file LUNs as this will require fewer reads from Oracle.

10


ESS Architecture

CHPIDs - the FICON Express card supportsFCP or FICON protocol

Host Adapter (HA) supporting FCP (FCP port)-16 Host Adapters, organized in4 bays, 4 ports each

the ESS is divided into two Clusters- Caches are organized per cluster!

Device Adapter Pairs (DA)- each one supports two loops

Disks are organized in ranks - each rank (8 physical disks) implements one RAID array (with logical disks)

Courtesy Martin Kammerer

ESS 2105 800

HA Bay 1 HA Bay 2 HA Bay 3 HA Bay 4

DeviceAdapter

Rank 1

Rank 9

Rank 2

Rank 10

DeviceAdapter

DeviceAdapter

Rank 3 Rank 4 DeviceAdapter

DeviceAdapter


DeviceAdapter


Rank 11 Rank 12

Rank 13 Rank 14

Rank 15 Rank 16

System z serverFC

P C

HP

ID

FICON/FCP Switch

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID


ESS Architecture






ESS 2105 800


DeviceAdapter

Rank 1

Rank 9

Rank 2

Rank 10

DeviceAdapter

DeviceAdapter


DeviceAdapter


DeviceAdapter


Rank 11 Rank 12

Rank 13 Rank 14

Rank 15 Rank 16

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

5100-510F

System z server

FICON/FCP Switch

11


Optimize the disk setup for an ESS






ESS 2105 800


DeviceAdapter

Rank 1

Rank 9

Rank 2

Rank 10

DeviceAdapter

DeviceAdapter


DeviceAdapter


DeviceAdapter


Rank 11 Rank 12

Rank 13 Rank 14

Rank 15 Rank 16

System z serverFC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

FC

P C

HP

ID

5100, 5200, 5180, 5280, 5300, 5400,

5380, 5480

FICON/FCP Switch


Raid 5 vs Raid 10

Raid 5 provides redundancy through a parity disk. Raid 10 (stripe and mirror) provides redundancy through mirroredstripes. RAID5 is up to 50% slower than RAID0 for an array of the same capacity for write intensive applications. e.g. 4 writes for Raid 5 versus 2 for Raid 10 if decoupled from san cache.For Read intensive databases Raid 5 and Raid 10 are similarRaid 10 is good for write intensive applications but is expensive as more disks are needed.Fon Non ASM, Oracle recommends the SAME approach (Mirror And Stripe Everywhere), essentially RAID 1+0 with segregation of undo/redo and rollbacks on separate spindles. Use filesystemio_options = setall to utilize asynchronous and directio, sizing the sga_target to utilize RAM, more efficient than linux file system buffer cache.

12


IBM System Storage Productivity Center

Enables end-to-end disk management on single screen

–Supports management of heterogeneous SMI-S-conforming systems and devices

Common console for DS8000 & SVC–Device configuration for DS8000, SVC

–Support for other IBM storage forthcoming

SSPC is preloaded with IBM TotalStorageProductivity Center products to ease install

–TPC Basic Edition – required license

–TPC Standard Edition - recommended license

• TPC for Disk• TPC for Data• TPC for Fabric

–Preload enables simpler install/configuration

New console offering integrated view for simplified storage management


Setting File PermissionsFor Suse 10 or Red Hat 5 you need to set the file permissions for the Shared Disk.

Create file /etc/udev/rules.d/99-raw.rules based on 50-udev-default.rules KERNEL=="raw[1-2]*", GROUP="oinstall", MODE="640"KERNEL=="raw[3-5]*", OWNER="oracle", GROUP="oinstall", MODE="660“

# /etc/init.d/boot.udev restartOr for RedHat 5

# udevcontrol reload_rules# start_udev

Red Hat 4 and SLES 9Create /etc/udev/permissions.d/49-oracle.permissions based on 50-udev.permissions

# Block OCR devicesdasdo1:root:oinstall:0640# Voting diskdasdp1:oracle:oinstall:0660• Then run # /sbin/start_udev or reboot for the file permissions to take effect

for /dev/mapper files (LVM or multipath) you can use /etc/rc.d/rc.local to set the file permissions

13


FCP Multipath

If there are multiple paths for the same volume, one device file is assigned to each path

FCP multipathing is implemented by software.– The implementation depends on distribution

and version.– Boot partitions are not supported

Linux

LUN

/dev/sda /dev/sdb

SAN Switch FCP


Parameters for FCP devices

L PS SDRAM

(32MB min.)

LPS

FLASHFLASH

LPS SDRAM

(32MB mi n. )

LPS

FLASHFLASH

System z

LUN

Storage devices

Device Number

World Wide Port Name(WWPN)

FCP Logical Unit Number(FCP LUN)

Ex) 0xd80 Ex). 0x500308c8fb6bf003 Ex). 0x5600000000000000Path to the LUN = ＋＋

You can collect the following parameters to configure FCP devices.

SAN Switch

Defined in IOCP

Get information via storage administration tools

14


z/VM basic operation (1)Attach an FCP device address– Attach dynamically

(Ex) Attach an FCP device address “0D80” to a VM guest user “LINUX1”

– Define USER DIRECT to attach the FCP device address to a VM guest user

ATTACH D80 LINUX1 D80Virtual device address of the FCP portVM user id the FCP port is attached to Physical device address of the FCP port

DEDICATE D80 D80Physical device address of the FCP portVirtual device address of the FCP port


Before Setting up FCP multipathing, stop CRS

And unbind raw devices

Set up FCP multipathing [both]

# /etc/init.d/init.crs stopShutting down Oracle Cluster Ready Services (CRS):Stopping resources. This could take several minutes.Successfully stopped CRS resources.Stopping CSSD.Shutting down CSS daemon.Shutdown request successfully issued.Shutdown has begun. The daemons should exit soon.

# raw /dev/raw/raw1 0 0/dev/raw/raw1: bound to major 0, minor 0# raw /dev/raw/raw2 0 0/dev/raw/raw2: bound to major 0, minor 0# raw –qa (No output)

Note you may need to disable multipathing to avoid installation error while installing Oracle. Once finished, configure multipath again. Bug 5504430

15


Set up FCP multipathing [both]Configure multipathing

– modprobe dm-multipath

– create /etc/multipath.conf with following lines

– configure multipathing

– edit /var/lib/multipath/bindings as the following

– reconfigure multipathing with alias

defaults {user_friendly_names yes

}

lun001 1IBM_2105_10627963lun002 1IBM_2105_10727963lun003 1IBM_2105_10827963lun004 1IBM_2105_20827963lun005 1IBM_2105_20927963

# modprobe dm-multipath

# multipath

# multipath

Modify the alias name only.Don’t change the wwid, which is different with LUNs.


Set the multipath services on

Configure raw bind– edit /etc/raw as the following

Reboot

Set up FCP multipathing

raw1:mapper/lun001-part1raw2:mapper/lun002-part1raw3:mapper/lun001-part2raw4:mapper/lun002-part2raw5:mapper/lun003-part1raw6:mapper/lun004-part1raw7:mapper/lun005-part1

# chkconfig boot.multipath on# chkconfig multipathd on

16


Client Example

Utilizing both ECKD and SCSI(FCP) storageOperating system, Oracle Binaries, secondary swap device on ECKDUtilizing VDISK for Primary swapVoting and OCR disk shared on ECKD to just RAC nodes in the cluster.SCSI (FCP) disk with Multipathing across several disk ranks of the same size and utilizing Oracle ASM to stripe across these devices.$cluvfy stage -pre crsinst -n all -verbose $cluvfy stage -post crsinst -n all –verboseAre good steps to run to check your setup before and after.


Backing Up ASM, Voting and OCR disks

Backup any *ASM* files in the Oracle ASM home dbs directory in order to backup the ASM instance. e.g. ab_+ASM1.dat, hc_+ASM1.dat, init+ASM1.ora, and orapw+ASM1

Use dd to backup Oracle voting disk (can be done online)dd if=/dev/dasdo1 of=backup_voting_disk_${ORACLE_SID}.dmp bs=4k

Use standard Oracle RMAN scripts to backup the database, recommend that archivelogs are on shared storage so that rman can read the archivelogs for all instances

OCR disk is backed up every 4 hours automatically, $ ocrconfig -showbackuporalin02 2008/04/21 16:08:56 /oracle/CRS/product/10gR2/cdata/crs

You can also use ocrconfig –export to export as well

17


Some Oracle 11g Upcoming Parameters and Features

– AU_SIZE = 1, 2, 4, 8, 16, 32, or 64 MB– DISK_REPAIR_TIME = 3.6 Hours– ASM fast resync keeps track of pending changes to extents on an

OFFLINE disk during an outage. The extents are resynced when the disk is brought back online or replaced.

By default, ASM drops a disk shortly after it is taken offline. You can set the DISK_REPAIR_TIME attribute to prevent this operationby specifying a time interval to repair the disk and bring it back online.

– Database replay (Capture workload in production, replay in test)– Hot Patching using opatch, Memory requirement of one OS page

per Oracle process


Without Fast Mirror ResyncASM redundancy used1 Disk access failure2

Disk added back:Extents rebalanced

4 Disk automatically dropped:All dropped extents re-created

3

Oracle Database 10g and 11g

18


With ASM Fast Mirror ResyncASM redundancy used1 Disk access failure2

Oracle Database 11g

Failure time < DISK_REPAIR_TIME Access other extents

3Disk again accessible:

Auto resync modified extents4


Other ASM Administration Features

Migrating Oracle ASM Disk groups per Note 438501.1, works well but be sure to switch the redo logs a couple of times.Be very careful when recreating controlfile and redo log threads, Need to use ALTER DATABASE ADD LOGFILE THREAD 2 command per Metalink Note:118931.1 or else all the logs will be assigned to one node.Bug when adding voting disk while clusterware is running - 329734.1, safer to keep clusterware down when modifying voting disks as patch is not available.When modifying or adding OCR disks, keep clusterware running on all nodes or else you will need to repair the OCR on the nodes that were down.You can utilize the Enterprise Manager AWR reports via scriptSnapshot creation:SQL> EXECUTE dbms_workload_repository.create_snapshot();Report creation:SQL> @?/rdbms/admin/awrrpt.sql

19


SLES 10,Red Hat 5.0 (when certified) make sure to edit vipca and srvct

Edit vipca and srvctl

if [ "$arch" = "i686" -o "$arch" = "ia64" -o "$arch" = "x86_64" -o "$arch" = "s390x" ]then

LD_ASSUME_KERNEL=2.4.19export LD_ASSUME_KERNEL

fiunset LD_ASSUME_KERNEL

#End workaround

Add this line

# vi /opt/oracle/product/10gR2/crs/bin/vipca

#Remove this workaround when the bug 3937317 is fixedLD_ASSUME_KERNEL=2.4.19export LD_ASSUME_KERNELunset LD_ASSUME_KERNEL Add this line

# vi /opt/oracle/product/10gR2/crs/bin/srvctl


Information Sourceshttp://www.ibm.com/redbooks

– SG24-6552-00 Experiences with Oracle9i for Linux on zSeries– SG24-6482-00 Experience with Oracle Database 10g on Linux for zSeries– SG24-7191-00 Experiences with Oracle 10gR2 Solutions on Linux for System z– SG24-6669-00 Linux for IBM System z9 and zSeries– SG24-7573 - Using Oracle Solutions on Linux for System z

http://www.ibm.com/developerworks/linux/linux390/perf/index.html

http://download.oracle.com/docs/cd/B19306_01/server.102/b14231/storeman.htm#sthref1677 – Oracle ASM Memory Sizing

http://download.oracle.com/docs/cd/B19306_01/server.102/b14211/memory.htm#i47856– 10gR2 Oracle Database Performance Tuning Guide– Linux on System z Oracle Hands-on materials Japan System z Technical Sales

Metalink Notes:Note 257643.1 - Oracle Database 10g Automated SGA Memory Tuning Note 295626.1 - Automatic Shared Memory Management (ASMM) Oracle10gNote 471165.1-Additional steps 10gR2 RAC IBM zSeries Based Linux (SLES10)

oracle rac on system z storage configurations for rac assignment of disk from z/vm oracle asm,...

Documents