DELL CONFIDENTIAL

MAXIMIZING PERFORMANCE ON ORACLE 11G SE RAC

Wendy Chen, Systems EngineerNaveen Iyengar, Systems EngineerGlobal Solutions Engineering, Dell Inc.

SE VS. EE

Oracle 11g database is available in multiple editions Enterprise Edition (EE)

– Single or clustered servers– No limit on the maximum number of CPU (up to the maximum

number of nodes supported in a RAC cluster)– Contains all Oracle database components, and can be further

enhanced with the purchase of options and packs Standard Edition (SE)

– Single or clustered servers– Up to a maximum of 4 CPU sockets in the single or clustered

servers– Includes selected, but not all the features come with EE – Built from the same code base as EE– Ideally suited to the needs to SMB with enterprise level

performance

FEATURE AVAILABILITY

A number of features that come with Oracle 11g EE are not available in SE. For example,– Data guard– Rolling upgrades– Online index and table organization– Parallel backup and recovery– Tablespaces point-in-time recovery– Flashback table / transaction / database– Parallel query / statistics gathering / index builds / data pump

export and import– Transportable tablespaces– Infiniband support

Before deploying SE, make sure that you do not require any of the non-supported features.

PAY-AS-YOU-GROW SCALABILITY

“Pay-as-you-grow” methodology to scale up to four single socket machines

Storage Group

RAID 10

Flash Recovery Area Volume

Storage Group

OCR and Voting Disk Volume

Data VolumeStorage Pool

RAID 10 RAID 10

Client Systems

Dell Gigabit Ethernet Switches for Oracle Cluster Private Network

Dell PowerEdge R610 Servers

Dell PowerConnect 6248 Gigabit Ethernet Switches* for

iSCSI Storage Area Network

Dell EqualLogic PS4000XV iSCSI Storage Arrays

TESTING PERFORMANCE CAPABILITIES OF 11G SE RAC

Quest Benchmark Factory TPC-C– A single node Oracle 11.1.0.7 SE database: 100 to 10,000 users

– Two-node Oracle 11.1.0.7 SE RAC database: 100 to 10,000 users

– Three-node Oracle 11.1.0.7 SE RAC database: 100 to 10,000 users

– Four-node Oracle 11.1.0.7 SE RAC database: 100 to 10,000 users

Server Up to four Dell PowerEdge R610 servers with:A single Intel® Xeon® X5560 quad-core 2.80 GHz CPU with 8M cache, 6.40 GT/s QPI and TURBO and HT mode enabled 

24 GB of RAM or 48 GB of RAM, with 4GB or 8GB DIMMs respectivelyFour 1Gb Broadcom NetXtreme II NIC ports for iSCSI traffic

External Storage Two DellTM EqualLogicTM PS4000XV iSCSI storage arrays, each with 16 15K RPM 146GB SAS hard drives

Volume Configuration Three 220 GB volumes for database files; One 150 GB volume for Flash Recovery area; One 2 GB volume for OCR, CSS, and SPFILE

OS and Device Driver Oracle Enterprise Linux 5 Update 3Open iSCSI initiator iscsi-initiator-utils-6.2.0.868-0.18.el5Device Mapper multipath driver device-mapper-multipath-0.4.7-23.el5

Storage Network Two stacked Dell PowerConnect 6248 gigabit Ethernet switches for iSCSI SAN

Test Software Quest Benchmark Factory 5.7.1 Oracle 64 bit 11.1.0.7 SE RAC

Database Configuration Up to four Oracle 11.1.0.7 SE RAC with:13 GB memory_target on each instance

PERFORMANCE MONITORING

CPU utilization: SAR (System Activity Report)

nohup sar 1 30000 > ~/sar-4nodes-run195-node1.txt &  CPU %user %nice %system %iowait %steal %idle

02:31:43 PM all 0.38 0.00 0.25 4.25 0.00 95.1202:31:44 PM all 0.25 0.00 0.25 4.12 0.00 95.3802:31:45 PM all 0.12 0.00 0.50 0.25 0.00 99.1302:31:46 PM all 0.62 0.00 0.62 1.12 0.00 97.6202:31:47 PM all 0.87 0.00 1.12 1.99 0.00 96.0102:31:48 PM all 0.13 0.00 0.38 5.63 0.00 93.8702:31:49 PM all 0.62 0.00 0.37 1.87 0.00 97.13...02:33:29 PM all 0.38 0.00 0.12 0.25 0.00 99.2502:33:30 PM all 0.00 0.00 0.25 0.00 0.00 99.7502:33:31 PM all 0.25 0.00 0.62 0.25 0.00 98.8802:33:32 PM all 0.37 0.00 0.25 0.00 0.00 99.3802:33:33 PM all 0.50 0.00 0.50 1.13 0.00 97.8702:33:34 PM all 0.25 0.00 0.25 0.25 0.00 99.25

PERFORMANCE MONITORING

Physical memory utilization: OS Watcher

nohup ./startOSW.sh 5 96 &

 zzz ***Sun Jun 21 23:04:59 CDT 2009MemTotal: 49434036 kBMemFree: 34270564 kBBuffers: 354892 kBCached: 12637136 kBSwapCached: 7732 kBActive: 10278772 kBInactive: 4483804 kBHighTotal: 0 kBHighFree: 0 kBLowTotal: 49434036 kBLowFree: 34270564 kBSwapTotal: 8388600 kBSwapFree: 8239712 kB

PERFORMANCE MONITORING

Storage utilization: EqualLogic SAN HeadQuarters

TUNING ORACLE MEMORY SIZE The optimal size of SGA is the point where the marginal benefit of physical

I/O reduction begins to decline The optimal size of PGA is when the estimated PGA over-allocation size is 0

SGA Target Size (M) SGA Size Factor Est DB Time (s) Est Physical Reads

3,168 0.38 1,327,870 37,779,644

4,224 0.50 895,776 20,459,187

5,280 0.63 728,220 13,742,069

6,336 0.75 641,710 10,272,675

7,392 0.88 598,910 8,558,231

8,448 1.00 569,142 7,364,453

9,504 1.13 549,509 6,576,457

10,560 1.25 534,597 5,979,936

11,616 1.38 524,011 5,555,743

12,672 1.50 519,685 5,381,942

13,728 1.63 516,046 5,236,863

14,784 1.75 516,112 5,236,863

15,840 1.88 516,117 5,236,863

16,896 2.00 516,119 5,236,863

PGA Target Est (MB)

Size FactrW/A MB

Processed

Estd Extra W/A MB Read/

Written to Disk

Estd PGA Cache Hit %

Estd PGA Overalloc Count

Estd Time

912 0.25 564.98 15.25 97.00 113 1,740

1,824 0.50 564.98 15.25 97.00 77 1,740

2,736 0.75 564.98 15.25 97.00 50 1,740

3,648 1.00 564.98 0.00 100.00 17 1,694

4,377 1.20 564.98 0.00 100.00 0 1,694

5,107 1.40 564.98 0.00 100.00 0 1,694

5,836 1.60 564.98 0.00 100.00 0 1,694

6,566 1.80 564.98 0.00 100.00 0 1,694

7,296 2.00 564.98 0.00 100.00 0 1,694

TESTING SE RAC SCALABILITY WITH 24 GB RAM PER NODE – CPU UTILIZATION

TESTING SE RAC SCALABILITY WITH 24 GB RAM PER NODE – MEMORY AND STORAGE UTILIZATION

TESTING SE RAC SCALABILITY WITH 24 GB RAM PER NODE – OVERALL PERFORMANCE ANALYSIS

As the systems near their maximum memory performance, they progressively utilize more swap space and the database starts to get unstable.

When comparing the CPU graph with the memory graph, at the maximum user load there is still ample CPU capability even once the test fails.

Given that the Oracle memory area (memory_target) is set to 13 GB to minimize physical I/O, even at the maximum 4-node cluster size the storage members were delivering IOPS well below their potential, suggesting that the storage disks were not the bottleneck for the overall performance.

TESTING SE RAC SCALABILITY WITH 48 GB RAM PER NODE – CPU UTILIZATION

TESTING SE RAC SCALABILITY WITH 48 GB RAM PER NODE – MEMORY AND STORAGE UTILIZATION

TESTING SE RAC SCALABILITY WITH 48 GB RAM PER NODE – OVERALL PERFORMANCE ANALYSIS

As the physical memory increased from 24 GB to 48 GB, the system performance was no longer limited

by memory, thus the database was able to handle more transactions at higher user loads. Consequently,

the PS4000XV members processed more I/O requests and showed much improved IOPS results

comparing to the 24 GB configuration.

The corresponding disk latency performance was still within the acceptable range of 20 ms in all test

durations of the 48 GB memory configurations, except towards the end of run on the 4-node RAC, when

disk latency started to have a delayed response of above 20 ms. This indicates that the PS4000XV

showed signs of stress after approximately 7000 user loads on the 4-node RAC.

PERFORMANCE CHARACTERISTICS SUMMARY

The 48 GB Oracle RAC cluster scales better than its 24 GB counterpart. For the 24 GB configuration, the TPS

measure scales in the range of 40% - 60% with the addition of each node. With 48 GB per node, the results

demonstrate a near linear scalability, in the range of 75%-95% and that the benefit of additional memory became

more apparent with each additional node.

With 48 GB of physical memory per node, both the CPU and memory resources become limited almost

simultaneously. In an ideal situation, all the resources would be constrained simultaneously. Extra CPU

capability without complementary memory may not lead to ideal scaling characteristics, and likewise extra

memory without CPU capability may be unnecessary.

By adequately balancing memory, CPU and storage you can help make the most out of scaling out an Oracle

RAC cluster. A bottleneck in any one component such as storage, CPU, or memory may result in non-optimal

scaling. Likewise, scaling up of one component when another component is the bottleneck for the system may

not add performance and may be unnecessary.

RESOURCES

Dell whitepaper: Maximizing Performance on a Cost Effective Oracle 11g SE RAC Database on Dell EqualLogic PS4000XV iSCSI Storage http://www.dell.com/downloads/global/solutions/Oracle_SE_RAC.pdf?c=us&cs=555&l=en&s=biz

Dell Oracle Solutions Engineering http://www.dell.com/oracle

Q & A