Building an Inexpensive Oracle RAC 10g R2 on Linux - (RHEL 5.3 /

iSCSI)

by Jeff Hunter, Sr. Database Administrator

Contents

1. Overview

2. Oracle RAC 10g Overview

3. Shared-Storage Overview

4. iSCSI Technology

5. Hardware and Costs

6. Install the Linux Operating System

7. Install Required Linux Packages for Oracle RAC

8. Network Configuration

9. Install Openfiler

10. Configure iSCSI Volumes using Openfiler

11. Configure iSCSI Volumes on Oracle RAC Nodes

12. Create "oracle" User and Directories

13. Configure the Linux Servers for Oracle

14. Configure the "hangcheck-timer" Kernel Module

15. Configure RAC Nodes for Remote Access using SSH

16. All Startup Commands for Both Oracle RAC Nodes

17. Install and Configure Oracle Cluster File System (OCFS2)

18. Install and Configure Automatic Storage Management (ASMLib 2.0)

19. Download Oracle RAC 10g Software

20. Pre-Installation Tasks for Oracle10g Release 2

21. Install Oracle Clusterware 10g Software

22. Install Oracle Database 10g Software

23. Install Oracle Database 10g Companion CD Software

24. Create TNS Listener Process

25. Create the Oracle Cluster Database

26. Post-Installation Tasks - (Optional)

27. Verify TNS Networking Files

28. Create / Alter Tablespaces

29. Verify the RAC Cluster & Database Configuration

30. Starting / Stopping the Cluster

31. Transparent Application Failover - (TAF)

32. Troubleshooting

33. Conclusion

34. Building an Oracle RAC Cluster Remotely

35. Acknowledgements

36. About the Author

Overview

One of the most efficient ways to become familiar with Oracle Real Application Cluster (RAC) 10g technology

is to have access to an actual Oracle RAC 10g cluster. In learning this new technology, you will soon start to

realize the benefits Oracle RAC 10g has to offer like fault tolerance, new levels of security, load balancing, and

the ease of upgrading capacity. The problem though is the price of the hardware required for a typical production

RAC configuration. A small two node cluster, for example, could run anywhere from US$10,000 to well over

US$20,000. This would not even include the heart of a production RAC environment, the shared storage. In most

cases, this would be a Storage Area Network (SAN), which generally start at US$10,000.

For those who want to become familiar with Oracle RAC 10g without a major cash outlay, this guide provides a

low-cost alternative to configuring an Oracle RAC 10g Release 2 system using commercial off-the-shelf

components and downloadable software at an estimated cost of US$2,500 to US$3,000. The system will consist

of a dual node cluster (two i386 nodes with a single processor), both running Linux (CentOS 5.3 or Red Hat

Enterprise Linux 5 Update 3), Oracle10g Release 2 for Linux x86, OCFS2, and ASMLib 2.0. All shared disk

storage for Oracle RAC will be based on iSCSI using Openfiler Release 2.3 (Final) x86_64 running on a third

node (known in this article as the Network Storage Server).

Powered by rPath Linux, Openfiler is a free browser-based network storage management utility that delivers file-

based Network Attached Storage (NAS) and block-based Storage Area Networking (SAN) in a single

framework. The entire software stack interfaces with open source applications such as Apache, Samba, LVM2,

ext3, Linux NFS and iSCSI Enterprise Target. Openfiler combines these ubiquitous technologies into a small,

easy to manage solution fronted by a powerful web-based management interface.

Openfiler supports CIFS, NFS, HTTP/DAV, FTP, however, we will only be making use of its iSCSI capabilities

to implement an inexpensive SAN for the shared storage components required by Oracle RAC 10g. The

operating system and Openfiler application will be installed on one internal SATA disk. A second internal 73GB

15K SCSI hard disk will be configured as a single "Volume Group" that will be used for all shared disk storage

requirements. The Openfiler server will be configured to use this volume group for iSCSI based storage and will

be used in our Oracle RAC 10g configuration to store the shared files required by Oracle Clusterware as well as

all Oracle ASM volumes.

This article is provided for educational purposes only, so the setup is kept simple to demonstrate ideas and

concepts. For example, the disk mirroring configured in this article will be setup on one physical disk only,

while in practice that should be performed on multiple physical drives.

In addition, each Linux node will only be configured with two network interfaces — one for the public network

(eth0) and one that will be used for both the Oracle RAC private interconnect "and" the network storage server

for shared iSCSI access (eth1). For a production RAC implementation, the private interconnect should be at

least Gigabit (or more) with redundant paths and "only" be used by Oracle to transfer Cluster Manager and

Cache Fusion related data. A third dedicated network interface (i.e. eth2) should be configured on another

redundant Gigabit network for access to the network storage server (Openfiler).

Finally, while this article provides detailed instructions for successfully installing a complete Oracle RAC 10g

system, it is by no means a substitute for the official Oracle documentation. In addition to this article, users

should also consult the following Oracle documents to gain a full understanding of alternative configuration

options, installation, and administration with Oracle RAC 10g. Oracle's official documentation site is

docs.oracle.com.

Oracle Clusterware and Oracle Real Application Clusters Installation Guide - 10g Release 2 (10.2) for

Linux

Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide - 10g

Release 2 (10.2)

2 Day + Real Application Clusters Guide - 10g Release 2 (10.2)

Although in past articles I used raw partitions for storing files on shared storage, here we will make use of the

Oracle Cluster File System V2 (OCFS2) and Oracle Automatic Storage Management (ASM). The two Oracle

RAC nodes will be configured as follows:

Oracle Database Files

RAC Node Name Instance Name Database

Name $ORACLE_BAS

E

File System -

Volume Manager for DB

Files

linux1 racdb1 racdb /u01/app/oracle ASM

linux2 racdb2 racdb /u01/app/oracle ASM

Oracle Clusterware Shared Files

File Type File Name iSCSI

Volume Name Mount Point File System

Oracle Cluster Registry

(OCR)

/u02/oradata/racdb/OCRFile

/u02/oradata/racdb/OCRFile_mirr

or crs /u02 OCFS2

Voting Disk

/u02/oradata/racdb/CSSFile

/u02/oradata/racdb/CSSFile_mirro

r1

/u02/oradata/racdb/CSSFile_mirro

r2

crs /u02 OCFS2

With Oracle Database 10g Release 2 (10.2), Cluster Ready Services, or CRS, is now

called Oracle Clusterware.

The Oracle Clusterware software will be installed to /u01/app/crs on both of the nodes that make up the RAC

cluster. Starting with Oracle Database 10g Release 2 (10.2), Oracle Clusterware should be installed in a separate

Oracle Clusterware home directory which is non-release specific (/u01/app/oracle/product/10.2.0/... for

example) and must never be a subdirectory of the ORACLE_BASE directory (/u01/app/oracle for example). This

is a change to the Optimal Flexible Architecture (OFA) rules. Note that the Oracle Clusterware and Oracle Real

Application Clusters installation documentation from Oracle incorrectly state that the Oracle Clusterware home

directory can be a subdirectory of the ORACLE_BASE directory. For example, in Chapter 2, "Preinstallation", in the

section "Oracle Clusterware home directory", it incorrectly lists the path /u01/app/oracle/product/crs as a

possible Oracle Clusterware home (or CRS home) path. This is incorrect. The default ORACLE_BASE path is

/u01/app/oracle, and the Oracle Clusterware home must never be a subdirectory of the ORACLE_BASE

directory. This issue is tracked with Oracle documentation bug "5843155" - (B14203-08 HAS CONFLICTING

CRS_HOME LOCATIONS ) and is fixed in Oracle 11g.

The Oracle Clusterware software will be installed to /u01/app/crs on both of the nodes that make up the RAC

cluster, however, the Clusterware software requires that two of its files, the "Oracle Cluster Registry (OCR)" file

and the "Voting Disk" file be shared with both nodes in the cluster. These two files will be installed on shared

storage using Oracle's Cluster File System, Release 2 (OCFS2). It is also possible to use RAW devices for these

files, however, it is not possible to use ASM for these two shared Clusterware files.

The Oracle Database 10g Release 2 software will be installed into a separate Oracle Home; namely

/u01/app/oracle/product/10.2.0/db_1 on both of the nodes that make up the RAC cluster. All of the Oracle

physical database files (data, online redo logs, control files, archived redo logs) will be installed to shared

volumes being managed by Automatic Storage Management (ASM).

The Oracle database files could have just as well been stored on the Oracle Cluster File

System (OFCS2). Using ASM, however, makes the article that much more interesting!

This article is only designed to work as documented with absolutely no substitutions!

The only exception here is the choice of vendor hardware (i.e. machines, networking

equipment, and internal / external hard drives). Ensure that the hardware you

purchase from the vendor is supported on Red Hat Linux 5. I tend to stick with Dell

hardware given their superb quality and compatibility with Linux. For a test system

of this nature, I highly recommend purchasing pre-owned or refurbished Dell hardware

from a reputable company like Stallard Technologies, Inc.. Stallard Technologies has a

proven track record of delivering the best value on pre-owned hardware combined with

a commitment to superior customer service. I base my recommendation on my own

outstanding personal experience with their organization. To learn more about Stallard

Technologies, visit their website or contact William Buchanan.

The following is a list of past articles which describe configuring a similar Oracle RAC Cluster

using various versions of Oracle, Operating System, and shared storage medium:

Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.2 / FireWire)

Building an Inexpensive Oracle RAC 10g Release 1 on Linux - (WBEL 3.0 / FireWire)

Building an Inexpensive Oracle RAC 9i on Linux - (Fedora Core 1 / FireWire)

Oracle RAC 10g Overview

Before introducing the details for building a RAC cluster, it might be helpful to first clarify what a cluster is. A

cluster is a group of two or more interconnected computers or servers that appear as if they are one server to end

users and applications and generally share the same set of physical disks. The key benefit of clustering is to

provide a highly available framework where the failure of one node (for example a database server) does not

bring down an entire application. In the case of failure with one of the servers, the other surviving server (or

servers) can take over the workload from the failed server and the application continue to function normally as if

nothing has happened.

The concept of clustering computers actually started several decades ago. The first successful cluster product

was developed by DataPoint in 1977 named ARCnet. The ARCnet product enjoyed much success by academia

types in research labs, but didn't really take off in the commercial market. It wasn't until the 1980's when Digital

Equipment Corporation (DEC) released its VAX cluster product for the VAX/VMS operating system.

With the release of Oracle 6 for the Digital VAX cluster product, Oracle Corporation was the first commercial

database to support clustering at the database level. It wasn't long, however, before Oracle realized the need for a

more efficient and scalable distributed lock manager (DLM) as the one included with the VAX/VMS cluster

product was not well suited for database applications. Oracle decided to design and write their own DLM for the

VAX/VMS cluster product which provided the fine-grain block level locking required by the database. Oracle's

own DLM was included in Oracle 6.2 which gave birth to Oracle Parallel Server (OPS) - the first database to run

the parallel server.

By Oracle 7, OPS was extended to included support for not only the VAX/VMS cluster product but also with

most flavors of UNIX. This framework required vendor-supplied clusterware which worked well, but made for a

complex environment to setup and manage given the multiple layers involved. By Oracle 8, Oracle introduced a

generic lock manager which was integrated into the Oracle kernel. In later releases of Oracle, this became known

as the Integrated Distributed Lock Manager (IDLM) and relied on an additional layer known as the Operating

System Dependant (OSD) layer. This new model paved the way for Oracle to not only have their own DLM, but

to also create their own clusterware product in future releases.

Oracle Real Application Clusters (RAC), introduced with Oracle9i, is the successor to Oracle Parallel Server.

Using the same IDLM, Oracle 9i could still rely on external clusterware but was the first release to include their

own clusterware product named Cluster Ready Services (CRS). With Oracle 9i, CRS was only available for

Windows and Linux. By Oracle 10g, Oracle's clusterware product was available for all operating systems. With

the release of Oracle Database 10g Release 2 (10.2), Cluster Ready Services was renamed to Oracle Clusterware.

When using Oracle 10g or higher, Oracle Clusterware is the only clusterware that you need for most platforms

on which Oracle RAC operates (except for Tru cluster, where you need vendor clusterware). You can still use

clusterware from other vendors if the clusterware is certified for Oracle RAC. This guide uses Oracle

Clusterware 10g.

Like OPS, Oracle RAC allows multiple instances to access the same database (storage) simultaneously. RAC

provides fault tolerance, load balancing, and performance benefits by allowing the system to scale out, and at the

same time since all instances access the same database, the failure of one node will not cause the loss of access to

the database.

At the heart of Oracle RAC is a shared disk subsystem. Each instance in the cluster must be able to access all of

the data, redo log files, control files and parameter file for all other instances in the cluster. The data disks must

be globally available in order to allow all instances to access the database. Each instance has its own redo log

files and UNDO tablespace that are locally read-writeable. The other instances in the cluster must be able to

access them (read-only) in order to recover that instance in the event of a system failure. The redo log files for an

instance are only writeable by that instance and will only be read from another instance during system failure.

The UNDO, on the other hand, is read all the time during normal database operation (e.g. for CR fabrication).

The biggest difference between Oracle RAC and OPS is the addition of Cache Fusion. With OPS a request for

data from one instance to another required the data to be written to disk first, then the requesting instance can

read that data (after acquiring the required locks). With cache fusion, data is passed along a high-speed

interconnect using a sophisticated locking algorithm.

Not all database clustering solutions use shared storage. Some vendors use an approach known as a Federated

Cluster, in which data is spread across several machines rather than shared by all. With Oracle RAC, however,

multiple instances use the same set of disks for storing data. Oracle's approach to clustering leverages the

collective processing power of all the nodes in the cluster and at the same time provides failover security.

Pre-configured Oracle RAC solutions are available from vendors such as Dell, IBM and HP for production

environments. This article, however, focuses on putting together your own Oracle RAC 10g environment for

development and testing by using Linux servers and a low cost shared disk solution; iSCSI.

For more background about Oracle RAC, visit the Oracle RAC Product Center on OTN.

Shared-Storage Overview

Today, fibre channel is one of the most popular solutions for shared storage. As mentioned earlier, fibre channel

is a high-speed serial-transfer interface that is used to connect systems and storage devices in either point-to-

point (FC-P2P), arbitrated loop (FC-AL), or switched topologies (FC-SW). Protocols supported by Fibre

Channel include SCSI and IP. Fibre channel configurations can support as many as 127 nodes and have a

throughput of up to 2.12 gigabits per second in each direction, and 4.25 Gbps is expected.

Fibre channel, however, is very expensive. Just the fibre channel switch alone can start at around US$1,000. This

does not even include the fibre channel storage array and high-end drives, which can reach prices of about

US$300 for a 36GB drive. A typical fibre channel setup which includes fibre channel cards for the servers is

roughly US$10,000, which does not include the cost of the servers that make up the cluster.

A less expensive alternative to fibre channel is SCSI. SCSI technology provides acceptable performance for

shared storage, but for administrators and developers who are used to GPL-based Linux prices, even SCSI can

come in over budget, at around US$2,000 to US$5,000 for a two-node cluster.

Another popular solution is the Sun NFS (Network File System) found on a NAS. It can be used for shared

storage but only if you are using a network appliance or something similar. Specifically, you need servers that

guarantee direct I/O over NFS, TCP as the transport protocol, and read/write block sizes of 32K. See the Certify

page on Oracle Metalink for supported Network Attached Storage (NAS) devices that can be used with Oracle

RAC. One of the key drawbacks that has limited the benefits of using NFS and NAS for database storage has

been performance degradation and complex configuration requirements. Standard NFS client software (client

systems that use the operating system provided NFS driver) is not optimized for Oracle database file I/O access

patterns. With the introduction of Oracle 11g, a new feature known as Direct NFS Client integrates the NFS

client functionality directly in the Oracle software. Through this integration, Oracle is able to optimize the I/O

path between the Oracle software and the NFS server resulting in significant performance gains. Direct NFS

Client can simplify, and in many cases automate, the performance optimization of the NFS client configuration

for database workloads. To learn more about Direct NFS Client, see the Oracle White Paper entitled "Oracle

Database 11g Direct NFS Client".

The shared storage that will be used for this article is based on iSCSI technology using a network storage server

installed with Openfiler. This solution offers a low-cost alternative to fibre channel for testing and educational

purposes, but given the low-end hardware being used, it is not recommended to be used in a production

environment.

iSCSI Technology

For many years, the only technology that existed for building a network based storage solution was a Fibre

Channel Storage Area Network (FC SAN). Based on an earlier set of ANSI protocols called Fiber Distributed

Data Interface (FDDI), Fibre Channel was developed to move SCSI commands over a storage network.

Several of the advantages to FC SAN include greater performance, increased disk utilization, improved

availability, better scalability, and most important to us — support for server clustering! Still today, however, FC

SANs suffer from three major disadvantages. The first is price. While the costs involved in building a FC SAN

have come down in recent years, the cost of entry still remains prohibitive for small companies with limited IT

budgets. The second is incompatible hardware components. Since its adoption, many product manufacturers

have interpreted the Fibre Channel specifications differently from each other which has resulted in scores of

interconnect problems. When purchasing Fibre Channel components from a common manufacturer, this is

usually not a problem. The third disadvantage is the fact that a Fibre Channel network is not Ethernet! It requires

a separate network technology along with a second set of skill sets that need to exist with the datacenter staff.

With the popularity of Gigabit Ethernet and the demand for lower cost, Fibre Channel has recently been given a

run for its money by iSCSI-based storage systems. Today, iSCSI SANs remain the leading competitor to FC

SANs.

Ratified on February 11, 2003 by the Internet Engineering Task Force (IETF), the Internet Small Computer

System Interface, better known as iSCSI, is an Internet Protocol (IP)-based storage networking standard for

establishing and managing connections between IP-based storage devices, hosts, and clients. iSCSI is a data

transport protocol defined in the SCSI-3 specifications framework and is similar to Fibre Channel in that it is

responsible for carrying block-level data over a storage network. Block-level communication means that data is

transferred between the host and the client in chunks called blocks. Database servers depend on this type of

communication (as opposed to the file level communication used by most NAS systems) in order to work

properly. Like a FC SAN, an iSCSI SAN should be a separate physical network devoted entirely to storage,

however, its components can be much the same as in a typical IP network (LAN).

While iSCSI has a promising future, many of its early critics were quick to point out some of its inherent

shortcomings with regards to performance. The beauty of iSCSI is its ability to utilize an already familiar IP

network as its transport mechanism. The TCP/IP protocol, however, is very complex and CPU intensive. With

iSCSI, most of the processing of the data (both TCP and iSCSI) is handled in software and is much slower than

Fibre Channel which is handled completely in hardware. The overhead incurred in mapping every SCSI

command onto an equivalent iSCSI transaction is excessive. For many the solution is to do away with iSCSI

software initiators and invest in specialized cards that can offload TCP/IP and iSCSI processing from a server's

CPU. These specialized cards are sometimes referred to as an iSCSI Host Bus Adaptor (HBA) or a TCP Offload

Engine (TOE) card. Also consider that 10-Gigabit Ethernet is a reality today!

As with any new technology, iSCSI comes with its own set of acronyms and terminology. For the purpose of this

article, it is only important to understand the difference between an iSCSI initiator and an iSCSI target.

iSCSI Initiator

Basically, an iSCSI initiator is a client device that connects and initiates requests to some service offered by a

server (in this case an iSCSI target). The iSCSI initiator software will need to exist on each of the Oracle RAC

nodes (linux1 and linux2).

An iSCSI initiator can be implemented using either software or hardware. Software iSCSI initiators are available

for most major operating system platforms. For this article, we will be using the free Linux Open-iSCSI software

driver found in the iscsi-initiator-utils RPM. The iSCSI software initiator is generally used with a

standard network interface card (NIC) — a Gigabit Ethernet card in most cases. A hardware initiator is an iSCSI

HBA (or a TCP Offload Engine (TOE) card), which is basically just a specialized Ethernet card with a SCSI

ASIC on-board to offload all the work (TCP and SCSI commands) from the system CPU. iSCSI HBAs are

available from a number of vendors, including Adaptec, Alacritech, Intel, and QLogic.

iSCSI Target

An iSCSI target is the "server" component of an iSCSI network. This is typically the storage device that contains

the information you want and answers requests from the initiator(s). For the purpose of this article, the node

openfiler1 will be the iSCSI target.

So with all of this talk about iSCSI, does this mean the death of Fibre Channel anytime soon? Probably not.

Fibre Channel has clearly demonstrated its capabilities over the years with its capacity for extremely high

speeds, flexibility, and robust reliability. Customers who have strict requirements for high performance storage,

large complex connectivity, and mission critical reliability will undoubtedly continue to choose Fibre Channel.

Before closing out this section, I thought it would be appropriate to present the following chart that shows speed

comparisons of the various types of disk interfaces and network technologies. For each interface, I provide the

maximum transfer rates in kilobits (kb), kilobytes (KB), megabits (Mb), megabytes (MB), gigabits (Gb), and

gigabytes (GB) per second with some of the more common ones highlighted in grey.

Disk Interface / Network / BUS Speed

Kb KB Mb MB Gb GB

Serial 115 14.375 0.115 0.014

Parallel (standard) 920 115 0.92 0.115

10Base-T Ethernet 10 1.25

IEEE 802.11b wireless Wi-Fi (2.4 GHz band) 11 1.375

USB 1.1 12 1.5

Parallel (ECP/EPP) 24 3

SCSI-1 40 5

IEEE 802.11g wireless WLAN (2.4 GHz band) 54 6.75

SCSI-2 (Fast SCSI / Fast Narrow SCSI) 80 10

100Base-T Ethernet (Fast Ethernet) 100 12.5

ATA/100 (parallel) 100 12.5

IDE 133.6 16.7

Fast Wide SCSI (Wide SCSI) 160 20

Ultra SCSI (SCSI-3 / Fast-20 / Ultra Narrow) 160 20

Ultra IDE 264 33

Wide Ultra SCSI (Fast Wide 20) 320 40

Ultra2 SCSI 320 40

FireWire 400 - (IEEE1394a) 400 50

USB 2.0 480 60

Wide Ultra2 SCSI 640 80

Ultra3 SCSI 640 80

FireWire 800 - (IEEE1394b) 800 100

Gigabit Ethernet 1000 125 1

PCI - (33 MHz / 32-bit) 1064 133 1.064

Serial ATA I - (SATA I) 1200 150 1.2

Wide Ultra3 SCSI 1280 160 1.28

Ultra160 SCSI 1280 160 1.28

PCI - (33 MHz / 64-bit) 2128 266 2.128

PCI - (66 MHz / 32-bit) 2128 266 2.128

AGP 1x - (66 MHz / 32-bit) 2128 266 2.128

Serial ATA II - (SATA II) 2400 300 2.4

Ultra320 SCSI 2560 320 2.56

FC-AL Fibre Channel 3200 400 3.2

PCI-Express x1 - (bidirectional) 4000 500 4

PCI - (66 MHz / 64-bit) 4256 532 4.256

AGP 2x - (133 MHz / 32-bit) 4264 533 4.264

Serial ATA III - (SATA III) 4800 600 4.8

PCI-X - (100 MHz / 64-bit) 6400 800 6.4

PCI-X - (133 MHz / 64-bit) 1064 8.512 1

AGP 4x - (266 MHz / 32-bit) 1066 8.528 1

10G Ethernet - (IEEE 802.3ae) 1250 10 1.25

PCI-Express x4 - (bidirectional) 2000 16 2

AGP 8x - (533 MHz / 32-bit) 2133 17.064 2.1

PCI-Express x8 - (bidirectional) 4000 32 4

PCI-Express x16 - (bidirectional) 8000 64 8

Hardware and Costs

The hardware used to build our example Oracle RAC 10g environment consists of three Linux servers (two

Oracle RAC nodes and one Network Storage Server) and components that can be purchased at many local

computer stores or over the Internet (i.e. Stallard Technologies, Inc.).

Oracle RAC Node 1 - (linux1)

Dell Dimension 2400 Series

- Intel(R) Pentium(R) 4 Processor at 2.80GHz - 2GB DDR SDRAM (at 333MHz)

- 40GB 7200 RPM Internal Hard Drive

- Integrated Intel 3D AGP Graphics

- Integrated 10/100 Ethernet - (Broadcom BCM4401)

- CDROM (48X Max Variable)

- 3.5" Floppy

- No Keyboard, Monitor, or Mouse - (Connected to KVM Switch) US$600

1 - Ethernet LAN Card

Used for RAC interconnect to linux2 and Openfiler networked storage.

Each Linux server for Oracle RAC should contain two NIC adapters. The Dell Dimension

includes an integrated 10/100 Ethernet adapter that will be used to connect to the public

network. The second NIC adapter will be used for the private network (RAC interconnect and

Openfiler networked storage). Select the appropriate NIC adapter that is compatible with the

maximum data transmission speed of the network switch to be used for the private network. For the purpose of this article, I used a Gigabit Ethernet switch (and 1Gb Ethernet cards) for

the private network.

Gigabit Ethernet

Intel 10/100/1000Mbps PCI Desktop Adapter - (PWLA8391GT) US$35

Oracle RAC Node 2 - (linux2)

Dell Dimension 2400 Series

- Intel(R) Pentium(R) 4 Processor at 2.80GHz

- 2GB DDR SDRAM (at 333MHz)

- 40GB 7200 RPM Internal Hard Drive - Integrated Intel 3D AGP Graphics

- Integrated 10/100 Ethernet - (Broadcom BCM4401)

- CDROM (48X Max Variable)

- 3.5" Floppy

- No Keyboard, Monitor, or Mouse - (Connected to KVM Switch) US$600

1 - Ethernet LAN Card

Used for RAC interconnect to linux1 and Openfiler networked storage.

Each Linux server for Oracle RAC should contain two NIC adapters. The Dell Dimension

includes an integrated 10/100 Ethernet adapter that will be used to connect to the public

network. The second NIC adapter will be used for the private network (RAC interconnect and

Openfiler networked storage). Select the appropriate NIC adapter that is compatible with the

maximum data transmission speed of the network switch to be used for the private network.

For the purpose of this article, I used a Gigabit Ethernet switch (and 1Gb Ethernet cards) for

the private network.

Gigabit Ethernet

Intel 10/100/1000Mbps PCI Desktop Adapter - (PWLA8391GT) US$35

Network Storage Server - (openfiler1)

Dell PowerEdge 1800

- Dual 3.0GHz Xeon / 1MB Cache / 800FSB (SL7PE) - 6GB of ECC Memory

- 500GB SATA Internal Hard Disk

- 73GB 15K SCSI Internal Hard Disk

- Integrated Graphics

- Single embedded Intel 10/100/1000 Gigabit NIC - 16x DVD Drive

- No Keyboard, Monitor, or Mouse - (Connected to KVM Switch)

Note: The operating system and Openfiler application will be installed on the 500GB internal

SATA disk. A second internal 73GB 15K SCSI hard disk will be configured for the database storage. The Openfiler server will be configured to use this second hard disk for iSCSI based

storage and will be used in our Oracle RAC 10g configuration to store the shared files required

by Oracle Clusterware as well as all Oracle ASM volumes.

Please be aware that any type of hard disk (internal or external) should work for database storage as long as it can be recognized by the network storage server (Openfiler) and has

adequate space. For example, I could have made an extra partition on the 500GB internal

SATA disk for the iSCSI target, but decided to make use of the faster SCSI disk for this US$800

example.

1 - Ethernet LAN Card

Used for networked storage on the private network.

The Network Storage Server (Openfiler server) should contain two NIC adapters. The Dell PowerEdge 1800 machine includes an integrated 10/100/1000 Ethernet adapter that will be

used to connect to the public network. The second NIC adapter will be used for the private

network (Openfiler networked storage). Select the appropriate NIC adapter that is compatible

with the maximum data transmission speed of the network switch to be used for the private

network. For the purpose of this article, I used a Gigabit Ethernet switch (and 1Gb Ethernet

cards) for the private network.

Gigabit Ethernet

Intel 10/100/1000Mbps PCI Desktop Adapter - (PWLA8391GT) US$35

Miscellaneous Components

1 - Ethernet Switch

Used for the interconnect between linux1-priv and linux2-priv. This switch will also be used for network storage traffic for Openfiler. For the purpose of this article, I used a Gigabit

Ethernet switch (and 1Gb Ethernet cards) for the private network.

Gigabit Ethernet

D-Link 8-port 10/100/1000 Desktop Switch - (DGS-2208) US$50

6 - Network Cables

Category 6 patch cable - (Connect linux1 to public network) Category 6 patch cable - (Connect linux2 to public network)

Category 6 patch cable - (Connect openfiler1 to public network)

Category 6 patch cable - (Connect linux1 to interconnect Ethernet switch)

Category 6 patch cable - (Connect linux2 to interconnect Ethernet switch)

Category 6 patch cable - (Connect openfiler1 to interconnect Ethernet switch)

US$10

US$10

US$10

US$10

US$10

US$10

Optional Components

KVM Switch

This article requires access to the console of all nodes (servers) in order to install the operating system and perform several of the configuration tasks. When managing a very small number of

servers, it might make sense to connect each server with its own monitor, keyboard, and mouse

in order to access its console. However, as the number of servers to manage increases, this

solution becomes unfeasible. A more practical solution would be to configure a dedicated

computer which would include a single monitor, keyboard, and mouse that would have direct

access to the console of each server. This solution is made possible using a Keyboard, Video,

Mouse Switch —better known as a KVM Switch. A KVM switch is a hardware device that

allows a user to control multiple computers from a single keyboard, video monitor and mouse. US$340

Avocent provides a high quality and economical 4-port switch which includes four 6' cables:

SwitchView® 1000 - (4SV1000BND1-001)

For a detailed explanation and guide on the use and KVM switches, please see the article

"KVM Switches For the Home and the Enterprise".

Total US$2,555

We are about to start the installation process. Now that we have talked about the hardware that will be used in

this example, let's take a conceptual look at what the environment would look like after connecting all of the

hardware components (click on the graphic below to view larger image):

Figure 1: Architecture

As we start to go into the details of the installation, it should be noted that most of the tasks within this document

will need to be performed on both Oracle RAC nodes (linux1 and linux2). I will indicate at the beginning of each

section whether or not the task(s) should be performed on both Oracle RAC nodes or on the network storage

server (openfiler1).

Install the Linux Operating System

Perform the following installation on both Oracle RAC nodes in the cluster!

This section provides a summary of the screens used to install the Linux operating system. This guide is

designed to work with CentOS Release 5 Update 3 for x86 and follows Oracle's suggestion of performing a

"default RPMs" installation type to ensure all expected Linux O/S packages are present for a successful Oracle

RDBMS installation.

Downloading CentOS

CentOS.org

Download and burn the following ISO images to CD/DVD for CentOS Release 5 Update 3 for either x86 or

x86_64 depending on your hardware architecture.

32-bit (x86) Installations

CentOS-5.3-i386-bin-1of6.iso (624 MB)

CentOS-5.3-i386-bin-2of6.iso (636 MB)

CentOS-5.3-i386-bin-3of6.iso (634 MB)

CentOS-5.3-i386-bin-4of6.iso (632 MB)

CentOS-5.3-i386-bin-5of6.iso (638 MB)

CentOS-5.3-i386-bin-6of6.iso (625 MB)

If the Linux RAC nodes have a DVD installed, you may find it more convenient to make use of the single DVD

image:

CentOS-5.3-i386-bin-DVD.iso (3.7 GB)

64-bit (x86_64) Installations

CentOS-5.3-x86_64-bin-1of7.iso (619 MB)

CentOS-5.3-x86_64-bin-2of7.iso (554 MB)

CentOS-5.3-x86_64-bin-3of7.iso (637 MB)

CentOS-5.3-x86_64-bin-4of7.iso (629 MB)

CentOS-5.3-x86_64-bin-5of7.iso (635 MB)

CentOS-5.3-x86_64-bin-6of7.iso (637 MB)

CentOS-5.3-x86_64-bin-7of7.iso (638 MB)

If the Linux RAC nodes have a DVD installed, you may find it more convenient to make use of the single DVD

image:

CentOS-5.3-x86_64-bin-DVD.iso (4.2 GB)

If you are downloading the above ISO files to a MS Windows machine, there are many

options for burning these images (ISO files) to a CD. You may already be familiar with

and have the proper software to burn images to CD. If you are not familiar with this

process and do not have the required software to burn images to CD, here are just two

(of many) software packages that can be used:

UltraISO

Magic ISO Maker

Installing CentOS

This section provides a summary of the screens used to install CentOS. For more detailed installation

instructions, it is possible to use the manuals from Red Hat Linux http://www.redhat.com/docs/manuals/. I would

suggest, however, that the instructions I have provided below be used for this Oracle RAC 10g configuration.

Before installing the Linux operating system on both nodes, you should have the two

NIC interfaces (cards) installed.

After downloading and burning the CentOS images (ISO files) to CD/DVD, insert CentOS Disk #1 into the first

server (linux1 in this example), power it on, and answer the installation screen prompts as noted below. After

completing the Linux installation on the first node, perform the same Linux installation on the second node while

substituting the node name linux1 for linux2 and the different IP addresses were appropriate.

Boot Screen

The first screen is the CentOS boot screen. At the boot: prompt, hit [Enter] to start the installation process.

Media Test

When asked to test the CD media, tab over to [Skip] and hit [Enter]. If there were any errors, the media burning

software would have warned us. After several seconds, the installer should then detect the video card, monitor,

and mouse. The installer then goes into GUI mode.

Welcome to CentOS

At the welcome screen, click [Next] to continue.

Language / Keyboard Selection

The next two screens prompt you for the Language and Keyboard settings. In almost all cases, you can accept

the defaults. Make the appropriate selection for your configuration and click [Next] to continue.

Disk Partitioning Setup

Select [Remove all partitions on selected drives and create default layout] and check the option to [Review and

modify partitioning layout]. Click [Next] to continue.

You will then be prompted with a dialog window asking if you really want to remove all Linux partitions. Click

[Yes] to acknowledge this warning.

Partitioning

The installer will then allow you to view (and modify if needed) the disk partitions it automatically selected. For

most automatic layouts, the installer will choose 100MB for /boot, double the amount of RAM (systems with <=

2,048MB RAM) or an amount equal to RAM (systems with > 2,048MB RAM) for swap, and the rest going to the

root (/) partition. Starting with RHEL 4, the installer will create the same disk configuration as just noted but

will create them using the Logical Volume Manager (LVM). For example, it will partition the first hard drive

(/dev/hda for my configuration) into two partitions — one for the /boot partition (/dev/hda1) and the

remainder of the disk dedicate to a LVM named VolGroup00 (/dev/hda2). The LVM Volume Group

(VolGroup00) is then partitioned into two LVM partitions - one for the root filesystem (/) and another for swap.

The main concern during the partitioning phase is to ensure enough swap space is allocated as required by

Oracle (which is a multiple of the available RAM). The following is Oracle's requirement for swap space:

Available RAM Swap Space Required

Between 1,024MB and 2,048MB 1.5 times the size of RAM

Between 2,049MB and 8,192MB Equal to the size of RAM

More than 8,192MB .75 times the size of RAM

For the purpose of this install, I will accept all automatically preferred sizes. (Including 4,096MB for swap since

I have 2,048MB of RAM installed.

If for any reason, the automatic layout does not configure an adequate amount of swap space, you can easily

change that from this screen. To increase the size of the swap partition, [Edit] the volume group VolGroup00.

This will bring up the "Edit LVM Volume Group: VolGroup00" dialog. First, [Edit] and decrease the size of the

root file system (/) by the amount you want to add to the swap partition. For example, to add another 512MB to

swap, you would decrease the size of the root file system by 512MB (i.e. 36,032MB - 512MB = 35,520MB). Now

add the space you decreased from the root file system (512MB) to the swap partition. When completed, click

[OK] on the "Edit LVM Volume Group: VolGroup00" dialog.

Once you are satisfied with the disk layout, click [Next] to continue.

Boot Loader Configuration

The installer will use the GRUB boot loader by default. To use the GRUB boot loader, accept all default values

and click [Next] to continue.

Network Configuration

I made sure to install both NIC interfaces (cards) in each of the Linux machines before starting the operating

system installation. This screen should have successfully detected each of the network devices. Since we will be

using this machine to host an Oracle instance, there will be several changes that need to be made to the network

configuration. The settings you make here will, of course, depend on your network configuration. The key point

to make is that the machine should never be configured with DHCP since it will be used to host an Oracle

instance. You will need to configure the machine with static IP addresses. You will also need to configure the

server with a real host name.

First, make sure that each of the network devices are checked to [Active on boot]. The installer may choose to

not activate eth1 by default.

Second, [Edit] both eth0 and eth1 as follows. Verify that the option "Enable IPv4 support" is selected. Click off

the option to use "Dynamic IP configuration (DHCP)" by selecting the "Manual configuration" radio button and

configure a static IP address and Netmask for your environment. Click off the option to "Enable IPv6 support".

You may choose to use different IP addresses for both eth0 and eth1 that I have documented in this guide and

that is OK. Put eth1 (the interconnect) on a different subnet than eth0 (the public network):

eth0: - Check ON the option to [Enable IPv4 support]

- Check OFF the option to use [Dynamic IP configuration (DHCP)] - (select Manual configuration)

IPv4 Address: 192.168.1.100

Prefix (Netmask): 255.255.255.0

- Check OFF the option to [Enable IPv6 support]

eth1:

- Check ON the option to [Enable IPv4 support]

- Check OFF the option to use [Dynamic IP configuration (DHCP)] - (select Manual configuration)

IPv4 Address: 192.168.2.100

Prefix (Netmask): 255.255.255.0

- Check OFF the option to [Enable IPv6 support]

Continue by manually setting your hostname. I used "linux1" for the first node and "linux2" for the second.

Finish this dialog off by supplying your gateway and DNS servers.

Time Zone Selection

Select the appropriate time zone for your environment and click [Next] to continue.

Set Root Password

Select a root password and click [Next] to continue.

Package Installation Defaults

By default, CentOS Linux installs most of the software required for a typical server. There are several other

packages (RPMs), however, that are required to successfully install the Oracle software. The installer includes a

"Customize software" selection that allows the addition of RPM groupings such as "Development Libraries" or

"Legacy Library Support". The addition of such RPM groupings is not an issue. De-selecting any "default RPM"

groupings or individual RPMs, however, can result in failed Oracle Clusterware and Oracle Database

installation attempts.

For the purpose of this article, select the radio button [Customize now] and click [Next] to continue.

This is where you pick the packages to install. Most of the packages required for the Oracle software are

grouped into "Package Groups" (i.e. Application -> Editors). Since these nodes will be hosting the Oracle

Clusterware and Oracle RAC software, verify that at least the following package groups are selected for install.

For many of the Linux package groups, not all of the packages associated with that group get selected for

installation. (Note the "Optional packages" button after selecting a package group.) So although the package

group gets selected for install, some of the packages required by Oracle do not get installed. In fact, there are

some packages that are required by Oracle that do not belong to any of the available package groups (i.e.

libaio-devel). Not to worry. A complete list of required packages for Oracle Clusterware 10g and Oracle RAC

10g for CentOS 5 will be provided at the end of this section. These packages will need to be manually installed

from the CentOS CDs after the operating system install. For now, install the following package groups:

Desktop Environments

o GNOME Desktop Environment

Applications

o Editors

o Graphical Internet

o Text-based Internet

Development

o Development Libraries

o Development Tools

o Legacy Software Development

Servers

o Server Configuration Tools

Base System

o Administration Tools

o Base

o Java

o Legacy Software Support

o System Tools

o X Window System

In addition to the above packages, select any additional packages you wish to install for this node keeping in

mind to NOT de-select any of the "default" RPM packages. After selecting the packages to install click [Next] to

continue.

About to Install

This screen is basically a confirmation screen. Click [Next] to start the installation. If you are installing CentOS

using CDs, you will be asked to switch CDs during the installation process depending on which packages you

selected.

Congratulations

And that's it. You have successfully installed CentOS on the first node (linux1). The installer will eject the

CD/DVD from the CD-ROM drive. Take out the CD/DVD and click [Reboot] to reboot the system.

Post Installation Wizard Welcome Screen

When the system boots into CentOS Linux for the first time, it will prompt you with another Welcome screen for

the "Post Installation Wizard". The post installation wizard allows you to make final O/S configuration settings.

On the "Welcome" screen, click [Forward] to continue.

Firewall

On this screen, make sure to select the [Disabled] option and click [Forward] to continue.

You will be prompted with a warning dialog about not setting the firewall. When this occurs, click [Yes] to

continue.

SELinux

On the SELinux screen, choose the [Disabled] option and click [Forward] to continue.

You will be prompted with a warning dialog warning that changing the SELinux setting will require rebooting

the system so the entire file system can be relabeled. When this occurs, click [Yes] to acknowledge a reboot of

the system will occur after firstboot (Post Installation Wizard) is completed.

Kdump

Accept the default setting on the Kdump screen (disabled) and click [Forward] to continue.

Date and Time Settings

Adjust the date and time settings if necessary and click [Forward] to continue.

Create User

Create any additional (non-oracle) operating system user accounts if desired and click [Forward] to continue.

For the purpose of this article, I will not be creating any additional operating system accounts. I will be creating

the "oracle" user account during the Oracle database installation later in this guide.

If you chose not to define any additional operating system user accounts, click [Continue] to acknowledge the

warning dialog.

Sound Card

This screen will only appear if the wizard detects a sound card. On the sound card screen click [Forward] to

continue.

Additional CDs

On the "Additional CDs" screen click [Finish] to continue.

Reboot System

Given we changed the SELinux option (to disabled), we are prompted to reboot the system. Click [OK] to reboot

the system for normal use.

Login Screen

After rebooting the machine, you are presented with the login screen. Log in using the "root" user account and

the password you provided during the installation.

Perform the same installation on the second node

After completing the Linux installation on the first node, repeat the above steps for the second node (linux2).

When configuring the machine name and networking, ensure to configure the proper values. For my installation,

this is what I configured for linux2:

First, make sure that each of the network devices are checked to [Active on boot]. The installer may choose to

not activate eth1.

Second, [Edit] both eth0 and eth1 as follows. Verify that the option "Enable IPv4 support" is selected. Click off

the option for "Use dynamic IP configuration (DHCP)" and configure a static IP address and Netmask for your

environment. Click off the option to "Enable IPv6 support". You may choose to use different IP addresses for

both eth0 and eth1 that I have documented in this guide and that is OK. Put eth1 (the interconnect) on a different

subnet than eth0 (the public network):

eth0: - Check ON the option to [Enable IPv4 support]

- Check OFF the option to [Use dynamic IP configuration (DHCP)] - (select Manual configuration)

IPv4 Address: 192.168.1.101

Prefix (Netmask): 255.255.255.0

- Check OFF the option to [Enable IPv6 support]

eth1:

- Check ON the option to [Enable IPv4 support]

- Check OFF the option to [Use dynamic IP configuration (DHCP)] - (select Manual configuration)

IPv4 Address: 192.168.2.101

Prefix (Netmask): 255.255.255.0

- Check OFF the option to [Enable IPv6 support]

Continue by setting your hostname manually. I used "linux2" for the second node. Finish this dialog off by

supplying your gateway and DNS servers.

Install Required Linux Packages for Oracle RAC

Install the following required Linux packages on both Oracle RAC nodes in

the cluster!

After installing CentOS Linux, the next step is to verify and install all packages (RPMs) required by both Oracle

Clusterware and Oracle RAC. The Oracle Universal Installer (OUI) performs checks on your machine during

installation to verify that it meets the appropriate operating system package requirements. To ensure that these

checks complete successfully, verify the software requirements documented in this section before starting the

Oracle installs.

Although many of the required packages for Oracle were installed in the section Install the Linux Operating

System, several will be missing either because they were considered optional within the package group or simply

didn't exist in any package group!

The packages listed in this section (or later versions) are required for Oracle Clusterware 10g Release 2 and

Oracle RAC 10g Release 2 running on the CentOS 5 or Red Hat Enterprise Linux 5 platform.

32-bit (x86) Installations

binutils-2.17.50.0.6-2.el5

compat-libstdc++-296-2.96-138

compat-libstdc++-33-3.2.3-61

elfutils-libelf-0.125-3.el5

elfutils-libelf-devel-0.125

gcc-4.1.1-52

gcc-c++-4.1.1-52

glibc-2.5-12

glibc-common-2.5-12

glibc-devel-2.5-12

glibc-headers-2.5-12

libaio-0.3.106

libaio-devel-0.3.106

libgcc-4.1.1-52

libstdc++-4.1.1

libstdc++-devel-4.1.1-52.e15

libXp-1.0.0-8

make-3.81-1.1

openmotif-2.2.3

sysstat-7.0.0

unixODBC-2.2.11

unixODBC-devel-2.2.11

Note that the openmotif RPM packages are required to install Oracle demos. This article does not cover the

installation of Oracle demos.

Each of the packages listed above can be found on CD #1, CD #2, and CD #3 on the CentOS 5 - (x86) CDs.

While it is possible to query each individual package to determine which ones are missing and need to be

installed, an easier method is to run the rpm -Uvh PackageName command from the five CDs as follows. For

packages that already exist and are up to date, the RPM command will simply ignore the install and print a

warning message to the console that the package is already installed.

# From CentOS 5.3 (x86)- [CD #1]

mkdir -p /media/cdrom

mount -r /dev/cdrom /media/cdrom

cd /media/cdrom/CentOS

rpm -Uvh binutils-2.*

rpm -Uvh elfutils-libelf-0.*

rpm -Uvh glibc-2.*

rpm -Uvh glibc-common-2.*

rpm -Uvh libaio-0.*

rpm -Uvh libgcc-4.*

rpm -Uvh libstdc++-4.*

rpm -Uvh make-3.*

cd /

eject

# From CentOS 5.3 (x86) - [CD #2]

mount -r /dev/cdrom /media/cdrom

cd /media/cdrom/CentOS

rpm -Uvh elfutils-libelf-devel-0.*

rpm -Uvh glibc-devel-2.*

rpm -Uvh glibc-headers-2.*

rpm -Uvh gcc-4.*

rpm -Uvh gcc-c++-4.*

rpm -Uvh libstdc++-devel-4.*

rpm -Uvh unixODBC-2.*

cd /

eject

# From CentOS 5.3 (x86) - [CD #3]

mount -r /dev/cdrom /media/cdrom

cd /media/cdrom/CentOS

rpm -Uvh compat-libstdc++-296*

rpm -Uvh compat-libstdc++-33*

rpm -Uvh libaio-devel-0.*

rpm -Uvh libXp-1.*

rpm -Uvh openmotif-2.*

rpm -Uvh unixODBC-devel-2.*

cd /

eject

# From CentOS 5.3 (x86) - [CD #4]

mount -r /dev/cdrom /media/cdrom

cd /media/cdrom/CentOS

rpm -Uvh sysstat-7.*

cd /

eject

64-bit (x86_64) Installations

binutils-2.17.50.0.6-2.el5

compat-libstdc++-296-2.96-138

compat-libstdc++-33-3.2.3-61

compat-libstdc++-33-3.2.3-61 (32 bit)

elfutils-libelf-0.125-3.el5

elfutils-libelf-0.125-3.el5 (32 bit)

elfutils-libelf-devel-0.125

gcc-4.1.1

gcc-c++-4.1.1

glibc-2.5-12

glibc-2.5-12 (32 bit)

glibc-common-2.5

glibc-devel-2.5-12

glibc-devel-2.5-12 (32 bit)

glibc-headers-2.5-12

libaio-0.3.106

libaio-0.3.106 (32 bit)

libaio-devel-0.3.106

libgcc-4.1.1

libgcc-4.1.1 (32 bit)

libstdc++-4.1.1

libstdc++-4.1.1 (32 bit)

libstdc++-devel-4.1.1-52.e15

libXp-1.0.0-8

libXp-1.0.0-8 (32 bit)

make-3.81-1.1

openmotif-2.2.3

sysstat-7.0.0

unixODBC-2.2.11

unixODBC-2.2.11 (32 bit)

unixODBC-devel-2.2.11

Note that the openmotif RPM packages are required to install Oracle demos. This article does not cover the

installation of Oracle demos.

Each of the packages listed above can be found on CD #1, CD #2, CD #3, and CD #4 on the CentOS 5 -

(x86_64) CDs. While it is possible to query each individual package to determine which ones are missing and

need to be installed, an easier method is to run the rpm -Uvh PackageName command from the five CDs as

follows. For packages that already exist and are up to date, the RPM command will simply ignore the install and

print a warning message to the console that the package is already installed.

# From CentOS 5.3 (x86_64)- [CD #1]

mkdir -p /media/cdrom

mount -r /dev/cdrom /media/cdrom

cd /media/cdrom/CentOS

rpm -Uvh binutils-2.*

rpm -Uvh elfutils-libelf-0.*

rpm -Uvh glibc-2.*

rpm -Uvh glibc-common-2.*

rpm -Uvh libaio-0.*

rpm -Uvh libgcc-4.*

rpm -Uvh libstdc++-4.*

rpm -Uvh make-3.*

cd /

eject

# From CentOS 5.3 (x86_64) - [CD #2]

mount -r /dev/cdrom /media/cdrom

cd /media/cdrom/CentOS

rpm -Uvh elfutils-libelf-devel-0.*

rpm -Uvh glibc-devel-2.*

rpm -Uvh glibc-headers-2.*

rpm -Uvh gcc-4.*

rpm -Uvh gcc-c++-4.*

rpm -Uvh libstdc++-devel-4.*

rpm -Uvh unixODBC-2.*

cd /

eject

# From CentOS 5.3 (x86_64) - [CD #3]

mount -r /dev/cdrom /media/cdrom

cd /media/cdrom/CentOS

rpm -Uvh compat-libstdc++-296*

rpm -Uvh compat-libstdc++-33*

rpm -Uvh libaio-devel-0.*

rpm -Uvh libXp-1.*

rpm -Uvh openmotif-2.*

rpm -Uvh unixODBC-devel-2.*

cd /

eject

# From CentOS 5.3 (x86_64) - [CD #4]

mount -r /dev/cdrom /media/cdrom

cd /media/cdrom/CentOS

rpm -Uvh sysstat-7.*

cd /

eject

Network Configuration

Perform the following network configuration on both Oracle RAC nodes in

the cluster!

Although we configured several of the network settings during the installation of CentOS, it is important to not

skip this section as it contains critical steps that are required for a successful RAC environment.

Introduction to Network Settings

During the Linux O/S install we already configured the IP address and host name for both of the Oracle RAC

nodes. We now need to configure the /etc/hosts file as well as adjusting several of the network settings for the

interconnect.

Both of the Oracle RAC nodes should have one static IP address for the public network and one static IP address

for the private cluster interconnect. Do not use DHCP naming for the public IP address or the interconnects; you

need static IP addresses! The private interconnect should only be used by Oracle to transfer Cluster Manager and

Cache Fusion related data along with data for the network storage server (Openfiler). Note that Oracle does not

support using the public network interface for the interconnect. You must have one network interface for the

public network and another network interface for the private interconnect. For a production RAC

implementation, the interconnect should be at least gigabit (or more) and only be used by Oracle as well as

having the network storage server (Openfiler) on a separate gigabit network.

Configuring Public and Private Network

In our two node example, we need to configure the network on both Oracle RAC nodes for access to the public

network as well as their private interconnect.

The easiest way to configure network settings in CentOS is with the program Network Configuration. This

application can be started from the command-line as the "root" user account as follows:

# /usr/bin/system-config-network &

Do not use DHCP naming for the public IP address or the interconnects - we need static

IP addresses!

Using the Network Configuration application, you need to configure both NIC devices as well as the

/etc/hosts file. Both of these tasks can be completed using the Network Configuration GUI. Notice that the

/etc/hosts settings are the same for both nodes and that I removed any entry that has to do with IPv6 (for

example, ::1 localhost6.localdomain6 localhost6).

Our example configuration will use the following settings:

Oracle RAC Node 1 - (linux1)

Device IP Address Subnet Gateway Purpose

eth0 192.168.1.100 255.255.255.0 192.168.1.1 Connects linux1 to the public network

eth1 192.168.2.100 255.255.255.0 Connects linux1 (interconnect) to linux2 (linux2-

priv)

/etc/hosts

127.0.0.1 localhost.localdomain localhost

# Public Network - (eth0)

192.168.1.100 linux1

192.168.1.101 linux2

# Private Interconnect - (eth1)

192.168.2.100 linux1-priv

192.168.2.101 linux2-priv

# Public Virtual IP (VIP) addresses - (eth0:1)

192.168.1.200 linux1-vip

192.168.1.201 linux2-vip

# Private Storage Network for Openfiler - (eth1)

192.168.1.195 openfiler1

192.168.2.195 openfiler1-priv

Oracle RAC Node 2 - (linux2)

Device IP Address Subnet Gateway Purpose

eth0 192.168.1.101 255.255.255.0 192.168.1.1 Connects linux2 to the public network

eth1 192.168.2.101 255.255.255.0 Connects linux2 (interconnect) to linux1 (linux1-

priv)

/etc/hosts

127.0.0.1 localhost.localdomain localhost

# Public Network - (eth0)

192.168.1.100 linux1

192.168.1.101 linux2

# Private Interconnect - (eth1)

192.168.2.100 linux1-priv

192.168.2.101 linux2-priv

# Public Virtual IP (VIP) addresses - (eth0:1)

192.168.1.200 linux1-vip

192.168.1.201 linux2-vip

# Private Storage Network for Openfiler - (eth1)

192.168.1.195 openfiler1

192.168.2.195 openfiler1-priv

Note that the virtual IP addresses only need to be defined in the /etc/hosts file (or

your DNS) for both Oracle RAC nodes. The public virtual IP addresses will be

configured automatically by Oracle Clusterware when you run the Oracle Universal

Installer, which starts Oracle's Virtual Internet Protocol Configuration Assistant

(VIPCA). All virtual IP addresses will be activated when the srvctl start nodeapps

-n <node_name> command is run. This is the Host Name/IP Address that will be

configured in the client(s) tnsnames.ora file (more details later).

Although I am getting ahead of myself, this is the Host Name/IP Address that will be

configured in the client(s) tnsnames.ora file for each Oracle Net Service Name. All of

this will be explained much later in this article!

In the screen shots below, only Oracle RAC Node 1 (linux1) is shown. Be sure to make all the proper network

settings to both Oracle RAC nodes.

Figure 2: Network Configuration Screen - Node 1 (linux1)

Figure 3: Ethernet Device Screen - eth0 (linux1)

Figure 4: Ethernet Device Screen - eth1 (linux1)

Figure 5: Network Configuration Screen - /etc/hosts (linux1)

Once the network is configured, you can use the ifconfig command to verify everything is working. The

following example is from linux1:

# /sbin/ifconfig -a

eth0 Link encap:Ethernet HWaddr 00:14:6C:76:5C:71

inet addr:192.168.1.100 Bcast:192.168.1.255 Mask:255.255.255.0

inet6 addr: fe80::214:6cff:fe76:5c71/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:3059 errors:0 dropped:0 overruns:0 frame:0

TX packets:1539 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:3460697 (3.3 MiB) TX bytes:145612 (142.1 KiB)

Interrupt:169 Base address:0xef00

eth1 Link encap:Ethernet HWaddr 00:0E:0C:64:D1:E5

inet addr:192.168.2.100 Bcast:192.168.2.255 Mask:255.255.255.0

inet6 addr: fe80::20e:cff:fe64:d1e5/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:11 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 b) TX bytes:782 (782.0 b)

Base address:0xddc0 Memory:fe9c0000-fe9e0000

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:16436 Metric:1

RX packets:1764 errors:0 dropped:0 overruns:0 frame:0

TX packets:1764 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:1991946 (1.8 MiB) TX bytes:1991946 (1.8 MiB)

sit0 Link encap:IPv6-in-IPv4

NOARP MTU:1480 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

About Virtual IP

Why do we have a Virtual IP (VIP) in 10g? Why does it just return a dead connection when its primary node

fails?

It's all about availability of the application. When a node fails, the VIP associated with it is supposed to be

automatically failed over to some other node. When this occurs, two things happen.

1. The new node re-arps the world indicating a new MAC address for the address. For directly

connected clients, this usually causes them to see errors on their connections to the old address.

2. Subsequent packets sent to the VIP go to the new node, which will send error RST packets back

to the clients. This results in the clients getting errors immediately.

This means that when the client issues SQL to the node that is now down, or traverses the address list while

connecting, rather than waiting on a very long TCP/IP time-out (~10 minutes), the client receives a TCP reset. In

the case of SQL, this is ORA-3113. In the case of connect, the next address in tnsnames is used.

Going one step further is making use of Transparent Application Failover (TAF). With TAF successfully

configured, it is possible to completely avoid ORA-3113 errors all together! TAF will be discussed in more

detail in the section "Transparent Application Failover - (TAF)".

Without using VIPs, clients connected to a node that died will often wait a 10 minute TCP timeout period before

getting an error. As a result, you don't really have a good HA solution without using VIPs.

Source - Metalink: "RAC Frequently Asked Questions" (Note:220970.1)

Confirm the RAC Node Name is Not Listed in Loopback Address

Ensure that the node names (linux1 or linux2) are not included for the loopback address in the /etc/hosts

file. If the machine name is listed in the in the loopback address entry as below:

127.0.0.1 linux1 localhost.localdomain localhost

it will need to be removed as shown below: 127.0.0.1 localhost.localdomain localhost

If the RAC node name is listed for the loopback address, you will receive the following

error during the RAC installation:

ORA-00603: ORACLE server session terminated by fatal error

or ORA-29702: error occurred in Cluster Group Service operation

Confirm localhost is defined in the /etc/hosts file for the loopback address

Ensure that the entry for localhost.localdomain and localhost are included for the loopback address in the

/etc/hosts file for each of the Oracle RAC nodes:

127.0.0.1 localhost.localdomain localhost

If an entry does not exist for localhost in the /etc/hosts file, Oracle Clusterware

will be unable to start the application resources — notably the ONS process. The error

would indicate "Failed to get IP for localhost" and will be written to the log file for

ONS. For example:

CRS-0215 could not start resource 'ora.linux1.ons'. Check log file

"/u01/app/crs/log/linux1/racg/ora.linux1.ons.log"

for more details.

The ONS log file will contain lines similar to the following:

Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996,

2005 Oracle. All rights reserved.

2007-04-14 13:10:02.729: [

RACG][3086871296][13316][3086871296][ora.linux1.ons]: Failed to get IP

for localhost (1)

Failed to get IP for localhost (1)

Failed to get IP for localhost (1)

onsctl: ons failed to start

...

Adjusting Network Settings

With Oracle 9.2.0.1 and later, Oracle makes use of UDP as the default protocol on Linux for inter-process

communication (IPC), such as Cache Fusion and Cluster Manager buffer transfers between instances within the

RAC cluster.

Oracle strongly suggests to adjust the default and maximum receive buffer size (SO_RCVBUF socket option) to

1024KB and the default and maximum send buffer size (SO_SNDBUF socket option) to 256KB.

The receive buffers are used by TCP and UDP to hold received data until it is read by the application. The

receive buffer cannot overflow because the peer is not allowed to send data beyond the buffer size window. This

means that datagrams will be discarded if they don't fit in the socket receive buffer, potentially causing the

sender to overwhelm the receiver.

The default and maximum window size can be changed in the /proc file system

without reboot:

# sysctl -w net.core.rmem_default=1048576

net.core.rmem_default = 1048576

# sysctl -w net.core.rmem_max=1048576

net.core.rmem_max = 1048576

# sysctl -w net.core.wmem_default=262144

net.core.wmem_default = 262144

# sysctl -w net.core.wmem_max=262144

net.core.wmem_max = 262144

The above commands made the changes to the already running OS. You should now make the above changes

permanent (for each reboot) by adding the following lines to the /etc/sysctl.conf file for both nodes in your

RAC cluster:

# +---------------------------------------------------------+

# | ADJUSTING NETWORK SETTINGS |

# +---------------------------------------------------------+

# | With Oracle 9.2.0.1 and onwards, Oracle now makes use |

# | of UDP as the default protocol on Linux for |

# | inter-process communication (IPC), such as Cache Fusion |

# | and Cluster Manager buffer transfers between instances |

# | within the RAC cluster. Oracle strongly suggests to |

# | adjust the default and maximum receive buffer size |

# | (SO_RCVBUF socket option) to 1024KB, and the default |

# | and maximum send buffer size (SO_SNDBUF socket option) |

# | to 256KB. The receive buffers are used by TCP and UDP |

# | to hold received data until it is read by the |

# | application. The receive buffer cannot overflow because |

# | the peer is not allowed to send data beyond the buffer |

# | size window. This means that datagrams will be |

# | discarded if they don't fit in the socket receive |

# | buffer. This could cause the sender to overwhelm the |

# | receiver. |

# +---------------------------------------------------------+

# +---------------------------------------------------------+

# | Default setting in bytes of the socket "receive" buffer |

# | which may be set by using the SO_RCVBUF socket option. |

# +---------------------------------------------------------+

net.core.rmem_default=1048576

# +---------------------------------------------------------+

# | Maximum setting in bytes of the socket "receive" buffer |

# | which may be set by using the SO_RCVBUF socket option. |

# +---------------------------------------------------------+

net.core.rmem_max=1048576

# +---------------------------------------------------------+

# | Default setting in bytes of the socket "send" buffer |

# | which may be set by using the SO_SNDBUF socket option. |

# +---------------------------------------------------------+

net.core.wmem_default=262144

# +---------------------------------------------------------+

# | Maximum setting in bytes of the socket "send" buffer |

# | which may be set by using the SO_SNDBUF socket option. |

# +---------------------------------------------------------+

net.core.wmem_max=262144

Check and turn off UDP ICMP rejections:

During the Linux installation process, I indicated to not configure the firewall option. By default the option to

configure a firewall is selected by the installer. This has burned me several times so I like to do a double-check

that the firewall option is not configured and to ensure udp ICMP filtering is turned off.

If UDP ICMP is blocked or rejected by the firewall, the Oracle Clusterware software will crash after several

minutes of running. When the Oracle Clusterware process fails, you will have something similar to the following

in the <machine_name>_evmocr.log file:

08/29/2005 22:17:19

oac_init:2: Could not connect to server, clsc retcode = 9

08/29/2005 22:17:19

a_init:12!: Client init unsuccessful : [32]

ibctx:1:ERROR: INVALID FORMAT

proprinit:problem reading the bootblock or superbloc 22

When experiencing this type of error, the solution is to remove the udp ICMP (iptables) rejection rule - or to

simply have the firewall option turned off. The Oracle Clusterware software will then start to operate normally

and not crash. The following commands should be executed as the root user account:

1. Check to ensure that the firewall option is turned off. If the firewall option is stopped (like it is in

my example below) you do not have to proceed with the following steps. 2. # /etc/rc.d/init.d/iptables status

Firewall is stopped.

3. If the firewall option is operating you will need to first manually disable UDP ICMP rejections: 4. # /etc/rc.d/init.d/iptables stop 5. 6. Flushing firewall rules: [ OK ] 7. Setting chains to policy ACCEPT: filter [ OK ]

Unloading iptables modules: [ OK ]

8. Then, to turn UDP ICMP rejections off for next server reboot (which should always be turned

off):

# chkconfig iptables off

Install Openfiler

Perform the following installation on the network storage server (openfiler1)!

With the network configured on both Oracle RAC nodes, the next step is to install the Openfiler software to the

network storage server (openfiler1). Later in this article, the network storage server will be configured as an

iSCSI storage device for all Oracle RAC 10g shared storage requirements.

Powered by rPath Linux, Openfiler is a free browser-based network storage management utility that delivers file-

based Network Attached Storage (NAS) and block-based Storage Area Networking (SAN) in a single

framework. The entire software stack interfaces with open source applications such as Apache, Samba, LVM2,

ext3, Linux NFS and iSCSI Enterprise Target. Openfiler combines these ubiquitous technologies into a small,

easy to manage solution fronted by a powerful web-based management interface.

Openfiler supports CIFS, NFS, HTTP/DAV, FTP, however, we will only be making use of its iSCSI capabilities

to implement an inexpensive SAN for the shared storage components required by Oracle RAC 10g. The

operating system and Openfiler application will be installed on one internal SATA disk. A second internal 73GB

15K SCSI hard disk will be configured as a single "Volume Group" that will be used for all shared disk storage

requirements. The Openfiler server will be configured to use this volume group for iSCSI based storage and will

be used in our Oracle RAC 10g configuration to store the shared files required by Oracle Clusterware as well as

all Oracle ASM volumes.

Please be aware that any type of hard disk (internal or external) should work for database storage as long as it

can be recognized by the network storage server (Openfiler) and has adequate space. For example, I could have

made an extra partition on the 500GB internal SATA disk for the iSCSI target, but decided to make use of the

faster SCSI disk for this example.

To learn more about Openfiler, please visit their website at http://www.openfiler.com/

Download Openfiler

Use the links below to download Openfiler NAS/SAN Appliance, version 2.3 (Final Release) for either x86 or

x86_64 depending on your hardware architecture. This example uses x86_64. After downloading Openfiler, you

will then need to burn the ISO image to CD.

openfiler-2.3-x86-disc1.iso

Processor Type x86

Size 322 MB

SHA1SUM 8343a9ae07a0590d88484b3dac117c843626bf9c

openfiler-2.3-x86_64-disc1.iso

Processor Type x86_64

Size 336 MB

SHA1SUM dae55d478ade41389cff3718579745b1a95f6d41

If you are downloading the above ISO file to a MS Windows machine, there are many

options for burning the ISO image (ISO file) to a CD. You may already be familiar with

and have the proper software to burn images to CD. If you are not familiar with this

process and do not have the required software to burn images to CD, here are just two

(of many) software packages that can be used:

UltraISO

Magic ISO Maker

Install Openfiler

This section provides a summary of the screens used to install the Openfiler software. For the purpose of this

article, I opted to install Openfiler with all default options. The only manual change required was for configuring

the local network settings.

Once the install has completed, the server will reboot to make sure all required components, services and drivers

are started and recognized. After the reboot, any external hard drives (if connected) will be discovered by the

Openfiler server.

For more detailed installation instructions, please visit http://www.openfiler.com/docs/. I would suggest,

however, that the instructions I have provided below be used for this Oracle RAC 10g configuration.

Before installing the Openfiler software to the network storage server, you should have

both NIC interfaces (cards) installed and any external hard drives connected and turned

on (if you will be using external hard drives).

After downloading and burning the Openfiler ISO image (ISO file) to CD, insert the CD into the network storage

server (openfiler1 in this example), power it on, and answer the installation screen prompts as noted below.

Boot Screen

The first screen is the Openfiler boot screen. At the boot: prompt, hit [Enter] to start the installation process.

Media Test

When asked to test the CD media, tab over to [Skip] and hit [Enter]. If there were any errors, the media burning

software would have warned us. After several seconds, the installer should then detect the video card, monitor,

and mouse. The installer then goes into GUI mode.

Welcome to Openfiler NSA

At the welcome screen, click [Next] to continue.

Keyboard Configuration

The next screen prompts you for the Keyboard settings. Make the appropriate selection for your configuration.

Disk Partitioning Setup

The next screen asks whether to perform disk partitioning using "Automatic Partitioning" or "Manual

Partitioning with Disk Druid". Although the official Openfiler documentation suggests to use Manual

Partitioning, I opted to use "Automatic Partitioning" given the simplicity of my example configuration.

Select [Automatically partition] and click [Next] continue.

Automatic Partitioning

If there were a previous installation of Linux on this machine, the next screen will ask if you want to "remove" or

"keep" old partitions. Select the option to [Remove all partitions on this system]. For my example configuration,

I selected ONLY the 500GB SATA internal hard drive [sda] for the operating system and Openfiler application

installation. I de-selected the 73GB SCSI internal hard drive since this disk will be used exclusively in the next

section to create a single "Volume Group" that will be used for all iSCSI based shared disk storage requirements

for Oracle RAC 10g.

I also keep the checkbox [Review (and modify if needed) the partitions created] selected. Click [Next] to

continue.

You will then be prompted with a dialog window asking if you really want to remove all partitions. Click [Yes] to

acknowledge this warning.

Partitioning

The installer will then allow you to view (and modify if needed) the disk partitions it automatically chose for

hard disks selected in the previous screen. In almost all cases, the installer will choose 100MB for /boot, an

adequate amount of swap, and the rest going to the root (/) partition for that disk (or disks). In this example, I

am satisfied with the installers recommended partitioning for /dev/sda.

The installer will also show any other internal hard disks it discovered. For my example configuration, the

installer found the 73GB SCSI internal hard drive as /dev/sdb. For now, I will "Delete" any and all partitions

on this drive (there was only one, /dev/sdb1). In the next section, I will create the required partition for this

particular hard disk.

Network Configuration

I made sure to install all NIC interfaces (cards) in the network storage server before starting the Openfiler

installation. This screen should have successfully detected each of the network devices.

First, make sure that each of the network devices are checked to [Active on boot]. The installer may choose to

not activate eth1 by default.

Second, [Edit] both eth0 and eth1 as follows. You may choose to use different IP addresses for both eth0 and

eth1 and that is OK. You must, however, configure eth1 (the storage network) to be on the same subnet you

configured for eth1 on linux1 and linux2:

eth0: - Check OFF the option to [Configure using DHCP]

- Leave the [Activate on boot] checked ON

- IP Address: 192.168.1.195

- Netmask: 255.255.255.0

eth1: - Check OFF the option to [Configure using DHCP]

- Leave the [Activate on boot] checked ON

- IP Address: 192.168.2.195

- Netmask: 255.255.255.0

Continue by setting your hostname manually. I used a hostname of "openfiler1". Finish this dialog off by

supplying your gateway and DNS servers.

Time Zone Selection

The next screen allows you to configure your time zone information. Make the appropriate selection for your

location.

Set Root Password

Select a root password and click [Next] to continue.

About to Install

This screen is basically a confirmation screen. Click [Next] to start the installation.

Congratulations

And that's it. You have successfully installed Openfiler on the network storage server. The installer will eject the

CD from the CD-ROM drive. Take out the CD and click [Reboot] to reboot the system.

If everything was successful after the reboot, you should now be presented with a text login screen and the

URL(s) to use for administering the Openfiler server.

Modify /etc/hosts File on Openfiler Server

Although not mandatory, I typically copy the contents of the /etc/hosts file from one of the Oracle RAC nodes to

the new Openfiler server. This allows convenient name resolution when testing the network for the cluster.

Configure iSCSI Volumes using Openfiler

Perform the following configuration tasks on the network storage server

(openfiler1)!

Openfiler administration is performed using the Openfiler Storage Control Center — a browser based tool over

an https connection on port 446. For example:

https://openfiler1.idevelopment.info:446/

From the Openfiler Storage Control Center home page, log in as an administrator. The default administration

login credentials for Openfiler are:

Username: openfiler

Password: password

The first page the administrator sees is the [Status] / [System Information] screen.

To use Openfiler as an iSCSI storage server, we have to perform six major tasks; set up iSCSI services,

configure network access, identify and partition the physical storage, create a new volume group, create all

logical volumes, and finally, create new iSCSI targets for each of the logical volumes.

Services

To control services, we use the Openfiler Storage Control Center and navigate to [Services] / [Manage Services]:

Figure 6: Enable iSCSI Openfiler Service

To enable the iSCSI service, click on the 'Enable' link under the 'iSCSI target server' service name. After that, the

'iSCSI target server' status should change to 'Enabled'.

The ietd program implements the user level part of iSCSI Enterprise Target software for building an iSCSI

storage system on Linux. With the iSCSI target enabled, we should be able to SSH into the Openfiler server and

see the iscsi-target service running:

[root@openfiler1 ~]# service iscsi-target status

ietd (pid 3839) is running...

Network Access Configuration

The next step is to configure network access in Openfiler to identify both Oracle RAC nodes (linux1 and

linux2) that will need to access the iSCSI volumes through the storage (private) network. Note that iSCSI

volumes will be created later on in this section. Also note that this step does not actually grant the appropriate

permissions to the iSCSI volumes required by both Oracle RAC nodes. That will be accomplished later in this

section by updating the ACL for each new logical volume.

As in the previous section, configuring network access is accomplished using the Openfiler Storage Control

Center by navigating to [System] / [Network Setup]. The "Network Access Configuration" section (at the bottom

of the page) allows an administrator to setup networks and/or hosts that will be allowed to access resources

exported by the Openfiler appliance. For the purpose of this article, we will want to add both Oracle RAC nodes

individually rather than allowing the entire 192.168.2.0 network to have access to Openfiler resources.

When entering each of the Oracle RAC nodes, note that the 'Name' field is just a logical name used for reference

only. As a convention when entering nodes, I simply use the node name defined for that IP address. Next, when

entering the actual node in the 'Network/Host' field, always use its IP address even though its host name may

already be defined in your /etc/hosts file or DNS. Lastly, when entering actual hosts in our Class C network,

use a subnet mask of 255.255.255.255.

It is important to remember that you will be entering the IP address of the private network (eth1) for each of the

RAC nodes in the cluster.

The following image shows the results of adding both Oracle RAC nodes:

Figure 7: Configure Openfiler Network Access for Oracle RAC Nodes

Physical Storage

In this section, we will be creating the five iSCSI volumes to be used as shared storage by both of the Oracle

RAC nodes in the cluster. This involves multiple steps that will be performed on the internal 73GB 15K SCSI

hard disk connected to the Openfiler server.

Storage devices like internal IDE/SATA/SCSI/SAS disks, storage arrays, external USB drives, external FireWire

drives, or ANY other storage can be connected to the Openfiler server and served to the clients. Once these

devices are discovered at the OS level, Openfiler Storage Control Center can be used to set up and manage all of

that storage.

In our case, we have a 73GB internal SCSI hard drive for our shared storage needs. On the Openfiler server this

drive is seen as /dev/sdb (MAXTOR ATLAS15K2_73SCA). To see this and to start the process of creating our

iSCSI volumes, navigate to [Volumes] / [Block Devices] from the Openfiler Storage Control Center:

Figure 8: Openfiler Physical Storage - Block Device Management

Partitioning the Physical Disk

The first step we will perform is to create a single primary partition on the /dev/sdb internal hard disk. By

clicking on the /dev/sdb link, we are presented with the options to 'Edit' or 'Create' a partition. Since we will be

creating a single primary partition that spans the entire disk, most of the options can be left to their default

setting where the only modification would be to change the 'Partition Type' from 'Extended partition' to

'Physical volume'. Here are the values I specified to create the primary partition on /dev/sdb:

Mode: Primary

Partition Type: Physical volume

Starting Cylinder: 1

Ending Cylinder: 8924

The size now shows 68.36 GB. To accept that we click on the "Create" button. This results in a new partition

(/dev/sdb1) on our internal hard disk:

Figure 9: Partition the Physical Volume

Volume Group Management

The next step is to create a Volume Group. We will be creating a single volume group named rac1 that contains

the newly created primary partition.

From the Openfiler Storage Control Center, navigate to [Volumes] / [Volume Groups]. There we would see any

existing volume groups, or none as in our case. Using the Volume Group Management screen, enter the name of

the new volume group (rac1), click on the checkbox in front of /dev/sdb1 to select that partition, and finally

click on the 'Add volume group' button.

After that we are presented with the list that now shows our newly created volume group named "rac1":

Figure 10: New Volume Group Created

Logical Volumes

We can now create the five logical volumes in the newly created volume group (rac1).

From the Openfiler Storage Control Center, navigate to [Volumes] / [Add Volume]. There we will see the newly

created volume group (rac1) along with its block storage statistics. Also available at the bottom of this screen is

the option to create a new volume in the selected volume group - (Create a volume in "rac1"). Use this screen to

create the following five logical (iSCSI) volumes. After creating each logical volume, the application will point

you to the "Manage Volumes" screen. You will then need to click back to the "Add Volume" tab to create the

next logical volume until all five iSCSI volumes are created:

iSCSI / Logical Volumes

Volume Name Volume Description Required Space (MB) Filesystem Type

racdb-crs racdb - Oracle Clusterware 2,048 iSCSI

racdb-asm1 racdb - ASM Volume 1 16,984 iSCSI

racdb-asm2 racdb - ASM Volume 2 16,984 iSCSI

racdb-asm3 racdb - ASM Volume 3 16,984 iSCSI

racdb-asm4 racdb - ASM Volume 4 16,984 iSCSI

In effect we have created five iSCSI disks that can now be presented to iSCSI clients (linux1 and linux2) on

the network. The "Manage Volumes" screen should look as follows:

Figure 11: New Logical (iSCSI) Volumes

iSCSI Targets

At this point we have five iSCSI logical volumes. Before an iSCSI client can have access to them, however, an

iSCSI target will need to be created for each of these five volumes. Each iSCSI logical volume will be mapped to

a specific iSCSI target and the appropriate network access permissions to that target will be granted to both

Oracle RAC nodes. For the purpose of this article, there will be a one-to-one mapping between an iSCSI logical

volume and an iSCSI target.

There are three steps involved in creating and configuring an iSCSI target; create a unique Target IQN (basically,

the universal name for the new iSCSI target), map one of the iSCSI logical volumes created in the previous

section to the newly created iSCSI target, and finally, grant both of the Oracle RAC nodes access to the new

iSCSI target. Please note that this process will need to be performed for each of the five iSCSI logical volumes

created in the previous section.

For the purpose of this article, the following table lists the new iSCSI target names (the Target IQN) and which

iSCSI logical volume it will be mapped to:

iSCSI Target / Logical Volume Mappings

Target IQN iSCSI Volume Name Volume Description

iqn.2006-01.com.openfiler:racdb.crs racdb-crs racdb - Oracle Clusterware

iqn.2006-01.com.openfiler:racdb.asm1 racdb-asm1 racdb - ASM Volume 1

iqn.2006-01.com.openfiler:racdb.asm2 racdb-asm2 racdb - ASM Volume 2

iqn.2006-01.com.openfiler:racdb.asm3 racdb-asm3 racdb - ASM Volume 3

iqn.2006-01.com.openfiler:racdb.asm4 racdb-asm4 racdb - ASM Volume 4

We are now ready to create the five new iSCSI targets - one for each of the iSCSI logical volumes. The example

below illustrates the three steps required to create a new iSCSI target by creating the Oracle Clusterware / racdb-

crs target (iqn.2006-01.com.openfiler:racdb.crs). This three step process will need to be repeated for each

of the five new iSCSI targets listed in the table above.

Create New Target IQN

From the Openfiler Storage Control Center, navigate to [Volumes] / [iSCSI Targets]. Verify the grey sub-tab

"Target Configuration" is selected. This page allows you to create a new iSCSI target. A default value is

automatically generated for the name of the new iSCSI target (better known as the "Target IQN"). An example

Target IQN is "iqn.2006-01.com.openfiler:tsn.ae4683b67fd3":

Figure 12: Create New iSCSI Target : Default Target IQN

I prefer to replace the last segment of the default Target IQN with something more meaningful. For the first

iSCSI target (Oracle Clusterware / racdb-crs), I will modify the default Target IQN by replacing the string

"tsn.ae4683b67fd3" with "racdb.crs" as shown in Figure 13 below:

Figure 13: Create New iSCSI Target : Replace Default Target IQN

Once you are satisfied with the new Target IQN, click the "Add" button. This will create a new iSCSI target and

then bring up a page that allows you to modify a number of settings for the new iSCSI target. For the purpose of

this article, none of settings for the new iSCSI target need to be changed.

LUN Mapping

After creating the new iSCSI target, the next step is to map the appropriate iSCSI logical volumes to it. Under

the "Target Configuration" sub-tab, verify the correct iSCSI target is selected in the section "Select iSCSI

Target". If not, use the pull-down menu to select the correct iSCSI target and hit the "Change" button.

Next, click on the grey sub-tab named "LUN Mapping" (next to "Target Configuration" sub-tab). Locate the

appropriate iSCSI logical volume (/dev/rac1/racdb-crs in this case) and click the "Map" button. You do not

need to change any settings on this page. Your screen should look similar to Figure 14 after clicking the "Map"

button for volume /dev/rac1/racdb-crs:

Figure 14: Create New iSCSI Target : Map LUN

Network ACL

Before an iSCSI client can have access to the newly created iSCSI target, it needs to be granted the appropriate

permissions. Awhile back, we configured network access in Openfiler for two hosts (the Oracle RAC nodes).

These are the two nodes that will need to access the new iSCSI targets through the storage (private) network. We

now need to grant both of the Oracle RAC nodes access to the new iSCSI target.

Click on the grey sub-tab named "Network ACL" (next to "LUN Mapping" sub-tab). For the current iSCSI

target, change the "Access" for both hosts from 'Deny' to 'Allow' and click the 'Update' button:

Figure 15: Create New iSCSI Target : Update Network ACL

Go back to the Create New Target IQN section and perform these three tasks for the remaining four iSCSI

logical volumes while substituting the values found in the "iSCSI Target / Logical Volume Mappings" table .

Configure iSCSI Volumes on Oracle RAC Nodes

Configure the iSCSI initiator on both Oracle RAC nodes in the cluster!

Creating partitions, however, should only be executed on one of nodes in the

RAC cluster.

An iSCSI client can be any system (Linux, Unix, MS Windows, Apple Mac, etc.) for which iSCSI support (a

driver) is available. In our case, the clients are two Linux servers, linux1 and linux2, running CentOS 5.3.

In this section we will be configuring the iSCSI software initiator on both of the Oracle RAC nodes. CentOS 5.3

includes the Open-iSCSI iSCSI software initiator which can be found in the iscsi-initiator-utils RPM.

This is a change from previous versions of CentOS (4.x) which included the Linux iscsi-sfnet software driver

developed as part of the Linux-iSCSI Project. All iSCSI management tasks like discovery and logins will use the

command-line interface iscsiadm which is included with Open-iSCSI.

The iSCSI software initiator will be configured to automatically log in to the network storage server

(openfiler1) and discover the iSCSI volumes created in the previous section. We will then go through the steps

of creating persistent local SCSI device names (i.e. /dev/iscsi/asm1) for each of the iSCSI target names

discovered using udev. Having a consistent local SCSI device name and which iSCSI target it maps to is

required in order to know which volume (device) is to be used for OCFS2 and which volumes belong to ASM.

Before we can do any of this, however, we must first install the iSCSI initiator software!

Installing the iSCSI (initiator) service

With CentOS 5.3, the Open-iSCSI iSCSI software initiator does not get installed by default. The software is

included in the iscsi-initiator-utils package which can be found on CD #1. To determine if this package is

installed (which in most cases, it will not be), perform the following on both Oracle RAC nodes:

# rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"| grep iscsi-initiator-

utils

If the iscsi-initiator-utils package is not installed, load CD #1 into each of the Oracle RAC nodes and

perform the following:

# mount -r /dev/cdrom /media/cdrom

# cd /media/cdrom/CentOS

# rpm -Uvh iscsi-initiator-utils-*

# cd /

# eject

Configure the iSCSI (initiator) service

After verifying that the iscsi-initiator-utils package is installed on both Oracle RAC nodes, start the

iscsid service and enable it to automatically start when the system boots. We will also configure the iscsi

service to automatically start which logs into iSCSI targets needed at system startup.

# service iscsid start

Turning off network shutdown. Starting iSCSI daemon: [ OK ]

[ OK ]

# chkconfig iscsid on

# chkconfig iscsi on

Now that the iSCSI service is started, use the iscsiadm command-line interface to discover all available targets

on the network storage server. This should be performed on both Oracle RAC nodes to verify the configuration

is functioning properly:

# iscsiadm -m discovery -t sendtargets -p openfiler1-priv

192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm1

192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm2

192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm3

192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm4

192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.crs

Manually Log In to iSCSI Targets

At this point the iSCSI initiator service has been started and each of the Oracle RAC nodes were able to discover

the available targets from the network storage server. The next step is to manually log in to each of the available

targets which can be done using the iscsiadm command-line interface. This needs to be run on both Oracle

RAC nodes. Note that I had to specify the IP address and not the host name of the network storage server

(openfiler1-priv) - I believe this is required given the discovery (above) shows the targets using the IP

address.

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm1 -p 192.168.2.195 -l

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm2 -p 192.168.2.195 -l

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm3 -p 192.168.2.195 -l

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm4 -p 192.168.2.195 -l

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.crs -p 192.168.2.195 -l

Configure Automatic Log In

The next step is to ensure the client will automatically log in to each of the targets listed above when the machine

is booted (or the iSCSI initiator service is started/restarted). As with the manual log in process described above,

perform the following on both Oracle RAC nodes:

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm1 -p 192.168.2.195 --op update -n

node.startup -v automatic

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm2 -p 192.168.2.195 --op update -n

node.startup -v automatic

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm3 -p 192.168.2.195 --op update -n

node.startup -v automatic

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm4 -p 192.168.2.195 --op update -n

node.startup -v automatic

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.crs -p 192.168.2.195 --op update -n

node.startup -v automatic

Create Persistent Local SCSI Device Names

In this section, we will go through the steps to create persistent local SCSI device names for each of the iSCSI

target names. This will be done using udev. Having a consistent local SCSI device name and which iSCSI target

it maps to is required in order to know which volume (device) is to be used for OCFS2 and which volumes

belong to ASM.

When either of the Oracle RAC nodes boot and the iSCSI initiator service is started, it will automatically log in

to each of the targets configured in a random fashion and map them to the next available local SCSI device

name. For example, the target iqn.2006-01.com.openfiler:racdb.asm1 may get mapped to /dev/sda. I can

actually determine the current mappings for all targets by looking at the /dev/disk/by-path directory:

# (cd /dev/disk/by-path; ls -l *openfiler* | awk '{FS=" "; print $9 " " $10 " " $11}')

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm1-lun-0 -> ../../sda

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm2-lun-0 -> ../../sdb

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm3-lun-0 -> ../../sdc

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm4-lun-0 -> ../../sdd

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.crs-lun-0 -> ../../sde

Using the output from the above listing, we can establish the following current mappings:

Current iSCSI Target Name to local SCSI Device Name Mappings

iSCSI Target Name SCSI Device Name

iqn.2006-01.com.openfiler:racdb.asm1 /dev/sda

iqn.2006-01.com.openfiler:racdb.asm2 /dev/sdb

iqn.2006-01.com.openfiler:racdb.asm3 /dev/sdc

iqn.2006-01.com.openfiler:racdb.asm4 /dev/sdd

iqn.2006-01.com.openfiler:racdb.crs /dev/sde

This mapping, however, may change every time the Oracle RAC node is rebooted. For example, after a reboot it

may be determined that the iSCSI target iqn.2006-01.com.openfiler:racdb.asm1 gets mapped to the local

SCSI device /dev/sdd. It is therefore impractical to rely on using the local SCSI device name given there is no

way to predict the iSCSI target mappings after a reboot.

What we need is a consistent device name we can reference (i.e. /dev/iscsi/asm1) that will always point to the

appropriate iSCSI target through reboots. This is where the Dynamic Device Management tool named udev

comes in. udev provides a dynamic device directory using symbolic links that point to the actual device using a

configurable set of rules. When udev receives a device event (for example, the client logging in to an iSCSI

target), it matches its configured rules against the available device attributes provided in sysfs to identify the

device. Rules that match may provide additional device information or specify a device node name and multiple

symlink names and instruct udev to run additional programs (a SHELL script for example) as part of the device

event handling process.

The first step is to create a new rules file. The file will be named /etc/udev/rules.d/55-openiscsi.rules

and contain only a single line of name=value pairs used to receive events we are interested in. It will also define

a call-out SHELL script (/etc/udev/scripts/iscsidev.sh) to handle the event.

Create the following rules file /etc/udev/rules.d/55-openiscsi.rules on both Oracle RAC nodes:

/etc/udev/rules.d/55-openiscsi.rules

# /etc/udev/rules.d/55-openiscsi.rules

KERNEL=="sd*", BUS=="scsi", PROGRAM="/etc/udev/scripts/iscsidev.sh

%b",SYMLINK+="iscsi/%c/part%n"

We now need to create the UNIX SHELL script that will be called when this event is received. Let's first create a

separate directory on both Oracle RAC nodes where udev scripts can be stored:

# mkdir -p /etc/udev/scripts

Next, create the UNIX shell script /etc/udev/scripts/iscsidev.sh on both Oracle RAC nodes:

/etc/udev/scripts/iscsidev.sh

#!/bin/sh

# FILE: /etc/udev/scripts/iscsidev.sh

BUS=${1}

HOST=${BUS%%:*}

[ -e /sys/class/iscsi_host ] || exit 1

file="/sys/class/iscsi_host/host${HOST}/device/session*/iscsi_session*/targetname"

target_name=$(cat ${file})

# This is not an open-scsi drive

if [ -z "${target_name}" ]; then

exit 1

fi

# Check if QNAP drive

check_qnap_target_name=${target_name%%:*}

if [ $check_qnap_target_name = "iqn.2004-04.com.qnap" ]; then

target_name=`echo "${target_name%.*}"`

fi

echo "${target_name##*.}"

After creating the UNIX SHELL script, change it to executable:

# chmod 755 /etc/udev/scripts/iscsidev.sh

Now that udev is configured, restart the iSCSI service on both Oracle RAC nodes:

# service iscsi stop

Logging out of session [sid: 1, target: iqn.2006-01.com.openfiler:racdb.asm1, portal:

192.168.2.195,3260]

Logging out of session [sid: 2, target: iqn.2006-01.com.openfiler:racdb.asm2, portal:

192.168.2.195,3260]

Logging out of session [sid: 3, target: iqn.2006-01.com.openfiler:racdb.asm3, portal:

192.168.2.195,3260]

Logging out of session [sid: 4, target: iqn.2006-01.com.openfiler:racdb.asm4, portal:

192.168.2.195,3260]

Logging out of session [sid: 5, target: iqn.2006-01.com.openfiler:racdb.crs, portal:

192.168.2.195,3260]

Logout of [sid: 1, target: iqn.2006-01.com.openfiler:racdb.asm1, portal:

192.168.2.195,3260]: successful

Logout of [sid: 2, target: iqn.2006-01.com.openfiler:racdb.asm2, portal:

192.168.2.195,3260]: successful

Logout of [sid: 3, target: iqn.2006-01.com.openfiler:racdb.asm3, portal:

192.168.2.195,3260]: successful

Logout of [sid: 4, target: iqn.2006-01.com.openfiler:racdb.asm4, portal:

192.168.2.195,3260]: successful

Logout of [sid: 5, target: iqn.2006-01.com.openfiler:racdb.crs, portal:

192.168.2.195,3260]: successful

Stopping iSCSI daemon: [ OK ]

# service iscsi start

iscsid dead but pid file exists

Turning off network shutdown. Starting iSCSI daemon: [ OK ]

[ OK ]

Setting up iSCSI targets: Logging in to [iface: default, target: iqn.2006-

01.com.openfiler:racdb.asm2, portal: 192.168.2.195,3260]

Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm1, portal:

192.168.2.195,3260]

Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.crs, portal:

192.168.2.195,3260]

Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm4, portal:

192.168.2.195,3260]

Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm3, portal:

192.168.2.195,3260]

Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm2, portal:

192.168.2.195,3260]: successful

Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm1, portal:

192.168.2.195,3260]: successful

Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.crs, portal:

192.168.2.195,3260]: successful

Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm4, portal:

192.168.2.195,3260]: successful

Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm3, portal:

192.168.2.195,3260]: successful

[ OK ]

Let's see if our hard work paid off:

# ls -l /dev/iscsi/*

/dev/iscsi/asm1:

total 0

lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> ../../sda

/dev/iscsi/asm2:

total 0

lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> ../../sdc

/dev/iscsi/asm3:

total 0

lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> ../../sdb

/dev/iscsi/asm4:

total 0

lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> ../../sde

/dev/iscsi/crs:

total 0

lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> ../../sdd

The listing above shows that udev did the job it was suppose to do! We now have a consistent set of local device

names that can be used to reference the iSCSI targets. For example, we can safely assume that the device name

/dev/iscsi/asm1/part will always reference the iSCSI target iqn.2006-01.com.openfiler:racdb.asm1.

We now have a consistent iSCSI target name to local device name mapping which is described in the following

table:

iSCSI Target Name to Local Device Name Mappings

iSCSI Target Name Local Device Name

iqn.2006-01.com.openfiler:racdb.asm1 /dev/iscsi/asm1/part

iqn.2006-01.com.openfiler:racdb.asm2 /dev/iscsi/asm2/part

iqn.2006-01.com.openfiler:racdb.asm3 /dev/iscsi/asm3/part

iqn.2006-01.com.openfiler:racdb.asm4 /dev/iscsi/asm4/part

iqn.2006-01.com.openfiler:racdb.crs /dev/iscsi/crs/part

Create Partitions on iSCSI Volumes

We now need to create a single primary partition on each of the iSCSI volumes that spans the entire size of the

volume. As mentioned earlier in this article, I will be using Oracle's Cluster File System, Release 2 (OCFS2) to

store the two files to be shared for Oracle's Clusterware software. We will then be using Automatic Storage

Management (ASM) to create four ASM volumes; two for all physical database files (data/index files, online

redo log files, and control files) and two for the Flash Recovery Area (RMAN backups and archived redo log

files).

The following table lists the five iSCSI volumes and what file systems they will support:

Oracle Shared Drive Configuration

File

System

Type

iSCSI

Target

(short)

Name

Size Mount

Point ASM Diskgroup

Name File

Types

OCFS2 crs 2GB /u02 Oracle Cluster Registry (OCR) File - (~250

MB)

Voting Disk - (~20MB)

ASM asm1 17.8GB ORCL:VOL

1 +RACDB_DATA1 Oracle Database Files

ASM asm2 17.8GB ORCL:VOL

2 +RACDB_DATA1 Oracle Database Files

ASM asm3 17.8GB ORCL:VOL

3 +FLASH_RECOVERY_ARE

A Oracle Flash Recovery Area

ASM asm4 17.8GB ORCL:VOL

4 +FLASH_RECOVERY_ARE

A Oracle Flash Recovery Area

Total 73.2G

B

As shown in the table above, we will need to create a single Linux primary partition on each of the five iSCSI

volumes. The fdisk command is used in Linux for creating (and removing) partitions. For each of the five

iSCSI volumes, you can use the default values when creating the primary partition as the default action is to use

the entire disk. You can safely ignore any warnings that may indicate the device does not contain a valid DOS

partition (or Sun, SGI or OSF disklabel).

In this example, I will be running the fdisk command from linux1 to create a single primary partition on each

iSCSI target using the local device names created by udev in the previous section:

/dev/iscsi/asm1/part /dev/iscsi/asm2/part /dev/iscsi/asm3/part /dev/iscsi/asm4/part /dev/iscsi/crs/part

Creating the single partition on each of the iSCSI volumes must only be run from one of

the nodes in the Oracle RAC cluster! (i.e. linux1)

# ---------------------------------------

# fdisk /dev/iscsi/asm1/part

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-16992, default 1): 1

Last cylinder or +size or +sizeM or +sizeK (1-16992, default 16992): 16992

Command (m for help): p

Disk /dev/iscsi/asm1/part: 17.8 GB, 17817403392 bytes

64 heads, 32 sectors/track, 16992 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System

/dev/iscsi/asm1/part1 1 16992 17399792 83 Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

# ---------------------------------------

# fdisk /dev/iscsi/asm2/part

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-16992, default 1): 1

Last cylinder or +size or +sizeM or +sizeK (1-16992, default 16992): 16992

Command (m for help): p

Disk /dev/iscsi/asm2/part: 17.8 GB, 17817403392 bytes

64 heads, 32 sectors/track, 16992 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System

/dev/iscsi/asm2/part1 1 16992 17399792 83 Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

# ---------------------------------------

# fdisk /dev/iscsi/asm3/part

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-16992, default 1): 1

Last cylinder or +size or +sizeM or +sizeK (1-16992, default 16992): 16992

Command (m for help): p

Disk /dev/iscsi/asm3/part: 17.8 GB, 17817403392 bytes

64 heads, 32 sectors/track, 16992 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System

/dev/iscsi/asm3/part1 1 16992 17399792 83 Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

# ---------------------------------------

# fdisk /dev/iscsi/asm4/part

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-16960, default 1): 1

Last cylinder or +size or +sizeM or +sizeK (1-16960, default 16960): 16960

Command (m for help): p

Disk /dev/iscsi/asm4/part: 17.7 GB, 17783848960 bytes

64 heads, 32 sectors/track, 16960 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System

/dev/iscsi/asm4/part1 1 16960 17367024 83 Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

# ---------------------------------------

# fdisk /dev/iscsi/crs/part

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-1009, default 1): 1

Last cylinder or +size or +sizeM or +sizeK (1-1009, default 1009): 1009

Command (m for help): p

Disk /dev/iscsi/crs/part: 2147 MB, 2147483648 bytes

67 heads, 62 sectors/track, 1009 cylinders

Units = cylinders of 4154 * 512 = 2126848 bytes

Device Boot Start End Blocks Id System

/dev/iscsi/crs/part1 1 1009 2095662 83 Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

Verify New Partitions

After creating all required partitions from linux1, you should now inform the kernel of the partition changes

using the following command as the "root" user account from all remaining nodes in the Oracle RAC cluster

(linux2). Note that the mapping of iSCSI target names discovered from Openfiler and the local SCSI device

name will be different on both Oracle RAC nodes. This is not a concern and will not cause any problems since

we will not be using the local SCSI device names but rather the local device names created by udev in the

previous section.

From linux2, run the following commands:

# partprobe

# fdisk -l

Disk /dev/hda: 40.0 GB, 40000000000 bytes

255 heads, 63 sectors/track, 4863 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/hda1 * 1 13 104391 83 Linux

/dev/hda2 14 4863 38957625 8e Linux LVM

Disk /dev/sda: 2147 MB, 2147483648 bytes

67 heads, 62 sectors/track, 1009 cylinders

Units = cylinders of 4154 * 512 = 2126848 bytes

Device Boot Start End Blocks Id System

/dev/sda1 1 1009 2095662 83 Linux

Disk /dev/sdb: 17.7 GB, 17783848960 bytes

64 heads, 32 sectors/track, 16960 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System

/dev/sdb1 1 16960 17367024 83 Linux

Disk /dev/sdc: 17.8 GB, 17817403392 bytes

64 heads, 32 sectors/track, 16992 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System

/dev/sdc1 1 16992 17399792 83 Linux

Disk /dev/sdd: 17.8 GB, 17817403392 bytes

64 heads, 32 sectors/track, 16992 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System

/dev/sdd1 1 16992 17399792 83 Linux

Disk /dev/sde: 17.8 GB, 17817403392 bytes

64 heads, 32 sectors/track, 16992 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System

/dev/sde1 1 16992 17399792 83 Linux

As a final step you should run the following command on both Oracle RAC nodes to verify that udev created the

new symbolic links for each new partition:

# (cd /dev/disk/by-path; ls -l *openfiler* | awk '{FS=" "; print $9 " " $10 " " $11}')

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm1-lun-0 -> ../../sdc

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm1-lun-0-part1 -> ../../sdc1

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm2-lun-0 -> ../../sdd

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm2-lun-0-part1 -> ../../sdd1

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm3-lun-0 -> ../../sde

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm3-lun-0-part1 -> ../../sde1

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm4-lun-0 -> ../../sdb

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm4-lun-0-part1 -> ../../sdb1

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.crs-lun-0 -> ../../sda

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.crs-lun-0-part1 -> ../../sda1

The listing above shows that udev did indeed create new device names for each of the new partitions. We will be

using these new device names when configuring the volumes for OCFS2 and ASMlib later in this guide:

/dev/iscsi/asm1/part1 /dev/iscsi/asm2/part1 /dev/iscsi/asm3/part1 /dev/iscsi/asm4/part1 /dev/iscsi/crs/part1

Create "oracle" User and Directories

Perform the following tasks on both Oracle RAC nodes in the cluster!

In this section we will create the oracle UNIX user account, recommended O/S groups, and all required

directories. The following O/S groups will be created:

Description Oracle Privilege Oracle Group Name UNIX Group name

Oracle Inventory and Software Owner oinstall

Database Administrator SYSDBA OSDBA dba

Database Operator SYSOPER OSOPER oper

We will be using the Oracle Cluster File System, Release 2 (OCFS2) to store the files required to be shared for

the Oracle Clusterware software. When using OCFS2, the UID of the UNIX user "oracle" and GID of the

UNIX group "oinstall" must be the same on both of the Oracle RAC nodes in the cluster. If either the UID or

GID are different, the files on the OCFS2 file system will show up as "unowned" or may even be owned by a

different user. For this article, I will use 501 for the "oracle" UID and 501 for the "oinstall" GID.

Note that members of the UNIX group oinstall are considered the "owners" of the Oracle software. Members

of the dba group can administer Oracle databases, for example starting up and shutting down databases.

Members of the optional group oper have a limited set of database administrative privileges such as managing

and running backups. The default name for this group is oper. To use this group, choose the "Custom"

installation type to install the Oracle database software. In this article, we are creating the oracle user account to

have all responsibilities!

This guide adheres to the Optimal Flexible Architecture (OFA) for naming conventions

used in creating the directory structure.

Create Group and User for Oracle

Lets start this section by creating the UNIX oinstall, dba, and oper group and oracle user account:

# groupadd -g 501 oinstall

# groupadd -g 502 dba

# groupadd -g 503 oper

# useradd -m -u 501 -g oinstall -G dba,oper -d /home/oracle -s /bin/bash -c "Oracle

Software Owner" oracle

# id oracle

uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper)

Set the password for the oracle account:

# passwd oracle

Changing password for user oracle.

New UNIX password: xxxxxxxxxxx

Retype new UNIX password: xxxxxxxxxxx

passwd: all authentication tokens updated successfully.

Verify That the User nobody Exists

Before installing the Oracle software, complete the following procedure to verify that the user nobody exists on

the system:

1. To determine if the user exists, enter the following command: 2. # id nobody

uid=99(nobody) gid=99(nobody) groups=99(nobody)

If this command displays information about the nobody user, then you do not have to create that

user.

3. If the user nobody does not exist, then enter the following command to create it:

# /usr/sbin/useradd nobody

4. Repeat this procedure on all the other Oracle RAC nodes in the cluster.

Create the Oracle Base Directory

The next step is to create a new directory that will be used to store the Oracle Database software. When

configuring the oracle user's environment (later in this section) we will be assigning the location of this

directory to the $ORACLE_BASE environment variable.

The following assumes that the directories are being created in the root file system. Please note that this is being

done for the sake of simplicity and is not recommended as a general practice. Normally, these directories would

be created on a separate file system.

After the directory is created, you must then specify the correct owner, group, and permissions for it. Perform the

following on both Oracle RAC nodes:

# mkdir -p /u01/app/oracle

# chown -R oracle:oinstall /u01/app/oracle

# chmod -R 775 /u01/app/oracle

At the end of this procedure, you will have the following:

/u01 owned by root.

/u01/app owned by root.

/u01/app/oracle owned by oracle:oinstall with 775 permissions. This ownership and

permissions enables the OUI to create the oraInventory directory, in the path

/u01/app/oracle/oraInventory.

Create the Oracle Clusterware Home Directory

Next, create a new directory that will be used to store the Oracle Clusterware software. When configuring the

oracle user's environment (later in this section) we will be assigning the location of this directory to the

$ORA_CRS_HOME environment variable.

As noted in the previous section, the following assumes that the directories are being created in the root file

system. This is being done for the sake of simplicity and is not recommended as a general practice. Normally,

these directories would be created on a separate file system.

After the directory is created, you must then specify the correct owner, group, and permissions for it. Perform the

following on both Oracle RAC nodes:

# mkdir -p /u01/app/crs

# chown -R oracle:oinstall /u01/app/crs

# chmod -R 775 /u01/app/crs

At the end of this procedure, you will have the following:

/u01 owned by root.

/u01/app owned by root.

/u01/app/crs owned by oracle:oinstall with 775 permissions. These permissions are

required for Oracle Clusterware installation and are changed during the installation process.

Create Mount Point for OCFS2 / Clusterware

Let's now create the mount point for the Oracle Cluster File System, Release 2 (OCFS2) that will be used to store

the two Oracle Clusterware shared files.

Perform the following on both Oracle RAC nodes:

# mkdir -p /u02

# chown -R oracle:oinstall /u02

# chmod -R 775 /u02

Create Login Script for oracle User Account

To ensure that the environment is setup correctly for the "oracle" UNIX userid on both Oracle RAC nodes, use

the following .bash_profile:

When you are setting the Oracle environment variables for each Oracle RAC node,

ensure to assign each RAC node a unique Oracle SID! For this example, I used:

For this example, I used:

linux1 : ORACLE_SID=racdb1

linux2 : ORACLE_SID=racdb2

Log in to each node as the oracle user account:

# su - oracle

.bash_profile for Oracle User

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

alias ls="ls -FA"

alias s="screen -DRRS iPad -t iPad"

export JAVA_HOME=/usr/local/java

# User specific environment and startup programs

export ORACLE_BASE=/u01/app/oracle

export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1

export ORA_CRS_HOME=/u01/app/crs

export ORACLE_PATH=$ORACLE_BASE/dba_scripts/common/sql:.:$ORACLE_HOME/rdbms/admin

export CV_JDKHOME=/usr/local/java

# Each RAC node must have a unique ORACLE_SID. (i.e. racdb1, racdb2,...)

export ORACLE_SID=racdb1

export PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin

export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin

export PATH=${PATH}:$ORACLE_BASE/dba_scripts/common/bin

export ORACLE_TERM=xterm

export TNS_ADMIN=$ORACLE_HOME/network/admin

export ORA_NLS10=$ORACLE_HOME/nls/data

export NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"

export LD_LIBRARY_PATH=$ORACLE_HOME/lib

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib

export CLASSPATH=$ORACLE_HOME/JRE

export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib

export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib

export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib

export THREADS_FLAG=native

export TEMP=/tmp

export TMPDIR=/tmp

Configure the Linux Servers for Oracle

Perform the following configuration procedures on both Oracle RAC nodes

in the cluster!

The kernel parameters and shell limits discussed in this section will need to be defined

on both Oracle RAC nodes in the cluster every time the machine is booted. This section

provides information about setting those kernel parameters required for Oracle.

Instructions for placing them in a startup script (/etc/sysctl.conf) are included in the

section "All Startup Commands for Both Oracle RAC Nodes".

Overview

This section focuses on configuring both Oracle RAC Linux servers - getting each one prepared for the Oracle

RAC 10g installation. This includes verifying enough memory and swap space, setting shared memory and

semaphores, setting the maximum number of file handles, setting the IP local port range, setting shell limits for

the oracle user, activating all kernel parameters for the system, and finally how to verify the correct date and

time for both nodes in the cluster.

There are several different ways to configure (set) these parameters. For the purpose of this article, I will be

making all changes permanent (through reboots) by placing all commands in the /etc/sysctl.conf file.

Memory and Swap Space Considerations

Installing Oracle Database 10g Release 2 on RHEL/OEL 5 requires a minimum of 1024MB of memory. (Note:

An inadequate amount of swap during the installation will cause the Oracle Universal Installer to either "hang"

or "die")

To check the amount of memory you have, type:

# cat /proc/meminfo | grep MemTotal

MemTotal: 2074084 kB

To check the amount of swap you have allocated, type:

# cat /proc/meminfo | grep SwapTotal

SwapTotal: 4128760 kB

If you have less than 2048MB of memory (between your RAM and SWAP), you can

add temporary swap space by creating a temporary swap file. This way you do not have

to use a raw device or even more drastic, rebuild your system.

As root, make a file that will act as additional swap space, let's say about 500MB:

# dd if=/dev/zero of=tempswap bs=1k count=500000

Now we should change the file permissions:

# chmod 600 tempswap

Finally we format the "partition" as swap and add it to the swap space:

# mke2fs tempswap

# mkswap tempswap

# swapon tempswap

Configuring Kernel Parameters and Shell Limits

The kernel parameters and shell limits presented in this section are recommended values only as documented by

Oracle. For production database systems, Oracle recommends that you tune these values to optimize the

performance of the system.

On both Oracle RAC nodes, verify that the kernel parameters described in this section are set to values greater

than or equal to the recommended values. Also note that when setting the four semaphore values that all four

values need to be entered on one line.

Configuring Kernel Parameters

Oracle Database 10g Release 2 on RHEL/OEL 5 requires the kernel parameter settings shown below. The values

given are minimums, so if your system uses a larger value, do not change it:

kernel.shmmax = 4294967295

kernel.shmall = 268435456

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 65536

net.ipv4.ip_local_port_range = 1024 65000

net.core.rmem_default=1048576

net.core.rmem_max=1048576

net.core.wmem_default=262144

net.core.wmem_max=262144

RHEL/OEL 5 already comes configured with default values defined for the following kernel parameters:

kernel.shmall

kernel.shmmax

Use the default values if they are the same or larger than the required values.

This article assumes a fresh new install of CentOS 5 and as such, many of the required kernel parameters are

already set (see above). This being the case, you can simply copy / paste the following to both Oracle RAC

nodes while logged in as root:

cat >> /etc/sysctl.conf <<EOF

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 65536

net.ipv4.ip_local_port_range = 1024 65000

EOF

The above command persisted the required kernel parameters through reboots by inserting them in the

/etc/sysctl.conf startup file. Linux allows modification of these kernel parameters to the current system

while it is up and running, so there's no need to reboot the system after making kernel parameter changes. To

activate the new kernel parameter values for the currently running system, run the following as root on both

Oracle RAC nodes in the cluster:

# sysctl -p

net.ipv4.ip_forward = 0

net.ipv4.conf.default.rp_filter = 1

net.ipv4.conf.default.accept_source_route = 0

kernel.sysrq = 0

kernel.core_uses_pid = 1

net.ipv4.tcp_syncookies = 1

kernel.msgmnb = 65536

kernel.msgmax = 65536

kernel.shmmax = 4294967295

kernel.shmall = 268435456

net.core.rmem_default = 1048576

net.core.rmem_max = 1048576

net.core.wmem_default = 262144

net.core.wmem_max = 262144

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 65536

net.ipv4.ip_local_port_range = 1024 65000

Verify the new kernel parameter values by running the following on both Oracle RAC nodes in the cluster:

# /sbin/sysctl -a | grep shm

vm.hugetlb_shm_group = 0

kernel.shmmni = 4096

kernel.shmall = 268435456

kernel.shmmax = 4294967295

# /sbin/sysctl -a | grep sem

kernel.sem = 250 32000 100 128

# /sbin/sysctl -a | grep file-max

fs.file-max = 65536

# /sbin/sysctl -a | grep ip_local_port_range

net.ipv4.ip_local_port_range = 1024 65000

# /sbin/sysctl -a | grep 'core\.[rw]mem'

net.core.rmem_default = 1048576

net.core.wmem_default = 262144

net.core.rmem_max = 1048576

net.core.wmem_max = 262144

Setting Shell Limits for the oracle User

To improve the performance of the software on Linux systems, Oracle recommends you increase the following

shell limits for the oracle user:

Shell Limit Item in limits.conf Hard Limit

Maximum number of open file descriptors nofile 65536

Maximum number of processes available to a single user nproc 16384

To make these changes, run the following as root:

cat >> /etc/security/limits.conf <<EOF

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536

EOF

cat >> /etc/pam.d/login <<EOF

session required /lib/security/pam_limits.so

EOF

Update the default shell startup file for the "oracle" UNIX account.

For the Bourne, Bash, or Korn shell, add the following lines to the /etc/profile file by running

the following command: cat >> /etc/profile <<EOF

if [ \$USER = "oracle" ]; then

if [ \$SHELL = "/bin/ksh" ]; then

ulimit -p 16384

ulimit -n 65536

else

ulimit -u 16384 -n 65536

fi

umask 022

fi

EOF

For the C shell (csh or tcsh), add the following lines to the /etc/csh.login file by running the

following command: cat >> /etc/csh.login <<EOF

if ( \$USER == "oracle" ) then

limit maxproc 16384

limit descriptors 65536

endif

EOF

Setting the Correct Date and Time on All Cluster Nodes

During the installation of Oracle Clusterware, the Database, and the Companion CD, the Oracle Universal

Installer (OUI) first installs the software to the local node running the installer (i.e. linux1). The software is then

copied remotely to all of the remaining nodes in the cluster (i.e. linux2). During the remote copy process, the

OUI will execute the UNIX "tar" command on each of the remote nodes to extract the files that were archived

and copied over. If the date and time on the node performing the install is greater than that of the node it is

copying to, the OUI will throw an error from the "tar" command indicating it is attempting to extract files

stamped with a time in the future:

Error while copying directory

/u01/app/crs with exclude file list 'null' to nodes 'linux2'.

[PRKC-1002 : All the submitted commands did not execute successfully]

---------------------------------------------

linux2:

/bin/tar: ./bin/lsnodes: time stamp 2009-08-16 09:21:34 is 735 s in the future

/bin/tar: ./bin/olsnodes: time stamp 2009-08-16 09:21:34 is 735 s in the future

...(more errors on this node)

Please note that although this would seem like a severe error from the OUI, it can safely be disregarded as a

warning. The "tar" command DOES actually extract the files; however, when you perform a listing of the files

(using ls -l) on the remote node, they will be missing the time field until the time on the remote server is

greater than the timestamp of the file.

Before starting any of the above noted installations, ensure that each member node of the cluster is set as closely

as possible to the same date and time. Oracle strongly recommends using the Network Time Protocol feature of

most operating systems for this purpose, with all nodes using the same reference Network Time Protocol server.

Accessing a Network Time Protocol server, however, may not always be an option. In this case, when manually

setting the date and time for the nodes in the cluster, ensure that the date and time of the node you are performing

the software installations from (linux1) is less than all other nodes in the cluster (linux2). I generally use a 20

second difference as shown in the following example:

Setting the date and time from linux1:

# date -s "8/16/2009 01:12:00"

Setting the date and time from linux2:

# date -s "8/16/2009 01:12:20"

The two-node RAC configuration described in this article does not make use of a Network Time Protocol server.

Configure the "hangcheck-timer" Kernel Module

Perform the following configuration procedures on both Oracle RAC nodes

in the cluster!

Oracle 9.0.1 and 9.2.0.1 used a userspace watchdog daemon called watchdogd to monitor the health of the

cluster and to restart a RAC node in case of a failure. Starting with Oracle 9.2.0.2 (and still available in

Oracle10g Release 2), the watchdog daemon has been deprecated by a Linux kernel module named hangcheck-

timer which addresses availability and reliability problems much better. The hang-check timer is loaded into

the Linux kernel and checks if the system hangs. It will set a timer and check the timer after a certain amount of

time. There is a configurable threshold to hang-check that, if exceeded will reboot the machine. Although the

hangcheck-timer module is not required for Oracle Clusterware (Cluster Manager) operation, it is highly

recommended by Oracle.

The hangcheck-timer.ko Module

The hangcheck-timer module uses a kernel-based timer that periodically checks the system task scheduler to

catch delays in order to determine the health of the system. If the system hangs or pauses, the timer resets the

node. The hangcheck-timer module uses the Time Stamp Counter (TSC) CPU register which is a counter that

is incremented at each clock signal. The TCS offers much more accurate time measurements since this register is

updated by the hardware automatically.

Much more information about the

hangcheck-timer project can be found here.

Installing the hangcheck-timer.ko Module

The hangcheck-timer was normally shipped only by Oracle, however, this module is now included with Red

Hat Linux AS starting with kernel versions 2.4.9-e.12 and higher. The hangcheck-timer should already be

included. Use the following to ensure that you have the module included:

# find /lib/modules -name "hangcheck-timer.ko"

/lib/modules/2.6.18-128.el5/kernel/drivers/char/hangcheck-timer.ko

In the above output, we care about the hangcheck timer object (hangcheck-timer.ko) in the

/lib/modules/2.6.18-128.el5/kernel/drivers/char directory.

Configuring and Loading the hangcheck-timer Module

There are two key parameters to the hangcheck-timer module:

hangcheck-tick: This parameter defines the period of time between checks of system health.

The default value is 60 seconds; Oracle recommends setting it to 30 seconds.

hangcheck-margin: This parameter defines the maximum hang delay that should be tolerated

before hangcheck-timer resets the RAC node. It defines the margin of error in seconds. The

default value is 180 seconds; Oracle recommends setting it to 180 seconds.

The two hangcheck-timer module parameters indicate how long a RAC node must

hang before it will reset the system. A node reset will occur when the following is true: system hang time > (hangcheck_tick + hangcheck_margin)

Configuring Hangcheck Kernel Module Parameters

Each time the hangcheck-timer kernel module is loaded (manually or by Oracle) it needs to know what value to

use for each of the two parameters we just discussed: (hangcheck-tick and hangcheck-margin).

These values need to be available after each reboot of the Linux server. To do this, make an entry with the

correct values to the /etc/modprobe.conf file as follows:

# echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >>

/etc/modprobe.conf

Each time the hangcheck-timer kernel module gets loaded, it will use the values defined by the entry I made in

the /etc/modprobe.conf file.

Manually Loading the Hangcheck Kernel Module for Testing

Oracle is responsible for loading the hangcheck-timer kernel module when required. It is for this reason that it is

not required to perform a modprobe or insmod of the hangcheck-timer kernel module in any of the startup files

(i.e. /etc/rc.local).

It is only out of pure habit that I continue to include a modprobe of the hangcheck-timer kernel module in the

/etc/rc.local file. Someday I will get over it, but realize that it does not hurt to include a modprobe of the

hangcheck-timer kernel module during startup.

So to keep myself sane and able to sleep at night, I always configure the loading of the hangcheck-timer kernel

module on each startup as follows:

# echo "/sbin/modprobe hangcheck-timer" >> /etc/rc.local

You don't have to manually load the hangcheck-timer kernel module using modprobe or

insmod after each reboot. The hangcheck-timer module will be loaded by Oracle

(automatically) when needed.

Now, to test the hangcheck-timer kernel module to verify it is picking up the correct parameters we defined in

the /etc/modprobe.conf file, use the modprobe command. Although you could load the hangcheck-timer

kernel module by passing it the appropriate parameters (e.g. insmod hangcheck-timer hangcheck_tick=30

hangcheck_margin=180), we want to verify that it is picking up the options we set in the /etc/modprobe.conf

file.

To manually load the hangcheck-timer kernel module and verify it is using the correct values defined in the

/etc/modprobe.conf file, run the following command:

# modprobe hangcheck-timer

# grep Hangcheck /var/log/messages | tail -2

Aug 16 16:46:59 linux1 kernel: Hangcheck: starting hangcheck timer 0.9.0 (tick is 30

seconds, margin is 180 seconds).

Aug 16 16:46:59 linux1 kernel: Hangcheck: Using get_cycles().

Configure RAC Nodes for Remote Access using SSH

Perform the following configuration procedures on both Oracle RAC nodes

in the cluster!

Before you can install Oracle RAC 10g, you must configure secure shell (SSH) for the UNIX user account you

plan to use to install Oracle Clusterware and the Oracle Database software. The installation and configuration

tasks described in this section will need to be performed on both Oracle RAC nodes. As configured earlier in this

article, the software owner for Oracle Clusterware and the Oracle Database software will be "oracle".

The goal here is to setup user equivalence for the oracle UNIX user account. User equivalence enables the

oracle UNIX user account to access all other nodes in the cluster (running commands and copying files)

without the need for a password. Oracle added support in 10g Release 1 for using the SSH tool suite for setting

up user equivalence. Before Oracle Database 10g, user equivalence had to be configured using remote shell

(RSH).

The SSH configuration described in this article uses SSH1. If SSH is not available, then

OUI attempts to use rsh and rcp instead. These services, however, are disabled by

default on most Linux systems. The use of RSH will not be discussed in this article.

You need either an RSA or a DSA key for the SSH protocol. RSA is used with the SSH 1.5 protocol, while DSA

is the default for the SSH 2.0 protocol. With OpenSSH, you can use either RSA or DSA. For the purpose of this

article, we will configure SSH using SSH1.

If you have an SSH2 installation, and you cannot use SSH1, then refer to your SSH distribution documentation

to configure SSH1 compatibility or to configure SSH2 with DSA. This type of configuration is beyond the scope

of this article and will not be discussed.

So, why do we have to setup user equivalence? Installing Oracle Clusterware and the Oracle Database software

is only performed from one node in a RAC cluster. When running the Oracle Universal Installer (OUI) on that

particular node, it will use the ssh and scp commands to run remote commands on and copy files (the Oracle

software) to all other nodes within the RAC cluster. The oracle UNIX user account on the node running the

OUI (runInstaller) must be trusted by all other nodes in your RAC cluster. This means that you must be able

to run the secure shell commands (ssh or scp) on the Linux server you will be running the OUI from against all

other Linux servers in the cluster without being prompted for a password.

Please note that the use of secure shell is not required for normal RAC operation. This

configuration, however, must to be enabled for RAC and patchset installations as well

as creating the clustered database.

The methods required for configuring SSH1, an RSA key, and user equivalence is described in the following

sections.

Configuring the Secure Shell

To determine if SSH is installed and running, enter the following command:

# pgrep sshd

2360

If SSH is running, then the response to this command is a list of process ID number(s). Run this command on

both Oracle RAC nodes in the cluster to verify the SSH daemons are installed and running!

To find out more about SSH, refer to the man page: # man ssh

Creating the RSA Keys on Both Oracle RAC Nodes

The first step in configuring SSH is to create an RSA public/private key pair on both Oracle RAC nodes in the

cluster. The command to do this will create a public and private key for RSA (for a total of two keys per node).

The content of the RSA public keys will then need to be copied into an authorized key file which is then

distributed to both Oracle RAC nodes in the cluster.

Use the following steps to create the RSA key pair. Please note that these steps will need to be completed on

both Oracle RAC nodes in the cluster:

1. Log on as the "oracle" UNIX user account.

# su - oracle

2. If necessary, create the .ssh directory in the "oracle" user's home directory and set the correct

permissions on it: 3. $ mkdir -p ~/.ssh

$ chmod 700 ~/.ssh

4. Enter the following command to generate an RSA key pair (public and private key) for the SSH

protocol:

$ /usr/bin/ssh-keygen -t rsa

At the prompts:

o Accept the default location for the key files.

o Enter and confirm a pass phrase. This should be different from the "oracle" UNIX user

account password however it is not a requirement.

This command will write the public key to the ~/.ssh/id_rsa.pub file and the private key to the

~/.ssh/id_rsa file. Note that you should never distribute the private key to anyone!

5. Repeat the above steps for each Oracle RAC node in the cluster.

Now that both Oracle RAC nodes contain a public and private key for RSA, you will need to create an

authorized key file on one of the nodes. An authorized key file is nothing more than a single file that contains a

copy of everyone's (every node's) RSA public key. Once the authorized key file contains all of the public keys, it

is then distributed to all other nodes in the cluster.

Complete the following steps on one of the nodes in the cluster to create and then distribute the authorized key

file. For the purpose of this article, I am using linux1:

1. First, determine if an authorized key file already exists on the node (~/.ssh/authorized_keys).

In most cases this will not exist since this article assumes you are working with a new install. If

the file doesn't exist, create it now: 2. $ touch ~/.ssh/authorized_keys 3. $ cd ~/.ssh 4. $ ls -l *.pub

-rw-r--r-- 1 oracle oinstall 395 Aug 16 17:05 id_rsa.pub

The listing above should show the id_rsa.pub public key created in the previous section.

5. In this step, use SCP (Secure Copy) or SFTP (Secure FTP) to copy the content of the

~/.ssh/id_rsa.pub public key from both Oracle RAC nodes in the cluster to the authorized key

file just created (~/.ssh/authorized_keys). Again, this will be done from linux1. You will be

prompted for the oracle UNIX user account password for both Oracle RAC nodes accessed.

The following example is being run from linux1 and assumes a two-node cluster, with nodes

linux1 and linux2:

$ ssh linux1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The authenticity of host 'linux1 (192.168.1.100)' can't be established.

RSA key fingerprint is d8:2e:03:d5:18:56:60:f3:5b:fa:ba:29:c9:f4:be:d5.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'linux1,192.168.1.100' (RSA) to the list of known

hosts.

oracle@linux1's password: xxxxx

$ ssh linux2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The authenticity of host 'linux2 (192.168.1.101)' can't be established.

RSA key fingerprint is 9d:dd:6d:7f:8c:0b:46:bb:93:77:05:a4:b8:d8:4b:31.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'linux2,192.168.1.101' (RSA) to the list of known

hosts.

oracle@linux2's password: xxxxx

The first time you use SSH to connect to a node from a particular system, you will see a

message similar to the following:

The authenticity of host 'linux1 (192.168.1.100)' can't be

established.

RSA key fingerprint is

d8:2e:03:d5:18:56:60:f3:5b:fa:ba:29:c9:f4:be:d5.

Are you sure you want to continue connecting (yes/no)? yes

Enter yes at the prompt to continue. You should not see this message again when you

connect from this system to the same node.

6. At this point, we have the RSA public key from every node in the cluster in the authorized key

file (~/.ssh/authorized_keys) on linux1. We now need to copy it to the remaining nodes in

the cluster. In our two-node cluster example, the only remaining node is linux2. Use the scp

command to copy the authorized key file to all remaining nodes in the RAC cluster: 7. $ scp ~/.ssh/authorized_keys linux2:.ssh/authorized_keys 8. oracle@linux2's password: xxxxx

authorized_keys 100% 790 0.8KB/s 00:00

9. Change the permission of the authorized key file for both Oracle RAC nodes in the cluster by

logging into the node and running the following:

$ chmod 600 ~/.ssh/authorized_keys

10. At this point, if you use ssh to log in to or run a command on another node, you are prompted for

the pass phrase that you specified when you created the RSA key. For example, test the following

from linux1: 11. $ ssh linux1 hostname

12. Enter passphrase for key '/home/oracle/.ssh/id_rsa': xxxxx

13. linux1

14.

15. $ ssh linux2 hostname

16. Enter passphrase for key '/home/oracle/.ssh/id_rsa': xxxxx

linux2

If you see any other messages or text, apart from the host name, then the Oracle

installation can fail. Make any changes required to ensure that only the host name is

displayed when you enter these commands. You should ensure that any part of a login

script(s) that generate any output, or ask any questions, are modified so that they act

only when the shell is an interactive shell.

Enabling SSH User Equivalency for the Current Shell Session

When running the OUI, it will need to run the secure shell tool commands (ssh and scp) without being

prompted for a pass phrase. Even though SSH is configured on both Oracle RAC nodes in the cluster, using the

secure shell tool commands will still prompt for a pass phrase. Before running the OUI, you need to enable user

equivalence for the terminal session you plan to run the OUI from. For the purpose of this article, all Oracle

installations will be performed from linux1.

User equivalence will need to be enabled on any new terminal shell session before attempting to run the OUI. If

you log out and log back in to the node you will be performing the Oracle installation from, you must enable

user equivalence for the terminal shell session as this is not done by default.

To enable user equivalence for the current terminal shell session, perform the following steps:

1. Log on to the node where you want to run the OUI from (linux1) as the "oracle" UNIX user

account.

# su - oracle

2. Enter the following commands: 3. $ exec /usr/bin/ssh-agent $SHELL 4. $ /usr/bin/ssh-add 5. Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx

Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

At the prompt, enter the pass phrase for each key that you generated.

6. If SSH is configured correctly, you will be able to use the ssh and scp commands without being

prompted for a password or pass phrase from this terminal session: 7. $ ssh linux1 "date;hostname" 8. Sun Aug 16 17:37:47 EDT 2009 9. linux1 10.

11. $ ssh linux2 "date;hostname"

12. Sun Aug 16 17:38:20 EDT 2009

linux2

The commands above should display the date set on each Oracle RAC node along with

its hostname. If any of the nodes prompt for a password or pass phrase then verify that

the ~/.ssh/authorized_keys file on that node contains the correct public keys.

Also, if you see any other messages or text, apart from the date and hostname, then the

Oracle installation can fail. Make any changes required to ensure that only the date is

displayed when you enter these commands. You should ensure that any part of a login

script(s) that generate any output, or ask any questions, are modified so that they act

only when the shell is an interactive shell.

13. The Oracle Universal Installer is a GUI interface and requires the use of an X Server. From the

terminal session enabled for user equivalence (the node you will be performing the Oracle

installations from), set the environment variable DISPLAY to a valid X Windows display:

Bourne, Korn, and Bash shells:

$ DISPLAY=<Any X-Windows Host>:0

$ export DISPLAY

C shell:

$ setenv DISPLAY <Any X-Windows Host>:0

After setting the DISPLAY variable to a valid X Windows display, you should perform another test

of the current terminal session to ensure that X11 forwarding is not enabled:

$ ssh linux1 hostname

linux1

$ ssh linux2 hostname

linux2

If you are using a remote client to connect to the node performing the installation, and

you see a message similar to: "Warning: No xauth data; using fake

authentication data for X11 forwarding." then this means that your authorized

keys file is configured correctly; however, your SSH configuration has X11 forwarding

enabled. For example:

$ export DISPLAY=melody:0

$ ssh linux2 hostname

Warning: No xauth data; using fake authentication data for X11

forwarding.

linux2

Note that having X11 Forwarding enabled will cause the Oracle installation to fail. To

correct this problem, create a user-level SSH client configuration file for the "oracle"

UNIX user account that disables X11 Forwarding:

o Using a text editor, edit or create the file ~/.ssh/config

o Make sure that the ForwardX11 attribute is set to no. For example, insert the

following into the ~/.ssh/config file: o Host *

ForwardX11 no

14. You must run the Oracle Universal Installer from this terminal session or remember to repeat the

steps to enable user equivalence (steps 2, 3, and 4 from this section) before you start the Oracle

Universal Installer from a different terminal session.

Remove any stty Commands

When installing the Oracle software, any hidden files on the system (i.e. .bashrc, .cshrc, .profile) will cause

the installation process to fail if they contain stty commands.

To avoid this problem, you must modify these files to suppress all output on STDERR as in the following

examples:

Bourne, Bash, or Korn shell: if [ -t 0 ]; then

stty intr ^C

fi

C shell: test -t 0

if ($status == 0) then

stty intr ^C

endif

If there are hidden files that contain stty commands that are loaded by the remote shell,

then OUI indicates an error and stops the installation.

All Startup Commands for Both Oracle RAC Nodes

Verify that the following startup commands are included on both of the

Oracle RAC nodes in the cluster!

Up to this point, we have talked in great detail about the parameters and resources that need to be configured on

both nodes in the Oracle RAC 10g configuration. This section will review those parameters, commands, and

entries from previous sections that need to occur on both Oracle RAC nodes when they are booted.

In this section, I provide all of the commands, parameters, and entries that have been discussed so far that will

need to be included in the startup scripts for each Linux node in the RAC cluster. For each of the startup files

below, I indicate in blue the entries that should be included in each of the startup files in order to provide a

successful RAC node.

/etc/modprobe.conf

All parameters and values to be used by kernel modules.

/etc/modprobe.conf

alias eth0 r8169

alias eth1 e1000

alias scsi_hostadapter ata_piix

alias snd-card-0 snd-intel8x0

options snd-card-0 index=0

options snd-intel8x0 index=0

remove snd-intel8x0 { /usr/sbin/alsactl store 0 >/dev/null 2>&1 || : ;

}; /sbin/modprobe -r --ignore-remove snd-intel8x0

options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180

/etc/sysctl.conf

We wanted to adjust the default and maximum send buffer size as well as the default and maximum receive

buffer size for the interconnect. This file also contains those parameters responsible for configuring shared

memory, semaphores, file handles, and local IP range for use by the Oracle instance.

/etc/sysctl.conf

# Kernel sysctl configuration file for Red Hat Linux

#

# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and

# sysctl.conf(5) for more details.

# Controls IP packet forwarding

net.ipv4.ip_forward = 0

# Controls source route verification

net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing

net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel

kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename

# Useful for debugging multi-threaded applications

kernel.core_uses_pid = 1

# Controls the use of TCP syncookies

net.ipv4.tcp_syncookies = 1

# Controls the maximum size of a message, in bytes

kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue

kernel.msgmax = 65536

# +---------------------------------------------------------+

# | ADJUSTING NETWORK SETTINGS |

# +---------------------------------------------------------+

# | With Oracle 9.2.0.1 and onwards, Oracle now makes use |

# | of UDP as the default protocol on Linux for |

# | inter-process communication (IPC), such as Cache Fusion |

# | and Cluster Manager buffer transfers between instances |

# | within the RAC cluster. Oracle strongly suggests to |

# | adjust the default and maximum receive buffer size |

# | (SO_RCVBUF socket option) to 1024MB, and the default |

# | and maximum send buffer size (SO_SNDBUF socket option) |

# | to 256KB. The receive buffers are used by TCP and UDP |

# | to hold received data until it is read by the |

# | application. The receive buffer cannot overflow because |

# | the peer is not allowed to send data beyond the buffer |

# | size window. This means that datagrams will be |

# | discarded if they don't fit in the socket receive |

# | buffer. This could cause the sender to overwhelm the |

# | receiver. |

# +---------------------------------------------------------+

# +---------------------------------------------------------+

# | Default setting in bytes of the socket "receive" buffer |

# | which may be set by using the SO_RCVBUF socket option. |

# +---------------------------------------------------------+

net.core.rmem_default=1048576

# +---------------------------------------------------------+

# | Maximum setting in bytes of the socket "receive" buffer |

# | which may be set by using the SO_RCVBUF socket option. |

# +---------------------------------------------------------+

net.core.rmem_max=1048576

# +---------------------------------------------------------+

# | Default setting in bytes of the socket "send" buffer |

# | which may be set by using the SO_SNDBUF socket option. |

# +---------------------------------------------------------+

net.core.wmem_default=262144

# +---------------------------------------------------------+

# | Maximum setting in bytes of the socket "send" buffer |

# | which may be set by using the SO_SNDBUF socket option. |

# +---------------------------------------------------------+

net.core.wmem_max=262144

# +---------------------------------------------------------+

# | ADJUSTING ADDITIONAL KERNEL PARAMETERS FOR ORACLE |

# +---------------------------------------------------------+

# | Configure the kernel parameters for all Oracle Linux |

# | servers by setting shared memory and semaphores, |

# | setting the maximum amount of file handles, and setting |

# | the IP local port range. |

# +---------------------------------------------------------+

# +---------------------------------------------------------+

# | SHARED MEMORY |

# +---------------------------------------------------------+

# Controls the maximum shared segment size, in bytes

kernel.shmmax = 4294967295

# Controls the maximum number of shared memory segments, in pages

kernel.shmall = 268435456

# Controls the maximum number of shared memory segments system wide

kernel.shmmni = 4096

# +---------------------------------------------------------+

# | SEMAPHORES |

# | ---------- |

# | |

# | SEMMSL_value SEMMNS_value SEMOPM_value SEMMNI_value |

# | |

# +---------------------------------------------------------+

kernel.sem=250 32000 100 128

# +---------------------------------------------------------+

# | FILE HANDLES |

# ----------------------------------------------------------+

fs.file-max=65536

# +---------------------------------------------------------+

# | LOCAL IP RANGE |

# ----------------------------------------------------------+

net.ipv4.ip_local_port_range=1024 65000

Verify that each of the required kernel parameters (above) are configured in the

/etc/sysctl.conf file. Then, ensure that each of these parameters are truly in effect

by running the following command on both Oracle RAC nodes in the cluster:

# sysctl -p

net.ipv4.ip_forward = 0

net.ipv4.conf.default.rp_filter = 1

net.ipv4.conf.default.accept_source_route = 0

kernel.sysrq = 0

kernel.core_uses_pid = 1

net.ipv4.tcp_syncookies = 1

kernel.msgmnb = 65536

kernel.msgmax = 65536

net.core.rmem_default = 1048576

net.core.rmem_max = 1048576

net.core.wmem_default = 262144

net.core.wmem_max = 262144

kernel.shmmax = 4294967295

kernel.shmall = 268435456

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 65536

net.ipv4.ip_local_port_range = 1024 65000

/etc/hosts

All machine/IP entries for nodes in the RAC cluster.

/etc/hosts

# Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1 localhost.localdomain localhost

# Public Network - (eth0)

192.168.1.100 linux1

192.168.1.101 linux2

# Private Interconnect - (eth1)

192.168.2.100 linux1-priv

192.168.2.101 linux2-priv

# Public Virtual IP (VIP) addresses - (eth0:1)

192.168.1.200 linux1-vip

192.168.1.201 linux2-vip

# Private Storage Network for Openfiler - (eth1)

192.168.1.195 openfiler1

192.168.2.195 openfiler1-priv

# Miscellaneous Nodes

192.168.1.1 router

192.168.1.102 alex

192.168.1.103 nascar

192.168.1.105 packmule

192.168.1.106 melody

192.168.1.120 cartman

192.168.1.121 domo

192.168.1.122 switch1

192.168.1.190 george

192.168.1.245 accesspoint

/etc/udev/rules.d/55-openiscsi.rules

Rules file to be used by udev to mount iSCSI volumes. This file contains all name=value pairs used to receive

events and the call-out SHELL script to handle the event.

/etc/udev/rules.d/55-openiscsi.rules

# /etc/udev/rules.d/55-openiscsi.rules

KERNEL=="sd*", BUS=="scsi", PROGRAM="/etc/udev/scripts/iscsidev.sh

%b",SYMLINK+="iscsi/%c/part%n"

/etc/udev/scripts/iscsidev.sh

Call-out SHELL script that handles the events passed to it from the udev rules file (above) and used to mount

iSCSI volumes.

/etc/udev/scripts/iscsidev.sh

#!/bin/sh

# FILE: /etc/udev/scripts/iscsidev.sh

BUS=${1}

HOST=${BUS%%:*}

[ -e /sys/class/iscsi_host ] || exit 1

file="/sys/class/iscsi_host/host${HOST}/device/session*/iscsi_session*/targetname"

target_name=$(cat ${file})

# This is not an open-scsi drive

if [ -z "${target_name}" ]; then

exit 1

fi

echo "${target_name##*.}"

/etc/rc.local

Loading the hangcheck-timer kernel module.

/etc/rc.local

#!/bin/sh

#

# This script will be executed *after* all the other init scripts.

# You can put your own initialization stuff in here if you don't

# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

# +---------------------------------------------------------+

# | HANGCHECK TIMER |

# | (I do not believe this is required, but doesn't hurt) |

# +---------------------------------------------------------+

/sbin/modprobe hangcheck-timer

Install and Configure Oracle Cluster File System (OCFS2)

Most of the installation and configuration procedures in this section should

be performed on both Oracle RAC nodes in the cluster! Creating the

OCFS2 filesystem, however, should only be executed on one of nodes in the

RAC cluster.

Overview

It is now time to install the Oracle Cluster File System, Release 2 (OCFS2). OCFS2, developed by Oracle

Corporation, is a Cluster File System which allows all nodes in a cluster to concurrently access a device via the

standard file system interface. This allows for easy management of applications that need to run across a cluster.

OCFS (Release 1) was released in December 2002 to enable Oracle Real Application Cluster (RAC) users to run

the clustered database without having to deal with RAW devices. The file system was designed to store database

related files, such as data files, control files, redo logs, archive logs, etc. OCFS2 is the next generation of the

Oracle Cluster File System. It has been designed to be a general purpose cluster file system. With it, one can

store not only database related files on a shared disk, but also store Oracle binaries and configuration files

(shared Oracle Home) making management of RAC even easier.

In this article, I will be using the latest release of OCFS2 to store the two files that are required to be shared by

the Oracle Clusterware software. Along with these two files, I will also be using this space to store the shared

ASM SPFILE for all Oracle RAC instances.

See the following page for more information on OCFS2 (including Installation Notes) for Linux:

OCFS2 Project Documentation

Download OCFS2

First, let's download the latest OCFS2 distribution. The OCFS2 distribution comprises of two sets of RPMs;

namely, the kernel module and the tools. The latest kernel module is available for download from

http://oss.oracle.com/projects/ocfs2/files/ and the tools from http://oss.oracle.com/projects/ocfs2-tools/files/.

Download the appropriate RPMs starting with the latest OCFS2 kernel module (the driver). With CentOS 5.3, I

am using kernel release 2.6.18-128.el5. The appropriate OCFS2 kernel module was found in the latest release of

OCFS2 at the time of this writing (OCFS2 Release 1.4.2-1 at the time of this writing).

The available OCFS2 kernel modules for Linux kernel 2.6.18-128.el5 are listed below. Always download the

latest OCFS2 kernel module that matches the distribution, platform, kernel version and the kernel flavor (smp,

hugemem, psmp, etc).

32-bit (x86) Installations

ocfs2-2.6.18-128.el5-1.4.2-1.el5.i686.rpm - (Package for default kernel)

ocfs2-2.6.18-128.el5PAE-1.4.2-1.el5.i686.rpm - (Package for PAE kernel)

ocfs2-2.6.18-128.el5xen-1.4.2-1.el5.i686.rpm - (Package for xen kernel)

Next, download both the OCFS2 tools and the OCFS2 console applications:

ocfs2-tools-1.4.2-1.el5.i386.rpm - (OCFS2 tools)

ocfs2console-1.4.2-1.el5.i386.rpm - (OCFS2 console)

64-bit (x86_64) Installations

ocfs2-2.6.18-128.el5-1.4.2-1.el5.x86_64.rpm - (Package for default kernel)

ocfs2-2.6.18-128.el5xen-1.4.2-1.el5.x86_64.rpm - (Package for xen kernel)

Next, download both the OCFS2 tools and the OCFS2 console applications:

ocfs2-tools-1.4.2-1.el5.x86_64.rpm - (OCFS2 tools)

ocfs2console-1.4.2-1.el5.x86_64.rpm - (OCFS2 console)

The OCFS2 Console is optional but highly recommended. The ocfs2console

application requires e2fsprogs, glib2 2.2.3 or later, vte 0.11.10 or later, pygtk2 (EL4) or

python-gtk (SLES9) 1.99.16 or later, python 2.3 or later and ocfs2-tools.

If you were curious as to which OCFS2 driver release you need, use the OCFS2 release

that matches your kernel version. To determine your kernel release:

$ uname -a

Linux linux1 2.6.18-128.el5 #1 SMP Wed Jan 21 10:44:23 EST 2009 i686

i686 i386 GNU/Linux

Install OCFS2

I will be installing the OCFS2 files onto two - single processor / x86 machines. The installation process is simply

a matter of running the following command on both Oracle RAC nodes in the cluster as the root user account: # rpm -Uvh ocfs2-2.6.18-128.el5-1.4.2-1.el5.i686.rpm \

ocfs2console-1.4.2-1.el5.i386.rpm \

ocfs2-tools-1.4.2-1.el5.i386.rpm

Preparing... ########################################### [100%]

1:ocfs2-tools ########################################### [ 33%]

2:ocfs2-2.6.18-128.el5 ########################################### [ 67%]

3:ocfs2console ########################################### [100%]

Disable SELinux (RHEL4 U2 and higher)

Users of RHEL4 U2 and higher (CentOS 5.3 is based on RHEL5 U3) are advised that OCFS2 currently does not

work with SELinux enabled. If you are using RHEL4 U2 or higher (which includes us since we are using

CentOS 5.3) you will need to disable SELinux (using tool system-config-securitylevel) to get the O2CB

service to execute.

During the installation of CentOS, we Disabled SELinux on the SELinux screen. If, however, you did not

disable SELinux during the installation phase, you can use the tool system-config-securitylevel to disable

SELinux.

If you did not follow the instructions to disable the SELinux option during the installation of CentOS (or if you

simply want to verify it is truly disable), run the "Security Level Configuration" GUI utility:

# /usr/bin/system-config-securitylevel &

This will bring up the following screen:

Figure 16: Security Level Configuration Opening Screen / Firewall Disabled

Now, click the SELinux tab and select the "Disabled" option. After clicking [OK], you will be presented with a

warning dialog. Simply acknowledge this warning by clicking "Yes". Your screen should now look like the

following after disabling the SELinux option:

Figure 17: SELinux Disabled

If you needed to disable SELinux in this section on any of the nodes, those nodes will need to be rebooted to

implement the change. SELinux must be disabled before you can continue with configuring OCFS2!

Configure OCFS2

OCFS2 will be configured to use the private network (192.168.2.0) for all of its network traffic as recommended

by Oracle. While OCFS2 does not take much bandwidth, it does require the nodes to be alive on the network and

sends regular keepalive packets to ensure that they are. To avoid a network delay being interpreted as a node

disappearing on the net which could lead to a node-self-fencing, a private interconnect is recommended. It is safe

to use the same private interconnect for both Oracle RAC and OCFS2.

A popular question then is what node name should be used when adding nodes to an OCFS2 cluster and should it

be related to the IP address? When adding nodes to an OCFS2 cluster, the node name being entered must match

the hostname of the machine or the OCFS2 console will throw an error. The IP address, however, need not be the

one associated with that hostname. In other words, any valid IP address on that node can be used. OCFS2 will

not attempt to match the node name (hostname) with the specified IP address.

The next step is to generate and configure the /etc/ocfs2/cluster.conf file on both Oracle RAC nodes in the

cluster. The easiest way to accomplish this is to run the GUI tool ocfs2console. In this section, we will not only

create and configure the /etc/ocfs2/cluster.conf file using ocfs2console, but will also create and start the

cluster stack O2CB. When the /etc/ocfs2/cluster.conf file is not present, (as will be the case in our

example), the ocfs2console tool will create this file along with a new cluster stack service (O2CB) with a

default cluster name of ocfs2. This will need to be done on both Oracle RAC nodes in the cluster as the root

user account:

# ocfs2console &

This will bring up the GUI as shown below:

Figure 18: ocfs2console Screen

Using the ocfs2console GUI tool, perform the following steps:

1. Select [Cluster] -> [Configure Nodes...]. This will start the OCFS2 Cluster Stack (Figure

19). Acknowledge this Information dialog box by clicking [Close]. You will then be presented

with the "Node Configuration" dialog.

2. On the "Node Configuration" dialog, click the [Add] button.

o This will bring up the "Add Node" dialog.

o In the "Add Node" dialog, enter the Host name and IP address for the first node in the

cluster. Leave the IP Port set to its default value of 7777. In my example, I added both

nodes using linux1 / 192.168.2.100 for the first node and linux2 / 192.168.2.101 for

the second node.

Note: The node name you enter "must" match the hostname of the machine and the IP

addresses will use the private interconnect.

o Click [Apply] on the "Node Configuration" dialog - All nodes should now be "Active" as

shown in Figure 20.

o Click [Close] on the "Node Configuration" dialog.

3. After verifying all values are correct, exit the application using [File] -> [Quit]. This needs to

be performed on both Oracle RAC nodes in the cluster.

Figure 19: Starting the OCFS2 Cluster Stack

The following dialog shows the OCFS2 settings I used for the node linux1 and linux2:

Figure 20: Configuring Nodes for OCFS2

See the Troubleshooting section if you get the error: o2cb_ctl: Unable to access cluster service while creating node

After exiting the ocfs2console, you will have a /etc/ocfs2/cluster.conf similar to the following. This

process needs to be completed on both Oracle RAC nodes in the cluster and the OCFS2 configuration file should

be exactly the same for both of the nodes:

/etc/ocfs2/cluster.conf

node:

ip_port = 7777

ip_address = 192.168.2.100

number = 0

name = linux1

cluster = ocfs2

node:

ip_port = 7777

ip_address = 192.168.2.101

number = 1

name = linux2

cluster = ocfs2

cluster:

node_count = 2

name = ocfs2

O2CB Cluster Service

Before we can do anything with OCFS2 like formatting or mounting the file system, we need to first have

OCFS2's cluster stack, O2CB, running (which it will be as a result of the configuration process performed

above). The stack includes the following services:

NM: Node Manager that keep track of all the nodes in the cluster.conf

HB: Heart beat service that issues up/down notifications when nodes join or leave the cluster

TCP: Handles communication between the nodes

DLM: Distributed lock manager that keeps track of all locks, its owners and status

CONFIGFS: User space driven configuration file system mounted at /config

DLMFS: User space interface to the kernel space DLM

All of the above cluster services have been packaged in the o2cb system service (/etc/init.d/o2cb). Here is a

short listing of some of the more useful commands and options for the o2cb system service.

The following commands are for documentation purposes only and do not need to be

run when installing and configuring OCFS2 for this article!

/etc/init.d/o2cb status Driver for "configfs": Loaded

Filesystem "configfs": Mounted

Driver for "ocfs2_dlmfs": Loaded

Filesystem "ocfs2_dlmfs": Mounted

Checking O2CB cluster ocfs2: Online

Heartbeat dead threshold = 31

Network idle timeout: 30000

Network keepalive delay: 2000

Network reconnect delay: 2000

Checking O2CB heartbeat: Not active

/etc/init.d/o2cb offline ocfs2

Stopping O2CB cluster ocfs2: OK

The above command will offline the cluster we created, ocfs2.

/etc/init.d/o2cb unload Unmounting ocfs2_dlmfs filesystem: OK

Unloading module "ocfs2_dlmfs": OK

Unmounting configfs filesystem: OK

Unloading module "configfs": OK

The above command will unload all OCFS2 modules.

/etc/init.d/o2cb load Loading module "configfs": OK

Mounting configfs filesystem at /config: OK

Loading module "ocfs2_nodemanager": OK

Loading module "ocfs2_dlm": OK

Loading module "ocfs2_dlmfs": OK

Mounting ocfs2_dlmfs filesystem at /dlm: OK

Loads all OCFS2 modules.

/etc/init.d/o2cb online ocfs2

Starting O2CB cluster ocfs2: OK

The above command will online the cluster we created, ocfs2.

Configure O2CB to Start on Boot and Adjust O2CB Heartbeat Threshold

You now need to configure the on-boot properties of the OC2B driver so that the cluster stack services will start

on each boot. You will also be adjusting the OCFS2 Heartbeat Threshold from its default setting of 31 to 61.

Perform the following on both Oracle RAC nodes in the cluster:

With releases of OCFS2 prior to 1.2.1, a bug existed where the driver would not get

loaded on each boot even after configuring the on-boot properties to do so. This bug

was fixed in release 1.2.1 of OCFS2 and does not need to be addressed in this article. If

however you are using a release of OCFS2 prior to 1.2.1, please see the

Troubleshooting section for a workaround to this bug.

Set the on-boot properties as follows:

# /etc/init.d/o2cb offline ocfs2

# /etc/init.d/o2cb unload

# /etc/init.d/o2cb configure

Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.

The following questions will determine whether the driver is loaded on

boot. The current values will be shown in brackets ('[]'). Hitting

without typing an answer will keep that current value. Ctrl-C

will abort.

Load O2CB driver on boot (y/n) [n]: y

Cluster stack backing O2CB [o2cb]: o2cb

Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2

Specify heartbeat dead threshold (>=7) [31]: 61

Specify network idle timeout in ms (>=5000) [30000]: 30000

Specify network keepalive delay in ms (>=1000) [2000]: 2000

Specify network reconnect delay in ms (>=2000) [2000]: 2000

Writing O2CB configuration: OK

Loading filesystem "configfs": OK

Mounting configfs filesystem at /sys/kernel/config: OK

Loading filesystem "ocfs2_dlmfs": OK

Mounting ocfs2_dlmfs filesystem at /dlm: OK

Starting O2CB cluster ocfs2: OK

Format the OCFS2 File System

Unlike the other tasks in this section, creating the OCFS2 file system should only be

executed on one of nodes in the RAC cluster. I will be executing all commands in this

section from linux1 only.

We can now start to make use of the iSCSI volume we partitioned for OCFS2 in the section "Create Partitions on

iSCSI Volumes".

If the O2CB cluster is offline, start it. The format operation needs the cluster to be online, as it needs to ensure

that the volume is not mounted on some other node in the cluster.

Earlier in this document, we created the directory /u02 under the section Create Mount Point for OCFS2 /

Clusterware which will be used as the mount point for the OCFS2 cluster file system. This section contains the

commands to create and mount the file system to be used for the Cluster Manager.

Note that it is possible to create and mount the OCFS2 file system using either the GUI tool ocfs2console or

the command-line tool mkfs.ocfs2. From the ocfs2console utility, use the menu [Tasks] - [Format]. The

instructions below, however, demonstrate how to create the OCFS2 file system using the command-line tool

mkfs.ocfs2.

To create the file system, we can use the Oracle executable mkfs.ocfs2. For the purpose of this example, I run

the following command only from linux1 as the root user account using the local SCSI device name mapped to

the iSCSI volume for crs — /dev/iscsi/crs/part1. Also note that I specified a label named "oracrsfiles"

which will be referred to when mounting or un-mounting the volume:

# mkfs.ocfs2 -b 4K -C 32K -N 4 -L oracrsfiles /dev/iscsi/crs/part1

mkfs.ocfs2 1.4.2

Cluster stack: classic o2cb

Filesystem label=oracrsfiles

Block size=4096 (bits=12)

Cluster size=32768 (bits=15)

Volume size=2145943552 (65489 clusters) (523912 blocks)

3 cluster groups (tail covers 977 clusters, rest cover 32256 clusters)

Journal size=67108864

Initial number of node slots: 4

Creating bitmaps: done

Initializing superblock: done

Writing system files: done

Writing superblock: done

Writing backup superblock: 1 block(s)

Formatting Journals: done

Formatting slot map: done

Writing lost+found: done

mkfs.ocfs2 successful

Mount the OCFS2 File System

Now that the file system is created, we can mount it. Let's first do it using the command-line, then I'll show how

to include it in the /etc/fstab to have it mount on each boot.

Mounting the cluster file system will need to be performed on both Oracle RAC nodes

in the cluster as the root user account using the OCFS2 label oracrsfiles!

First, here is how to manually mount the OCFS2 file system from the command-line. Remember that this needs

to be performed as the root user account on both nodes in the cluster:

# mount -t ocfs2 -o datavolume,nointr -L "oracrsfiles" /u02

If the mount was successful, you will simply get your prompt back. We should, however, run the following

checks to ensure the file system is mounted correctly.

Use the mount command to ensure that the new file system is really mounted. This should be performed on both

nodes in the RAC cluster:

# mount

/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

/dev/hda1 on /boot type ext3 (rw)

tmpfs on /dev/shm type tmpfs (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

domo:Public on /domo type nfs (rw,addr=192.168.1.121)

configfs on /sys/kernel/config type configfs (rw)

ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)

/dev/sdd1 on /u02 type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)

Please take note of the datavolume option I am using to mount the new file system.

Oracle database users must mount any volume that will contain the Voting Disk file,

Cluster Registry (OCR), Data files, Redo logs, Archive logs and Control files with the

datavolume mount option so as to ensure that the Oracle processes open the files with

the o_direct flag. The nointr option ensures that the I/O's are not interrupted by

signals.

Any other type of volume, including an Oracle home (which I will not be using for this

article), should not be mounted with this mount option.

Why does it take so much time to mount the volume? It takes around 5 seconds for a

volume to mount. It does so as to let the heartbeat thread stabilize. In a later release,

Oracle plans to add support for a global heartbeat, which will make most mounts

instant.

Configure OCFS2 to Mount Automatically at Startup

Let's take a look at what you've have done so far. You installed the OCFS2 software packages which will be

used to store the shared files needed by Cluster Manager. After going through the install, you loaded the OCFS2

module into the kernel and then formatted the clustered file system. Finally, you mounted the newly created file

system using the OCFS2 label "oracrsfiles". This section walks through the steps responsible for mounting

the new OCFS2 file system each time the machine(s) are booted using its label.

Start by adding the following line to the /etc/fstab file on both Oracle RAC nodes in the cluster:

LABEL=oracrsfiles /u02 ocfs2 _netdev,datavolume,nointr 0 0

Notice the "_netdev" option for mounting this file system. The _netdev mount option

is a must for OCFS2 volumes. This mount option indicates that the volume is to be

mounted after the network is started and dismounted before the network is shutdown.

Now, let's make sure that the ocfs2.ko kernel module is being loaded and that the file system will be mounted

during the boot process.

If you have been following along with the examples in this article, the actions to load the kernel module and

mount the OCFS2 file system should already be enabled. However, you should still check those options by

running the following on both Oracle RAC nodes in the cluster as the root user account:

# chkconfig --list o2cb

o2cb 0:off 1:off 2:on 3:on 4:on 5:on 6:off

The flags that I have marked in bold should be set to "on".

Check Permissions on New OCFS2 File System

Use the ls command to check ownership. The permissions should be set to 0775 with owner "oracle" and

group "oinstall".

The following tasks only need to be executed on one of nodes in the RAC cluster. I will be executing all

commands in this section from linux1 only.

Let's first check the permissions:

# ls -ld /u02

drwxr-xr-x 3 root root 3896 Aug 16 21:17 /u02

As you can see from the listing above, the oracle user account (and the oinstall group) will not be able to

write to this directory. Let's fix that:

# chown oracle:oinstall /u02

# chmod 775 /u02

Let's now go back and re-check that the permissions are correct for both Oracle RAC nodes in the cluster:

# ls -ld /u02

drwxrwxr-x 3 oracle oinstall 3896 Aug 16 21:17 /u02

Create Directory for Oracle Clusterware Files

The last mandatory task is to create the appropriate directory on the new OCFS2 file system that will be used for

the Oracle Clusterware shared files. We will also modify the permissions of this new directory to allow the

"oracle" owner and group "oinstall" read/write access.

The following tasks only need to be executed on one of nodes in the RAC cluster. I will be executing all

commands in this section from linux1 only.

# mkdir -p /u02/oradata/racdb

# chown -R oracle:oinstall /u02/oradata

# chmod -R 775 /u02/oradata

# ls -l /u02/oradata

total 0

drwxrwxr-x 2 oracle oinstall 3896 Aug 16 21:33 racdb

Install and Configure Automatic Storage Management (ASMLib 2.0)

Most of the installation and configuration procedures in this section should

be performed on both of the Oracle RAC nodes in the cluster! Creating the

ASM disks, however, will only need to be performed on a single node within

the cluster.

Introduction

In this section, we will install and configure ASMLib 2.0 which is a support library for the Automatic Storage

Management (ASM) feature of the Oracle Database. In this article, ASM will be used as the shared file system

and volume manager for Oracle Database files (data, online redo logs, control files, archived redo logs) as well

as the Flash Recovery Area.

Automatic Storage Management simplifies database administration by eliminating the need for the DBA to

directly manage potentially thousands of Oracle database files requiring only the management of groups of disks

allocated to the Oracle Database. ASM is built into the Oracle kernel and can be used for both single and

clustered instances of Oracle. All of the files and directories to be used for Oracle will be contained in a disk

group — (or for the purpose of this article, two disk groups). ASM automatically performs load balancing in

parallel across all available disk drives to prevent hot spots and maximize performance, even with rapidly

changing data usage patterns. ASMLib allows an Oracle Database using ASM more efficient and capable access

to the disk groups it is using.

Keep in mind that ASMLib is only a support library for the ASM software. The ASM software gets installed as

part of the Oracle Database installation which occurs later in this guide.

So, is ASMLib required for ASM? Not at all. In fact, there are two different methods to configure ASM on

Linux:

ASM with ASMLib I/O: This method creates all Oracle database files on raw block devices

managed by ASM using ASMLib calls. RAW devices are not required with this method as

ASMLib works with block devices.

ASM with Standard Linux I/O: This method does not make use of ASMLib. Oracle database

files are created on raw character devices managed by ASM using standard Linux I/O system

calls. You will be required to create RAW devices for all disk partitions used by ASM.

In this article, I will be using the "ASM with ASMLib I/O" method. Oracle states in Metalink Note 275315.1 that

"ASMLib was provided to enable ASM I/O to Linux disks without the limitations of the standard UNIX I/O API".

I plan on performing several tests in the future to identify the performance gains in using ASMLib. Those

performance metrics and testing details are out of scope of this article and therefore will not be discussed.

We start this section by first downloading the ASMLib drivers (ASMLib Release 2.0) specific to our Linux

kernel. We will then install and configure the ASMLib 2.0 drivers while finishing off the section with a

demonstration of how to create the ASM disks.

If you would like to learn more about Oracle ASMLib 2.0, visit

http://www.oracle.com/technology/tech/linux/asmlib/

Download the ASMLib 2.0 Packages

We start this section by downloading the latest ASMLib 2.0 libraries and the driver from OTN. At the time of

this writing, the latest release of the ASMLib driver is 2.0.5-1. Like the Oracle Cluster File System, we need to

download the appropriate version of the ASMLib driver for the Linux kernel which in my case is kernel 2.6.18-

128.el5 running on two single processor / x86 machines:

# uname -a

Linux linux1 2.6.18-128.el5 #1 SMP Wed Jan 21 10:44:23 EST 2009 i686 i686 i386 GNU/Linux

If you do not currently have an account with Oracle OTN, you will need to create one.

This is a FREE account!

Oracle ASMLib Downloads for Red Hat Enterprise Linux Server 5

32-bit (x86) Installations

oracleasm-2.6.18-128.el5-2.0.5-1.el5.i686.rpm - (Package for default kernel)

oracleasm-2.6.18-128.el5PAE-2.0.5-1.el5.i686.rpm - (Package for PAE kernel)

oracleasm-2.6.18-128.el5xen-2.0.5-1.el5.i686.rpm - (Package for xen kernel)

Next, download the following ASMLib tools:

oracleasm-support-2.1.3-1.el5.i386.rpm - (Driver support files)

oracleasmlib-2.0.4-1.el5.i386.rpm - (Userspace library)

64-bit (x86_64) Installations

oracleasm-2.6.18-128.el5-2.0.5-1.el5.x86_64.rpm - (Package for default kernel)

oracleasm-2.6.18-128.el5xen-2.0.5-1.el5.x86_64.rpm - (Package for xen kernel)

Next, download the following ASMLib tools:

oracleasm-support-2.1.3-1.el5.x86_64.rpm - (Driver support files)

oracleasmlib-2.0.4-1.el5.x86_64.rpm - (Userspace library)

Install ASMLib 2.0 Packages

This installation needs to be performed on both nodes in the RAC cluster as the root user account:

# rpm -Uvh oracleasm-2.6.18-128.el5-2.0.5-1.el5.i686.rpm \

oracleasmlib-2.0.4-1.el5.i386.rpm \

oracleasm-support-2.1.3-1.el5.i386.rpm

Preparing... ########################################### [100%]

1:oracleasm-support ########################################### [ 33%]

2:oracleasm-2.6.18-128.el########################################### [ 67%]

3:oracleasmlib ########################################### [100%]

Configure and Load the ASMLib 2.0 Packages

Now that we downloaded and installed the ASMLib 2.0 Packages for Linux, we now need to configure and load

the ASM kernel module. This task needs to be run on both nodes in the RAC cluster as the root user account:

# /etc/init.d/oracleasm configure

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library

driver. The following questions will determine whether the driver is

loaded on boot and what permissions it will have. The current values

will be shown in brackets ('[]'). Hitting without typing an

answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle

Default group to own the driver interface []: oinstall

Start Oracle ASM library driver on boot (y/n) [n]: y

Scan for Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: done

Initializing the Oracle ASMLib driver: [ OK ]

Scanning the system for Oracle ASMLib disks: [ OK ]

Create ASM Disks for Oracle

Creating the ASM disks only needs to be performed from one node in the RAC cluster

as the root user account. I will be running these commands on linux1. On the other

Oracle RAC node, you will need to perform a scandisk to recognize the new volumes.

When that is complete, you should then run the oracleasm listdisks command on

both Oracle RAC nodes to verify that all ASM disks were created and available.

In the section "Create Partitions on iSCSI Volumes", we configured (partitioned) four iSCSI volumes to be used

by ASM. ASM will be used for storing Oracle database files like online redo logs, database files, control files,

archived redo log files, and the flash recovery area. Use the local device names that were created by udev when

configuring the four ASM volumns.

If you are repeating this article using the same hardware (actually, the same shared

logical drives), you may get a failure when attempting to create the ASM disks. If you

do receive a failure, try listing all ASM disks that were used by the previous install

using:

# /etc/init.d/oracleasm listdisks

VOL1

VOL2

VOL3

VOL4

As you can see, the results show that I have four ASM volumes already defined. If you

have the four volumes already defined from a previous run, go ahead and remove them

using the following commands. After removing the previously created volumes, use the

"oracleasm createdisk" commands (below) to create the new volumes.

# /etc/init.d/oracleasm deletedisk VOL1

Removing ASM disk "VOL1" [ OK ]

# /etc/init.d/oracleasm deletedisk VOL2

Removing ASM disk "VOL2" [ OK ]

# /etc/init.d/oracleasm deletedisk VOL3

Removing ASM disk "VOL3" [ OK ]

# /etc/init.d/oracleasm deletedisk VOL4

Removing ASM disk "VOL4" [ OK ]

To create the ASM disks using the iSCSI target names to local device name mappings, type the following:

# /etc/init.d/oracleasm createdisk VOL1 /dev/iscsi/asm1/part1

Marking disk "VOL1" as an ASM disk: [ OK ]

# /etc/init.d/oracleasm createdisk VOL2 /dev/iscsi/asm2/part1

Marking disk "VOL2" as an ASM disk: [ OK ]

# /etc/init.d/oracleasm createdisk VOL3 /dev/iscsi/asm3/part1

Marking disk "VOL3" as an ASM disk: [ OK ]

# /etc/init.d/oracleasm createdisk VOL4 /dev/iscsi/asm4/part1

Marking disk "VOL4" as an ASM disk: [ OK ]

On all other nodes in the RAC cluster, you must perform a scandisk to recognize the new volumes:

# /etc/init.d/oracleasm scandisks

Scanning the system for Oracle ASMLib disks: [ OK ]

We can now test that the ASM disks were successfully created by using the following command on both nodes

in the RAC cluster as the root user account:

# /etc/init.d/oracleasm listdisks

VOL1

VOL2

VOL3

VOL4

Download Oracle RAC 10g Software

The following download procedures only need to be performed on one node

in the cluster!

Overview

The next step is to download and extract the required Oracle software packages from the Oracle Technology

Network (OTN):

If you do not currently have an account with Oracle OTN, you will need to create one.

This is a FREE account!

Oracle offers a development and testing license free of charge. No support, however, is

provided and the license does not permit production use. A full description of the

license agreement is available on OTN.

32-bit (x86) Installations

http://www.oracle.com/technology/software/products/database/oracle10g/htdocs/10201linuxsoft.html

64-bit (x86_64) Installations

http://www.oracle.com/technology/software/products/database/oracle10g/htdocs/10201linx8664soft.html

You will be downloading and extracting the required software from Oracle to only one of the Linux nodes in the

cluster—namely, linux1. You will perform all Oracle software installs from this machine. The Oracle installer

will copy the required software packages to all other nodes in the RAC configuration using the remote access

method we setup in the section Configure RAC Nodes for Remote Access using SSH.

Log in to the node that you will be performing all of the Oracle installations from (linux1) as the "oracle" user

account. In this example, you will be downloading the required Oracle software to linux1 and saving them to

/home/oracle/orainstall.

Downloading and Extracting the Software

First, download the Oracle Clusterware Release 2 (10.2.0.1.0), Oracle Database 10g Release 2 (10.2.0.1.0), and

Oracle Database 10g Companion CD Release 2 (10.2.0.1.0) software for either Linux x86 or Linux x86-64. All

downloads are available from the same page.

As the oracle user account, extract the three packages you downloaded to a temporary directory. In this

example, we will use /home/oracle/orainstall.

Extract the Oracle Clusterware package as follows:

# su - oracle

$ mkdir -p /home/oracle/orainstall

$ cd /home/oracle/orainstall

$ unzip 10201_clusterware_linux32.zip

Then extract the Oracle Database Software:

$ cd /home/oracle/orainstall

$ unzip 10201_database_linux32.zip

Finally, extract the (optional) Oracle Companion CD Software:

$ cd /home/oracle/orainstall

$ unzip 10201_companion_linux32.zip

Pre-Installation Tasks for Oracle10g Release 2

Perform the following checks on both Oracle RAC nodes in the cluster!

Before installing the Oracle Clusterware and Oracle RAC software, it is highly recommended to run the Cluster

Verification Utility (CVU) to verify the hardware and software configuration. CVU is a command-line utility

provided on the Oracle Clusterware installation media. It is responsible for performing various system checks to

assist you with confirming the Oracle RAC nodes are properly configured for Oracle Clusterware and Oracle

Real Application Clusters installation. The CVU only needs to be run from the node you will be performing the

Oracle installations from (linux1 in this article). Note that the CVU is also run automatically at the end of the

Oracle Clusterware installation as part of the Configuration Assistants process.

Prerequisites for Using Cluster Verification Utility

Install cvuqdisk RPM (RHEL Users Only)

The first pre-requisite for running the CVU pertains to users running Oracle Enterprise Linux, Red Hat Linux,

CentOS, and SuSE. If you are using any of the above listed operating systems, then you must download and

install the package cvuqdisk to both of the Oracle RAC nodes in the cluster. This means you will need to install

the cvuqdisk RPM to both linux1 and linux2. Without cvuqdisk, CVU will be unable to discover shared

disks and you will receive the error message "Package cvuqdisk not installed" when you run CVU.

The cvuqdisk RPM can be found on the Oracle Clusterware installation media in the rpm directory. For the

purpose of this article, the Oracle Clusterware media was extracted to the

/home/oracle/orainstall/clusterware directory on linux1. Note that before installing the cvuqdisk RPM,

we need to set an environment variable named CVUQDISK_GRP to point to the group that will own the cvuqdisk

utility. The default group is oinstall which is the group we are using for the oracle UNIX user account in this

article.

Locate and copy the cvuqdisk RPM from linux1 to linux2 as the "oracle" user account:

$ ssh linux2 "mkdir -p /home/oracle/orainstall/clusterware/rpm"

$ scp /home/oracle/orainstall/clusterware/rpm/cvuqdisk-1.0.1-1.rpm

linux2:/home/oracle/orainstall/clusterware/rpm

Perform the following steps as the "root" user account on both Oracle RAC nodes to install the cvuqdisk RPM:

# cd /home/oracle/orainstall/clusterware/rpm

# CVUQDISK_GRP=oinstall; export CVUQDISK_GRP

# rpm -iv cvuqdisk-1.0.1-1.rpm

Preparing packages for installation...

cvuqdisk-1.0.1-1

# ls -l /usr/sbin/cvuqdisk

-rwsr-x--- 1 root oinstall 4168 Jun 2 2005 /usr/sbin/cvuqdisk

Verify Remote Access / User Equivalence

The CVU should be run from linux1 — the node we will be performing all of the Oracle installations from.

Before running CVU, log in as the oracle user account and verify remote access / user equivalence is

configured to all nodes in the cluster. When using the secure shell method, user equivalence will need to be

enabled for the terminal shell session before attempting to run the CVU. To enable user equivalence for the

current terminal shell session, perform the following steps remembering to enter the pass phrase for each key that

you generated when prompted:

# su - oracle

$ exec /usr/bin/ssh-agent $SHELL

$ /usr/bin/ssh-add

Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx

Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

Checking Pre-Installation Tasks for CRS with CVU

Once all prerequisites for running the CVU utility have been met, we can now check that all pre-installation

tasks for Oracle Clusterware are completed by executing the following command as the "oracle" UNIX user

account (with user equivalence enabled) from linux1:

$ cd /home/oracle/orainstall/clusterware/cluvfy

$ mkdir -p jdk14

$ unzip jrepack.zip -d jdk14

$ CV_HOME=/home/oracle/orainstall/clusterware/cluvfy; export CV_HOME

$ CV_JDKHOME=/home/oracle/orainstall/clusterware/cluvfy/jdk14; export CV_JDKHOME

$ ./runcluvfy.sh stage -pre crsinst -n linux1,linux2 -verbose

Review the CVU report. Note that there are several errors you may ignore in this report.

If your system only has 1GB of RAM memory, you may receive an error during the "Total memory" check:

Check: Total memory

Node Name Available Required Comment

------------ ------------------------ ------------------------ ----------

linux2 1009.65MB (1033880KB) 1GB (1048576KB) failed

linux1 1009.65MB (1033880KB) 1GB (1048576KB) failed

Result: Total memory check failed.

As you can see from the output above, the requirement is for 1GB of memory (1048576 KB). Although your

system may have 1GB of memory installed in each of the Oracle RAC nodes, the Linux kernel is calculating it to

be 1033880 KB which comes out to be 14696 KB short. This can be considered close enough and safe to

continue with the installation. As I mentioned earlier in this article, I highly recommend both Oracle RAC nodes

have 2GB of RAM memory or higher for performance reasons.

The first error is with regards to finding a suitable set of interfaces for VIPs which can be safely ignored. This is

a bug documented in Metalink Note 338924.1:

Suitable interfaces for the private interconnect on subnet "192.168.2.0":

linux2 eth1:192.168.2.101

linux1 eth1:192.168.2.100

ERROR:

Could not find a suitable set of interfaces for VIPs.

Result: Node connectivity check failed.

As documented in the note, this error can be safely ignored.

The last set of errors that can be ignored deal with specific RPM package versions that are not required with

CentOS 5. For example:

compat-db-4.0.14-5

compat-gcc-7.3-2.96.128

compat-gcc-c++-7.3-2.96.128

compat-libstdc++-7.3-2.96.128

compat-libstdc++-devel-7.3-2.96.128

compat-libstdc++-devel-7.3-2.96.128

While these specific packages are listed as missing in the CVU report, please ensure that the correct versions of

the compat-* packages are installed on both of the Oracle RAC nodes in the cluster. For example, in CentOS 5

Update 3, these would be:

compat-gcc-34-3.4.6-4

compat-gcc-34-c++-3.4.6-4

compat-libstdc++-33-3.2.3-61

Checking the Hardware and Operating System Setup with CVU

The next CVU check to run will verify the hardware and operating system setup. Again, run the following as the

"oracle" UNIX user account from linux1:

$ cd /home/oracle/orainstall/clusterware/cluvfy

$ ./runcluvfy.sh stage -post hwos -n linux1,linux2 -verbose

Review the CVU report. As with the previous check (pre-installation tasks for CRS), the check for finding a

suitable set of interfaces for VIPs will fail and can be safely ignored

Also note you may receive warnings in the "Checking shared storage accessibility..." portion of the report:

Checking shared storage accessibility...

WARNING:

Unable to determine the sharedness of /dev/sde on nodes:

linux2,linux2,linux2,linux2,linux2,linux1,linux1,linux1,linux1,linux1

Shared storage check failed on nodes "linux2,linux1".

If this occurs, this too can be safely ignored. While we know the disks are visible and shared from both of our

Oracle RAC nodes in the cluster, the check itself may fail. Several reasons for this have been documented. The

first came from Metalink indicating that cluvfy currently does not work with devices other than SCSI devices.

This would include devices like EMC PowerPath and volume groups like those from Openfiler. At the time of

this writing, no workaround exists other than to use manual methods for detecting shared devices. Another

reason for this error was documented by Bane Radulovic at Oracle Corporation. His research shows that CVU

calls smartclt on Linux, and the problem is that smartclt does not return the serial number from our iSCSI

devices. For example, a check against /dev/sde shows:

# /usr/sbin/smartctl -i /dev/sde

smartctl version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen

Home page is http://smartmontools.sourceforge.net/

Device: Openfile Virtual disk Version: 0

Serial number:

Device type: disk

Local Time is: Mon Sep 3 02:02:53 2007 EDT

Device supports SMART and is Disabled

Temperature Warning Disabled or Not Supported

At the time of this writing, it is unknown if the Openfiler developers have plans to fix this.

Install Oracle Clusterware 10g Software

Perform the following installation procedures from only one of the Oracle

RAC nodes in the cluster (linux1)! The Oracle Clusterware software will be

installed to both of the Oracle RAC nodes in the cluster by the Oracle

Universal Installer.

Overview

We are ready to install the Cluster part of the environment - the Oracle Clusterware. In a previous section, we

downloaded and extracted the install files for Oracle Clusterware to linux1 in the directory

/home/oracle/orainstall/clusterware. This is the only node we need to perform the install from.

During the installation of Oracle Clusterware, you will be asked for the nodes involved and to configure in the

RAC cluster. During the installation phase, the OUI will copy the required software to all nodes using the remote

access we configured in the section "Configure RAC Nodes for Remote Access using SSH".

So, what exactly is the Oracle Clusterware responsible for? It contains all of the cluster and database

configuration metadata along with several system management features for RAC. It allows the DBA to register

and invite an Oracle instance (or instances) to the cluster. During normal operation, Oracle Clusterware will send

messages (via a special ping operation) to all nodes configured in the cluster, often called the "heartbeat." If the

heartbeat fails for any of the nodes, it checks with the Oracle Clusterware configuration files (on the shared disk)

to distinguish between a real node failure and a network failure.

After installing Oracle Clusterware, the Oracle Universal Installer (OUI) used to install the Oracle Database

software (next section) will automatically recognize these nodes. Like the Oracle Clusterware install you will be

performing in this section, the Oracle Database software only needs to be run from one node. The OUI will copy

the software packages to all nodes configured in the RAC cluster.

Oracle Clusterware Shared Files

The two shared files (actually file groups) used by Oracle Clusterware will be stored on the Oracle Cluster File

System, Release 2 (OFCS2) we created earlier. The two shared Oracle Clusterware file groups are:

Oracle Cluster Registry (OCR)

o File 1 : /u02/oradata/racdb/OCRFile

o File 2 : /u02/oradata/racdb/OCRFile_mirror

o Size : (2 * 100MB) = 200M

CRS Voting Disk

o File 1 : /u02/oradata/racdb/CSSFile

o File 2 : /u02/oradata/racdb/CSSFile_mirror1

o File 3 : /u02/oradata/racdb/CSSFile_mirror2

o Size : (3 * 20MB) = 60MB

It is not possible to use Automatic Storage Management (ASM) for the two shared

Oracle Clusterware files: Oracle Cluster Registry (OCR) or the CRS Voting Disk files.

The problem is that these files need to be in place and accessible BEFORE any Oracle

instances can be started. For ASM to be available, the ASM instance would need to be

run first.

Also note that the two shared files could be stored on the OCFS2, shared RAW devices,

or another vendor's clustered file system.

Verifying Terminal Shell Environment

Before starting the Oracle Universal Installer, you should first verify you are logged onto the server you will be

running the installer from (i.e. linux1) then run the xhost command as root from the console to allow X Server

connections. Next, log in as the oracle user account. If you are using a remote client to connect to the node

performing the installation (SSH or Telnet to linux1 from a workstation configured with an X Server), you will

need to set the DISPLAY variable to point to your local workstation. Finally, verify remote access / user

equivalence to all nodes in the cluster:

Verify Server and Enable X Server Access # hostname

linux1

# xhost +

access control disabled, clients can connect from any host

Log in as the oracle User Account and Set DISPLAY (if necessary) # su - oracle

$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE

$ # NODE PERFORMING THE INSTALL

$ DISPLAY=<your local workstation>:0.0

$ export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the

Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a

password. When using the secure shell method, user equivalence will need to be enabled for the terminal shell

session before attempting to run the OUI. To enable user equivalence for the current terminal shell session,

perform the following steps remembering to enter the pass phrase for the RSA key you generated when

prompted:

$ exec /usr/bin/ssh-agent $SHELL

$ /usr/bin/ssh-add

Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx

Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

$ ssh linux1 "date;hostname"

Tue Aug 18 11:44:16 EDT 2009

linux1

$ ssh linux2 "date;hostname"

Tue Aug 18 11:44:49 EDT 2009

linux2

Installing Oracle Clusterware

Perform the following tasks to install the Oracle Clusterware: $ cd ~oracle

$ /home/oracle/orainstall/clusterware/runInstaller -ignoreSysPrereqs

Screen Name Response

Welcome Screen Click Next

Specify Inventory directory and credentials Accept the default values: Inventory directory: /u01/app/oracle/oraInventory

Operating System group name: oinstall

Specify Home Details

Set the Name and Path for the ORACLE_HOME (actually the $ORA_CRS_HOME

that I will be using in this article) as follows:

Name: OraCrs10g_home

Path: /u01/app/crs

Product-Specific Prerequisite Checks

The installer will run through a series of checks to determine if the node meets the

minimum requirements for installing and configuring the Oracle Clusterware

software. If any of the checks fail, you will need to manually verify the check that

failed by clicking on the checkbox. For my installation, all checks passed with no

problems.

Click Next to continue.

Specify Cluster Configuration

Cluster Name: crs

Public Node Name Private Node Name Virtual Node Name

linux1 linux1-priv linux1-vip

linux2 linux2-priv linux2-vip

Specify Network Interface Usage

Interface Name Subnet Interface Type

eth0 192.168.1.0 Public

eth1 192.168.2.0 Private

Specify Oracle Cluster Registry (OCR) Starting with Oracle Database 10g Release 2 (10.2) with RAC, Oracle Clusterware

Location provides for the creation of a mirrored Oracle Cluster Registry (OCR) file, enhancing

cluster reliability. For the purpose of this example, I did choose to mirror the OCR file by keeping the default option of "Normal Redundancy":

Specify OCR Location: /u02/oradata/racdb/OCRFile

Specify OCR Mirror Location: /u02/oradata/racdb/OCRFile_mirror

Specify Voting Disk Location

Starting with Oracle Database 10g Release 2 (10.2) with RAC, CSS has been modified to allow you to configure CSS with multiple voting disks. In 10g Release 1

(10.1), you could configure only one voting disk. By enabling multiple voting disk

configuration, the redundant voting disks allow you to configure a RAC database with

multiple voting disks on independent shared physical disks. This option facilitates the

use of the iSCSI network protocol, and other Network Attached Storage (NAS)

storage solutions. Note that to take advantage of the benefits of multiple voting disks,

you must configure at least three voting disks. For the purpose of this example, I did

choose to mirror the voting disk by keeping the default option of "Normal

Redundancy":

Voting Disk Location: /u02/oradata/racdb/CSSFile Additional Voting Disk 1 Location: /u02/oradata/racdb/CSSFile_mirror1

Additional Voting Disk 2 Location: /u02/oradata/racdb/CSSFile_mirror2

Summary Click Install to start the installation!

Execute Configuration Scripts

orainstRoot.sh

After the installation has completed, you will be prompted to run the orainstRoot.sh

and root.sh script. Open a new console window on both Oracle RAC nodes in the

cluster, (starting with the node you are performing the install from), as the root user

account.

Navigate to the /u01/app/oracle/oraInventory directory and run orainstRoot.sh on

both nodes in the RAC cluster.

Note: After executing the orainstRoot.sh on both nodes, verify the permissions of the

file "/etc/oraInst.loc" are 644 (-rw-r--r--) and owned by root. Problems can occur

during the installation of Oracle if the oracle user account does not have read

permissions to this file - (the location of the oraInventory directory cannot be

determined). For example, during the Oracle Clusterware post-installation process

(while running the Oracle Clusterware Verification Utility), the following error will occur: "CRS is not installed on any of the nodes." If the permissions to /etc/oraInst.loc

are not set correctly, it is possible you didn't run orainstRoot.sh on both nodes before

running root.sh. Also, the umask setting may be off - it should be 0022. Run the

following on both nodes in the RAC cluster to correct this problem:

# chmod 644 /etc/oraInst.loc

# ls -l /etc/oraInst.loc

-rw-r--r-- 1 root root 63 Aug 19 10:55 /etc/oraInst.loc

root.sh

Within the same new console window on both Oracle RAC nodes in the cluster,

(starting with the node you are performing the install from), stay logged in as the root

user account.

Navigate to the /u01/app/crs directory and locate the root.sh file for each node in the

cluster - (starting with the node you are performing the install from). Run the root.sh

file on both nodes in the RAC cluster one at a time.

If the Oracle Clusterware home directory is a sub-directory of the ORACLE_BASE

directory (which should never be!), you will receive several warnings regarding

permission while running the root.sh script on both nodes. These warnings can be

safely ignored.

The root.sh script may take awhile to run.

Running root.sh on the Last Node will Fail!

After receiving any of the errors described in this section, please leave the OUI up.

Do not hit the OK button on the "Execute Configuration Scripts" dialog until all of

the issues described in this section have been resolved.

Issue #1

At the end of root.sh on the last node, vipca will fail to run with the following error:

Oracle CRS stack installed and running under init(1M)

Running vipca(silent) for configuring nodeapps

/u01/app/crs/jdk/jre//bin/java: error while loading

shared libraries: libpthread.so.0:

cannot open shared object file: No such file or directory

Note that srvctl will produce similar output until the workaround described below is

performed.

This error occurs because these releases of the Linux kernel fix an old bug in the

Linux threading that Oracle worked around using LD_ASSUME_KERNEL settings

in both vipca and srvctl, this workaround is no longer valid on OEL5 or RHEL5 or

SLES10 hence the failures.

To workaround Issue #1 above, edit vipca (in the CRS bin directory on both nodes)

to undo the setting of LD_ASSUME_KERNEL. After the IF statement around line

120, add an unset command to ensure LD_ASSUME_KERNEL is not set as follows:

if [ "$arch" = "i686" -o "$arch" = "ia64" ]

then

LD_ASSUME_KERNEL=2.4.19

export LD_ASSUME_KERNEL

fi

unset LD_ASSUME_KERNEL <<== Line to be added

Similarly for srvctl (in both the CRS and, when installed, RDBMS and ASM bin directories on both nodes), unset LD_ASSUME_KERNEL by adding one line,

around line 168 should look like this:

LD_ASSUME_KERNEL=2.4.19

export LD_ASSUME_KERNEL

unset LD_ASSUME_KERNEL <<== Line to be added

Note: Remember to re-edit these files on both nodes:

<ORA_CRS_HOME>/bin/vipca

<ORA_CRS_HOME>/bin/srvctl

<RDBMS_HOME>/bin/srvctl

<ASM_HOME>/bin/srvctl # (If exists)

after applying the 10.2.0.2 or 10.2.0.3 patch sets, as these patchset will still include

those settings unnecessary for OEL5 or RHEL5 or SLES10. This issue was raised

with development and is fixed in the 10.2.0.4 patchsets.

Also note that we are explicitly unsetting LD_ASSUME_KERNEL and not merely

commenting out its setting to handle a case where the user has it set in their

environment (login shell).

Issue #2

After resolving the issue above, manually re-run vipca (GUI) as root from the last

node in which the error occurred. Please keep in mind that vipca is a GUI and will

need to set your DISPLAY variable accordingly to your X server.

After working around Issue #1 above, vipca will now fail to run with the following

error if the VIP IP's are in a non-routable range [10.x.x.x, 172.(16-31).x.x, or

192.168.x.x]:

[root@linux2 ~]# $ORA_CRS_HOME/bin/vipca

Error 0(Native: listNetInterfaces:[3])

[Error 0(Native: listNetInterfaces:[3])]

There are several ways to workaround Issue #2. The goal to this workaround is to get

the output of "$ORA_CRS_HOME/bin/oifcfg getif" to include both public and

cluster_interconnect interfaces. If you try to run the above command, you will notice

it provides nothing which means we have some work to do!

[root@linux2 bin]# /u01/app/crs/bin/oifcfg getif

The first step is to identify the current interfaces and IP addresses:

[root@linux2 ~]# /u01/app/crs/bin/oifcfg iflist

eth0 192.168.1.0

eth1 192.168.2.0

Remember during the Oracle Clusterware install that 192.168.1.0 is my public

interface while 192.168.2.0 is the cluster_interconnect interface.

Using this information, we can manually set the public / private interfaces

accordingly using the "setif" option of the $ORA_CRS_HOME/bin/oifcfg command:

[root@linux2 ~]# /u01/app/crs/bin/oifcfg setif -global eth0/192.168.1.0:public

[root@linux2 ~]# /u01/app/crs/bin/oifcfg setif -global

eth1/192.168.2.0:cluster_interconnect

Let's know run the "/u01/app/crs/bin/oifcfg getif" command again to verify its output:

[root@linux2 ~]# /u01/app/crs/bin/oifcfg getif

eth0 192.168.1.0 global public

eth1 192.168.2.0 global cluster_interconnect

Manually Run vipca from Last Node

After resolving all of the issues above, manually re-run vipca (GUI) as root from the

last node in which the errors occurred. Please keep in mind that vipca is a GUI and

will need to set your DISPLAY variable accordingly to your X server:

[root@linux2 ~]# /u01/app/crs/bin/vipca

When the "VIP Configuration Assistant" appears, this is how I answered the screen

prompts:

Welcome: Click Next

Network interfaces: Select only the public interface - eth0

Virtual IPs for cluster nodes:

Node Name: linux1

IP Alias Name: linux1-vip IP Address: 192.168.1.200

Subnet Mask: 255.255.255.0

Node Name: linux2

IP Alias Name: linux2-vip IP Address: 192.168.1.201

Subnet Mask: 255.255.255.0

Summary: Click Finish

Configuration Assistant Progress Dialog: Click OK after configuration is complete.

Configuration Results: Click Exit

Go back to the OUI and acknowledge the "Execute Configuration scripts" dialog window.

End of installation At the end of the installation, exit from the OUI.

Verify Oracle Clusterware Installation

After the installation of Oracle Clusterware, we can run through several tests to verify the install was successful.

Run the following commands on both nodes in the RAC cluster as the oracle user account.

Check Cluster Nodes

$ $ORA_CRS_HOME/bin/olsnodes -n

linux1 1

linux2 2

Confirm Oracle Clusterware Function $ $ORA_CRS_HOME/bin/crs_stat -t -v

Name Type R/RA F/FT Target State Host

----------------------------------------------------------------------

ora.linux1.gsd application 0/5 0/0 ONLINE ONLINE linux1

ora.linux1.ons application 0/3 0/0 ONLINE ONLINE linux1

ora.linux1.vip application 0/0 0/0 ONLINE ONLINE linux1

ora.linux2.gsd application 0/5 0/0 ONLINE ONLINE linux2

ora.linux2.ons application 0/3 0/0 ONLINE ONLINE linux2

ora.linux2.vip application 0/0 0/0 ONLINE ONLINE linux2

Check CRS Status $ $ORA_CRS_HOME/bin/crsctl check crs

CSS appears healthy

CRS appears healthy

EVM appears healthy

Check Oracle Clusterware Auto-Start Scripts $ ls -l /etc/init.d/init.*

-r-xr-xr-x 1 root root 1951 Aug 19 11:03 /etc/init.d/init.crs

-r-xr-xr-x 1 root root 4699 Aug 19 11:03 /etc/init.d/init.crsd

-r-xr-xr-x 1 root root 35379 Aug 19 11:03 /etc/init.d/init.cssd

-r-xr-xr-x 1 root root 3175 Aug 19 11:03 /etc/init.d/init.evmd

Install Oracle Database 10g Software

Perform the following installation procedures from only one of the Oracle

RAC nodes in the cluster (linux1)! The Oracle Database software will be

installed to both of Oracle RAC nodes in the cluster by the Oracle Universal

Installer.

Overview

After successfully installing the Oracle Clusterware software, the next step is to install Oracle Database 10g

Release 2 (10.2.0.1.0) with RAC.

For the purpose of this example, you will forgo the "Create Database" option when

installing the software. You will, instead, create the database using the Database

Configuration Assistant (DBCA) after the install.

Like the Oracle Clusterware install (previous section), the Oracle 10g database software only needs to be run

from one node. The OUI will copy the software packages to all nodes configured in the RAC cluster.

Verifying Terminal Shell Environment

As discussed in the previous section, (Install Oracle Clusterware 10g Software), the terminal shell environment

needs to be configured for remote access and user equivalence to all nodes in the cluster before running the

Oracle Universal Installer. Note that you can utilize the same terminal shell session used in the previous section

which in this case, you do not have to perform any of the actions described below with regards to setting up

remote access and the DISPLAY variable:

Log in as the oracle User Account and Set DISPLAY (if necessary) # su - oracle

$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE

$ # NODE PERFORMING THE INSTALL

$ DISPLAY=<your local workstation>:0.0

$ export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the

Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a

password. When using the secure shell method, user equivalence will need to be enabled for the terminal shell

session before attempting to run the OUI. To enable user equivalence for the current terminal shell session,

perform the following steps remembering to enter the pass phrase for the RSA key you generated when

prompted:

$ exec /usr/bin/ssh-agent $SHELL

$ /usr/bin/ssh-add

Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx

Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

$ ssh linux1 "date;hostname"

Wed Aug 19 22:07:13 EDT 2009

linux1

$ ssh linux2 "date;hostname"

Wed Aug 19 22:07:44 EDT 2009

linux2

Run the Oracle Cluster Verification Utility

Before installing the Oracle Database Software, we should run the following database pre-installation check

using the Cluster Verification Utility (CVU).

Instructions for configuring CVU can be found in the section "Prerequisites for Using

Cluster Verification Utility discussed earlier in this article.

$ cd /home/oracle/orainstall/clusterware/cluvfy

$ CV_HOME=/home/oracle/orainstall/clusterware/cluvfy; export CV_HOME

$ CV_JDKHOME=/home/oracle/orainstall/clusterware/cluvfy/jdk14; export CV_JDKHOME

$ ./runcluvfy.sh stage -pre dbinst -n linux1,linux2 -r 10gR2 -verbose

Review the CVU report. Note that this report will contain the same errors we received when checking pre-

installation tasks for CRS — failure to find a suitable set of interfaces for VIPs and the failure to find specific

RPM packages that are not required with CentOS 5. These two errors can be safely ignored.

Install Oracle Database 10g Release 2 Software

Install the Oracle Database 10g Release 2 software as follows: $ cd ~oracle

$ /home/oracle/orainstall/database/runInstaller -ignoreSysPrereqs

Screen Name Response

Welcome Screen Click Next

Select Installation Type

Select the type of installation you would like to perform: "Enterprise Edition",

"Standard Edition", or "Custom".

If you need advanced components like Oracle Label Security or if you simply want to

customize the environment, select "Custom".

For the purpose of this article, I selected the "Custom" option.

Specify Home Details Set the Name and Path for the ORACLE_HOME as follows:

Name: OraDb10g_home1

Location: /u01/app/oracle/product/10.2.0/db_1

Specify Hardware Cluster Installation Mode

Select the Cluster Installation option then select all nodes available. Click Select All to

select all servers: linux1 and linux2.

If the installation stops here and the status of any of the RAC nodes is "Node not

reachable", perform the following checks:

Ensure the Oracle Clusterware is running on the node in question.

Ensure you are able to reach the node in question from the node you are

performing the installation from.

Available Product Components

(Custom Database Installs Only) Select the components that you plan on using for your database environment.

Product-Specific Prerequisite Checks

The installer will run through a series of checks to determine if the node meets the

minimum requirements for installing and configuring the Oracle database software. If

any of the checks fail, you will need to manually verify the check that failed by clicking on the checkbox.

It is possible to receive an error about the available swap space not meeting its

minimum requirements:

Checking available swap space requirements...

Expected result: 3036MB

Actual Result: 1983MB

In most cases, you will have the minimum required swap space (as shown above) and

this can be safely ignored. Simply click the check-box for "Checking available swap

space requirements..." and click Next to continue.

Privileged Operating System Groups

(Custom Database Installs Only)

Select the UNIX groups that will be used for each of the Oracle group names as

follows:

Database Administrator (OSDBA) Group: dba

Database Operator (OSOPER) Group: oper

Create Database Select the option to Install database Software only.

Remember that we will create the clustered database as a separate step using dbca.

Summary Click Install to start the installation!

Execute Configuration Scripts

After the installation has completed, you will be prompted to run the root.sh script. It

is important to keep in mind that the root.sh script will need to be run on both nodes in the RAC cluster one at a time starting with the node you are running the database

installation from.

First, open a new console window on the node you are installing the Oracle Database

10g software from as the root user account. For me, this was "linux1".

Navigate to the /u01/app/oracle/product/10.2.0/db_1 directory and run root.sh.

After running the root.sh script on both nodes in the cluster, go back to the OUI and

acknowledge the "Execute Configuration scripts" dialog window.

End of installation At the end of the installation, exit from the OUI.

Unset LD_ASSUME_KERNEL in SRVCTL

During the Oracle Clusterware installation in the previous section, we needed to modify both vipca and srvctl

in the CRS bin directory. Those same modifications need to be performed with the new srvctl in the Database

bin directory:

$ $ORACLE_HOME/bin/srvctl

/u01/app/oracle/product/10.2.0/db_1/jdk/jre/bin/java: error while loading

shared libraries: libpthread.so.0:

cannot open shared object file: No such file or directory

This error occurs because these releases of the Linux kernel fix an old bug in the Linux threading that Oracle

worked around using LD_ASSUME_KERNEL settings in both vipca and srvctl, this workaround is no longer valid

on OEL5 or RHEL5 or SLES10 hence the failures.

To workaround this issue, edit $ORACLE_HOME/bin/srvctl (in the Database bin directory on all nodes) to undo

the setting of LD_ASSUME_KERNEL by adding one line, around line 168:

LD_ASSUME_KERNEL=2.4.19

export LD_ASSUME_KERNEL

unset LD_ASSUME_KERNEL <<== Line to be added

Note: Remember to re-edit these files on all nodes:

<ORA_CRS_HOME>/bin/vipca

<ORA_CRS_HOME>/bin/srvctl

<RDBMS_HOME>/bin/srvctl

<ASM_HOME>/bin/srvctl # (If exists)

... after applying the 10.2.0.2 or 10.2.0.3 patch sets, as these patchset will still include those settings unnecessary

for OEL5 or RHEL5 or SLES10. This issue was raised with development and is fixed in the 10.2.0.4 patchsets.

Also note that we are explicitly unsetting LD_ASSUME_KERNEL and not merely commenting out its setting to

handle a case where the user has it set in their environment (login shell).

Install Oracle Database 10g Companion CD Software

Perform the following installation procedures from only one of the Oracle

RAC nodes in the cluster (linux1)! The Oracle Database 10g Companion

CD software will be installed to both of Oracle RAC nodes in the cluster by

the Oracle Universal Installer.

Overview

After successfully installing the Oracle Database software, the next step is to install the Oracle Database 10g

Companion CD Release 2 software (10.2.0.1.0).

Please keep in mind that this is an optional step. For the purpose of this article, my testing database will often

make use of the Java Virtual Machine (Java VM) and Oracle interMedia and therefore will require the

installation of the Oracle Database 10g Companion CD. The type of installation to perform will be the Oracle

Database 10g Products installation type.

This installation type includes the Natively Compiled Java Libraries (NCOMP) files to improve Java

performance. If you do not install the NCOMP files, the ORA-29558:JAccelerator (NCOMP) not installed

error occurs when a database that uses Java VM is upgraded to the patch release.

Like the Oracle Clusterware and Database install (previous sections), the Oracle Database 10g Companion CD

software only needs to be run from one node. The OUI will copy the software packages to all nodes configured

in the RAC cluster.

Verifying Terminal Shell Environment

As discussed in the previous section, (Install Oracle Database 10g Software), the terminal shell environment

needs to be configured for remote access and user equivalence to all nodes in the cluster before running the

Oracle Universal Installer. Note that you can utilize the same terminal shell session used in the previous section

which in this case, you do not have to perform any of the actions described below with regards to setting up

remote access and the DISPLAY variable:

Log in as the oracle User Account and Set DISPLAY (if necessary) # su - oracle

$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE

$ # NODE PERFORMING THE INSTALL

$ DISPLAY=<your local workstation>:0.0

$ export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the

Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a

password. When using the secure shell method, user equivalence will need to be enabled for the terminal shell

session before attempting to run the OUI. To enable user equivalence for the current terminal shell session,

perform the following steps remembering to enter the pass phrase for the RSA key you generated when

prompted:

$ exec /usr/bin/ssh-agent $SHELL

$ /usr/bin/ssh-add

Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx

Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

$ ssh linux1 "date;hostname"

Wed Aug 19 22:07:13 EDT 2009

linux1

$ ssh linux2 "date;hostname"

Wed Aug 19 22:07:44 EDT 2009

linux2

Install Oracle Database 10g Companion CD Software

Install the Oracle Database 10g Companion CD Software as follows:

$ cd ~oracle

$ /home/oracle/orainstall/companion/runInstaller -ignoreSysPrereqs

Screen Name Response

Welcome Screen Click Next

Select a Product to Install Select the Oracle Database 10g Products 10.2.0.1.0 option.

Specify Home Details

Set the destination for the ORACLE_HOME Name and Path to that of the previous

Oracle Database 10g software install as follows:

Name: OraDb10g_home1

Path: /u01/app/oracle/product/10.2.0/db_1

Specify Hardware Cluster Installation Mode

The Cluster Installation option will be selected along with all of the available nodes in

the cluster by default. Stay with these default options and click Next to continue.

If the installation stops here and the status of any of the RAC nodes is "Node not

reachable", perform the following checks:

Ensure the Oracle Clusterware is running on the node in question.

Ensure you are able to reach the node in question from the node you are

performing the installation from.

Product-Specific Prerequisite Checks

The installer will run through a series of checks to determine if the node meets the

minimum requirements for installing and configuring the Oracle Database 10g

Companion CD Software. If any of the checks fail, you will need to manually verify

the check that failed by clicking on the checkbox. For my installation, all checks passed with no problems.

Click Next to continue.

Summary On the Summary screen, click Install to start the installation!

End of installation At the end of the installation, exit from the OUI.

Create TNS Listener Process

Perform the following configuration procedures from only one of the Oracle

RAC nodes in the cluster (linux1)! The Network Configuration Assistant

(NETCA) will setup the TNS listener in a clustered configuration on both of

Oracle RAC nodes in the cluster.

Overview

The DBCA requires the Oracle TNS Listener process to be configured and running on all nodes in the RAC

cluster before it can create the clustered database.

The process of creating the TNS listener only needs to be performed on one node in the cluster. All changes will

be made and replicated to all nodes in the cluster. On one of the nodes (I will be using linux1) bring up the

NETCA and run through the process of creating a new TNS listener process and also configure the node for local

access.

Verifying Terminal Shell Environment

As discussed in the previous section, (Install Oracle Database 10g Companion CD Software), the terminal shell

environment needs to be configured for remote access and user equivalence to all nodes in the cluster before

running the Network Configuration Assistant (NETCA). Note that you can utilize the same terminal shell session

used in the previous section which in this case, you do not have to perform any of the actions described below

with regards to setting up remote access and the DISPLAY variable:

Log in as the oracle User Account and Set DISPLAY (if necessary) # su - oracle

$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE

$ # NODE PERFORMING THE INSTALL

$ DISPLAY=<your local workstation>:0.0

$ export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the

Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a

password. When using the secure shell method, user equivalence will need to be enabled for the terminal shell

session before attempting to run the OUI. To enable user equivalence for the current terminal shell session,

perform the following steps remembering to enter the pass phrase for the RSA key you generated when

prompted:

$ exec /usr/bin/ssh-agent $SHELL

$ /usr/bin/ssh-add

Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx

Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

$ ssh linux1 "date;hostname"

Wed Aug 19 22:07:13 EDT 2009

linux1

$ ssh linux2 "date;hostname"

Wed Aug 19 22:07:44 EDT 2009

linux2

Run the Network Configuration Assistant

To start the NETCA, run the following:

$ netca &

The following table walks you through the process of creating a new Oracle listener for our RAC environment.

Screen Name Response

Select the Type of Oracle

Net Services Configuration Select Cluster configuration

Select the nodes to configure Select all of the nodes: linux1 and linux2.

Type of Configuration Select Listener configuration.

Listener Configuration -

Next 6 Screens

The following screens are now like any other normal listener configuration. You

can simply accept the default parameters for the next six screens:

What do you want to do: Add

Listener name: LISTENER

Selected protocols: TCP

Port number: 1521 Configure another listener: No

Listener configuration complete! [ Next ]

You will be returned to this Welcome (Type of Configuration) Screen.

Type of Configuration Select Naming Methods configuration.

Naming Methods Configuration

The following screens are:

Selected Naming Methods: Local Naming

Naming Methods configuration complete! [ Next ]

You will be returned to this Welcome (Type of Configuration) Screen.

Type of Configuration Click Finish to exit the NETCA.

Verify TNS Listener Configuration

The Oracle TNS listener process should now be running on both nodes in the RAC cluster: $ hostname

linux1

$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'

LISTENER_LINUX1

$ $ORA_CRS_HOME/bin/crs_stat ora.linux1.LISTENER_LINUX1.lsnr

NAME=ora.linux1.LISTENER_LINUX1.lsnr

TYPE=application

TARGET=ONLINE

STATE=ONLINE on linux1

=====================

$ hostname

linux2

$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'

LISTENER_LINUX2

$ $ORA_CRS_HOME/bin/crs_stat ora.linux2.LISTENER_LINUX2.lsnr

NAME=ora.linux2.LISTENER_LINUX2.lsnr

TYPE=application

TARGET=ONLINE

STATE=ONLINE on linux2

Create the Oracle Cluster Database

The database creation process should only be performed from one of the

Oracle RAC nodes in the cluster (linux1)!

Overview

We will be using the Oracle Database Configuration Assistant (DBCA) to create the clustered database.

Before executing the Database Configuration Assistant, make sure that $ORACLE_HOME and $PATH are set

appropriately for the $ORACLE_BASE/product/10.2.0/db_1 environment.

You should also verify that all services we have installed up to this point (Oracle TNS listener, Oracle

Clusterware processes, etc.) are running before attempting to start the clustered database creation process.

Verifying Terminal Shell Environment

As discussed in the previous section, (Install Oracle Database 10g Companion CD Software), the terminal shell

environment needs to be configured for remote access and user equivalence to all nodes in the cluster before

running the Network Configuration Assistant (NETCA). Note that you can utilize the same terminal shell session

used in the previous section which in this case, you do not have to perform any of the actions described below

with regards to setting up remote access and the DISPLAY variable:

Log in as the oracle User Account and Set DISPLAY (if necessary) # su - oracle

$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE

$ # NODE PERFORMING THE INSTALL

$ DISPLAY=<your local workstation>:0.0

$ export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the

Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a

password. When using the secure shell method, user equivalence will need to be enabled for the terminal shell

session before attempting to run the OUI. To enable user equivalence for the current terminal shell session,

perform the following steps remembering to enter the pass phrase for the RSA key you generated when

prompted:

$ exec /usr/bin/ssh-agent $SHELL

$ /usr/bin/ssh-add

Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx

Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

$ ssh linux1 "date;hostname"

Wed Aug 19 22:07:13 EDT 2009

linux1

$ ssh linux2 "date;hostname"

Wed Aug 19 22:07:44 EDT 2009

linux2

Run the Oracle Cluster Verification Utility

Before creating the Oracle clustered database, we should run the following database configuration check using

the Cluster Verification Utility (CVU).

Instructions for configuring CVU can be found in the section "Prerequisites for Using

Cluster Verification Utility discussed earlier in this article.

$ cd /home/oracle/orainstall/clusterware/cluvfy

$ CV_HOME=/home/oracle/orainstall/clusterware/cluvfy; export CV_HOME

$ CV_JDKHOME=/home/oracle/orainstall/clusterware/cluvfy/jdk14; export CV_JDKHOME

$ ./runcluvfy.sh stage -pre dbcfg -n linux1,linux2 -d ${ORACLE_HOME} -verbose

Review the CVU report. Note that this report will contain the same error we received when checking pre-

installation tasks for CRS — failure to find a suitable set of interfaces for VIPs. This error can be safely ignored.

Create the Clustered Database

To start the database creation process, run the following:

$ dbca &

Screen Name Response

Welcome Screen Select Oracle Real Application Clusters database.

Operations Select Create a Database.

Node Selection Click the Select All button to select all servers: linux1 and linux2.

Database Templates Select Custom Database

Database Identification Select: Global Database Name: racdb.idevelopment.info

SID Prefix: racdb

I used idevelopment.info for the database domain. You may use any domain. Keep in mind that this domain does not have to be a valid DNS domain.

Management Option Leave the default options here which is to Configure the Database with Enterprise

Manager / Use Database Control for Database Management

Database Credentials I selected to Use the Same Password for All Accounts. Enter the password (twice) and

make sure the password does not start with a digit number.

Storage Options For this article, we will select to use Automatic Storage Management (ASM).

Create ASM Instance

Supply the SYS password to use for the new ASM instance.

Also, starting with Oracle10g Release 2, the ASM instance server parameter file (SPFILE) needs to be on a shared disk. You will need to modify the default entry for

"Create server parameter file (SPFILE)" to reside on the OCFS2 partition as follows:

/u02/oradata/racdb/dbs/spfile+ASM.ora. All other options can stay at their defaults.

You will then be prompted with a dialog box asking if you want to create and start the

ASM instance. Select the OK button to acknowledge this dialog.

The OUI will now create and start the ASM instance on all nodes in the RAC cluster.

ASM Disk Groups

To start, click the Create New button. This will bring up the "Create Disk Group"

window with the four volumes we configured earlier using ASMLib.

If the volumes we created earlier in this article do not show up in the "Select Member Disks" window: (ORCL:VOL1, ORCL:VOL2, ORCL:VOL3, and ORCL:VOL4) then

click on the "Change Disk Discovery Path" button and input "ORCL:VOL*".

For the first "Disk Group Name" I used the string RACDB_DATA1. Select the first two

ASM volumes (ORCL:VOL1 and ORCL:VOL2) in the "Select Member Disks" window.

Keep the "Redundancy" setting to Normal.

After verifying all values in this window are correct, click the OK button. This will

present the "ASM Disk Group Creation" dialog. When the ASM Disk Group Creation

process is finished, you will be returned to the "ASM Disk Groups" windows.

Click the Create New button again. For the second "Disk Group Name", I used the string

FLASH_RECOVERY_AREA. Select the last two ASM volumes (ORCL:VOL3 and

ORCL:VOL4) in the "Select Member Disks" window. Keep the "Redundancy" setting to

Normal.

After verifying all values in this window are correct, click the OK button. This will

present the "ASM Disk Group Creation" dialog.

When the ASM Disk Group Creation process is finished, you will be returned to the "ASM Disk Groups" window with two disk groups created and selected. Select only one

of the disk groups by using the checkbox next to the newly created Disk Group Name

RACDB_DATA1 (ensure that the disk group for FLASH_RECOVERY_AREA is not

selected) and click Next to continue.

Database File Locations I selected to use the default which is Use Oracle-Managed Files:

Database Area: +RACDB_DATA1

Recovery Configuration

Check the option for Specify Flash Recovery Area.

For the Flash Recovery Area, click the [Browse] button and select the disk group name

+FLASH_RECOVERY_AREA.

My disk group has a size of about 17GB. When defining the Flash Recovery Area size,

use the entire volume minus 10% for overhead — (17-10%=15 GB). I used a "Flash

Recovery Area Size" of 15 GB (15360 MB).

Database Content I left all of the Database Components (and destination tablespaces) set to their default

value, although it is perfectly OK to select the Sample Schemas. This option is available

since we installed the Oracle Companion CD software.

Database Services For this test configuration, click Add, and enter the Service Name: racdb_srvc. Leave both instances set to Preferred and for the "TAF Policy" select Basic.

Initialization Parameters Change any parameters for your environment. I left them all at their default settings.

Database Storage Change any parameters for your environment. I left them all at their default settings.

Creation Options

Keep the default option Create Database selected.

I also always select to "Generate Database Creation Scripts".

Click Finish to start the database creation process. After acknowledging the database

creation report and script generation dialog, the database creation will start.

Click OK on the "Summary" screen.

End of Database Creation

At the end of the database creation, exit from the DBCA.

Note: When exiting the DBCA you will not receive any feedback from the dialog window for around 30-60 seconds. After awhile, another dialog will come up indicating

that it is starting all Oracle instances and HA service "racdb_srvc". This may take

several minutes to complete. When finished, all windows and dialog boxes will

disappear.

When the Oracle Database Configuration Assistant has completed, you will have a fully functional Oracle RAC

cluster running!

$ $ORA_CRS_HOME/bin/crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora....SM1.asm application ONLINE ONLINE linux1

ora....X1.lsnr application ONLINE ONLINE linux1

ora.linux1.gsd application ONLINE ONLINE linux1

ora.linux1.ons application ONLINE ONLINE linux1

ora.linux1.vip application ONLINE ONLINE linux1

ora....SM2.asm application ONLINE ONLINE linux2

ora....X2.lsnr application ONLINE ONLINE linux2

ora.linux2.gsd application ONLINE ONLINE linux2

ora.linux2.ons application ONLINE ONLINE linux2

ora.linux2.vip application ONLINE ONLINE linux2

ora.racdb.db application ONLINE ONLINE linux1

ora....b1.inst application ONLINE ONLINE linux1

ora....b2.inst application ONLINE ONLINE linux2

ora....srvc.cs application ONLINE ONLINE linux1

ora....db1.srv application ONLINE ONLINE linux1

ora....db2.srv application ONLINE ONLINE linux2

Create the racdb_srvc Service

During the creation of the Oracle clustered database, we added a service named "racdb_srvc" that will be used

to connect to the database with TAF enabled. During several of my installs, the service was added to the

tnsnames.ora, but was never updated as a service for each Oracle instance.

Use the following to verify the racdb_srvc service was successfully added:

SQL> show parameter service

NAME TYPE VALUE

-------------------- ----------- --------------------------------

service_names string racdb.idevelopment.info, racdb_srvc

If the only service defined was for racdb.idevelopment.info, then you will need to manually add the service

to both instances:

SQL> show parameter service

NAME TYPE VALUE

-------------------- ----------- --------------------------

service_names string racdb.idevelopment.info

SQL> alter system set service_names =

2 'racdb.idevelopment.info, racdb_srvc.idevelopment.info' scope=both;

Post-Installation Tasks - (Optional)

This chapter describes several optional tasks that can be applied to your new Oracle 10g environment in order to

enhance availability as well as database management.

Re-compile Invalid Objects

Run the utlrp.sql script to recompile all invalid PL/SQL packages now instead of when the packages are

accessed for the first time. This step is optional but recommended.

$ sqlplus / as sysdba

SQL> @?/rdbms/admin/utlrp.sql

Enabling Archive Logs in a RAC Environment

Whether a single instance or clustered database, Oracle tracks and logs all changes to database blocks in online

redolog files. In an Oracle RAC environment, each instance will have its own set of online redolog files known

as a thread. Each Oracle instance will use its group of online redologs in a circular manner. Once an online

redolog fills, Oracle moves to the next one. If the database is in "Archive Log Mode", Oracle will make a copy

of the online redo log before it gets reused. A thread must contain at least two online redologs (or online redolog

groups). The same holds true for a single instance configuration. The single instance must contain at least two

online redologs (or online redolog groups).

The size of an online redolog file is completely independent of another intances' redolog size. Although in most

configurations the size is the same, it may be different depending on the workload and backup / recovery

considerations for each node. It is also worth mentioning that each instance has exclusive write access to its own

online redolog files. In a correctly configured RAC environment, however, each instance can read another

instance's current online redolog file to perform instance recovery if that instance was terminated abnormally. It

is therefore a requirement that online redo logs be located on a shared storage device (just like the database files).

As already mentioned, Oracle writes to its online redolog files in a circular manner. When the current online

redolog fills, Oracle will switch to the next one. To facilitate media recovery, Oracle allows the DBA to put the

database into "Archive Log Mode" which makes a copy of the online redolog after it fills (and before it gets

reused). This is a process known as archiving.

The Database Configuration Assistant (DBCA) allows users to configure a new database to be in archive log

mode, however most DBA's opt to bypass this option during initial database creation. In cases like this where the

database is in no archive log mode, it is a simple task to put the database into archive log mode. Note however

that this will require a short database outage. From one of the nodes in the Oracle RAC configuration, use the

following tasks to put a RAC enabled database into archive log mode. For the purpose of this article, I will use

the node linux1 which runs the racdb1 instance:

1. Log in to one of the nodes (i.e. linux1) and disable the cluster instance parameter by setting

cluster_database to FALSE from the current instance: 2. $ sqlplus / as sysdba

SQL> alter system set cluster_database=false scope=spfile sid='racdb1';

3. Shutdown all instances accessing the clustered database:

$ srvctl stop database -d racdb

4. Using the local instance, MOUNT the database: 5. $ sqlplus / as sysdba

SQL> startup mount

6. Enable archiving:

SQL> alter database archivelog;

7. Re-enable support for clustering by modifying the instance parameter cluster_database to TRUE

from the current instance:

SQL> alter system set cluster_database=true scope=spfile sid='racdb1';

8. Shutdown the local instance:

SQL> shutdown immediate

9. Bring all instance back up using srvctl:

$ srvctl start database -d racdb

10. (Optional) Bring any services (i.e. TAF) back up using srvctl:

$ srvctl start service -d racdb

11. Log in to the local instance and verify Archive Log Mode is enabled: 12. $ sqlplus / as sysdba

13. SQL> archive log list

14. Database log mode Archive Mode

15. Automatic archival Enabled

16. Archive destination USE_DB_RECOVERY_FILE_DEST

17. Oldest online log sequence 83

18. Next log sequence to archive 84

Current log sequence 84

After enabling Archive Log Mode, each instance in the RAC configuration can automatically archive redologs!

Download and Install Custom Oracle Database Scripts

DBA's rely on Oracle's data dictionary views and dynamic performance views in order to support and better

manage their databases. Although these views provide a simple and easy mechanism to query critical

information regarding the database, it helps to have a collection of accurate and readily available SQL scripts to

query these views.

In this section you will download and install a collection of Oracle DBA scripts that can be used to manage

many aspects of your database including space management, performance, backups, security, and session

management. The Oracle DBA scripts archive can be downloaded using the following link

http://www.idevelopment.info/data/Oracle/DBA_scripts/dba_scripts_archive_Oracle.zip. As the oracle user

account, download the dba_scripts_archive_Oracle.zip archive to the $ORACLE_BASE directory of each

node in the cluster. For the purpose of this example, the dba_scripts_archive_Oracle.zip archive will be

copied to /u01/app/oracle. Next, unzip the archive file to the $ORACLE_BASE directory.

For example, perform the following on both nodes in the Oracle RAC cluster as the oracle user account:

$ mv dba_scripts_archive_Oracle.zip /u01/app/oracle

$ cd /u01/app/oracle

$ unzip dba_scripts_archive_Oracle.zip

The final step is to verify (or set) the appropriate environment variable for the current UNIX shell to ensure the

Oracle SQL scripts can be run from SQL*Plus while in any directory. For UNIX verify the following

environment variable is set and included in your login shell script:

ORACLE_PATH=$ORACLE_BASE/dba_scripts/common/sql:.:$ORACLE_HOME/rdbms/admin

export ORACLE_PATH

Note that the ORACLE_PATH environment variable should already be set in the

.bash_profile login script that was created in the section Create Login Script for

oracle User Account.

Now that the Oracle DBA scripts have been unzipped and the UNIX environment variable ($ORACLE_PATH) has

been set to the appropriate directory, you should now be able to run any of the SQL scripts in your

$ORACLE_BASE/dba_scripts/common/sql while logged into SQL*Plus. For example, to query tablespace

information while logged into the Oracle database as a DBA user:

SQL> @dba_tablespaces

Status Tablespace Name TS Type Ext. Mgt. Seg. Mgt. Tablespace Size Used (in

bytes) Pct. Used

--------- --------------- ------------ ---------- --------- ------------------ -----------

------- ---------

ONLINE UNDOTBS1 UNDO LOCAL MANUAL 1,283,457,024

9,043,968 1

ONLINE SYSAUX PERMANENT LOCAL AUTO 524,288,000

378,732,544 72

ONLINE USERS PERMANENT LOCAL AUTO 2,147,483,648

321,257,472 15

ONLINE SYSTEM PERMANENT LOCAL MANUAL 838,860,800

505,544,704 60

ONLINE INDX PERMANENT LOCAL AUTO 1,073,741,824

65,536 0

ONLINE UNDOTBS2 UNDO LOCAL MANUAL 1,283,457,024

22,282,240 2

ONLINE TEMP TEMPORARY LOCAL MANUAL 1,073,741,824

92,274,688 9

------------------ -----------

------- ---------

avg

23

sum 8,225,030,144

1,329,201,152

7 rows selected.

To obtain a list of all available Oracle DBA scripts while logged into SQL*Plus, run the help.sql script:

SQL> @help.sql

========================================

Automatic Shared Memory Management

========================================

asmm_components.sql

========================================

Automatic Storage Management

========================================

asm_alias.sql

asm_clients.sql

asm_diskgroups.sql

asm_disks.sql

asm_disks_perf.sql

asm_drop_files.sql

asm_files.sql

asm_files2.sql

asm_templates.sql

< --- SNIP --- >

perf_top_sql_by_buffer_gets.sql

perf_top_sql_by_disk_reads.sql

========================================

Workspace Manager

========================================

wm_create_workspace.sql

wm_disable_versioning.sql

wm_enable_versioning.sql

wm_freeze_workspace.sql

wm_get_workspace.sql

wm_goto_workspace.sql

wm_merge_workspace.sql

wm_refresh_workspace.sql

wm_remove_workspace.sql

wm_unfreeze_workspace.sql

wm_workspaces.sql

Create Shared Oracle Password Files

In this section, I present the steps required to configure a shared Oracle password file between all instances in the

Oracle clustered database. The password file for the database in UNIX is located at

$ORACLE_HOME/dbs/orapw<ORACLE_SID> for each instance and contains a list of all database users that have

SYSDBA privileges. When a database user is granted the SYSDBA role, the instance records this in the database

password file for the instance you are logged into. But what about the other instances in the cluster? The

database password file on other instances do not get updated and will not contain the user who was just granted

the SYSDBA role. Therefore a program (like RMAN) that tries to log in as this new user with SYSDBA privileges

will fail if it tries to use an instance with a password file that does not contain his or her name.

To resolve this problem, a common solution is to place a single database password file on a shared / clustered

file system and then create symbolic links from each of the instances to this single version of the database

password file. Since the environment described in this article makes use of the Oracle Clustered File System

(OCFS2), we will use it to store the single version of the database password file.

In this section, we will also be including the Oracle password file for the ASM instance.

1. Create the database password directory on the clustered file system mounted on

/u02/oradata/racdb. Perform the following from only one node in the cluster as the oracle

user account - (linux1):

$ mkdir -p /u02/oradata/racdb/dbs

2. From one node in the cluster (linux1), move the database password files to the database

password directory on the clustered file system. Chose a node that contains a database password

file that has the most recent SYSDBA additions. In most cases, this will not matter since any

missing entries can be easily added by granting them the SYSDBA role - (plus the fact that this is a

fresh install and unlikely you created any SYSDBA users at this point!). Note that the database

server does not need to be shutdown while performing the following actions. From linux1 as the

oracle user account: 3. $ mv $ORACLE_HOME/dbs/orapw+ASM1 /u02/oradata/racdb/dbs/orapw+ASM 4. $ mv $ORACLE_HOME/dbs/orapwracdb1 /u02/oradata/racdb/dbs/orapwracdb 5. 6. $ ln -s /u02/oradata/racdb/dbs/orapw+ASM $ORACLE_HOME/dbs/orapw+ASM1

$ ln -s /u02/oradata/racdb/dbs/orapwracdb $ORACLE_HOME/dbs/orapwracdb1

7. From the second node in the cluster (linux2): 8. $ rm $ORACLE_HOME/dbs/orapw+ASM2 9. $ rm $ORACLE_HOME/dbs/orapwracdb2 10.

11. $ ln -s /u02/oradata/racdb/dbs/orapw+ASM $ORACLE_HOME/dbs/orapw+ASM2

$ ln -s /u02/oradata/racdb/dbs/orapwracdb $ORACLE_HOME/dbs/orapwracdb2

Now, when a user is granted the SYSDBA role, all instances will have access to the same password file:

SQL> GRANT sysdba TO scott;

Verify TNS Networking Files

Ensure that the TNS networking files are configured on both Oracle RAC

nodes in the cluster!

listener.ora

We already covered how to create a TNS listener configuration file (listener.ora) for a clustered environment

in the section Create TNS Listener Process. The listener.ora file should be properly configured and no

modifications should be needed.

For clarity, I included a copy of the listener.ora file from my node linux1:

listener.ora

# listener.ora.linux1 Network Configuration File:

# /u01/app/oracle/product/10.2.0/db_1/network/admin/listener.ora.linux1

# Generated by Oracle configuration tools.

LISTENER_LINUX1 =

(DESCRIPTION_LIST =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521)(IP =

FIRST))

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.100)(PORT = 1521)(IP

= FIRST))

)

)

SID_LIST_LISTENER_LINUX1 =

(SID_LIST =

(SID_DESC =

(SID_NAME = PLSExtProc)

(ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1)

(PROGRAM = extproc)

)

)

tnsnames.ora

Here is a copy of my tnsnames.ora file that was configured by Oracle and can be used for testing the

Transparent Application Failover (TAF). This file should already be configured on both of the Oracle RAC

nodes in the RAC cluster.

You can include any of these entries on other client machines that need access to the clustered database.

tnsnames.ora

# tnsnames.ora Network Configuration File:

# /u01/app/oracle/product/10.2.0/db_1/network/admin/tnsnames.ora

# Generated by Oracle configuration tools.

RACDB =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521))

(ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521))

(LOAD_BALANCE = yes)

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = racdb.idevelopment.info)

)

)

RACDB1 =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = racdb.idevelopment.info)

(INSTANCE_NAME = racdb1)

)

)

RACDB2 =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = racdb.idevelopment.info)

(INSTANCE_NAME = racdb2)

)

)

RACDB_SRVC =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521))

(ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521))

(LOAD_BALANCE = yes)

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = racdb_srvc.idevelopment.info)

(FAILOVER_MODE =

(TYPE = SELECT)

(METHOD = BASIC)

(RETRIES = 180)

(DELAY = 5)

)

)

)

LISTENERS_RACDB =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521))

(ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521))

)

EXTPROC_CONNECTION_DATA =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC0))

)

(CONNECT_DATA =

(SID = PLSExtProc)

(PRESENTATION = RO)

)

)

Connecting to Clustered Database From an External Client

This is an optional step, but I like to perform it in order to verify my TNS files are configured correctly. Use

another machine (i.e. a Windows machine connected to the network) that has Oracle installed and add the TNS

entries (in the tnsnames.ora) from either of the nodes in the cluster that were created for the clustered database.

Important Note: Verify that the machine you are connecting from can resolve all host names exactly how they

appear in the listener.ora and tnsnames.ora files. For the purpose of this document, the machine you are

connecting from should be able to resolve the following host names in the local hosts file or through DNS:

192.168.1.100 linux1

192.168.1.101 linux2

192.168.1.200 linux1-vip

192.168.1.201 linux2-vip

Try to connect to the clustered database using all available service names defined in the tnsnames.ora file:

C:\> sqlplus system/manager@racdb2

C:\> sqlplus system/manager@racdb1

C:\> sqlplus system/manager@racdb_srvc

C:\> sqlplus system/manager@racdb

Create / Alter Tablespaces

When creating the clustered database, we left all tablespaces set to their default size. If you are using a large

drive for the shared storage, you may want to make a sizable testing database.

Below are several optional SQL commands for modifying and creating all tablespaces for the test database.

Note: Please keep in mind that the database file names (OMF files) being listed in these examples may differ

from what the Oracle Database Configuration Assistant (DBCA) creates for your environment. When working

through this section, substitute the data file names that were created in your environment where appropriate. The

following query can be used to determine the file names for your environment:

SQL> select tablespace_name, file_name

2 from dba_data_files

3 union

4 select tablespace_name, file_name

5 from dba_temp_files;

TABLESPACE_NAME FILE_NAME

--------------- --------------------------------------------------

EXAMPLE +RACDB_DATA1/racdb/datafile/example.263.695345467

SYSAUX +RACDB_DATA1/racdb/datafile/sysaux.261.695345451

SYSTEM +RACDB_DATA1/racdb/datafile/system.259.695345415

TEMP +RACDB_DATA1/racdb/tempfile/temp.262.695345459

UNDOTBS1 +RACDB_DATA1/racdb/datafile/undotbs1.260.695345441

UNDOTBS2 +RACDB_DATA1/racdb/datafile/undotbs2.264.695345475

USERS +RACDB_DATA1/racdb/datafile/users.265.695345485

$ sqlplus "/ as sysdba"

SQL> create user scott identified by tiger default tablespace users;

SQL> grant dba, resource, connect to scott;

SQL> alter database datafile '+RACDB_DATA1/racdb/datafile/users.265.695345485' resize

1024m;

SQL> alter tablespace users add datafile '+RACDB_DATA1' size 1024m autoextend off;

SQL> create tablespace indx datafile '+RACDB_DATA1' size 1024m

2 autoextend on next 50m maxsize unlimited

3 extent management local autoallocate

4 segment space management auto;

SQL> alter database datafile '+RACDB_DATA1/racdb/datafile/system.259.695345415' resize

800m;

SQL> alter database datafile '+RACDB_DATA1/racdb/datafile/sysaux.261.695345451' resize

500m;

SQL> alter tablespace undotbs1 add datafile '+RACDB_DATA1' size 1024m

2 autoextend on next 50m maxsize 2048m;

SQL> alter tablespace undotbs2 add datafile '+RACDB_DATA1' size 1024m

2 autoextend on next 50m maxsize 2048m;

SQL> alter database tempfile '+RACDB_DATA1/racdb/tempfile/temp.262.695345459' resize

1024m;

Here is a snapshot of the tablespaces I have defined for my test database environment:

Status Tablespace Name TS Type Ext. Mgt. Seg. Mgt. Tablespace Size Used (in

bytes) Pct. Used

--------- --------------- ------------ ---------- --------- ------------------ -----------

------- ---------

ONLINE UNDOTBS1 UNDO LOCAL MANUAL 1,283,457,024

132,972,544 10

ONLINE SYSAUX PERMANENT LOCAL AUTO 524,288,000

270,336,000 52

ONLINE USERS PERMANENT LOCAL AUTO 2,147,483,648

131,072 0

ONLINE SYSTEM PERMANENT LOCAL MANUAL 838,860,800

503,971,840 60

ONLINE EXAMPLE PERMANENT LOCAL AUTO 157,286,400

83,820,544 53

ONLINE INDX PERMANENT LOCAL AUTO 1,073,741,824

65,536 0

ONLINE UNDOTBS2 UNDO LOCAL MANUAL 1,283,457,024

1,966,080 0

ONLINE TEMP TEMPORARY LOCAL MANUAL 1,073,741,824

25,165,824 2

------------------ -----------

------- ---------

avg

22

sum 8,382,316,544

1,018,429,440

8 rows selected.

Verify the RAC Cluster & Database Configuration

The following RAC verification checks should be performed on both Oracle

RAC nodes in the cluster! For this article, however, I will only be

performing checks from linux1.

Overview

This section provides several srvctl commands and SQL queries that can be used to validate your Oracle RAC

configuration.

There are five node-level tasks defined for SRVCTL:

Adding and deleting node level applications.

Setting and unsetting the environment for node-level applications.

Administering node applications.

Administering ASM instances.

Starting and stopping a group of programs that includes virtual IP addresses,

listeners, Oracle Notification Services, and Oracle Enterprise Manager agents

(for maintenance purposes).

Status of all instances and services

$ srvctl status database -d racdb

Instance racdb1 is running on node linux1

Instance racdb2 is running on node linux2

Status of a single instance

$ srvctl status instance -d racdb -i racdb2

Instance racdb2 is running on node linux2

Status of a named service globally across the database

$ srvctl status service -d racdb -s racdb_srvc

Service racdb_srvc is running on instance(s) racdb1, racdb2

Status of node applications on a particular node

$ srvctl status nodeapps -n linux1

VIP is running on node: linux1

GSD is running on node: linux1

Listener is running on node: linux1

ONS daemon is running on node: linux1

Status of an ASM instance

$ srvctl status asm -n linux1

ASM instance +ASM1 is running on node linux1.

List all configured databases

$ srvctl config database

racdb

Display configuration for our RAC database

$ srvctl config database -d racdb

linux1 racdb1 /u01/app/oracle/product/10.2.0/db_1

linux2 racdb2 /u01/app/oracle/product/10.2.0/db_1

Display all services for the specified cluster database

$ srvctl config service -d racdb

racdb_srvc PREF: racdb1 racdb2 AVAIL:

Display the configuration for node applications - (VIP, GSD, ONS, Listener)

$ srvctl config nodeapps -n linux1 -a -g -s -l

VIP exists.: /linux1-vip/192.168.1.200/255.255.255.0/eth0

GSD exists.

ONS daemon exists.

Listener exists.

Display the configuration for the ASM instance(s)

$ srvctl config asm -n linux1

+ASM1 /u01/app/oracle/product/10.2.0/db_1

All running instances in the cluster

SELECT

inst_id

, instance_number inst_no

, instance_name inst_name

, parallel

, status

, database_status db_status

, active_state state

, host_name host

FROM gv$instance

ORDER BY inst_id;

INST_ID INST_NO INST_NAME PAR STATUS DB_STATUS STATE HOST

-------- -------- ---------- --- ------- ------------ --------- -------

1 1 racdb1 YES OPEN ACTIVE NORMAL linux1

2 2 racdb2 YES OPEN ACTIVE NORMAL linux2

All data files which are in the disk group

select name from v$datafile

union

select member from v$logfile

union

select name from v$controlfile

union

select name from v$tempfile;

NAME

-------------------------------------------

+FLASH_RECOVERY_AREA/racdb/controlfile/current.256.695345395

+FLASH_RECOVERY_AREA/racdb/onlinelog/group_1.257.695345403

+FLASH_RECOVERY_AREA/racdb/onlinelog/group_2.258.695345409

+FLASH_RECOVERY_AREA/racdb/onlinelog/group_3.259.695350869

+FLASH_RECOVERY_AREA/racdb/onlinelog/group_4.260.695350875

+RACDB_DATA1/racdb/controlfile/current.256.695345391

+RACDB_DATA1/racdb/datafile/example.263.695345467

+RACDB_DATA1/racdb/datafile/indx.270.695352447

+RACDB_DATA1/racdb/datafile/sysaux.261.695345451

+RACDB_DATA1/racdb/datafile/system.259.695345415

+RACDB_DATA1/racdb/datafile/undotbs1.260.695345441

+RACDB_DATA1/racdb/datafile/undotbs1.271.695352519

+RACDB_DATA1/racdb/datafile/undotbs2.264.695345475

+RACDB_DATA1/racdb/datafile/undotbs2.272.695352571

+RACDB_DATA1/racdb/datafile/users.265.695345485

+RACDB_DATA1/racdb/datafile/users.269.695352363

+RACDB_DATA1/racdb/onlinelog/group_1.257.695345399

+RACDB_DATA1/racdb/onlinelog/group_2.258.695345405

+RACDB_DATA1/racdb/onlinelog/group_3.266.695350865

+RACDB_DATA1/racdb/onlinelog/group_4.267.695350871

+RACDB_DATA1/racdb/tempfile/temp.262.695345459

21 rows selected.

All ASM disk that belong to the 'RACDB_DATA1' disk group

SELECT path

FROM v$asm_disk

WHERE group_number IN (select group_number

from v$asm_diskgroup

where name = 'RACDB_DATA1');

PATH

----------------------------------

ORCL:VOL1

ORCL:VOL2

Starting / Stopping the Cluster

At this point, everything has been installed and configured for Oracle RAC 10g. We have all of the required

software installed and configured plus we have a fully functional clustered database.

With all of the work we have done up to this point, a popular question might be, "How do we start and stop

services?". If you have followed the instructions in this article, all services should start automatically on each

reboot of the Linux nodes. This would include Oracle Clusterware, all Oracle instances, Enterprise Manager

Database Console, etc.

There are times, however, when you might want to shutdown a node and manually start it back up. Or you may

find that Enterprise Manager is not running and need to start it. This section provides the commands responsible

for starting and stopping the cluster environment.

Ensure that you are logged in as the "oracle" UNIX user. I will be running all of the commands in this section

from linux1:

# su - oracle

$ hostname

linux1

Stopping the Oracle RAC 10g Environment

The first step is to stop the Oracle instance. Once the instance (and related services) is down, then bring down the

ASM instance. Finally, shutdown the node applications (Virtual IP, GSD, TNS Listener, and ONS). $ export ORACLE_SID=racdb1

$ emctl stop dbconsole

$ srvctl stop instance -d racdb -i racdb1

$ srvctl stop asm -n linux1

$ srvctl stop nodeapps -n linux1

Starting the Oracle RAC 10g Environment

The first step is to start the node applications (Virtual IP, GSD, TNS Listener, and ONS). Once the node

applications are successfully started, then bring up the ASM instance. Finally, bring up the Oracle instance (and

related services) and the Enterprise Manager Database console. $ export ORACLE_SID=racdb1

$ srvctl start nodeapps -n linux1

$ srvctl start asm -n linux1

$ srvctl start instance -d racdb -i racdb1

$ emctl start dbconsole

Start / Stop All Instances with SRVCTL

Start / Stop all of the instances and its enabled services. I just included this for fun as a way to bring down all

instances! $ srvctl start database -d racdb

$ srvctl stop database -d racdb

Transparent Application Failover - (TAF)

Overview

It is not uncommon for businesses of today to demand 99.99% or even 99.999% availability for their enterprise

applications. Think about what it would take to ensure a downtime of no more than .5 hours or even no

downtime during the year. To answer many of these high availability requirements, businesses are investing in

mechanisms that provide for automatic failover when one participating system fails. When considering the

availability of the Oracle database, Oracle RAC 10g provides a superior solution with its advanced failover

mechanisms. Oracle RAC 10g includes the required components that all work within a clustered configuration

responsible for providing continuous availability - when one of the participating systems fail within the cluster,

the users are automatically migrated to the other available systems.

A major component of Oracle RAC 10g that is responsible for failover processing is the Transparent Application

Failover (TAF) option. All database connections (and processes) that loose connections are reconnected to

another node within the cluster. The failover is completely transparent to the user.

This final section provides a short demonstration on how automatic failover works in Oracle RAC 10g. Please

note that a complete discussion on failover in Oracle RAC 10g would be an article in of its own. My intention

here is to present a brief overview and example of how it works.

One important note before continuing is that TAF happens automatically within the OCI libraries. This means

that your application (client) code does not need to change in order to take advantage of TAF. Certain

configuration steps, however, will need to be done on the Oracle TNS file tnsnames.ora.

Keep in mind that at the time of this article, using the Java thin client will not be able to

participate in TAF since it never reads the tnsnames.ora file.

Setup tnsnames.ora File

Before demonstrating TAF, we need to verify that a valid entry exists in the tnsnames.ora file on a non-RAC

client machine (if you have a Windows machine lying around). Ensure that you have Oracle RDBMS software

installed. (Actually, you only need a client install of the Oracle software.)

During the creation of the clustered database in this article, I created a new service that will be used for testing

TAF named racdb_srvc. It provides all of the necessary configuration parameters for load balancing and

failover. You can copy the contents of this entry to the %ORACLE_HOME%\network\admin\tnsnames.ora file on

the client machine (my Windows laptop is being used in this example) in order to connect to the new Oracle

clustered database:

tnsnames.ora File Entry for Clustered Database

...

RACDB_SRVC =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521))

(ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521))

(LOAD_BALANCE = yes)

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = racdb_srvc.idevelopment.info)

(FAILOVER_MODE =

(TYPE = SELECT)

(METHOD = BASIC)

(RETRIES = 180)

(DELAY = 5)

)

)

)

...

SQL Query to Check the Session's Failover Information

The following SQL query can be used to check a session's failover type, failover method, and if a failover has

occurred. We will be using this query throughout this example.

COLUMN instance_name FORMAT a13

COLUMN host_name FORMAT a9

COLUMN failover_method FORMAT a15

COLUMN failed_over FORMAT a11

SELECT

instance_name

, host_name

, NULL AS failover_type

, NULL AS failover_method

, NULL AS failed_over

FROM v$instance

UNION

SELECT

NULL

, NULL

, failover_type

, failover_method

, failed_over

FROM v$session

WHERE username = 'SYSTEM';

Transparent Application Failover Demonstration

From a Windows machine (or other non-RAC client machine), log in to the clustered database using the

racdb_srvc service as the SYSTEM user:

C:\> sqlplus system/manager@racdb_srvc

COLUMN instance_name FORMAT a13

COLUMN host_name FORMAT a9

COLUMN failover_method FORMAT a15

COLUMN failed_over FORMAT a11

SELECT

instance_name

, host_name

, NULL AS failover_type

, NULL AS failover_method

, NULL AS failed_over

FROM v$instance

UNION

SELECT

NULL

, NULL

, failover_type

, failover_method

, failed_over

FROM v$session

WHERE username = 'SYSTEM';

INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD FAILED_OVER

------------- --------- ------------- --------------- -----------

racdb1 linux1

SELECT BASIC NO

DO NOT logout of the above SQL*Plus session! Now that we have run the query (above), we should now

shutdown the instance racdb1 on linux1 using the abort option. To perform this operation, we can use the

srvctl command-line utility as follows:

# su - oracle

$ srvctl status database -d racdb

Instance racdb1 is running on node linux1

Instance racdb2 is running on node linux2

$ srvctl stop instance -d racdb -i racdb1 -o abort

$ srvctl status database -d racdb

Instance racdb1 is not running on node linux1

Instance racdb2 is running on node linux2

Now let's go back to our SQL session and rerun the SQL statement in the buffer:

COLUMN instance_name FORMAT a13

COLUMN host_name FORMAT a9

COLUMN failover_method FORMAT a15

COLUMN failed_over FORMAT a11

SELECT

instance_name

, host_name

, NULL AS failover_type

, NULL AS failover_method

, NULL AS failed_over

FROM v$instance

UNION

SELECT

NULL

, NULL

, failover_type

, failover_method

, failed_over

FROM v$session

WHERE username = 'SYSTEM';

INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD FAILED_OVER

------------- --------- ------------- --------------- -----------

racdb2 linux2

SELECT BASIC YES

SQL> exit

From the above demonstration, we can see that the above session has now been failed over to instance racdb2

on linux2.

Troubleshooting

Confirm the RAC Node Name is Not Listed in Loopback Address

Ensure that the node names (linux1 or linux2) are not included for the loopback address in the /etc/hosts

file. If the machine name is listed in the in the loopback address entry as below:

127.0.0.1 linux1 localhost.localdomain localhost

it will need to be removed as shown below: 127.0.0.1 localhost.localdomain localhost

If the RAC node name is listed for the loopback address, you will receive the following

error during the RAC installation: ORA-00603: ORACLE server session terminated by fatal error

or ORA-29702: error occurred in Cluster Group Service operation

Confirm localhost is defined in the /etc/hosts file for the loopback address

Ensure that the entry for localhost.localdomain and localhost are included for the loopback address in the

/etc/hosts file for each of the Oracle RAC nodes:

127.0.0.1 localhost.localdomain localhost

If an entry does not exist for localhost in the /etc/hosts file, Oracle Clusterware

will be unable to start the application resources — notably the ONS process. The error

would indicate "Failed to get IP for localhost" and will be written to the log file for

ONS. For example: CRS-0215 could not start resource 'ora.linux1.ons'. Check log file

"/u01/app/crs/log/linux1/racg/ora.linux1.ons.log"

for more details.

The ONS log file will contain lines similar to the following:

Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996,

2005 Oracle. All rights reserved.

2007-04-14 13:10:02.729: [

RACG][3086871296][13316][3086871296][ora.linux1.ons]: Failed to get IP

for localhost (1)

Failed to get IP for localhost (1)

Failed to get IP for localhost (1)

onsctl: ons failed to start

...

Setting the Correct Date and Time on Both Oracle RAC Nodes

During the installation of Oracle Clusterware, the Database, and the Companion CD, the Oracle Universal

Installer (OUI) first installs the software to the local node running the installer (i.e. linux1). The software is then

copied remotely to all of the remaining nodes in the cluster (i.e. linux2). During the remote copy process, the

OUI will execute the UNIX "tar" command on each of the remote nodes to extract the files that were archived

and copied over. If the date and time on the node performing the install is greater than that of the node it is

copying to, the OUI will throw an error from the "tar" command indicating it is attempting to extract files

stamped with a time in the future:

Error while copying directory

/u01/app/crs with exclude file list 'null' to nodes 'linux2'.

[PRKC-1002 : All the submitted commands did not execute successfully]

---------------------------------------------

linux2:

/bin/tar: ./bin/lsnodes: time stamp 2009-08-16 09:21:34 is 735 s in the future

/bin/tar: ./bin/olsnodes: time stamp 2009-08-16 09:21:34 is 735 s in the future

...(more errors on this node)

Please note that although this would seem like a severe error from the OUI, it can safely be disregarded as a

warning. The "tar" command DOES actually extract the files; however, when you perform a listing of the files

(using ls -l) on the remote node, they will be missing the time field until the time on the remote server is

greater than the timestamp of the file.

Before starting any of the above noted installations, ensure that each member node of the cluster is set as closely

as possible to the same date and time. Oracle strongly recommends using the Network Time Protocol feature of

most operating systems for this purpose, with all nodes using the same reference Network Time Protocol server.

Accessing a Network Time Protocol server, however, may not always be an option. In this case, when manually

setting the date and time for the nodes in the cluster, ensure that the date and time of the node you are performing

the software installations from (linux1) is less than all other nodes in the cluster (linux2). I generally use a 20

second difference as shown in the following example:

Setting the date and time from linux1:

# date -s "8/16/2009 01:12:00"

Setting the date and time from linux2:

# date -s "8/16/2009 01:12:20"

The two-node RAC configuration described in this article does not make use of a Network Time Protocol server.

Openfiler - Logical Volumes Not Active on Boot

One issue that I have run into several times occurs when using a USB drive connected to the Openfiler server.

When the Openfiler server is rebooted, the system is able to recognize the USB drive however, it is not able to

load the logical volumes and writes the following message to /var/log/messages - (also available through

dmesg):

iSCSI Enterprise Target Software - version 0.4.14

iotype_init(91) register fileio

iotype_init(91) register blockio

iotype_init(91) register nullio

open_path(120) Can't open /dev/rac1/crs -2

fileio_attach(268) -2

open_path(120) Can't open /dev/rac1/asm1 -2

fileio_attach(268) -2

open_path(120) Can't open /dev/rac1/asm2 -2

fileio_attach(268) -2

open_path(120) Can't open /dev/rac1/asm3 -2

fileio_attach(268) -2

open_path(120) Can't open /dev/rac1/asm4 -2

fileio_attach(268) -2

Please note that I am not suggesting that this only occurs with USB drives connected to

the Openfiler server. It may occur with other types of drives, however I have only seen

it with USB drives!

If you do receive this error, you should first check the status of all logical volumes using the lvscan command

from the Openfiler server:

# lvscan

inactive '/dev/rac1/crs' [2.00 GB] inherit

inactive '/dev/rac1/asm1' [115.94 GB] inherit

inactive '/dev/rac1/asm2' [115.94 GB] inherit

inactive '/dev/rac1/asm3' [115.94 GB] inherit

inactive '/dev/rac1/asm4' [115.94 GB] inherit

Notice that the status for each of the logical volumes is set to inactive - (the status for each logical volume on a

working system would be set to ACTIVE).

I currently know of two methods to get Openfiler to automatically load the logical volumes on reboot, both of

which are described below.

Method 1

One of the first steps is to shutdown both of the Oracle RAC nodes in the cluster - (linux1 and linux2). Then,

from the Openfiler server, manually set each of the logical volumes to ACTIVE for each consecutive reboot: # lvchange -a y /dev/rac1/crs

# lvchange -a y /dev/rac1/asm1

# lvchange -a y /dev/rac1/asm2

# lvchange -a y /dev/rac1/asm3

# lvchange -a y /dev/rac1/asm4

Another method to set the status to active for all logical volumes is to use the Volume

Group change command as follows: # vgscan

Reading all physical volumes. This may take a while...

Found volume group "rac1" using metadata type lvm2

# vgchange -ay

5 logical volume(s) in volume group "rac1" now active

After setting each of the logical volumes to active, use the lvscan command again to verify the status:

# lvscan

ACTIVE '/dev/rac1/crs' [2.00 GB] inherit

ACTIVE '/dev/rac1/asm1' [115.94 GB] inherit

ACTIVE '/dev/rac1/asm2' [115.94 GB] inherit

ACTIVE '/dev/rac1/asm3' [115.94 GB] inherit

ACTIVE '/dev/rac1/asm4' [115.94 GB] inherit

As a final test, reboot the Openfiler server to ensure each of the logical volumes will be set to ACTIVE after the

boot process. After you have verified that each of the logical volumes will be active on boot, check that the

iSCSI target service is running: # service iscsi-target status

ietd (pid 2668) is running...

Finally, restart each of the Oracle RAC nodes in the cluster - (linux1 and linux2).

Method 2

This method was kindly provided by Martin Jones. His workaround includes amending the /etc/rc.sysinit

script to basically wait for the USB disk (/dev/sda in my example) to be detected. After making the changes to

the /etc/rc.sysinit script (described below), verify the external drives are powered on and then reboot the

Openfiler server.

The following is a small portion of the /etc/rc.sysinit script on the Openfiler server with the changes

(highlighted in blue) proposed by Martin:

Make Modifications to /etc/rc.sysinit

# LVM2 initialization, take 2

if [ -c /dev/mapper/control ]; then

if [ -x /sbin/multipath.static ]; then

modprobe dm-multipath >/dev/null 2>&1

/sbin/multipath.static -v 0

if [ -x /sbin/kpartx ]; then

/sbin/dmsetup ls --target multipath --exec "/sbin/kpartx -

a"

fi

fi

if [ -x /sbin/dmraid ]; then

modprobe dm-mirror > /dev/null 2>&1

/sbin/dmraid -i -a y

fi

#-----

#----- MJONES - Customisation Start

#-----

# Check if /dev/sda is ready

while [ ! -e /dev/sda ]

do

echo "Device /dev/sda for first USB Drive is not yet ready."

echo "Waiting..."

sleep 5

done

echo "INFO - Device /dev/sda for first USB Drive is ready."

#-----

#----- MJONES - Customisation END

#-----

if [ -x /sbin/lvm.static ]; then

if /sbin/lvm.static vgscan > /dev/null 2>&1 ; then

action $"Setting up Logical Volume

Management:" /sbin/lvm.static vgscan --mknodes --ignorelockingfailure &&

/sbin/lvm.static vgchange -a y --ignorelockingfailure

fi

fi

fi

# Clean up SELinux labels

if [ -n "$SELINUX" ]; then

for file in /etc/mtab /etc/ld.so.cache ; do

[ -r $file ] && restorecon $file >/dev/null 2>&1

done

fi

Finally, restart each of the Oracle RAC nodes in the cluster - (linux1 and linux2).

OCFS2 - Configure O2CB to Start on Boot

With the releases of OCFS2 prior to 1.2.1, there is a bug that exists where the driver does not get loaded on each

boot even after configuring the on-boot properties to do so.

Note that this section does not apply here since the version of OCFS2 used in this article

is greater than release 1.2.1.

After attempting to configure the on-boot properties to start on each boot according to the official OCFS2

documentation, you will still get the following error on each boot:

...

Mounting other filesystems:

mount.ocfs2: Unable to access cluster service

Cannot initialize cluster mount.ocfs2:

Unable to access cluster service Cannot initialize cluster [FAILED]

...

Red Hat changed the way the service is registered between chkconfig-1.3.11.2-1 and chkconfig-1.3.13.2-1. The

O2CB script used to work with the former.

Before attempting to configure the on-boot properties:

REMOVE the following lines in /etc/init.d/o2cb ### BEGIN INIT INFO

# Provides: o2cb

# Required-Start:

# Should-Start:

# Required-Stop:

# Default-Start: 2 3 5

# Default-Stop:

# Description: Load O2CB cluster services at system boot.

### END INIT INFO

Re-register the o2cb service. # chkconfig --del o2cb

# chkconfig --add o2cb

# chkconfig --list o2cb

o2cb 0:off 1:off 2:on 3:on 4:on 5:on 6:off

# ll /etc/rc3.d/*o2cb*

lrwxrwxrwx 1 root root 14 Sep 29 11:56 /etc/rc3.d/S24o2cb -> ../init.d/o2cb

The service should be S24o2cb in the default runlevel.

After resolving the bug I listed above, we can now continue to set the on-boot properties as follows: # /etc/init.d/o2cb offline ocfs2

# /etc/init.d/o2cb unload

# /etc/init.d/o2cb configure

Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.

The following questions will determine whether the driver is loaded on

boot. The current values will be shown in brackets ('[]'). Hitting

<ENTER> without typing an answer will keep that current value. Ctrl-C

will abort.

Load O2CB driver on boot (y/n) [n]: y

Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2

Writing O2CB configuration: OK

Loading module "configfs": OK

Mounting configfs filesystem at /config: OK

Loading module "ocfs2_nodemanager": OK

Loading module "ocfs2_dlm": OK

Loading module "ocfs2_dlmfs": OK

Mounting ocfs2_dlmfs filesystem at /dlm: OK

Starting cluster ocfs2: OK

OCFS2 - o2cb_ctl: Unable to access cluster service while creating node

While configuring the nodes for OCFS2 using ocfs2console, it is possible to run into the error:

o2cb_ctl: Unable to access cluster service while creating node

This error does not show up when you startup ocfs2console for the first time. This message comes up when

there is a problem with the cluster configuration or if you do not save the cluster configuration initially while

setting it up using ocfs2console. This is a bug!

The work-around is to exit from the ocfs2console, unload the o2cb module and remove the ocfs2 cluster

configuration file /etc/ocfs2/cluster.conf. I also like to remove the /config directory. After removing the

ocfs2 cluster configuration file, restart the ocfs2console program.

For example:

# /etc/init.d/o2cb offline ocfs2

# /etc/init.d/o2cb unload

Unmounting ocfs2_dlmfs filesystem: OK

Unloading module "ocfs2_dlmfs": OK

Unmounting configfs filesystem: OK

Unloading module "configfs": OK

# rm -f /etc/ocfs2/cluster.conf

# rm -rf /config

# ocfs2console &

This time, it will add the nodes!

OCFS2 - Adjusting the O2CB Heartbeat Threshold

With previous versions of this article, (using FireWire as opposed to iSCSI for the shared storage), I was able to

install and configure OCFS2, format the new volume, and finally install Oracle Clusterware (with its two

required shared files; the voting disk and OCR file), located on the new OCFS2 volume. While I was able to

install Oracle Clusterware and see the shared drive using FireWire, however, I was receiving many lock-ups and

hanging after about 15 minutes when the Clusterware software was running on both nodes. It always varied on

which node would hang (either linux1 or linux2 in my example). It also didn't matter whether there was a high

I/O load or none at all for it to crash (hang).

After looking through the trace files for OCFS2, it was apparent that access to the voting disk was too slow

(exceeding the O2CB heartbeat threshold) and causing the Oracle Clusterware software (and the node) to crash.

On the console would be a message similar to the following:

...

Index 0: took 0 ms to do submit_bio for read

Index 1: took 3 ms to do waiting for read completion

Index 2: took 0 ms to do bio alloc write

Index 3: took 0 ms to do bio add page write

Index 4: took 0 ms to do submit_bio for write

Index 5: took 0 ms to do checking slots

Index 6: took 4 ms to do waiting for write completion

Index 7: took 1993 ms to do msleep

Index 8: took 0 ms to do allocating bios for read

Index 9: took 0 ms to do bio alloc read

Index 10: took 0 ms to do bio add page read

Index 11: took 0 ms to do submit_bio for read

Index 12: took 10006 ms to do waiting for read completion

(13,3):o2hb_stop_all_regions:1888 ERROR: stopping heartbeat on all active regions.

Kernel panic - not syncing: ocfs2 is very sorry to be fencing this system by panicing

The solution I used was to increase the O2CB heartbeat threshold from its default value of 7, to 61. Some setups

may require an even higher setting. This is a configurable parameter that is used to compute the time it takes for

a node to "fence" itself. During the installation and configuration of OCFS2, we adjusted this value in the section

"Configure O2CB to Start on Boot and Adjust O2CB Heartbeat Threshold". If you encounter a kernel panic from

OCFS2 and need to increase the heartbeat threshold, use the same procedures described in the section "Configure

O2CB to Start on Boot and Adjust O2CB Heartbeat Threshold". If you are using an earlier version of OCFS2

tools (prior to ocfs2-tools release 1.2.2-1), the following describes how to manually adjust the O2CB heartbeat

threshold.

First, let's see how to determine what the O2CB heartbeat threshold is currently set to. This can be done by

querying the /proc file system as follows:

# cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold

7

We see that the value is 7, but what does this value represent? Well, it is used in the formula below to determine

the fence time (in seconds): [fence time in seconds] = (O2CB_HEARTBEAT_THRESHOLD - 1) * 2

So, with an O2CB heartbeat threshold of 7, we would have a fence time of: (7 - 1) * 2 = 12 seconds

If we want a larger threshold (say 120 seconds), we would need to adjust O2CB_HEARTBEAT_THRESHOLD

to 61 as shown below: (61 - 1) * 2 = 120 seconds

Let's see now how to manually increase the O2CB heartbeat threshold from 7 to 61. This task will need to be

performed on all Oracle RAC nodes in the cluster. We first need to modify the file /etc/sysconfig/o2cb and

set O2CB_HEARTBEAT_THRESHOLD to 61:

/etc/sysconfig/o2cb

#

# This is a configuration file for automatic startup of the O2CB

# driver. It is generated by running /etc/init.d/o2cb configure.

# Please use that method to modify this file

#

# O2CB_ENABELED: 'true' means to load the driver on boot.

O2CB_ENABLED=true

# O2CB_BOOTCLUSTER: If not empty, the name of a cluster to start.

O2CB_BOOTCLUSTER=ocfs2

# O2CB_HEARTBEAT_THRESHOLD: Iterations before a node is considered

dead.

O2CB_HEARTBEAT_THRESHOLD=61

# O2CB_IDLE_TIMEOUT_MS: Time in ms before a network connection is

considered dead.

O2CB_IDLE_TIMEOUT_MS=

# O2CB_KEEPALIVE_DELAY_MS: Max time in ms before a keepalive

packet is sent

O2CB_KEEPALIVE_DELAY_MS=

# O2CB_RECONNECT_DELAY_MS: Min time in ms between connection

attempts

O2CB_RECONNECT_DELAY_MS=

After modifying the file /etc/sysconfig/o2cb, we need to alter the o2cb configuration. Again, this should be

performed on all Oracle RAC nodes in the cluster.

# umount /u02

# /etc/init.d/o2cb offline ocfs2

# /etc/init.d/o2cb unload

# /etc/init.d/o2cb configure

Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.

The following questions will determine whether the driver is loaded on

boot. The current values will be shown in brackets ('[]'). Hitting

<ENTER> without typing an answer will keep that current value. Ctrl-C

will abort.

Load O2CB driver on boot (y/n) [n]: y

Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2

Specify heartbeat dead threshold (>=7) [7]: 61

Specify network idle timeout in ms (>=5000) [10000]: 10000

Specify network keepalive delay in ms (>=1000) [5000]: 5000

Specify network reconnect delay in ms (>=2000) [2000]: 2000

Writing O2CB configuration: OK

Loading module "configfs": OK

Mounting configfs filesystem at /config: OK

Loading module "ocfs2_nodemanager": OK

Loading module "ocfs2_dlm": OK

Loading module "ocfs2_dlmfs": OK

Mounting ocfs2_dlmfs filesystem at /dlm: OK

Starting O2CB cluster ocfs2: OK

We can now check again to make sure the settings took place in for the o2cb cluster stack: # cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold

61

It is important to note that the value of 61 I used for the O2CB heartbeat threshold will

not work for all configurations. In some cases, the O2CB heartbeat threshold value had

to be increased to as high as 601 in order to prevent OCFS2 from panicking the kernel.

Oracle Clusterware Installation: Running root.sh Fails on the Last Node

After the Oracle Clusterware install process, running root.sh on the last node will fail while attempting to

configure vipca at the end of the script:

Oracle CRS stack installed and running under init(1M)

Running vipca(silent) for configuring nodeapps

/u01/app/crs/jdk/jre//bin/java: error while loading

shared libraries: libpthread.so.0:

cannot open shared object file: No such file or directory

After receiving this error, please leave the OUI up. Do not hit the OK button on the "Execute Configuration

Scripts" dialog until all of the issues described in this section have been resolved.

Note that srvctl will produce similar output until the workaround described below is performed.

This error occurs because these releases of the Linux kernel fix an old bug in the Linux threading that Oracle

worked around using LD_ASSUME_KERNEL settings in both vipca and srvctl, this workaround is no longer valid

on OEL5 or RHEL5 or SLES10 hence the failures.

To workaround this issue, edit vipca (in the CRS bin directory on all nodes) to undo the setting of

LD_ASSUME_KERNEL. After the IF statement around line 120, add an unset command to ensure

LD_ASSUME_KERNEL is not set as follows:

if [ "$arch" = "i686" -o "$arch" = "ia64" ]

then

LD_ASSUME_KERNEL=2.4.19

export LD_ASSUME_KERNEL

fi

unset LD_ASSUME_KERNEL <<== Line to be added

Similarly for srvctl (in both the CRS and, when installed, RDBMS and ASM bin directories on all nodes),

unset LD_ASSUME_KERNEL by adding one line, around line 168 should look like this:

LD_ASSUME_KERNEL=2.4.19

export LD_ASSUME_KERNEL

unset LD_ASSUME_KERNEL <<== Line to be added

Note: Remember to re-edit these files on all nodes:

<ORA_CRS_HOME>/bin/vipca

<ORA_CRS_HOME>/bin/srvctl

<RDBMS_HOME>/bin/srvctl

<ASM_HOME>/bin/srvctl # (If exists)

... after applying the 10.2.0.2 or 10.2.0.3 patch sets, as these patchset will still include those settings unnecessary

for OEL5 or RHEL5 or SLES10. This issue was raised with development and is fixed in the 10.2.0.4 patchsets.

Also note that we are explicitly unsetting LD_ASSUME_KERNEL and not merely commenting out its setting to

handle a case where the user has it set in their environment (login shell).

After working around the LD_ASSUME_KERNEL issue above, vipca will now fail to run with the following

error if the VIP IP's are in a non-routable range [10.x.x.x, 172.(16-31).x.x, or 192.168.x.x]:

[root@linux2 ~]# $ORA_CRS_HOME/bin/vipca

Error 0(Native: listNetInterfaces:[3])

[Error 0(Native: listNetInterfaces:[3])]

There are several ways to workaround this issue. The goal to this workaround is to get the output of

"$ORA_CRS_HOME/bin/oifcfg getif" to include both public and cluster_interconnect interfaces. If you try to

run the above command, you will notice it provides nothing which means we have some work to do!

The first step is to identify the current interfaces and IP addresses:

[root@linux2 ~]# $ORA_CRS_HOME/bin/oifcfg iflist

eth1 192.168.2.0

eth0 192.168.1.0

Remember during the Oracle Clusterware install that 192.168.1.0 is my public interface while 192.168.2.0 is

the cluster_interconnect interface.

Using this information, we can manually set the public / private interfaces accordingly using the setif option of

the $ORA_CRS_HOME/bin/oifcfg command:

# $ORA_CRS_HOME/bin/oifcfg setif -global eth0/192.168.1.0:public

# $ORA_CRS_HOME/bin/oifcfg setif -global eth1/192.168.2.0:cluster_interconnect

Let's know run the "$ORA_CRS_HOME/bin/oifcfg getif" command again to verify its output:

[root@linux2 ~]# $ORA_CRS_HOME/bin/oifcfg getif

eth0 192.168.1.0 global public

eth1 192.168.2.0 global cluster_interconnect

After resolving all of the issues above, manually re-run vipca (GUI) as root from the last node in which the

errors occurred. Please keep in mind that vipca is a GUI and will need to set your DISPLAY variable

accordingly to your X server:

# $ORA_CRS_HOME/bin/vipca

Conclusion

Oracle10g RAC allows the DBA to configure a database solution with superior fault tolerance and load

balancing. For those DBAs, however, that want to become more familiar with the features and benefits of

Oracle10g RAC will find the costs of configuring even a small RAC cluster costing in the range of US$15,000 to

US$20,000.

This article has hopefully given you an economical solution to setting up and configuring an inexpensive

Oracle10g Release 2 RAC Cluster using CentOS 5 and iSCSI technology. The RAC solution presented in this

article can be put together for around US$2,700 and will provide the DBA with a fully functional Oracle10g

Release 2 RAC cluster. While the hardware used for this article should be stable enough for educational

purposes, it should never be considered for a production environment.

Building an Oracle RAC Cluster Remotely

An iDevelopment.info reader is now offering 3 computers that will allow you to REMOTELY build an Oracle

RAC Cluster from scratch using a VNC enabled KVM. The cost is $14 for 7 days full access. Please email him

at "Bryan" AT "AHCCINC" DOT "COM" with the words "RAC LAB" as the subject. If the lab is available, you

can start building your RAC now!

Acknowledgements

An article of this magnitude and complexity is generally not the work of one person alone. Although I was able

to author and successfully demonstrate the validity of the components that make up this configuration, there are

several other individuals that deserve credit in making this article a success.

First, I would like to thank Bane Radulovic from the Server BDE Team at Oracle. Bane not only introduced me

to Openfiler, but shared with me his experience and knowledge of the product and how to best utilize it for

Oracle RAC. His research and hard work made the task of configuring Openfiler seamless. Bane was also

involved with hardware recommendations and testing.

A special thanks to K Gopalakrishnan for his assistance in delivering the Oracle RAC 10g Overview section of

this article. In this section, much of the content regarding the history of Oracle RAC can be found in his very

popular book Oracle Database 10g Real Application Clusters Handbook. This book comes highly recommended

for both DBAs and Developers wanting to successfully implement Oracle RAC and fully understand how many

of the advanced services like Cache Fusion and Global Resource Directory operate.

I would next like to thank Oracle ACE Werner Puschitz for his outstanding work on "Installing Oracle Database

10g with Real Application Cluster (RAC) on Red Hat Enterprise Linux Advanced Server 3". This article, along

with several others of his, provided information on Oracle RAC that could not be found in any other Oracle

documentation. Without his hard work and research into issues like configuring OCFS2 and ASMLib, this article

may have never come to fruition. If you are interested in examining technical articles on Linux internals and in-

depth Oracle configurations written by Werner Puschitz, please visit his website at www.puschitz.com.

Also, thanks to Tzvika Lemel for his comments and suggestions on using Oracle's Cluster Verification Utility

(CVU).

Lastly, I would like to express my appreciation to the following vendors for generously supplying the hardware

for this article; Stallard Technologies, Inc., Seagate, Avocent Corporation, Intel, D-Link, SIIG, and LaCie.

About the Author

Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an

Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in

Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming,

developing high availability solutions, capacity planning, database security, and physical / logical database

design in a UNIX, Linux, and Windows server environment. Jeff's other interests include mathematical

encryption theory, programming language processors (compilers and interpreters) in Java and C, LDAP, writing

web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and

Software Engineer for over 17 years and maintains his own website site at: http://www.iDevelopment.info. Jeff

graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science.