a quick reference for oracle database 10g rac on linux and unix platforms.pdf

7/29/2019 A Quick Reference for Oracle Database 10g RAC on Linux and Unix Platforms.pdf

1/8

uick Reference for Oracle Database 10g RAC on Linux and Unix Platforms

//www.dbspecialists.com/files/presentations/rac_quick_reference.html[7/8/2010 12:10:44 PM]

A Quick Reference for Orac le Database 10

RAC on Linux and Unix Plat forms

y Rich Headrickatabase Specialists, Inc.

About Database Specialists, Inc.Database Specialists, Inc. provides remote DBA services and onsite database support for your mission criticalOracle systems. Since 1995, we have been providing Oracle database consulting in Solaris, HP-UX, Linux, AIX,and Windows environments. We are DBAs, speakers, educators, and authors. Our team is continually recognizedby Oracle, at national conferences and by leading trade publications. Learn more about our remote DBA,database tuning, and consultingservices. Or, call us at 415-344-0500 or 888-648-0500.

ntroduction

This paper will discuss and demonstrate some of the basic RAC management commands you might use to manageour Oracle 10g RAC components. The material presented will be applicable to most if not all Linux and Unixlatforms. It will not cover RAC on the Microsoft Windows operating system. We will begin with the basics ofhecking out a RAC system to identify whether or not the appropriate services and resources are running or not. T

we will go through the basic startup and shutdown commands for all services and resources. Lastly, we will go ovetartup of individual resources and checking the status of each step as we progress. The shutdown of individualesources can be done by reversing the individual startup order.

About our Environment

The RAC environment that we will reference throughout this paper is composed of two nodes running Red Hatnterprise Linux 4 ES with a shared disk, and has two ASM instances named ASM1 and ASM2, two database

nstances named orcl1 and orcl2, and a service named RAC that is used for transparent application failover (TAF)

oad balancing.

Overview of Basic RAC Managem ent Commands

The commands we will use are listed below. Remember that this document is a quick reference, and not an exhausst of all commands for managing your RAC environment.

Cluster Related Commands

crs_stat -t Shows HA resource status (hard to read)

crsstat Ouptut of crs_stat -t formatted nicely

ps -ef|grep d.bin crsd.bin evmd.bin ocssd.bincrsctl check crs CSS,CRS,EVM appears healthy

crsctl stop crs Stop crs and all other services

crsctl disable crs* Prevents CRS from starting on reboot

crsctl enable crs* Enables CRS start on reboot

crs_stop -all Stops all registered resources

crs_start -all Starts all registered resources

These commands update the file /etc/oracle/scls_scr//root/crsstart which contains the string enable ordisable as appropriate.
http://www.dbspecialists.com/dbapro.htmlhttp://www.dbspecialists.com/performance.htmlhttp://www.dbspecialists.com/consulting.htmlhttp://www.dbspecialists.com/consulting.htmlhttp://www.dbspecialists.com/consulting.htmlhttp://www.dbspecialists.com/performance.htmlhttp://www.dbspecialists.com/dbapro.html


2/8



Database Related Commands

srvctl start instance -d -i Starts an instance

srvctl start database -d Starts all instances

srvctl stop database -d Stops all instances, closes database

srvctl stop instance -d -i Stops an instance

srvctl start service -d -s Starts a service

srvctl stop service -d -s Stops a service

srvctl status service -d Checks status of a servicesrvctl status instance -d -i Checks an individual instance

srvctl status database -d Checks status of all instances

srvctl start nodeapps -n Starts gsd, vip, listener, and ons

srvctl stop nodeapps -n Stops gsd, vip and listener

Keep in mind that some resources will not start unless other resources are already online. We will now look at theeneral dependency list in greater detail.

There are three main background processes you can see when doing a ps ef|grep d.bin. They are normally startednit during the operating system boot process. They can be started and stopped manually by issuing the commandetc/init.d/init.crs {start|stop|enable|disable}

1. /etc/rc.d/init.d/init.evmd2. /etc/rc.d/init.d/init.cssd3. /etc/rc.d/init.d/init.crsd

Once the above processes are running, they will automatically start the following services in the following order if re enabled. This list assumes you are using ASM and have a service set up for TAF/load balancing.

1. The nodeapps (gsd, VIP, ons, listener) are brought online.2. The ASM instances are brought online.

3. The database instances are brought online.4. Any defined services are brought online.

Basic RAC Management Commands

Now that we know the dependency tree and have some commands at our disposal, lets have a look at them one atme, starting with the cluster commands and processes.

rs_sta t -t

This command shows us the status of each registered resource in the cluster. I generally avoid this command beca

ts output is hard to read since the names are truncated as you can see in the sample output below. You can downlhelpful script called crsstat fromhttp://www.dbspecialists.com/specialists/specialist2007-05.htmlto make it easy our eyes.

[ oracl e@gr een ~] $ cr s_s t at - tName Type Target St at e Host- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -ora. . . . SM1. asm appl i cat i on ONLI NE ONLI NE gr eenor a. . . . EN. l snr appl i cat i on ONLI NE ONLI NE gr eenora. gr een. gsd appl i cat i on ONLI NE ONLI NE greenora. gr een. ons appl i cat i on ONLI NE ONLI NE greenora. gr een. vi p appl i cat i on ONLI NE ONLI NE gr eenor a. . . . . RAC. cs appl i cat i on ONLI NE ONLI NE r edor a. . . . cl 1. sr v appl i cat i on ONLI NE ONLI NE gr een
http://www.dbspecialists.com/specialists/specialist2007-05.htmlhttp://www.dbspecialists.com/specialists/specialist2007-05.htmlhttp://www.dbspecialists.com/specialists/specialist2007-05.html


3/8



or a. . . . cl 2. sr v appl i cat i on ONLI NE ONLI NE r edora. orcl . db appl i cat i on ONLI NE ONLI NE r edor a. . . . l 1. i nst appl i cat i on ONLI NE ONLI NE gr eenor a. . . . l 2. i nst appl i cat i on ONLI NE ONLI NE r edor a. . . . SM2. asm appl i cat i on ONLI NE ONLI NE r edor a. . . . ED. l snr appl i cat i on ONLI NE ONLI NE r edora. r ed. gsd appl i cat i on ONLI NE ONLI NE r edora. r ed. ons appl i cat i on ONLI NE ONLI NE r edora. r ed. vi p appl i cat i on ONLI NE ONLI NE r ed[ oracl e@green ~] $

rsstat

The output of this script is much better. You can learn more about this script and download it atttp://www.dbspecialists.com/specialists/specialist2007-05.html.

[ oracl e@gr een ~] $ crsst atHA Resour ce Target Stat e- - - - - - - - - - - - - - - - - - - - - -or a. green. ASM1. asm ONLI NE ONLI NE on greenor a. green. LI STENER_GREEN. l snr ONLI NE ONLI NE on greenor a. green. gsd ONLI NE ONLI NE on greenor a. green. ons ONLI NE ONLI NE on greenora. green. vi p ONLI NE ONLI NE on greenora. orcl . RAC. cs ONLI NE ONLI NE on r ed

ora. orcl . RAC. orcl 1. sr v ONLI NE ONLI NE on gr eenora. orcl . RAC. orcl 2. sr v ONLI NE ONLI NE on r edora. orcl . db ONLI NE ONLI NE on r edor a. or cl . or cl 1. i nst ONLI NE ONLI NE on gr eenor a. or cl . or cl 2. i nst ONLI NE ONLI NE on r edora. r ed. ASM2. asm ONLI NE ONLI NE on r edora. r ed. LI STENER_RED. l snr ONLI NE ONLI NE on r edora. r ed. gsd ONLI NE ONLI NE on r edora. r ed. ons ONLI NE ONLI NE on r edora. r ed. vi p ONLI NE ONLI NE on r ed[ oracl e@green ~] $

ps -ef|grep d.bin

We can use this command to verify that the CRS background processes are actually running. It is implicit that theunning if the crs_stat command and crsstat script work. If they do not work, you will want to verify the backgrounrocesses are really running.

[ r oot @gr een ~] # ps - ef | gr ep d. bi noracl e 5335 3525 0 J ul 11 ? 00: 00: 05/ u01/ app/ or acl e/ pr oduct / 10. 2. 0/ crs/ bi n/ evmd. bi nr oot 5487 3817 0 J ul 11 ? 00: 00: 00/ u01/ app/ or acl e/ pr oduct / 10. 2. 0/ crs/ bi n/ crsd. bi n r ebootoracl e 5932 5392 0 J ul 11 ? 00: 00: 00/ u01/ app/ or acl e/ pr oduct/ 10. 2. 0/ crs/ bi n/ ocssd. bi nr oot 30486 30177 0 18: 23 pt s/ 1 00: 00: 00 gr ep d. bi n[ r oot @green ~] #

rsctl check crs

This command verifies that the above background daemons are functioning.

[ oracl e@gr een ~] $ crsct l check cr sCSS appear s heal t hyCRS appears heal t hyEVM appear s heal t hy[ oracl e@green ~] $
http://www.dbspecialists.com/specialists/specialist2007-05.htmlhttp://www.dbspecialists.com/specialists/specialist2007-05.html


4/8



rsctl stop crs

Well need to be logged onto the server as the root user to run this command. It will stop all HA resources on the lode, and it will also stop the above mentioned background daemons.

[ oracl e@gr een ~] $ crsct l st op cr sI nsuf f i c i ent user pr i vi l eges.[ oracl e@green ~] $ suPasswor d:[ r oot @gr een or acl e] # cr sct l st op crsSt oppi ng r esour ces. Thi s coul d take sever al mi nut es.Successf ul l y st opped CRS resour ces.St oppi ng CSSD.Shutt i ng down CSS daemon.Shut down r equest successf ul l y i ssued.[ r oot @gr een or acl e] #

rsctl disable crs

This command will prevent CRS from starting on a reboot. Note there is no return output from the command.

[ r oot @gr een or acl e] # cr sct l di sabl e cr s

[ r oot @gr een or acl e] #

We did a reboot after this and verified that CRS did not come back online because we wanted to do some operatingystem maintenance. Lets check the status by running some of the commands weve just discussed.

[ oracl e@gr een ~] $ crsst atHA Resour ce Tar get Stat e- - - - - - - - - - - - - - - - - - - - - -er r or connect i ng t o CRSD at [ ( ADDRESS=( PROTOCOL=i pc) ( KEY=ora_cr sqs) ) ] cl sccon 184

[ oracl e@gr een ~] $ crsct l check cr sFai l ure 1 cont act i ng CSS daemonCannot communi cat e wi t h CRS

Cannot communi cat e wi t h EVM

[ oracl e@gr een ~] $ ps - ef | gr ep d. bi noracl e 6149 5582 0 15: 54 pt s/ 1 00: 00: 00 gr ep d. bi n[ oracl e@green ~] $

verything appears to be down on this node as expected.

Now lets start everything back up. We will need to be root for this, unless you have been given permissions or sudo run crsctl start crs.

[ r oot @gr een or acl e] # crsct l st ar t crs

At t empt i ng t o st art CRS stackThe CRS st ack wi l l be st ar t ed shor t l y[ r oot @gr een or acl e] #

After a few minutes the registered resources for this node should come online. Lets check to be sure:

[ oracl e@gr een ~] $ crsst atHA Resour ce Tar get Stat e- - - - - - - - - - - - - - - - - - - - - -or a. green. ASM1. asm ONLI NE ONLI NE on greenor a. green. LI STENER_GREEN. l snr ONLI NE ONLI NE on greenor a. green. gsd ONLI NE ONLI NE on greenor a. green. ons ONLI NE ONLI NE on green


5/8



ora. green. vi p ONLI NE ONLI NE on greenora. orcl . RAC. cs ONLI NE ONLI NE on r edora. orcl . RAC. orcl 1. sr v ONLI NE ONLI NE on gr eenora. orcl . RAC. orcl 2. sr v ONLI NE ONLI NE on r edora. orcl . db ONLI NE ONLI NE on r edor a. or cl . or cl 1. i nst ONLI NE ONLI NE on gr eenor a. or cl . or cl 2. i nst ONLI NE ONLI NE on r edora. r ed. ASM2. asm ONLI NE ONLI NE on r edora. r ed. LI STENER_RED. l snr ONLI NE ONLI NE on r edora. r ed. gsd ONLI NE ONLI NE on r edora. r ed. ons ONLI NE ONLI NE on r edora. r ed. vi p ONLI NE ONLI NE on r ed[ oracl e@green ~] $

ets not forget to enable CRS on reboot:

[ r oot @gr een oracl e] # cr sct l enabl e cr s

rs_stop -all

This is a handy script that stops the registered resources and leaves the CRS running. This includes all services inluster, so it will bring down all registered resources on all nodes.

[ oracl e@gr een ~] $ cr s_s t op - al lAt t empt i ng t o st op òra. green. gsd` on member `gr eenÀt t empt i ng t o st op òra. orcl . RAC. orcl 2. sr v` on member `r ed`St op of òra. orcl . TEST. orcl 1. sr v` on member `green` succeeded.At t empt i ng t o st op òr a. orcl . RAC. orcl 1. sr v` on member `greenÀt t empt i ng t o st op òra. green. ons` on member `gr eenÀt t empt i ng t o st op òr a. orcl . RAC. cs` on member `r edStop of òr a. green. gsd` on member `gr een` succeeded.St op of òr a. orcl . RAC. orcl 1. sr v` on member `gr een` succeeded.St op of òra. orcl . RAC. orcl 2. sr v` on member `r ed` succeeded.St op of òra. orcl . TEST. orcl 2. sr v` on member `r ed` succeeded.Stop of òr a. green. ons` on member `gr een` succeeded.- - sni p- -CRS- 0216: Coul d not st op resour ce ' or a. or cl . or cl 2. i nst ' .[ oracl e@green ~] $

Occasionally you will get the CRS-0216 error message shown above. This is usually bogus, but you should re-chewith crsstat and ps ef|grep smon or similar to be sure everything has died off.

ets verify that crs_stop -all worked as expected:

[ r oot @gr een or acl e] # cr sst atHA Resour ce Tar get Stat e- - - - - - - - - - - - - - - - - - - - - -ora. green. ASM1. asm OFFLI NE OFFLI NEora. green. LI STENER_GREEN. l snr OFFLI NE OFFLI NEora. green. gsd OFFLI NE OFFLI NEora. gr een. ons OFFLI NE OFFLI NEora. gr een. vi p OFFLI NE OFFLI NEora. orcl . RAC. cs OFFLI NE OFFLI NEor a. or cl . RAC. or cl 1. sr v OFFLI NE OFFLI NEor a. or cl . RAC. or cl 2. sr v OFFLI NE OFFLI NEora. orcl . db OFFLI NE OFFLI NEor a. or cl . or cl 1. i nst OFFLI NE OFFLI NEor a. or cl . or cl 2. i nst OFFLI NE OFFLI NEora. r ed. ASM2. asm OFFLI NE OFFLI NEora. r ed. LI STENER_RED. l snr OFFLI NE OFFLI NEora. r ed. gsd OFFLI NE OFFLI NEora. r ed. ons OFFLI NE OFFLI NEora. r ed. vi p OFFLI NE OFFLI NE[ r oot @gr een or acl e] #


6/8



ets move on to working with srvctl and managing individual resources. We will begin with the crs backgroundaemons already running, and all registered resources being offline from the last step above. We will first start theodeapps, then the ASM instances, followed by the database instances, and lastly the services for TAF and loadalancing. This is the dependency order in our particular environment. You may or may not have ASM or TAF a

oad balancing services to start in your environment.

rvct l start nodeapps -n (node)

This will bring up the gsd, ons, listener, and vip. The same command can shut down the nodeapps by replacing sta

with stop.

[ oracl e@gr een ~] $ sr vct l st art nodeapps - n gr een[ oracl e@gr een ~] $ srvct l st art nodeapps - n r ed

Now we will check with crsstat again to be sure the nodeapps have started.

[ oracl e@gr een ~] $ crsst atHA Resour ce Tar get Stat e- - - - - - - - - - - - - - - - - - - - - -ora. green. ASM1. asm OFFLI NE OFFLI NE

or a. green. LI STENER_GREEN. l snr ONLI NE ONLI NE on greenor a. green. gsd ONLI NE ONLI NE on greenor a. green. ons ONLI NE ONLI NE on greenora. green. vi p ONLI NE ONLI NE on greenora. orcl . RAC. cs OFFLI NE OFFLI NEor a. or cl . RAC. or cl 1. sr v OFFLI NE OFFLI NEor a. or cl . RAC. or cl 2. sr v OFFLI NE OFFLI NEora. orcl . db OFFLI NE OFFLI NEor a. or cl . or cl 1. i nst OFFLI NE OFFLI NEor a. or cl . or cl 2. i nst OFFLI NE OFFLI NEora. r ed. ASM2. asm OFFLI NE OFFLI NEora. r ed. LI STENER_RED. l snr ONLI NE ONLI NE on r edora. r ed. gsd ONLI NE ONLI NE on r edora. r ed. ons ONLI NE ONLI NE on r edora. r ed. vi p ONLI NE ONLI NE on r ed[ r oot @gr een or acl e] #

Now we need to start our ASM instances before we bring up our database and services.

rvctl start asm -n (node)

This will bring up our ASM instances on nodes green and red. Again, the same command will stop the ASM instany replacing start with stop.

[ oracl e@gr een ~] $ srvct l st ar t asm - n gr een[ or acl e@gr een ~] $ srvctl st ar t asm - n red[ oracl e@gr een ~] $ crsst atHA Resour ce Tar get Stat e

- - - - - - - - - - - - - - - - - - - - - -or a. green. ASM1. asm ONLI NE ONLI NE on greenor a. green. LI STENER_GREEN. l snr ONLI NE ONLI NE on greenor a. green. gsd ONLI NE ONLI NE on greenor a. green. ons ONLI NE ONLI NE on greenora. green. vi p ONLI NE ONLI NE on greenora. orcl . RAC. cs OFFLI NE OFFLI NEor a. or cl . RAC. or cl 1. sr v OFFLI NE OFFLI NEor a. or cl . RAC. or cl 2. sr v OFFLI NE OFFLI NEora. orcl . db OFFLI NE OFFLI NEor a. or cl . or cl 1. i nst OFFLI NE OFFLI NEor a. or cl . or cl 2. i nst OFFLI NE OFFLI NEora. r ed. ASM2. asm ONLI NE ONLI NE on r edora. r ed. LI STENER_RED. l snr ONLI NE ONLI NE on r edora. r ed. gsd ONLI NE ONLI NE on r edora. r ed. ons ONLI NE ONLI NE on r ed


7/8



ora. r ed. vi p ONLI NE ONLI NE on r ed[ oracl e@green ~] $

Now lets bring up our orcl1 and orcl2 instances, and verify they are up with crsstat. Once more we can replace stawith stop and shutdown an individual instance if we so choose.

rvctl start instance -d (data base) -I (instance )

[ or acl e@gr een ~] $ sr vct l st ar t i nst ance d or cl i or cl 1[ or acl e@gr een ~] $ sr vctl st ar t i nst ance - d or cl - i or cl 2[ oracl e@gr een ~] $ crsst atHA Resour ce Tar get Stat e- - - - - - - - - - - - - - - - - - - - - -or a. green. ASM1. asm ONLI NE ONLI NE on greenor a. green. LI STENER_GREEN. l snr ONLI NE ONLI NE on greenor a. green. gsd ONLI NE ONLI NE on greenor a. green. ons ONLI NE ONLI NE on greenora. green. vi p ONLI NE ONLI NE on greenora. orcl . RAC. cs OFFLI NE OFFLI NEor a. or cl . RAC. or cl 1. sr v OFFLI NE OFFLI NEor a. or cl . RAC. or cl 2. sr v OFFLI NE OFFLI NEora. orcl . db ONLI NE ONLI NE on r edor a. or cl . or cl 1. i nst ONLI NE ONLI NE on gr eenor a. or cl . or cl 2. i nst ONLI NE ONLI NE on r edora. r ed. ASM2. asm ONLI NE ONLI NE on r ed

ora. r ed. LI STENER_RED. l snr ONLI NE ONLI NE on r edora. r ed. gsd ONLI NE ONLI NE on r edora. r ed. ons ONLI NE ONLI NE on r edora. r ed. vi p ONLI NE ONLI NE on r ed[ oracl e@green ~] $

rvct l sta rt service -d (database ) -s (service )

Now we will finish up by bringing our load balanced/TAF service named RAC online.

[ or acl e@gr een ~] $ srvctl st ar t ser vi ce - d or cl - s RAC[ oracl e@gr een ~] $ crsst at

HA Resour ce Tar get Stat e- - - - - - - - - - - - - - - - - - - - - -or a. green. ASM1. asm ONLI NE ONLI NE on greenor a. green. LI STENER_GREEN. l snr ONLI NE ONLI NE on greenor a. green. gsd ONLI NE ONLI NE on greenor a. green. ons ONLI NE ONLI NE on greenora. green. vi p ONLI NE ONLI NE on greenora. orcl . RAC. cs ONLI NE ONLI NE on r edora. orcl . RAC. orcl 1. sr v ONLI NE ONLI NE on gr eenora. orcl . RAC. orcl 2. sr v ONLI NE ONLI NE on r edora. orcl . db ONLI NE ONLI NE on r edor a. or cl . or cl 1. i nst ONLI NE ONLI NE on gr eenor a. or cl . or cl 2. i nst ONLI NE ONLI NE on r edora. r ed. ASM2. asm ONLI NE ONLI NE on r edora. r ed. LI STENER_RED. l snr ONLI NE ONLI NE on r edora. r ed. gsd ONLI NE ONLI NE on r edora. r ed. ons ONLI NE ONLI NE on r ed

ora. r ed. vi p ONLI NE ONLI NE on r ed[ oracl e@green ~] $

There we have it; all of our resources are now online. The next steps would be to verify you can connect via SQL*r your favorite application.

Conclusion

When a product or process is new to you, as Oracle 10g RAC is to many people, it can be an intimidating and posisastrous experience. This paper has hopefully given you the elementary commands you will need to manage your

Oracle 10g RAC system. While it is not a complete dissection of RAC and its total command set, it should be enou


8/8


o get you on your feet. You can always get the basic syntax of srvctl by typing srvctl help. For a complete list options, type srvctl h. You can also get the complete syntax for crsctl by typing crsctl at the command line. Also, ave a peek at the Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guidublication number B14197-04. You can find it on Oracles website atttp://download.oracle.com/docs/cd/B19306_01/rac.102/b14197/toc.htm.

About t he Author

Rich Headrick has worked in IT for over seventeen years and has worked with Oracle for over thirteen years. Hetarted out at Hewlett Packard Corporation working with their enterprise hardware and operating systems, and movnto the roles of development and production DBA at various companies in the San Francisco Bay Area andacramento. He has been implementing and managing RAC systems since 2001.

Still Looking for Help on this Subject?

Get a ConsultationWe would be happy to talk with you about our services and how our senior-level database team might helpyou. Call Database Specialists at 415-344-0500 or 888-648-0500 or fill out a free consultation request form.Complimentary NewsletterIf you'd like to receive our complimentary monthly newsletter with database tips and new white paper

announcements, sign up for The Specialist.

opyright 2007 Database Specialists, Inc.http://www.dbspecialists.com/
http://download.oracle.com/docs/cd/B19306_01/rac.102/b14197/toc.htmhttp://www.dbspecialists.com/consultation.htmlhttp://www.dbspecialists.com/specialist.htmlhttp://www.dbspecialists.com/http://www.dbspecialists.com/http://www.dbspecialists.com/specialist.htmlhttp://www.dbspecialists.com/consultation.htmlhttp://download.oracle.com/docs/cd/B19306_01/rac.102/b14197/toc.htm

a quick reference for oracle database 10g rac on linux and unix platforms.pdf

Documents