spotlight on oracle getting started 9-0

Spotlight® on Oracle 9.0

Getting Started Guide

© 2012 Quest Software, Inc.ALL RIGHTS RESERVED.

This guide contains proprietary information protected by copyright. The software described inthis guide is furnished under a software license or nondisclosure agreement. This software may beused or copied only in accordance with the terms of the applicable agreement. No part of thisguide may be reproduced or transmitted in any form or by any means, electronic or mechanical,including photocopying and recording for any purpose other than the purchaser’s personal usewithout the written permission of Quest Software, Inc.

The information in this document is provided in connection with Quest products. No license,express or implied, by estoppel or otherwise, to any intellectual property right is granted by thisdocument or in connection with the sale of Quest products. EXCEPT AS SET FORTH INQUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FORTHIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMSANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTSINCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OFMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT,INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES(INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESSINTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE ORINABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THEPOSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respectto the accuracy or completeness of the contents of this document and reserves the right to makechanges to specifications and product descriptions at any time without notice. Quest does notmake any commitment to update the information contained in this document.

If you have any questions regarding your potential use of this material, contact:

Quest Software World HeadquartersLEGAL Dept5 Polaris WayAliso Viejo, CA 92656email: [email protected]

Refer to our Web site (www.quest.com) for regional and international office information.

Patents

Protected by U.S. Patents #6,901,582, 7,784,027 and 8,015,454. Additional patents pending.

Trademarks

Quest, Quest Software and the Quest Software logo are trademarks and registered trademarks ofQuest Software, Inc in the United States of America and other countries. For a complete list ofQuest Software’s trademarks, please see http://www.quest.com/legal/trademark-information.aspx.Other trademarks and registered trademarks are property of their respective owners.

http://www.quest.com/

http://www.quest.com/legal/trademark-information.aspx

http://www.quest.com/legal/trademark-information.aspx

Spotlight on Oracle 9.0Getting Started GuideThursday, 6 September 2012

Table of Contents

Welcome 7

Oracle Architecture Overview 7

Spotlight on Oracle Overview 8

Home Page 8

Alarms 9

Drilldowns 9

Features of Spotlight 10

Features Common to Spotlight Applications 10

Features Specific to Spotlight on Oracle 10

Features Specific to Spotlight on Oracle RAC 10

Predictive Diagnostics 10

DTrace Analysis (Unix Solaris 10 And Later) 11

Install Spotlight 12

Requirements 12

Steps To Install 13

Multiple Concurrent Installations 14

MSI Installers for Spotlight on Oracle 14

Uninstall Spotlight 14

Start Spotlight 15

Project Lucy 15

Authorize Spotlight 16

Spotlight Users 17

Oracle User Wizard 17

Spotlight on Oracle 20

Connect to Oracle Instance 20

Toad DBA Suite for Oracle 25

Spotlight on Oracle Getting Started GuideTable of Contents

5

Install And Configure DTrace Scripts 26

Spotlight on Oracle Home Page 26

Spotlight on Oracle Alarms 31

Spotlight on Oracle Drilldowns 39

Spotlight Tools 42

Tune Your Database 43

Predict Database Performance 45

Open Predictive Diagnostics 45

Predictive Diagnostics for Beginners 45

Predictive Diagnostics And Spotlight 46

Requirements for Predictive Diagnostics 46

Spotlight on Oracle RAC 48

What is Spotlight on Oracle RAC? 48

What Is Oracle RAC? 48

Connect to Spotlight on Oracle RAC 49

Spotlight on Oracle RAC Home 4

Interconnect and Global Status Panel 55

Instance Panels 56

IO Subsystem Panel 57

Exadata Storage Panel 57

Spotlight on Oracle RAC Alarms 57

Alarms on A Database Instance 58

Alarms on The Oracle RAC 58

Spotlight on Oracle RAC Drilldowns 61

Troubleshooting Spotlight on Oracle 64

Product Authorization Errors 64

Connection Problems 65

Spotlight on Oracle Getting Started GuideTable of Contents

6

Cannot Connect to Database 65

Error Attempting to Connect 65

Retrieve Data from Unix 66

Cannot Connect to ASM 66

Drilldown Problems 66

Alarm Help Displayed Instead of Drilldown 66

Dead Sessions in Top Sessions Drilldown 67

What Happens to The Operating System Drilldown when Connection Is Lost to TheServer? 67

Operating System Statistics Not Reported 67

Predictive Diagnostics Unavailable 68

High Spotlight Load on Database Server 68

Appendix: Contact Quest 69

Contact Quest Support 69

Contact Spotlight Support 69

Contact Quest Software 70

About Quest Software 70

Index 71

WelcomeSpotlight on Oracle is Quest Software’s powerful diagnostic and problem-resolution tool forOracle databases. Its unique user interface provides you with an intuitive, visual representation ofthe activity on your database and operating system.

Oracle Architecture OverviewThe following diagram illustrates some of the basic components of an Oracle server at thememory, database and disk layers. This architecture has been used to design the Spotlight onOracle home page.

The numbers indicate the order of the flow of information.

The numbered labels in the Oracle architecture diagram correspond to the following activities:

1. The client program (for example, SQL*PLUS, Oracle Power Objects, or some other tool)sends a SELECT statement to the server process.

2. The server process looks in the shared pool for a matching SQL statement. If none isfound, the server process parses the SQL and inserts the SQL statement into theshared pool.

3. The server process looks in the buffer cache for the data blocks required. If found, thedata block must be moved on to the most recently used end of the Least RecentlyUsed (LRU) list.

1

Spotlight on Oracle Getting Started GuideWelcome

8

4. If the block cannot be found in the buffer cache the server process must fetch it from thedisk file. This requires a disk I/O.

5. The server process returns the rows retrieved to the client process. This may involve somenetwork or communications delay.

6. When the client issues the UPDATE statement the process of parsing the SQL andretrieving the rows to be updated must occur. The update statement then changes therelevant blocks in shared memory and updates entries in the rollback segment buffers.

7. The update statement also makes an entry in the redo log buffer that records thetransaction details.

8. The database-writer background process copies modified blocks from the buffer cache tothe database files. The Oracle session performing the update does not have to wait forthis to occur.

9. When the COMMIT statement is issued the log writer process must copy the contents ofthe redo log buffer to the redo log file. The COMMIT statement does not return control tothe Oracle session issuing the commit until this write is complete.

10. If running in ARCHIVELOG mode, the archiver process copies full redo logs to thearchive destination. A redo log is not eligible for re-use until it has been archived.

11. At regular intervals, or when a redo log switch occurs, Oracle performs a checkpoint. Acheckpoint requires all modified blocks in the buffer cache to be written to disk. A redolog file cannot be re-used until the checkpoint completes.

Recommended Reading

The best information to help you tune and configure Oracle databases can be found online. Useas your first point of reference web sites such as:

l The Quest Software Oracle Solutions Web site.

l The Oracle home page.

Spotlight on Oracle OverviewSpotlight on Oracle makes it possible for you to view your database at different levels of detail.

Home Page

The Spotlight home page is a visual representation of the status of the major components of thedatabase. The detail at this level is designed to help you locate and identify bottlenecks as theydevelop. Additionally, Spotlight provides tools that allow you to identify and rectifyinefficiency in your database’s configuration.

If Spotlight detects that some component of the Oracle database constitutes a performancebottleneck, the corresponding component generates a visual or auditory alarm. The exactappearance of the alarm depends on how you have configured alarm severities. When an alarm iscurrent you can press F1 to display help for the alarm. Clicking the component (on the homepage ) displays the relevant drilldown.

http://www.quest.com/oracle/index.html







http://oracle.com/index.html





9

See "Spotlight on Oracle Home Page" (page 26) for more information.

The home page display is based on the Oracle database architecture. See "Oracle ArchitectureOverview" (page 7) for more information.

Alarms

Spotlight alarms indicate when the Oracle database exceeds acceptable performance thresholds.

See "Spotlight on Oracle Alarms" (page 31) for more information.

Drilldowns

Once a problem is isolated you can display a drilldown page with charts and tables that providea detailed breakdown of the underlying statistics.

For example, use the Activity drilldown to display overall efficiency and resource usage. Inparticular, the Event waits chart shows the amount of time database sessions spend waiting forvarious resources, as well as the amount of CPU being utilized. Efforts at tuning the database aremost likely to succeed when they concentrate on the resources being used most heavily and theWait activity topic contains a discussion of the meaning and implications of various wait events.

Other pages help you examine detailed activity. For instance, the Top Sessions drilldownidentifies the users who contribute most to database activity. You can display various details forthese sessions, including the SQL statements, locks and resource usage.

See "Spotlight on Oracle Drilldowns" (page 39) for more information.


10

Features of Spotlight

Features Common to Spotlight Applications

Spotlight is designed for ease of installation and use, and offers a range of tools for problemdiagnosis. Features that are common to most Spotlight applications include:

l Easy installation.

l The display of real-time data.

l Comprehensive information in charts and tables.

l Visible, audible and configurable alarms.

l Context-sensitive help.

l Calibration.

l Recording and playback of historical data.

l Common Spotlight technology. Spotlight on Oracle can run within the same Spotlightconsole as several other Quest Spotlight products.

Features Specific to Spotlight on Oraclel The ability to add new Spotlight on Oracle users, and to convert the accounts of existing

Oracle users so they can use Spotlight on Oracle.

l Access to a number of other Quest applications for viewing detailed information on thespecified Oracle database.

Features Specific to Spotlight on Oracle RACl The ability to verify connection details for Oracle RAC instances.

l The ability to open Spotlight on Oracle from Spotlight on Oracle RAC.

Predictive Diagnostics

Predictive Diagnostics allows Spotlight to collect and analyze performance metrics for:

l SQL statements as they are executed on the database instance.

l System bottlenecks as they affect database throughput and response time.

l Database resources as they reach their full capacity and affect database throughput andconcurrency.

When enough performance data is collected, Spotlight may be able to predict the futureperformance for individual SQL statements on that database, for potential bottlenecks in thesystem, and for database CPU, memory, and disk I/O resources.


11

Note: When you start using Predictive Diagnostics, Spotlight will take about 10 days to collectenough data to make valid predictions on the future performance of the database.

DTrace Analysis (Unix Solaris 10 And Later)

DTrace is a dynamic analysis tool for Sun Microsystems operating systems. It was firstintroduced in the Solaris 10 operating system, and can be used ONLY on Sun platforms fromSolaris 10 onwards.

Spotlight and DTrace

Spotlight on Oracle uses DTrace to capture details of the interaction between Oracle and theoperating system.

When Oracle sessions use CPU time, most of that elapsed time is spent executing Oracle code.However, Oracle sometimes uses system calls to perform functions such as:

l Allocating or deallocating private (PGA) memory.

l Synchronizing activities with other processes through semaphores or other IPCmechanisms.

l Allocating shared memory (SGA).

l Performing I/O operations on datafiles, redo logs, raw devices, and so on.

l Capturing CPU or elapsed timing to support the time model and wait interfaces.

Spotlight uses DTrace to investigate these system calls to calculate where that elapsed time isbeing spent.

Install and Configure DTrace scripts

Before you can capture the details of the interaction between Oracle and the Solaris operatingsystem, you need to Install And Configure DTrace Scripts (page 26).

Install Spotlight

RequirementsWindows 7 / Windows Vista / Windows Server 2008 :

Section Requirement

User Privileges Ensure the Windows user installing Spotlight hasWindows Administrator privileges.

Folder Permissions During installation, Spotlight requires write access tothe following:

l The Spotlight installation folder (by default,Program Files\Quest Software\Spotlight)

l Program Files\Common Files\Quest Shared

l ProgramData\Quest Software\Spotlight

l Users\<current user>\AppData\Roaming\QuestSoftware\Spotlight

Note: For 64-bit operating systems the Program Filesdirectory is Program Files (x86).

Permissions to Windowsregistry keys and sub-keys

During installation, Spotlight requires write access tothe following:

l HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\QuestSoftware\Products

l HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\QuestSoftware\Spotlight

l HKEY_CLASSES_ROOT

2

Spotlight on Oracle Getting Started GuideInstall Spotlight

13

Windows XP / Windows Server 2003:

Section Requirement

User Privileges Ensure the Windows user installing Spotlight hasWindows Administrator privileges.

Folder Permissions During installation, Spotlight requires write access tothe following:

l The Spotlight installation folder (by default,Program Files\Quest Software\Spotlight)

l Program Files\Common Files\Quest Shared

l Documents and Settings\All Users\ApplicationData\Quest Software\Spotlight

l Documents and Settings\<currentuser>\Application Data\QuestSoftware\Spotlight

Permissions to Windows registrykeys and sub-keys

During installation, Spotlight requires write access tothe following:

l HKEY_LOCAL_MACHINE\SOFTWARE\QuestSoftware\Products

l HKEY_LOCAL_MACHINE\SOFTWARE\QuestSoftware\Spotlight

l HKEY_CLASSES_ROOT

Steps To InstallTo install Spotlight from CD:

1. Close all open applications on your PC.

2. Insert the Spotlight Installer CD into your CD-ROM drive.

If the installation program does not start select Windows Start | Run

l For Spotlight delivered as part of the Toad DBA Suite, enter the commandd:\TOAD DBA Suite for Oracle vvv.exe (where d: is the CD-ROM identifier andvvv is the Spotlight version).

l For Spotlight delivered as a standalone installer, enter the commandd:\SpotlightonOracle_vvv.exe (where d: is the CD-ROM identifier and vvv is theSpotlight version).

3. Follow the instructions on the screen to complete the installation.

Spotlight on Oracle Getting Started GuideInstall Spotlight

14

To install Spotlight from the downloaded Quest Spotlight installer from the Quest Web site:

1. Close all open applications on your PC.

2. Execute the downloaded installer.

3. Follow the instructions on the screen to complete the installation.

Note: To cancel installation at any time, press Esc or click Cancel.

Multiple Concurrent InstallationsMany Spotlight applications can run within a single Spotlight installation and you can installand separately upgrade multiple copies of Spotlight on your computer.

Note: If Spotlight is delivered as part of the Toad DBA Suite and you require multiple Spotlightinstallations on your computer then contact Quest Software for more information.

MSI Installers for Spotlight on OracleMSI installers are available for Spotlight on Oracle. These installers can be run silently, andproduce an installation that can be captured and re-used via automatic deployment techniques.

Contact Quest Software for more information.

Uninstall SpotlightTo remove the application and all associated files from your machine, if required.

To uninstall Spotlight

1. Open Windows Control Panel.

2. Select Programs and Features. (On Windows XP select Add/Remove Programs.)

3. Click Spotlight on Oracle | Remove | Yes.

Spotlight on Oracle Getting Started GuideStart Spotlight

15

Start SpotlightTo start Spotlight:

» Double click on the Spotlight desktop icon

or…

» On the Windows task bar, click Start | Programs | Quest Software | Spotlight |Spotlight.

During operation

Spotlight requires write access to the following:

Windows Server 2008, Windows Server2003, Windows Vista, Windows 7

Users\<current user>\AppData\Roaming\QuestSoftware\Shared

Users\<current user>\AppData\Roaming\QuestSoftware\Spotlight

Windows XP Documents and Settings\<currentuser>\Application Data\Quest Software\Shared

Documents and Settings\<currentuser>\Application Data\QuestSoftware\Spotlight

You should also verify that you can run the required Oracle client on the Spotlight machine.

Project LucyThe first time you run Spotlight following install you will be prompted to upload data to ProjectLucy. Project Lucy is a free service provided by Quest Software that analyzes your performancedata, gives you reports and insight into your performance and allows you to compare yourexperience to everyone else that uses the service. Performance data is uploaded to Project Lucyonce a day.

Actions Description

To open the Project Lucy home page, in the SpotlightBrowser click Welcome to Spotlight | Spotlight on theWeb | Project Lucy.

l Learn more aboutProject Lucy and howit can benefit you.


16

Actions Description

l Login to your ProjectLucy account and useProject Lucy services.

l Create a Project Lucyaccount.

To configure uploading data from Spotlight on Oracle toProject Lucy, click View | Options | Project Lucy.

See "Project Lucy" in theonline help for moreinformation.

Authorize SpotlightSpotlight on Oracle is supplied with a time-limited trial key so you may test the product. Whenthe trial key expires Spotlight on Oracle reverts to a preview mode, which limits access to theSpotlight on Oracle home page and one connection at a time. If you select to open a drilldownin preview mode the following message is displayed:

You do not have a license for “Spotlight on Oracle” that enablesaccess to this functionality. Please contact Quest Software forinformation about obtaining the correct license.

Contact your Quest Software representative to obtain the necessary authorization keys to ensureSpotlight remains fully functional when the trial key expires.

To enter a new authorization key:

While Spotlight is running

1. Click Help | About Spotlight.

2. Ensure Installed Products is to the front.

3. Locate any Spotlight on Oracle in the list of installed products.

4. Select View / change product license.

5. Click Change this license.

6. Enter the Authorization key.

Notes:

l The Windows user that updates the authorization key must have administrator access tothe Windows registry and write access to the Console\Licenses sub-folder of theSpotlight installation folder.

l Make sure you enter the authorization key exactly as stated on your ProductAuthorization sheet or as provided by your Quest Software distributor. The site message


17

is case sensitive. If you enter the authorization key incorrectly, the following message isdisplayed: Invalid authorization key

l If Spotlight is in preview mode and the trial period has not expired, check your computersystem date is set correctly.

Spotlight UsersSpotlight on Oracle connects to the Oracle database system through the Spotlight user.

l You should not use any system accounts. We recommend creating a separate user forSpotlight or using a personal account.

l The Spotlight user must have sufficient privileges, views, and tables to diagnose theOracle database. You can view the privileges, views and tables required by Spotlightusers in the Database Objects and Privileges document in the Spotlight installationfolder: \Plug-ins\SoO\.

The Oracle User Wizard configures Spotlight users.

Oracle User Wizard

The Oracle User Wizard configures Spotlight users. Use the Oracle User Wizard to:

l Add a new user to the Oracle database

l Convert the account of an existing user on the Oracle database to use Spotlight

l Reconfigure an existing Spotlight user to work with new versions of Spotlight onOracle and Spotlight on Oracle RAC. Use this procedure if you have upgraded yourversion of Spotlight.

Notes:

l Spotlight users must be configured for each database they are to diagnose.

l You cannot configure Spotlight users for a database while Spotlight is diagnosingthat database.

l You must have the user ID and password of the DBA that manages the database to whichyou are adding the Spotlight user.

l Select Don't show this page in future to hide the first page of the Oracle User Wizard.

To fill in the Oracle User Wizard

The Oracle User Wizard runs automatically when required or to initiate it:

1. Click File | Oracle User Wizard...

2. At the DBA connection credentials page:


18

Field Description

Enter theSQL*Netconnectionstring…

Enter the database instance on which you wish to create the user.

Enter the username of aDBAaccount…

Enter a DBA user name for that database instance (so that theSpotlight user and the relevant objects can be created). The username requires SYSDBA access.

Enter thepassword…

Enter the DBA user password.

3. Select Create a new user

4. At the New oracle user page:

Field Description

User name Enter a name for the new user.

Password Enter a password for the new user.

Confirm password Confirm the password.

Select any table Select to allow the user to generate explain plans.

Alter system Select to allow the user to trace and terminate sessions.

5. At the Select tablespace page:

Field Description

…Spotlight tables The default tablespace is USERS, if available.

…temporary segments The default temporary tablespace is TEMP, if available.

6. At the Predictive Diagnostic Settings page:

Select Create/Upgrade Predictive Diagnostics schema to employ Predictive Diagnosticson this database instance.

7. Select to use an existing Predictive Diagnostics schema (recommended) or installa new schema.

8. Select to install a set of sample data (which you can use to experiment with the PredictiveDiagnostics feature).


19

9. Select the settings Spotlight will use to schedule the Oracle jobs that collect data on SQLperformance and impending bottlenecks.

Spotlight on Oracle

Connect to Oracle Instance1. From the Spotlight Browser select All Connections | Spotlight on Oracle

2. Select the database. If the database is not on the list then add a new connection.

Notes: (Now the database is listed in the Spotlight Browser | Live connections.)

l Click Set timed statistics now if you are prompted to do so. You see this promptwhen for the Spotlight user the timed_statistics parameter is set to false in theinit.ora file. Spotlight requires this parameter be true for the Spotlight user toaccurately monitor the database. Clicking Set timed statistics now corrects theproblem only if the Spotlight user has Alter System privileges. Ensure theSpotlight user has sufficient privileges. See "Spotlight Users" (page 17) for moreinformation.

l Spotlight operates at reduced functionality once the initial trial period has expired.See "Authorize Spotlight" (page 16) for more information.

Add A New Connection

Notes: (Before you add a new connection)

l Ensure you create a Spotlight user with sufficient privileges to diagnose the Oracledatabase. See "Spotlight Users" (page 17) for more information.

l Ensure the Unix or Windows server on which the Oracle database is installed isaccessible to Spotlight on Oracle. See the Spotlight on Unix or Spotlight on WindowsGetting Started Guide.

1. Click File | Connect

3

Spotlight on Oracle Getting Started Guide 21

2. Select Spotlight on Oracle on the Connections menu.

3. Double-click Add new connection.

4. Ensure Details is selected. Fill in the Properties window.

Oracle connection details

Field Description

ConnectionString

Select the name of the database that you wish to connect to.Note:When selected, the Connection name field resets to this value.You can then edit the Connection name field if required.

OracleUsername

Enter the Spotlight user name. See "Spotlight Users" (page 17) formore information.Tip: If you enter a Username that does not exist, Spotlight willprompt you when you complete this dialog to create a new Spotlightuser.

OraclePassword

Enter the Spotlight user password. See "Spotlight Users" (page 17) formore information.

OracleHome

Select an appropriate Oracle Home. Select a location where Oracleclient software is installed.Note: Choose a home that contains a version of the Oracle client thatis compatible with the Oracle server.

ASM connection details

ASM is built into the Oracle database kernel. It simplifies the creation and spacemanagement in Oracle databases.

Note: Available only in Oracle 10g and later.

Field Description

MonitorASM

Select to monitor the performance of Oracle's Automated StorageManagement (ASM) technology.

ASMConnectionString

Enter the connection string required to link to the ASM databaseinstance that is used for storage management for this Oracle database.


Field Description

Note: The TNSNAMES entry for the ASM instance must include theoption (UR = A) for the Spotlight connection to work properly.

ASMPassword

Enter the user password for Spotlight to login to the ASM databaseinstance.The user that Spotlight uses to login to the ASM database is SYS.This is the only user name that can provide access to ASM-specificinformation.Providing SYS access to an ASM instance does NOT create a securityissue for the Oracle database, but you should limit access to the SYSaccount to trusted individuals.Because the SYS password is stored in encrypted format, non-privileged users can still access Spotlight on Oracle. To do so:

l Use a pre-configured or shared Spotlight on Oracleconnection, OR

l Use a Spotlight connection without ASM access. Spotlightwill display Oracle-related data, but the ASM drilldown willcontain no data.

Server connection details

Field Description

Monitor OS Select to monitor the operating system on the database server.

OS Server Enter the name of the server.Tip: Use a period “.” to monitor the local database.

OS Type Select the type of operating system.

l Windows

l Unix(REXEC)

l Unix(SSH)

l Unix(SSH using Public/Private Keys)

If you select one of the Unix systems, make sure the SSH or REXECdaemon is running on the database server and is configured toreceive remote connections. Spotlight supports both SSH1 and SSH2protocols. Public-key encryption is supported under SSH2 only.


Field Description

SSH Port Enter the port number for SSH (secure shell) connection to thedatabase server. The default value is 22.Note: Applicable when OS Type is Unix(SSH) or Unix(SSH usingPublic/Private Keys).

OSUsername

Type the user name to login to the database server.Note:When OS Type is Windows:

l Not required if your login details have been assigned remoteaccess to the registry of the database server.

l Ensure the user has access to the registry of the server.

SSH KeyType

Select the key type from the options: RSA and DSA.Note: Applicable when OS Type is Unix(SSH using Public/PrivateKeys).

SSH PrivateKeyFilename

Locate the filename that contains the private key for the Spotlightconnection.Hint: Use the "..." button.Note: Applicable when OS Type is Unix(SSH using Public/PrivateKeys).

SSHPassphrase

Type the passphrase used to decrypt the private key.Note: Applicable when OS Type is Unix(SSH using Public/PrivateKeys).

OSPassword

Type the user password to login to the database server (if required).Note: Not applicable when OS Type is Unix(SSH usingPublic/Private Keys).

Connect toOSSpotlightwhenstarting thisconnection

Select to have the database server connection open automaticallywhen you open the connection to the Oracle database.

ESX connection details

Note: Requires Server connection details | Monitor OS selected.


Field Description

MonitorESX

Select to monitor the virtual machine that hosts the database.

ESXWebService

Type or select the URL of the ESX Web Service that manages thevirtual machine.

ESXUsername

Type the username required to connect to the ESX Web Service.Note: The username specified must have read only permissions for boththe ESX virtual machine and the ESX host that the virtual machineresides on, or may reside on.

ESXPassword

Type the password required to connect the specified user to the ESXWeb Service.

VirtualMachineName

Click ... to select the virtual machine.

Stealth Collect

StealthCollect is Quest technology designed to collect SQL statements and performanceindicators from Oracle databases and host operating systems without imposing overheadson the target database itself.

Note: Requires StealthCollect installed. StealthCollect is installed with PerformanceAnalysis.

Field Description

UseStealthCollect

Select Use StealthCollect to collect SQL statements andperformance data using StealthCollect.

SC Port No. Enter the port number where Spotlight can access StealthCollectdata.If required, click the … button to enter the StealthCollect usernameand password. (The default StealthCollect username and passwordis quest.)

SC OracleSID

Enter the StealthCollect Oracle SID for the database.

5. Click Test to verify Spotlight can connect successfully with the details provided.

Note: For connections to virtual machines, testing the connection validates that Spotlightcan connect to the Web Service using the specified username and password combination,


and that the virtual machine exists. It does not check permissions, that the virtual machineis turned on, or that it is the same machine as the one specified in the Server ConnectionDetails section.

6. Click Save password details (for this connection) to save all the enteredpassword details.

Toad DBA Suite for Oracle

Note: This section is relevant for systems with the following software installed: Toad DBA Suitefor Oracle with Spotlight on Oracle.

To connect to an Oracle database from within Toad

1. Click the Spotlight button where it appears in Toad.

2. Spotlight on Oracle opens at the Oracle database connection associated with the Toad forOracle connection.

The first time you open the Oracle database connection from within Toad

You are required to fill in the Details page with the following additions. See "Connect to OracleInstance" (page 20) for more information.

l The Oracle database and username fields are pre filled in as per the Toad connection.Security requires that you re-enter your Oracle password.

l The default connection name is Toad followed by the name of the Oracle database. Youmay choose to replace this with any connection name of your choice. Note that alimitation of the integration of Toad with Spotlight on Oracle is that this is a newSpotlight connection, ignoring all other pre-existing Spotlight connections.

l You may choose a different Oracle username to connect to Spotlight. For example, in anenvironment with multiple database administrators there may be a communal username forSpotlight connections. Simply replace the filled in Oracle username with the appropriateOracle username and password details.

l You can enter additional parameters to monitor ASM, the operating system and forvirtualization.

Using Spotlight on Oracle

l Spotlight opens at the Home Page or Drilldown relevant to the location in Toad fromwhere Spotlight was opened.

l You are prompted to create a new Spotlight user if the Oracle username has not beencreated in Spotlight. See "Spotlight Users" (page 17) for more information.

l This Spotlight connection can be found in the Spotlight Connection Manager. It can bemanaged the same as any other Spotlight connection. For example, you can close theconnection, edit the connection details and re-open the connection in the SpotlightConnection Manager.


Caution: When you click on the Spotlight button in Toad to connect to the Oracle database,a new Spotlight connection to the Oracle database is created, regardless of any pre-existingSpotlight connections to the Oracle database. To avoid the annoyance of setting up Spotlightconnections in Spotlight that will be set up identically through Toad later, you may chooseto create all connections through Toad if Toad integration is desired.

Install And Configure DTrace Scripts

DTrace is a dynamic analysis tool for Sun platforms. Spotlight on Oracle uses DTrace to capturedetails of the interaction between Oracle and the operating system (Solaris 10 onwards).

To use DTrace within Spotlight, you need to install and configure a number of DTrace scripts onthe server. Once installed, use the DTrace drilldown to view the DTrace analysis. See "Spotlighton Oracle Drilldowns" (page 39) for more information.

To install and configure DTrace scripts

1. Locate in the Spotlight installation folder: Plug-ins\SoO\SoO_DTrace_scripts.tar

2. Unpack SoO_DTrace_scripts.tar to a directory on the database server.

3. Ensure the unpacked files have execute permission.

4. For the account that will be used to run operating system commands from Spotlight:

l Edit .profile to define a QUEST_SOO_HOME variable that points to the directory onthe database server where the unpacked files are.

l Provide DTrace authority. Add this line to /etc/user_attr:

<<username>>::::defaultpriv=basic,dtrace_proc,dtrace_kernel

Spotlight on Oracle Home Page

The Spotlight home page shows the flow of information and commands between various sub-components and the size and status of internal resources such as processes, disk files and memorystructures.

Related statistics are grouped together on panels that are connected by a series of graphical flowsand icons. Spotlight updates these flows in real time so you can see how quickly data is movingthrough the system. The icons change color as Spotlight alarms are raised, upgraded, downgradedand canceled.

The panels on the Spotlight on Oracle home page are described in more detail next. For a fulldiscussion of the panels, see the Spotlight online help.

Tip: Hover the mouse pointer over a panel component for more information.


Sessions Panel

The Sessions panel indicates the status of the Oracle communicationsbetween client and server. Information displayed here includes:

l The time (in milliseconds) that it takes for the query select user fromdual to be submitted, executed and returned.

l The total number of users connected to the database.

l The number of users who are currently active.

l The sum of DB time over all sessions, divided by elapsed time.

Note: The Average Active Sessions gauge is available for (Oracle 10g andlater).

The flows between the Sessions panel and the Server Processes panel represent the rate of datatransfer in both directions across the SQL*NET interface.


Host Panel

The Host panel shows metrics for the operating system:

l The total amount of CPU being used.

l The number of tasks waiting for CPU resources.

l The percentage of CPU that is unavailable to the virtual machine (ifapplicable).

l The available physical RAM.

l The amount of memory that has been reclaimed from the virtualmachine by ESX (if applicable).

Server Processes Panel

The Server Processes panel indicates the status of Oracle server processesthat perform database activities on behalf of end users, and mediate databaseconnections:

l Statistics for the PGA memory allocated by server processes.

l Dedicated server processes that perform work on behalf of a singleclient process.

l Shared or multi-threaded servers (MTS) that perform work on behalfof more than one client process.

l MTS dispatchers that coordinate the allocation of shared servers toclient tasks.

l Server processes that support parallel execution of queries.

l The current length of the Parallel Execution Queue.Note: The Parallel Exec Queue gauge applies only to (Oracle 11gRelease 2).

l Job queue server processes that run PL/SQL commands submitted tothe Oracle job queue via the DBMS_JOB package.

l The state of the EM agent connection between Oracle EnterpriseManager (if installed) and the database.

l The number of server processes that are currently active.

The flow to the Server Processes panel from the Disk Storage panel represents the rate of directphysical reads from disk.

The flows between the Server Processes panel and the SGA panel represent the rates of:


l Blocks reads from the SGA by all server processes.

l Block changes in the SGA by all server processes.

l Redo buffer entries made by all server processes.

l SQL parse requests per second by all server processes.

l SQL execution requests per second by all server processes.

SGA Panel

The SGA panel shows details of specific memory areas within the systemglobal area (SGA). The SGA is an area of shared or common process memorythat is used to cache frequently used data, SQL statements, procedures andother structures:

l The size of the buffer cache and its components. These hold data thatis accessed frequently.

l The current size of the SGA.

l The size of the Java Pool, which caches class definitions, Javamethods and Java objects.

l The size of the Large Pool allocation heap, which is used in MTS forsession memory. It can be used by parallel execution and backupprocesses.

l The size of the redo buffer. This contains redo entries that areeventually written to the redo log.

l The size of the Shared Pool, which caches SQL statements, PL/SQLprograms, object definitions and session memory for MTS sessions.

l The size and activity of the result cache (Oracle 11g and later).

l Whether the database is in manual or auto memory managementmode (Oracle 10g and later).

The flow between the SGA panel and the Disk Storage panel represents the rate of blockreads from disk.

The flows between the SGA panel and the Background Processes panel represent the rates of:

l Block writes by the Database writer (DBWR) process.

l I/O writes by the Recovery writer (DBWR) process.

l Redo log block writes by the Redo Log Writer (LGWR) process.


Background Processes Panel

The Background Processes panel displays the following key Oraclebackground processes:

l Database Writer (DBWRn).

l The Recovery Writer (RVWRn).

l The Log Writer (LGWRn).

l The Archiver (ARCHn).

l Data Guard Overhead and Apply Lag

l Predictive Diagnostics.

Note: The n value in each process icon indicates the number of processesthere are of that type.The flows between the Background Processes panel and the Disk Storagepanel represent:

l The rate at which modified blocks are written from the SGA to diskby the DBWR processes.

l The rate of I/O write operations to the flash recovery area by theRVWR processes.

l The rate at which redo log entries are written to the redo log files bythe LGWR processes.

l The rate at which the archiver copies from redo log files to archivedlogs.

The Background Processes panel also contains a status indicator forPredictive Diagnostics (if installed). See "Predict Database Performance"(page 45) for more information.


Disk Storage Panel

The Disk Storage panel represents Oracle database and log files on disk,including:

l Oracle database files.

l Size of the flash cache.

l Oracle flashback recovery files.

l Oracle redo logs.

l The archive log destinations.

Right-click the Disk Storage icon to open Space Manager. See "Tools| Space Manager" (page 43) for more information.

Spotlight on Oracle Alarms

Spotlight alerts you to problems with your system by issuing an alarm. You can configureSpotlight in the level of severity that constitutes an alarm, to disable an alarm, and the actionsSpotlight takes on raising the alarm.

When an alarm is raised Spotlight performs one or more of the following actions:

l Changes the color or intensity of relevant components.

l Gives audible warnings.

l Sends email notifications.

Actions you can take on an alarm being raised include:

l View details of the statistics that caused the alarm in a Spotlight drilldown page. See"Spotlight on Oracle Drilldowns" (page 39) for more information.

l View the Spotlight online help.

l View details of the alarm in the Spotlight Alarm Log drilldown.

l Snooze the alarm.

l Save the alarm details.

l Filter the list of alarms.


Following are the alarms specific to Spotlight on Oracle. For possible solutions to the problemsindicated by these alarms or for information on how to diagnose problems further, see Spotlighton Oracle alarms in the online help.

Alarm Description

Agent Statusalarm

Spotlight raises the Agent Status alarm when the status of the EM agent isdown.OEM is a Web-based database administration and monitoring tool thatmanages Oracle components and administers Oracle databases. Wheninstalled and active, OEM communicates with the databases it administersvia a set of EM agent processes under the automated DBSNMP useraccount.

ArchiveCriticalFailure alarm

An Archive Critical Failure alarm indicates that the number of workingarchive destinations for the database instance is about to fall below theminimum number needed to archive Oracle redo logs successfully. Whenyou see this alarm, you should make space in the archive log destinationby purging – or by backing up to an offline medium – any archived filesthat are not immediately needed.

ArchiveDestinationFailed alarm

Spotlight raises an Archive Destination Failed alarm when at least one ofthe archive destinations specified for redo log files has become full. SeeArchive Destination Failure alarm below.

ArchiveDestinationFailure alarm

An Archive Destination Failure alarm when at least one of the archivedestinations specified for redo log files is about to become full. When yousee this alarm, you should make space in the archive log destination bypurging it, or by backing it up to an offline medium.

Array FetchSize alarm

When more than one row is retrieved from an Oracle database in a singleSELECT (an array fetch), the amount of network traffic associated withSELECT operations is radically reduced. The Array Fetch Size alarm israised when the array fetch size for the database may be too low.

ASM Balancealarm

This alarm indicates that there is unbalanced access to one or more of thedisks in the ASM (Automatic Storage Management) disk group that hasbeen allocated to the Oracle instance.

ASM ServiceTime alarm

This alarm indicates that the average time taken to perform I/O on theASM instance has exceeded a specified threshold value. (Service timerepresents the response time of a disk plus any wait time experiencedduring disk operations.)

ASM Statusalarm

When a Spotlight connection is lost, Spotlight automatically tries to re-establish the connection. The ASM Status alarm indicates that Spotlight isre-establishing its connection to the ASM diskgroup for the Oracleinstance.


Alarm Description

AverageTime to Synca Redo LogEntry alarm

The Average Time to Sync a Redo Log Entry alarm (formerly the AverageRedo Write Time alarm) is activated when the time taken to write redo logentries exceeds a threshold. Unlike most other Oracle write I/Os, Oraclesessions must wait for redo log writes to complete before they cancontinue processing.

Buffer BusyWait alarm

The Buffer Busy Wait alarm occurs when a session cannot access a blockbecause it is in use by another session. The two most common causes areinsufficient free lists for a table or insufficient rollback segments.

Cache BufferChains LatchWait alarm

The Cache Buffer Chains Latch Wait alarm occurs because contention forthe cache buffer chains latch exceeds a threshold.

Cache BufferLRU ChainsLatch Waitalarm

The Cache Buffer LRU Chains Latch Wait alarm occurs becausecontention for the cache buffer chains latch exceeds a threshold.

CollectionStatus alarm

Spotlight raises the Collection Status alarm when collection of data for theOEM (Oracle Enterprise Manager) stops. Perhaps the server is running lowon disk space.The Collection Status alarm often occurs with the Files Pending UploadAlarm.

ContinuedFetch Ratealarm

Oracle sometimes needs to migrate a row in a database from its originalblock to a new block, or needs to chain an updated row across multipleblocks. These migrated and chained rows cause additional I/O, and theContinued Fetch alarm occurs when the number of these "continued row"fetches exceeds a threshold.

CPU Busyalarm

The CPU Busy alarm occurs when the total CPU utilization of the systemexceeds a threshold.

CPUVirtualizationOverheadalarm

The CPU Virtualization Overhead alarm is activated when VirtualizationOverhead is greater than a threshold and CPU Usage is 95% or higher.

Datafile ReadTime alarm

The Datafile Read Time alarm occurs if the average time for a randomdatafile read exceeds a given threshold.

Data GuardConfigurationAlarm

The Data Guard Configuration Alarm occurs on a Primary database in anOracle Data Guard environment. Please supply details of Standbyconnections to Spotlight so Spotlight can monitor the Oracle Data Guardenvironment. In monitoring the Oracle Data Guard environment Spotlightcan alert you if key elements of the Oracle Data Guard environment are


Alarm Description

not working, or if Standby instances are lagging behind.

DispatchersBusy alarm

The Dispatchers Busy alarm occurs when all or most dispatcher processesare busy. In an MTS environment, multiple sessions share a smaller numberof shared server processes. The servers are usually allocated by a smallernumber of dispatcher processes.

EstimatedShared PoolSize alarm

This alarm occurs when the Shared Pool is set too small. A possible causeis the use of literals rather than bind variables in SQL statements. You canalso increase the size of the Shared Pool.

ExcessiveRBSActivitiesalarm

The Excessive RBS Activities alarm occurs when there are high rollbackextent/shrink/wrap activities.

Files PendingUpload alarm

Spotlight raises the Files Pending Upload alarm when a significant numberof XML files are pending upload indicating a problem with the uploadprocess. The strategy for dealing with this alarm is dependent on the OEMcontrol. The OEM control is defined in the Spotlight configuration forOEM.This alarm may occur in conjunction with the Collection Status Alarm.

Flash CacheInsert Skip:DBWROverloadedalarm

The Flash Cache Insert Skip: DBWR Overloaded alarm is raised when theratio of insert skips from the DBWR being overloaded exceeds a threshold.

Flash CacheRead Latency

The Flash Cache Read Latency alarm is raised when the time taken to reada single block from flash cache exceeds a threshold. This may be causedby a low performance flash device.

Flash CacheWriteCompleteWaits

The Flash Cache Write Complete Waits alarm is raised when the ratio of'write complete waits: flash cache' to DB time exceeds a threshold.

FlashbackBuffer Waitalarm

The Flashback Buffer Wait alarm indicates that the percentage of availabletime spent waiting for the Recovery Writer (RVWR) to write data to theflashback logs has exceeded a specified threshold.

FlashbackSpace Usagealarm

This alarm is raised when the amount of used space in the FlashbackRecovery Area exceeds a threshold.

Free Buffer The Free Buffer Waits alarm occurs when free buffer waits (as a proportion


Alarm Description

Waits alarm of total waits) exceeds a threshold.

In MemoryUndo LatchWait alarm

The in memory undo latch is associated with Oracle's relatively new inmemory undo (IMU) structures. These structures hold in memoryinformation formerly maintained in rollback (undo) segments.

InstanceFailed alarm

Spotlight raises an Instance Failed alarm when a specified instance hasfailed because of an Archiver error. See Archive Critical Failure alarmabove.

InsufficientFlashbackLogs alarm

This alarm fires when the flashback logs in the flashback area do not coverthe amount of time specified in the parameter DB_FLASHBACK_RETENTION_TARGET.

Job ProcessesBusy alarm

The Job Processes Busy alarm occurs when all or most job queue processesare busy.

Large PoolFull alarm

This alarm fires when the Large Pool allocation heap is full. The LargePool is used in MTS for session memory, and can be used by parallelexecution and backup processes.

Latch FreeWaits alarm

If a session needs to acquire a latch that is held by another session, a latchfree wait may occur. The presence of latch free waits may indicate abottleneck within the SGA. (The specific action depends on the latch.)

LibraryCacheLatchesalarms

These alarm fire when the time spent waiting on library cache latches orlibrary cache pin latches exceeds a threshold. Library cache latches (ormutexes in 10g R2 and above) protect the cached SQL statements andobject definitions held in the library cache within the Shared Pool.

LibraryCache MissRate alarm

The Library Cache Miss Rate alarm occurs when the library cache hit ratiofalls below a given threshold.

Lock Waitalarm

The Lock Wait alarm occurs when the proportion of time sessions spendwaiting for locks exceeds a threshold.

Log BufferSpace Waitalarm

The Log Buffer Space Wait alarm occurs if the amount of time spentwaiting for space in the redo log buffer exceeds a threshold.

Log SwitchTime alarm

The Log Switch Time alarm occurs when the amount of time spent waitingfor log switch events exceeds a threshold.

Low FreePhysicalRAM alarm

The Low Free Physical RAM alarm occurs when the server’s availableRAM is low.


Alarm Description

MaximumQueued Timealarm

The Maximum Queued Time alarm becomes active when a session hasbeen in the parallel execution queue for longer than the specifiedthreshold.Note: Applies only to Oracle 11g Release 2.

Multi-threadedServer alarm

The Multi-threaded Server alarm occurs when all or most multi-threadedserver (MTS) or dispatcher processes are busy.

Non-reclaimableFlashbackSpace alarm

This alarm is raised when the amount of non-reclaimable used space in theRecovery Area (flashback area) exceeds a threshold. Non-reclaimable filesinclude archived redo logs, which must be backed up before being deleted.

OracleMemory onDisk alarm

The Oracle Memory on Disk alarm is activated when the amount of guestphysical memory mapped to machine memory is less than the currentcombined size of Oracle's SGA and PGA.

OracleParameters forPredictiveDiagnosticsalarm

The Predictive Diagnostics feature cannot predict the behavior of Oracledatabases without the access to Oracle data that is granted by severalOracle parameters.

Oracle Statusalarm

When a Spotlight connection is lost, Spotlight automatically tries to re-establish the connection. The Oracle Status alarm indicates that Spotlightis re-establishing its connection to the Oracle instance.

ParallelQuery Serveralarms

The Parallel Query Server alarm occurs when all or most parallel serversare busy. Parallel server processes perform work to resolve SQL statementsthat are being executed in parallel. If all parallel server processes are busy,parallel SQL may be downgraded, or even executed serially, thusincreasing its execution time. Spotlight can raise these Parallel QueryServer alarms: Parallel Query Servers Busy, Parallel Queries Downgraded,and Parallel Queries Serialized.

Parse Ratioalarm

The Parse Ratio alarm is raised when the ratio of parse calls to executecalls exceeds a threshold.Note This alarm is initially disabled; see Total Parse Wait Time Ratioalarm.

PredictiveDiagnosticsalarms

Predictive Diagnostics alarm (SQL Collector, Bottleneck Collector orPerformance Modeling Collector alarm) is raised when the appropriatecollection of predictive data has not been scheduled.

PredictiveDiagnostics

This alarm indicates that the customized SQL query use to representdatabase concurrency within Predictive Diagnostics has returned an error.


Alarm Description

Custom SQLError alarm

PredictiveDiagnosticsOracleParametersalarm

The Predictive Diagnostics feature cannot predict the behavior of Oracledatabases without the access to Oracle data that is granted by severalOracle parameters.

RedoAllocationand RedoCopy LatchWait alarms

The Redo Allocation and Redo Copy Latch Wait alarms are raised whenthere is contention for the redo allocation latch and redo copy latch.

Result CacheFind/CreateRatio alarm

When Oracle executes SQL queries, the result cache stores the result setsof specified queries in shared memory for later use. The Find/Create Ratioalarm occurs when the rate of find operations on the result cache dropsbelow the rate of create operations, and indicates that the contents of theresult cache are under-used.

Result CacheLatch SleepRatio alarm

Spotlight raises the Result Cache Latch Sleep Ratio alarm when the rate oflatch sleeps for the Result Cache latch indicates that there is contention onthe Result Cache latch, where multiple processes are competing to use thelatch and are causing a significant degradation in response time.

Row Cacheobjects LatchWait

The row cache objects latch protects the contents of the row cache; thesection of the shared pool to cache dictionary information.

SequentialRead Timealarm

This alarm fires when the average time that Oracle takes to read a singleblock of data — typically an index-based lookup — is significantly higherthan the average time recorded during calibration.

SessionsQueued alarm

The Sessions Queued alarm becomes active when the number of sessionsin the parallel execution queue exceeds a threshold.Note: Applies only to Oracle 11g Release 2.

Shared PoolLatch Waitalarm

Shared pool latches control access to the shared pool memory map.

Shared PoolLockPercentagealarm

This alarm is raised when one user session holds a lock on object(s) in theShared Pool for a prolonged period, and causes other sessions to wait onthe same resource. This "blocking lock" happens rarely. When it doesoccur, use a SQL query to find the affected sessions.


Alarm Description

Shared ServerConvertedintoDedicatedServer alarm

Spotlight raises an alarm when many shared servers are converted, leavingtoo few shared servers to meet the demand. Converted shared servers willhave a status of WAIT(RECEIVE) in v$shared_server.

Status Erroralarm

This alarm is raised when there is a problem with the way the OEMconfiguration has been defined in Spotlight. Correct the values entered inthe dialog View | Options | OEM Monitoring.

Temporary IOWait Ratioalarm

Spotlight raises the Temporary IO Wait Ratio alarm when the time spent intemporary IO waits is too high as a percentage of the time spentperforming user calls on the Oracle database.

Total ParseWait TimeRatio alarm

Spotlight raises this alarm when the amount of time spent parsing SQLstatements as a percentage of the active database time exceeds a level thatmay cause latch contention on the shared pool and library cache latches.Note: The Total Parse Wait Time Ratio alarm is available for (Oracle 10.2and later). When monitoring earlier Oracle systems, consider reactivatingthe (disabled) Parse Ratio alarm.

UnarchivedLogs alarm

The Unarchived Logs alarm occurs when the number of unarchived redologs exceeds a threshold. If all online logs need archiving, database updateactivity can be suspended while the redo log writer waits for the next logto be archived.

Unix Statusalarm

When a Spotlight connection is lost, Spotlight automatically attempts tore-establish the connection. Where the Oracle database is hosted on a Unixserver, this alarm indicates that Spotlight is re-establishing its connectionto the Unix server.

WMI Statusalarm

When a Spotlight connection is lost, Spotlight automatically attempts tore-establish the connection. Where the Oracle database is hosted on aWindows server, this alarm indicates that Spotlight is re-establishing itsWMI connection to the Windows server.

WriteCompleteWait alarm

Spotlight raises the a Write Complete Wait alarm when write completewaits (as a proportion of total waits) exceed a threshold. Write completewaits occur when a session tries to modify a block that is currently beingwritten to disk by the database writer process.

Note: Spotlight on Oracle Data Guard alarms occur in the Alarm Log of the Primary Databasein the Oracle Data Guard environment. See Spotlight on Oracle Data Guard Alarms in the onlinehelp for more information.


Spotlight on Oracle Drilldowns

When you have isolated a problem, you can display a drilldown page, whose charts and tablesprovide a detailed breakdown of the underlying statistics.

Following are Spotlight on Oracle drilldowns.

Drilldown Clicktoopen

Description

Top Sessions The Sessions drilldown displays information about theusers connected to the Oracle database. The pages andsub-pages in the drilldown display information on:

l All users connected to the Oracle database.

l Session details for those users, including waits,SQL statements executed, locks, resource usage,and Oracle statistics.

l Parallel queries executed for all users connected tothe Oracle database.

l Oracle jobs currently scheduled for the database.

SQL & ApplicationWorkload

The SQL & Application Workload drilldown allows youto select, sort and display SQL statements executedaccording to specified criteria. The drilldown includes:

l The Top SQL page, which displays the SQLstatements that consume the most system resourcesfor the current connection.

l The Array Fetch page, which displays informationon array processing for the Oracle database underinvestigation.

l The Parse Activity page, which displaysinformation on SQL statements associated withhigh parse rates (Oracle 10.2 and later).

l The Sort Activity page, which displaysinformation on SQL statements associated withhigh sort activity (Oracle 10.2 and later).

l The Analyze Trace page (formerly a separatedrilldown), which displays the contents of tracefiles that store information on how Oracleprocesses the execution of SQL statements, and onthe resources it uses to do so (Oracle 10.2 andlater).



Description

l The Result Cache page (Oracle 11g and later)shows information about the result cache, whichstores the result sets of specified queries so theycan be re-used.

Activity The Activity drilldown displays the activity on thedatabase. The pages and sub-pages in the drilldowndisplay information on waits, waiting events, blockinglocks, lock activity, latches, query servers, transactions,rollbacks, and buffer busy waits.

I/O The I/O drilldown displays the input and output activityfor the database. The pages and sub-pages in thedrilldown display information on I/O by category, session,SQL, tablespaces, datafiles, segments and row chaining.

Configuration &Memory

The Configuration and Memory drilldown shows thememory utilization for the Shared Pool and the SGA, aswell as shared pool and buffer cache hit ratios, Oraclealert log and Oracle parameters.

Operating System The Operating System drilldown provides informationabout the performance, processes, and storage on thedatabase machine under investigation.

Disk Storage The Disk Storage drilldown displays information aboutonline redo logs, tablespace utilization, archivedestinations, and Flashback recovery.Right-click the tables in the Disk Storage drilldown toopen Space Manager. See "Tools | Space Manager" (page43) for more information.

Tuning You can use the Tuning drilldown to view informationabout Oracle memory resources in the Oracle SGA andPGA (System and Program Global Areas), to tune memoryresources where feasible, and to view and tune latchactivity for systems with multiple CPUs.

ASM If your system uses Automatic Storage Management(ASM) to manage disk storage, you can use the ASMdrilldown within Spotlight on Oracle to diagnose ASMperformance. ASM is a storage management solution fromOracle Corporation that automatically reorganizes databased on statistics collected from usage and user accesspatterns over time.



Description

Analyze Trace To understand how well SQL is being processed by thesystem under investigation, and how to improve thatprocessing, you must trace the execution of SQLstatements for:

l How Oracle is processing the statements.

l The resources that are required to execute thosestatements.

Oracle stores that information in trace files. The pages inSpotlight's Analyze Trace page allow you to view thecontents of those trace files and save the contents to theclient's computer.

DTrace DTrace is a dynamic analysis tool for Sun platforms.Spotlight on Oracle uses DTrace to capture details of theinteraction between Oracle and the operating system(Solaris 10 onwards). See "Install And Configure DTraceScripts" (page 26) for more information.

PredictiveDiagnostics (ifenabled)

Predictive Diagnostics (if available) allows Spotlight tocollect and analyze performance metrics for SQLstatements as they are executed on the database instance.These analyses can be used to identify SQL statementswhose performance may not scale adequately as datavolumes and SQL execution rates increase.Predictive Diagnostics allows Spotlight to collect andanalyze performance metrics for SQL statements, and maybe used to predict the future performance for SQLstatements on that database.Predictive Diagnostics (if available) allows Spotlight tocollect and analyze performance metrics for an Oracledatabase in order to predict its future behavior. In doingso, it identifies:

l SQL statements ("degrading SQL") whoseperformance may not scale adequately in the futureas data volumes and SQL execution rates increase.

l Waiting events ("bottlenecks") that may in thefuture affect database throughput and responsetime.

l Database resources (CPU, memory, and disk I/O)



Description

whose limitations may in the future affect databaseperformance.

See "Predict Database Performance" (page 45) for moreinformation.

Virtualization Use the Virtualization drilldown to view detailedinformation about the CPU and memory use of a virtualmachine.

Data Guard Monitor an Oracle Data Guard environment withSpotlight on Oracle Data Guard.Note: The Data Guard icon is enabled for Primarydatabases configured in the Oracle Data Guardenvironment.

Reports The Report wizard generates printable reports of fourtypes of monitoring data: alarms by percentage of time,waits, Top SQL and IO. The reports are designed anddelivered through SAP Crystal Reports®.

Alarm Log The Alarm Log drilldown displays information on thealarms associated with the Oracle database.You can filter the Alarm Log to display only the alarmsthat meet specified criteria.

Spotlight Tools

Tools | Explain Plan

Tools | Explain Plan allows you to determine the execution plan that Oracle applies to aparticular SQL statement. Spotlight on Oracle allows you to view graphical representations ofthe execution plan for SQL statements being executed by a user, or for those identified by theTop SQL Page.

To open the Explain Plan

1. Open a connection to an Oracle Database.

See "Connect to Oracle Instance" (page 20) for more information.

2. Click Tools | Explain Plan.


Tools | SQL Optimizer

SQL Optimizer provides context-sensitive tuning advice for SQL statements based on the Oracleexecution plan and the database structure.

Before you open SQL Optimizer

l Ensure SQL Optimizer is installed on your machine.

l Ensure you have DBA privileges for the database you are diagnosing.

To open SQL Optimizer



2. Click Tools | SQL Optimizer.

Tools | Space Manager

Space Manager provides a comprehensive solution for space management and reorganization inOracle databases. Space Manager offers preventive maintenance, problem detection andresolution, and capacity planning across any number of databases.

Before you open Space Manager

l Ensure Space Manager (version 4.5 or later) is installed on your machine.

l Ensure you have DBA privileges for the database you are diagnosing.

To open the Space Manager



2. Click Tools | Space Manager.

Tune Your Database

Spotlight supports an approach to Oracle performance tuning that could be described as "tuningby bottleneck". Spotlight alerts you if any component of the Oracle architecture is forming abottleneck. Additionally, Spotlight provides tools that allow you to identify and rectify anyinefficiency in your database’s configuration.

In general, you tune your Oracle instance by reiteratively identifying bottlenecks, contention andcritical resources, and by using Spotlight's online advice to remove the bottleneck or contentionto improve the performance of the resource.

The following is a summary of steps that you can undertake to tune Oracle using Spotlight.


l Use the Spotlight home page to alert you to any obvious bottlenecks. If Spotlight detectsthat some component of Oracle constitutes a performance bottleneck, the correspondingcomponent generates a visual or auditory alarm. The exact appearance of the alarmdepends on how you have configured alarm severities. When an alarm is current you canpress F1 to display help for the alarm. Clicking the object (in the main window) displaysthe relevant drilldown.

See "Spotlight on Oracle Home Page" (page 26) for more information.

l Use the Activity drilldown to display overall efficiency and resource usage. In particular,the Event waits chart shows the amount of time database sessions are spending waitingfor various resources, as well as the amount of CPU being utilized. Tuning efforts aremost likely to succeed if they are concentrated on the resources being used most heavily.The Wait activity topic contains a discussion of the meaning and implications of variouswait events.

See "Spotlight on Oracle Drilldowns" (page 39) for more information.

l Other pages help you examine detailed activity. For instance, the Top Sessions drilldownshows the users who contribute most to database activity. You can display various detailsfor these sessions, including the SQL statements, locks and resource usage.

For further information on tuning your database, see the online help.

Spotlight on Oracle Getting Started GuidePredict Database Performance

45

Predict Database PerformanceUse Predictive Diagnostics to predict the future performance of a database (and identify how toimprove that performance):

l Identify SQL statements whose performance may not scale adequately as data volumesand SQL execution rates increase.

l Identify bottlenecks that may affect database throughput and response time.

l Predict when database resources such as CPU, memory and disk I/O are expected to reachtheir full capacity, and recommend strategies to enable the database to handle high loads.

Note: It takes about 10 days for Spotlight to collect enough data to make valid predictions onthe future performance of the database.

Open Predictive Diagnostics1. Open a Spotlight on Oracle connection. See "Connect to Oracle Instance" (page 20) for

more information.

2. Click Predictive Diagnostics or CTRL+G

Predictive Diagnostics for BeginnersAlthough you can use any feature in Predictive Diagnostics at any time, we recommend thefollowing procedure for first-time users.

Follow the suggestions inWhat To Do Next.

The quality of the predictions depends on the quality of thedata.What To Do Next tracks the current data and indicateswhen it is incomplete or outdated.

Wait until enough realdata is available.

This usually occurs about 10 days after you begin monitoringthe database. Before that time, become familiar with howPredictive Diagnostics works by using the supplied sampledata.

Improve how thedatabase is used now.

Employ the Identify Degrading SQL and Identify DatabaseBottlenecks features to optimize data processing.Note: Any changes you make MAY cause the data collectedearlier to become outdated. If so, you may need to collect newdata for another 10 days.

Use the Configurationpages to identify the

Set parameters for the CPU, memory and disk I/O resourcesavailable to the database.


46

resources available to thedatabase.

In particular, use Peak Processing Periods to identify WHENdatabase resources are in greatest demand. This will improvethe precision of your predictions.Note: The default settings we provide for these parameters maynot be appropriate for your database.

Use the PerformanceModeling pages.

Predict when and why database resources will run out, whatwill cause performance to suffer, and what you need to do tocounter those problems.

Predictive Diagnostics And SpotlightFor detailed information on Predictive Diagnostics, see the Spotlight online help.

When... Do this...

Predictive Diagnosticsidentifies a SQL statement thatmay cause problems

Open the Top SQL drilldown to view further detailsabout the SQL statement.

Open Explain Plan to view further information aboutthe statement's execution plan.

Open SQL Optimizer to view context-sensitive tuningadvice for the SQL statement based on the Oracleexecution plan and the structure of the database.

Predictive Diagnosticsidentifies a bottleneck that maycause problems

Open the Activity | Waiting Events drilldown to viewfurther details about the current behavior of potentialbottlenecks.

Predictive Diagnosticsidentifies a database resourcewhose limitations may causeproblems

Open the I/O, Configuration & Memory, and Tuningdrilldowns to view further details about the currentbehavior of CPU, memory, and disk I/O resources.

Requirements for Predictive DiagnosticsThe disk storage requirements for Predictive Diagnostics schemas comprise:

Situation Requirement

Sample data Approximately 70 MB of disk space.

SQLstatements

A fixed overhead of approximately 10 MB per schema.An additional requirement of approximately 1 KB per snapshot for each


47

Situation Requirement

SQL statement.The number of snapshots collected depends on the application workload andsize of the shared pool, the collection interval for snapshots, and the lengthof time each snapshot is retained.ExampleIf there are 500 eligible SQL statements per snapshot, and snapshots arecollected twice a day, and each snapshot is kept for one year, the diskstorage requirement for SQL statements is:10MB + (500 x .001MB x 2 x 365) = 375 MB

Impendingbottlenecks

Disk storage requirements depend on how frequently data is collected, andthe length of time that snapshots are retained.ExamplesIf data is collected every 10 minutes and retained for six months:approximately 200 MB.If data is collected every 30 minutes and retained for six months:approximately 65 MB.If data is collected every 60 minutes and retained for six months:approximately 35 MB.

Performancemodeling

Disk storage requirements depend on how frequently data is collected, andthe length of time that performance data is retained.ExampleIf data is collected every 10 minutes and retained for one year:approximately 130 MB.

Note: Parameters set in View | Options | Predictive Diagnostics and the SQL Statement TrendAnalysis wizard affect the storage requirements for Predictive Diagnostics data. See the onlinehelp more for more information.

Spotlight on Oracle Getting Started GuideSpotlight on Oracle RAC

48

Spotlight on Oracle RAC

What is Spotlight on Oracle RAC?Spotlight on Oracle RAC is an optional add-on to Spotlight on Oracle. Spotlight on OracleRAC is installed automatically on your system via the Spotlight on Oracle or TOAD DBASuite installer.

Use Spotlight on Oracle RAC to diagnose Oracle databases on multiple host machines under theReal Application Clusters (RAC) architecture.

When monitoring the behavior of an Oracle RAC cluster, a database administrator needsto know:

1. How is each of the instances in my cluster performing?

2. How is the database as a whole performing?

3. Is my database experiencing cluster-related contention?

To further investigate the behavior of individual database instances within the cluster, thedatabase administrator can use Spotlight on Oracle RAC to launch Spotlight on Oracle on anyparticular instance within a cluster. See "Instance Panels" (page 56) for more information.

What Is Oracle RAC?Real Application Clusters (RAC) is a clustering technology introduced by Oracle in release 9i. Itsupersedes Oracle’s previous clustering technology Oracle Parallel Server (OPS).

Oracle RAC is a “shared disk” clustering solution that is significantly different from thearchitectures offered by Microsoft SQL Server and IBM DB2. Each node in an Oracle RACcluster has equal access to all of the database data through a shared disk subsystem. Data isneither partitioned to specific nodes, nor replicated across nodes. A high-speed networkinterconnect allows each node to keep its in-memory view of data consistent.

An Oracle RAC configuration has the following architectural characteristics:

1. Many instances of Oracle run across several nodes.

2. Many instances share a single physical copy of a shared Oracle database.

3. All instances have common data and control files.

4. Every instance has individual redo logs and undo segments.

5. Every instance can simultaneously execute transactions against the same database.

The following diagram illustrates the basic components of an Oracle RAC cluster.


49

A user should find an Oracle RAC database operationally identical to a database hosted on asingle server. From top to bottom, the important features represented are:

l Cluster interconnect - A high-speed, high-bandwidth communication facility thatconnects the nodes in an Oracle RAC cluster.

l Multiple nodes/instances - Oracle instances run on the nodes (host machines) in theOracle RAC cluster. Each instance comprises an Oracle System Global Area (SGA)plus the corresponding Oracle background processes that retain and process Oracledatabase requests.

l Shared disk subsystem - Database files in Oracle RAC systems are stored on multipledisks that are shared by all the nodes in the cluster, and all nodes must be able read andwrite to those disks.

Connect to Spotlight on Oracle RACUse this procedure to connect to an Oracle instance running on an Oracle RAC cluster.

1. From the Spotlight Browser select All Connections | Spotlight on Oracle RAC

2. Select the RAC to connect to. If the RAC is not on the list then Add a new connection


50

(page 50).

3. The Verify Cluster Connection Details dialog is displayed if Spotlight encounters anerror during the connection process. See Verify Cluster Connection Details Dialog formore information.

Note: You can request Spotlight displays this dialog every time you connect to theOracle RAC. On the connection Properties | Details dialog select Display ConnectionDetails Verification Dialog. Alternatively, on the Verify Cluster Connection Detailsdialog select Display this dialog for the current connection.

Add a new connection

1. Ensure you create a Spotlight user with sufficient privileges to diagnose the Oracle RAC.See "Spotlight Users" in the online help for more information.

2. Click File | Connect

3. Select Spotlight on Oracle RAC on the Connections menu.

4. Double-click Add new connection.

5. Ensure Details is selected. Fill in the Properties window.

Oracle Connection Details

Field Description

ConnectionString

Select the name (the SQL*Net alias) of the Oracle RAC instance thatyou wish to connect to.Notes:


51

Field Description

l The database name is defined within SQL*Net configurationutilities.

l While this can be the name of any instance in the Oracle RACcluster, we recommend using the name of the load-balancedinstance – the instance whose TNSNAMES entry includes theoption (LOAD_BALANCE = yes)

l When selected, the Connection name field resets to this value.You can then edit the Connection name field if required.

OracleUsername

Enter the Spotlight user name.

OraclePassword

Enter the Spotlight password.

OracleHome

Select an appropriate Oracle Home – a location where Oracle clientsoftware is installed.Notes:

l Choose a home that contains a version of the Oracle clientthat is compatible with the Oracle RAC server.

l You CANNOT change this value while an active Spotlightconnection is already using an Oracle Home.

l When you specify an Oracle Home for a particular Spotlightconnection, the connection-based Oracle Home overrides anydefault values.

l The first active Spotlight connection in a session forces allsubsequent connections to use its specified Oracle Home.

ASM connection details

ASM is built into the Oracle database kernel. It simplifies the creation and spacemanagement in Oracle databases.

Note: Available for Oracle 10g and later.

Field Description

MonitorASM

Select to monitor the performance of Oracle's Automated StorageManagement (ASM) technology.

ASMConnectionString

Enter the connection string required to link to the ASM databaseinstance that is used for storage management by this Oracle database.Note: The TNSNAMES entry for the ASM instance must include theoption (UR = A) for the Spotlight connection to work properly.


52

Field Description

ASMPassword

Enter the password to login to the ASM database instance.Note: There is no ASM Username field. The ASM Password is to theASM user name of SYS. SYS is the only username that can provideaccess to ASM-specific information.The user that Spotlight uses to login to the ASM database is SYS.This is the only user name that can provide access to ASM-specificinformation.Providing SYS access to an ASM instance does NOT create a securityissue for the Oracle RAC, but you should limit access to the SYSaccount to trusted individuals.Because the SYS password is stored in encrypted format, non-privileged users can still access Spotlight on Oracle RAC. To do so:

l Use a pre-configured or shared Spotlight on Oracleconnection, OR

l Use a Spotlight connection without ASM access. Spotlightwill display Oracle-related data, but the ASM drilldown willcontain no data.

Server connection details

Field Description

MonitorOS

Select to monitor the operating system on the database server.Note: Clear this option if the database is running on an operatingsystem NOT supported by Spotlight on Oracle RAC. If you leave thisoption selected and Spotlight is unable to collect statistics from theserver operating system, you may experience errors.

OS Server Enter the name of the server.Tip: Use a period “.” to monitor the local database.

OS Type Select the type of operating system.

l Windows

l Unix(REXEC)

l Unix(SSH)

l Unix(SSH using Public/Private Keys)

If you select one of the Unix systems, make sure the SSH or REXECdaemon is running on the database server and is configured to receiveremote connections.Note: Spotlight supports both SSH1 and SSH2 protocols; public-key


53

Field Description

encryption is supported under SSH2 only.

SSH Port Enter the port number for SSH (secure shell) connection to thedatabase servercluster. The default value is 22.Note: Applicable when OS Type is Unix (SSH) or Unix(SSH usingPublic/Private Keys).

OSUsername

Type the user name to login to the database servercluster.Notes:

l Spotlight will connect successfully to all nodes in the clusteronly if every node employs the SAME username and password.

l If the username — and password if required — are notcompleted, you will not be able to view the operating systemperformance statistics for the database servercluster.

l When OS Type is Windows:

l Not required if your login details have been assignedremote access to the registry of the nodes in the databaseservercluster.

l If your login details have NOT been set up on thedatabase .servercluster, you must enter a username andpassword that has access to the registry of each node.

SSH KeyType

Choose the type of key to use when making the Spotlightconnection: RSA or DSA.Note: Applicable when OS Type is Unix (SSH using Public/PrivateKeys).

SSHPrivateKeyFilename

Locate the filename that contains the private key for the Spotlightconnection.Hint: Use the "..." button.Note: Applicable when OS Type is Unix (SSH using Public/PrivateKeys).

SSHPassphrase

Type the passphrase used to decrypt the private key.Note: Applicable when OS Type is Unix (SSH using Public/PrivateKeys).

OSPassword

Type the user password to login to the database servercluster (ifrequired).Note: Not applicable when OS Type is Unix (SSH usingPublic/Private Keys).

6. Select Launch Spotlight on Oracle to open a Spotlight on Oracle connection to the


54

database when you open the Spotlight on Oracle RAC connection.

7. Select Display Connection Details Verification Dialog to display the ConnectionDetails Verification dialog when you connect to a specified Oracle RAC cluster. Thedialog allows you to specify and test connections to ALL instances in the cluster.

8. Select Save password details to save all the password details that have been entered.

Spotlight on Oracle RAC Home PageThe Spotlight home page shows an overview of the activity in an Oracle RAC database cluster.

Related operating system statistics are grouped together on panels that are connected by a seriesof graphical flows and icons. Spotlight updates these flows in real time so you can see howquickly data is moving through the system. The icons change color as Spotlight alarms areraised, upgraded, downgraded and canceled.

The panels on the Spotlight on Oracle RAC home page are described in more detail next. For afull discussion of the panels, see the Spotlight online help.

Tip: Hover the mouse pointer over a panel component for more information.

Where an Oracle Exadata System is installed and licensed the Spotlight on Oracle RAC HomePage looks slightly different. An additional Exadata Storage Panel (page 57) monitors theExadata Storage System (Cells).


55

Interconnect and Global Status Panel

This panel shows a combination of Interconnect and Global Cache Service information, andinformation about the Oracle RAC database overall. It consists of information aggregated fromqueries sent out to the individual instances. This provides information on how well data is beingshared and transferred between nodes in the cluster.

Some information displayed here is easily calculated: for example, the Total Logical Readsspinner is the sum of the individual Logical Reads values. Other components, such as ClusterLatency and Cluster Overhead, have a more complex calculation.

Spotlight uses the Global Database label to indicate alarms in one or more Oracle instances thatmay affect the performance of the Oracle RAC database as a whole.

The flows between the Interconnect panel and the Instances panel represent the flow of datafrom one instance to another across the interconnect.


56

Instance Panels

This panels show the participating instances of a cluster. Together with separate panels for allparticipating instances, this panel provides information on the performance of individualinstances in the cluster.

Spotlight on Oracle RAC supports a theoretical maximum of 36 instances, with up to 6 instancesshown on the home page. When there are more than 6 instances (not shown here), you can use anavigation control in the Instances panel to navigate between groups of instances.

The color of the border and label on each instance represents the Spotlight on Oracle severity ofthe instance. You can move the mouse over a named instance to display a list of all the alarmscurrently raised against that instance.

Note: You can launch Spotlight on Oracle from any Instance panel on the Spotlight on OracleRAC home page.

The overall status of the cluster does not necessarily reflect the status of individual instances,as problems with individual instances do not necessarily translate to a problem with thecluster as a whole.

Two further metrics — Balance and Availability— are shown in the instances panel. Balanceindicates the comparative load on the instances. Availability shows the percentage of instancesfor which a SQL*Net connection is currently open and working.


57

IO Subsystem Panel

This panel shows information relating to the shared storage subsystem for the OracleRAC cluster.

Some of this information is aggregated from queries sent to the individual instances; otherinformation is obtained from a single query to one of the instances.

The ASM spinner is populated only when ASM (Automatic Storage Management) is being used.See "Connect to Spotlight on Oracle RAC" (page 49) for more information.

The flows between the I/O Subsystem panel and the individual instances represent the flow ofphysical reads and writes between the instances and the physical database.

Exadata Storage Panel

Note: Spotlight monitoring of Exadata systems requires additional licensing.

This panel shows information relating to the Exadata Storage Servers (Cells).

l Cell Averages - Measure how much the Exadata Storage Servers are used.

l Smart Scans - Exadata's Smart Scans optimize SQL statements executed in Exadata'sStorage Servers. Smart Scans are typically for scan operations like Full Table Scans.

l Smart Flash Cache - Speed up I/O performance for frequently accessed data. ExadataSmart Flash Cache acts like a large, database-aware disk cache for the storage cell.

l Cell Summary - Summary statistics for Exadata's Storage Servers.

Spotlight on Oracle RAC AlarmsSpotlight alerts you to problems with your system by issuing an alarm. You can configureSpotlight in the level of severity that constitutes an alarm, to disable an alarm, and the actionsSpotlight takes on raising the alarm.

When an alarm is raised Spotlight performs one or more of the following actions:


58

l Changes the color or intensity of relevant components.

l Gives audible warnings.

l Sends email notifications.

Actions you can take on an alarm being raised include:

l View details of the statistics that caused the alarm in a Spotlight drilldown page. See"Spotlight on Oracle RAC Drilldowns" (page 61) for more information.

l View the Spotlight online help.

l View details of the alarm in the Spotlight Alarm Log drilldown.

l Snooze the alarm.

l Save the alarm details.

l Filter the list of alarms.

Alarms on A Database Instance

Problems in a single instance may not always affect the operation of the cluster as a whole. Insuch cases, Spotlight on Oracle RAC can display an alarm in the relevant instance panel withoutreporting a specific Spotlight on Oracle RAC alarm.

However, if a problem in a single instance DOES affect the operation of the whole cluster – or isrepeated in multiple instances so that the combined effect is significant – Spotlight on OracleRAC reports the problem as an aggregated alarm.

These alarms are displayed on the Spotlight on Oracle RAC home page – usually in the GlobalDatabase heading of the Interconnect and Global Status panel, or on the individual componentsof the IO Subsystem panel.

Notes:

l Aggregated alarms within Spotlight on Oracle RAC are the same as the correspondingalarms in Spotlight on Oracle.

l Not all Spotlight on Oracle alarms are aggregated in Spotlight on Oracle RAC.

Alarms on The Oracle RAC

Following are the alarms specific to Spotlight on Oracle RAC. For possible solutions to theproblems indicated by these alarms or for information on how to diagnose problems further, seeSpotlight on Oracle RAC alarms in the online help.

Alarm Description

ArchiveCriticalFailure alarm

An Archive Critical Failure alarm indicates that the number of workingarchive destinations for the database cluster is about to fall below theminimum number needed to archive Oracle redo logs successfully. Whenyou see this alarm, you should make space in the archive log destination


59

Alarm Description

by purging – or by backing up to an offline medium – any archived filesthat are not immediately needed.

ArchiveDestinationFailed alarm

Spotlight raises an Archive Destination Failed alarm when at least one ofthe archive destinations specified for redo log files has become full. SeeArchive Destination Failure alarm below.

ArchiveDestinationFailure alarm

An Archive Destination Failure alarm when at least one of the archivedestinations specified for redo log files is about to become full. When yousee this alarm, you should make space in the archive log destination bypurging it, or by backing it up to an offline medium.

ASM Statusalarm

When a Spotlight connection is lost, Spotlight automatically tries to re-establish the connection. The ASM Status alarm indicates that Spotlight isre-establishing its connection to the ASM diskgroup for the Oracle RACcluster.

Availabilityalarm

This alarm indicates that the instances participating in the cluster are NOTall currently available for access. The database administrator shoulddetermine why the instances are unavailable, and restart them after takingthe required actions.

(Global)Cache MissRate alarm

See Miss Rate alarm below.

ClusterLatencyalarm

Spotlight raises the Cluster Latency alarm when cluster latency from allcauses (interconnect, GES, LMS overhead) exceeds a specified threshold.Cluster latency is defined as the average elapsed time from when arequesting instance requests a block of data to the time when therequesting instance receives the block.

ClusterOverheadalarm

Spotlight raises the Cluster Overhead alarm when the percentage of timededicated to cluster co-ordination activities exceeds a specified threshold.Cluster overhead is the amount of time spent waiting for cluster relatedactivities as a percentage of the total time spent waiting for all activities.

Corrupt orLost Blocksalarms

Spotlight raises separate alarms for corrupt and lost blocks when aspecified percentage of the blocks transferred across the interconnect arecorrupted or lost during global cache consistent read requests.High levels of lost or corrupt blocks should not happen in normalcircumstances. If these alarms fire at all, it indicates either very seriousoverloading of the Oracle RAC cluster, or software or hardware faults inthe cluster.

Data GuardConfiguration

The Data Guard Configuration Alarm occurs on a Primary database in anOracle Data Guard environment. Please supply details of Standby


60

Alarm Description

Alarm connections to Spotlight so Spotlight can monitor the Oracle Data Guardenvironment. In monitoring the Oracle Data Guard environment Spotlightcan alert you if key elements of the Oracle Data Guard environment arenot working, or if Standby instances are lagging behind.

InstanceFailed alarm

Spotlight raises an Instance Failed alarm when a specified instance hasfailed because of an Archiver error. See Archive Critical Failure alarmabove.

LoadImbalancealarm

Spotlight raises the Load Imbalance alarm when:

l The load across instances in the cluster is significantly unbalanced,and

l The imbalance is proven to be statistically significant, beyond a95% probability.

Cluster balance is a fundamental indication of the health and productivityof an Oracle RAC environment. If load is not reasonably evenlydistributed across the instances in a cluster, then the DBA is likely toregret having chosen a clustered environment.Spotlight on Oracle RAC calculates balance by recording a user-definablemetric of load over time on each instance (the default is Logical Reads,but you can switch this to CPU usage or Physical Writes via theSpotlight on Oracle RAC Options window) and calculating the relativebalance of load across the cluster.In the normal course of events, there are always minor variations in loadbetween instances within the cluster, so Spotlight uses a statisticaltechnique to determine if a variation is significant.Note: The activation of specific Oracle Services can unbalance theworkload across the Oracle RAC cluster. In these cases, Spotlight triggersthe Load Imbalance alarm only when a more significant threshold isreached.

Miss Ratealarm

This alarm, more specifically known as the Cache Miss Rate alarm orGlobal Cache Miss Rate alarm, is raised when the global cache local missrate exceeds a specified threshold. This rate is the percentage of time thata block required for a logical read is located in an inconsistent mode onANOTHER instance in the cluster and must be transferred.


61

Alarm Description

AveragePercentageCPU LoadAcross TheExadataStorage Cellsalarm

The Average Percentage CPU Load Across The Exadata Storage Cellsalarm occurs when the total CPU utilization of the Exadata Storage Serversexceeds a threshold. One or more of the Server CPU may encounter a largenumber of requests, or you may have un-tuned SQL, which uses excessiveamounts of CPU.

MaximumPercentageCPU Load onThe ExadataStorage Cellsalarm

The Maximum Percentage CPU Load on The Exadata Storage Cells alarmoccurs when the CPU utilization of one or more Exadata Storage Servers(Cells) exceeds a threshold.. One or more of the Server CPU may encountera large number of requests, or you may have un-tuned SQL, which usesexcessive amounts of CPU.

Single BlockRead alarm

The Single Block Read alarm is raised if the average time taken tocomplete a request for a single block from the Exadata Storage Serversexceeds a given threshold. The source of the latency may be any databaseusing the Exadata Storage Servers.

Cell Balancealarm

The Cell Balance alarm is raised when there is an unequal distribution ofwork-load between the Exadata Storage Servers.

ExadataStorageServer (Cell)NotConnected

Spotlight has lost its connection to at least one Exadata Storage Server,and is unable to re-connect. Check the status of your network connection.Check the status of the hosts of each Exadata Storage Server.

No NodesAvailablealarm

This alarm indicates that Spotlight has lost its connection to the OracleRAC cluster, and cannot re-connect to any of the nodes in the cluster.Check the status of your network connection, your host machines, andyour database instances.

Note: Spotlight on Data Guard alarms occur in the Alarm Log of the Primary Database in theOracle Data Guard environment. See Spotlight on Data Guard Alarms in the online help for moreinformation.

Spotlight on Oracle RAC DrilldownsWhen you have isolated a problem, you can display a drilldown page, whose charts and tablesprovide a detailed breakdown of the underlying statistics.

Following are Spotlight on Oracle RAC drilldowns.


62


Description

Top Sessions As per Spotlight on Oracle.

SQL & ApplicationWorkload

As per Spotlight on Oracle. The SQL & ApplicationWorkload drilldown in Spotlight on Oracle RAC doesNOT contain the following pages:

l Analyze Trace

l Result Cache

Activity As per Spotlight on Oracle. The Activity drilldown inSpotlight on Oracle RAC does NOT contain the followingpages:

l Lock Activity

l Server

l Rollback

I/O As per Spotlight on Oracle.

Configuration &Memory

As per Spotlight on Oracle. The Configuration & Memorydrilldown in Spotlight on Oracle RAC does NOT containthe following pages:

l Shared Pool Utilization

l Buffer Cache

Disk Storage As per Spotlight on Oracle. The Disk Storage drilldown inSpotlight on Oracle RAC does NOT contain the followingpages:

l Online Redo

l Recovery Area

Cluster The pages in the Cluster drilldown display informationabout the performance of the Oracle RAC cluster, includingdetails on cluster latency, balance, overhead, miss ratios,and corrupt or lost blocks. For further information see“Cluster drilldown” in the online help.

ASM As per Spotlight on Oracle.

Services The Services drilldown displays details of the servicesdefined and deployed for the Oracle RAC cluster underinvestigation.Note: Available for Oracle 10.2 and later.


63


Description

PredictiveDiagnostics (ifenabled)

As per Spotlight on Oracle.

Data Guard If the system is a Primary database in the Oracle DataGuard environment, you can use Spotlight on Data Guardto monitor the system's Standby databases. Standbydatabases are alternative/supplementary repositories to theproduction Primary database.Note: The Data Guard icon is enabled once the system isconfigured as a Primary database in the Oracle Data Guardenvironment.

Exadata Detect and help resolve performance issues on Oracle'sExadata machine.Note: The Exadata icon is enabled on connection to anOracle Exadata system. Spotlight monitoring of Exadatasystems requires additional licensing.

Alarm Log As per Spotlight on Oracle.

Troubleshooting Spotlight on OracleThis section identifies general problems that you may encounter when using Spotlight on Oracle,and details how to address those problems. If you do not find a solution in this section thencheck the Spotlight on Oracle Release Notes.

Product Authorization ErrorsWhen first installed, Spotlight can be used with full functionality for a specified trial period.When the trial period expires, contact your Quest Software representative to obtain anauthorization key to continue. See "Authorize Spotlight" (page 16) for more information.

Notes:

l Make sure that you enter the authorization key and site message exactly as stated on yourProduct Authorization sheet or as provided by your Quest Software distributor. The sitemessage is case sensitive. If you enter the authorization key incorrectly, the followingmessage is displayed:

Invalid authorization key

l Once the trial period has expired, if an authorization key is not supplied, only theSpotlight on Oracle home page will be visible. If you click any of the drilldown buttons,the following message is displayed:

You do not have a license for “Spotlight on Oracle” that enables accessto this functionality. Please contact Quest Software for informationabout obtaining the correct license.

If the trial date has not passed, check that the date on your system is set correctly.

4

Spotlight on Oracle Getting Started GuideTroubleshooting Spotlight on Oracle

65

Connection Problems

Cannot Connect to Database

Possible cause… Solution…

Oracle is not running. Verify that the Oracle instance is runningand can be accessed from a host or clientworkstation.

The connect string and alias of the databaseare incorrect or have changed.

Check your SQL* Net configuration andverify the connect string and alias you areusing are correct.

The user ID you are using to log intoSpotlight does not exist in the database.

Run the User Wizard to establish the userID in the database.

The Spotlight user has the wrongpermissions in the database.

Contact your DBA to add relevantpermissions to the user.

Remote commands do not work whenmonitoring Unix OS with a userID/password that exists on the machine.

Remote commands may not work unless theuser ID is added to the /etc/hosts.equiv fileon that Unix machine.

Spotlight takes too long to open the Oracleconnection so it appears to be unable toconnect.

Turn on Fast Initialize.More information on Fast Initialize isavailable in the online help.

1. Click View | Options.

2. Select Data Collection | OracleInitialization | Turn Fast Initializeon.

3. Restart Spotlight.

Error Attempting to Connect

The following error message is displayed:

Connection has been refused by the server

This means that Spotlight cannot access the operating statistics of the (Windows) databasemachine. Users must have access to the registry so that Spotlight can retrieve the operatingsystem statistics. A user account with administrative rights to the target machine will allow this.


66

Retrieve Data from Unix

If the database is running on Unix, Spotlight on Oracle may be using the REXEC or SSHprotocols to submit various Unix commands (for example, netstat, vmstat, iostat, sar) to analyzesystem activity.

Example

If Spotlight uses REXEC, these commands must be accessible to an REXEC session forSpotlight on Oracle to be able to analyze Unix activity. If these commands are not located in thestandard search path available to REXEC, Spotlight on Oracle displays an error.

Ensure that all Unix commands are located in the standard search path available to REXEC, orone of the following paths:

/usr/bin

/usr/sbin

/usr/ucb

If the command is located elsewhere in the file system, you must create a symbolic link in one ofthe areas listed above.

When observing a Unix operating system with a user ID/password that exists on the machine, itis possible that remote commands will not work on the machine unless the user ID is added tothe /etc/hosts.equiv file on that Unix machine.

Cannot Connect to ASM

When installed and available, Automatic Storage Management (ASM) stores Oracle data in aseparate ASM database instance. Spotlight on Oracle displays performance metrics for ASM onthe home page and in the ASM drilldown.

If the database under investigation uses ASM but Spotlight cannot connect to it to retrieve therequired information, you may need to edit the TNSNAMES entry for the ASM instance.

1. Locate the tnsnames.ora file in Oracle’s network/admin sub-folder of the Oracleinstallation folder on your Spotlight client.

2. Edit tnsnames.ora so that the TNSNAMES entry for the target ASM instance contains theoption (UR = A).

Drilldown Problems

Alarm Help Displayed Instead of Drilldown

Spotlight can be configured to show the help or jump directly to the related drilldown onclicking a home-page control.


67

To show the drilldown

1. Click View | Options.

2. Select Alarms and notifications | Balloon help.

3. Select Always jump directly to the related drilldown.

Dead Sessions in Top Sessions Drilldown

The solution is to reset the Top Sessions drilldown.

To reset the Top Sessions drilldown

From the Top Sessions drilldown

1. Click Filter.

2. Select Reset

3. Select the appropriate frequency.

What Happens to The Operating System Drilldown whenConnection Is Lost to The Server?

The Operating System drilldown may report inaccurate data if Spotlight looses access to thedatabase server.

1. Reconnect to the database. When the Connect dialog is displayed ensure that the OSserver and OS type fields (and if necessary the OS Username and OS Password) arecompleted correctly.

2. Ensure the Unix or Windows server on which the Oracle database is installed isaccessible to Spotlight on Oracle. See the Spotlight on Unix or Spotlight on WindowsGetting Started Guide.

Operating System Statistics Not Reported

Differences between versions of operating systems mean that Spotlight is unable to collect allO/S statistics in all environments. Refer to the Spotlight on Oracle Release Notes for moreinformation on the Operating System Drilldown.

Where the operating system is Unix refer to the Spotlight on Unix Release Notes and Spotlighton Unix online help.

Where the operating system is Windows refer to the Spotlight on Windows Release Notes andSpotlight on Windows online help.


68

Predictive Diagnostics Unavailable

You can use Predictive Diagnostics to collect and analyze performance metrics for SQLstatements as they are executed on the database instance, system bottlenecks as they affectdatabase throughput and response time, and database resources whose limitations affect databaseperformance.

If Predictive Diagnostics is unavailable for your Spotlight connections, re-run the Oracle userwizard to install Predictive Diagnostics.

1. Close and re-open Spotlight on Oracle.

2. Click File | Oracle User Wizard...

3. Create a spotlight user. Make sure that you install Predictive Diagnostics as part of theprocedure. See "Spotlight Users" (page 17) for more information.

Note:When you start using Predictive Diagnostics, Spotlight will take about 10 days to collectenough data to make valid predictions on the future performance of the database. During thattime, you can use the supplied sample data to view Predictive Diagnostics at work.

High Spotlight Load on Database ServerThe queries that Spotlight uses to collect data can place an additional load on the systemunder diagnosis.

Most Spotlight queries consume a measurable but not significant amount of system resources.However, queries that are complex or collect large amounts of data may put a significant load onthe system.

If you find that the load that Spotlight places on your server is too high, you can adjustcollection rates for different categories of Spotlight data, and so limit the load that Spotlightplaces on the system under analysis.

To choose a standard Spotlight setting or create your own Custom settings

1. Close and re-open Spotlight on Oracle.

2. Select File | Connect | Spotlight on Oracle.

3. Right click the Spotlight on Oracle connection experiencing the high load and selectProperties.

4. Select Overhead.

5. Adjust the Spotlight load for the connection:

l Use the Overhead Management slider to adjust the overall Spotlight load.

l Alternatively, click Custom… to adjust specific load categories.

6. Open the Spotlight connection.

Note: The procedure for adjusting the Spotlight load is covered in more detail in the online help.

Appendix: Contact Quest

Contact Quest SupportQuest Support is available to customers who have a trial version of a Quest product or who havepurchased a Quest product and have a valid maintenance contract. Quest Support providesunlimited 24x7 access to our Support Portal at www.quest.com/support.

From our Support Portal, you can do the following:

l Retrieve thousands of solutions from our Knowledge Base

l Download the latest releases and service packs

l Create, update and review Support cases

View the Global Support Guide for a detailed explanation of support programs, online services,contact information, policies and procedures. The guide is available at: www.quest.com/support.

Contact Spotlight Support

Whenever you have a question about Spotlight, click Support Bundle or Contact Support on theHelp menu. This creates a file called spotlightsupport.zip in one of the following folders:

Windows XP In a sub-folder under the Documents and Settings folder.

Windows Vista / Windows 7 In a sub-folder under the Users folder.

This file contains a snapshot of your Spotlight installation. Send this file and your request forassistance to Quest Software ([email protected]).

Notes:

l Support bundles contain information for the active instance only. If you are usingSpotlight to diagnose multiple instances ensure that the correct instance is displayedbefore creating the support bundle.

l Support bundles cannot be created when the History Browser is open. For furtherinformation on the History Browser, see the Spotlight online help under Conceptsand Features.

http://www.quest.com/support

http://www.quest.com/support

Spotlight on Oracle Getting Started GuideAppendix: Contact Quest

70

Contact Quest Software

Email: [email protected]

Mail: Quest Software, Inc.World Headquarters5 Polaris WayAliso Viejo, CA 92656USA

Web site: www.quest.com

Refer to our Web site for regional and international office information.

About Quest SoftwareNow more than ever, organizations need to work smart and improve efficiency. Quest Softwarecreates and supports smart systems management products—helping our customers solve everydayIT challenges easier and faster. Learn more at www.quest.com.



IndexA

activity drilldown 9, 40

aggregated alarms 58

alarm log drilldown 31, 42, 58

alarms 9, 31, 57

analyze trace 39

architecture diagram, Oracle 7

architecture diagram, Oracle RAC 49

array fetch 39

ASM 21, 32, 40, 51, 57, 59, 66

ASM drilldown

no data 52

authorize Spotlight 16

Automated Storage Management 21, 32, 40,51, 57, 59,

66

B

background processes panel flows 29-30

C

configuration and memory drilldown 40,46

connect to Oracle database 20

connect to Oracle RAC 49

contact support 69

D

disk storage drilldown 40

disk storage panel 31

disk storage panel flows 28-30

drilldowns 9, 39, 61, 66

Spotlight on Windows 43

DSA 23

DTrace 11, 26, 41

E

ESX 23, 28

event waits 9

Exadata Storage 57

explain plan 46

F

fast initialize 65

H

home page 8, 26

Spotlight on Oracle RAC 54

host panel 28

I

I/O drilldown 40, 46

install 12

instance panels 56

instance panels flows 55

interconnect and global status panel 55

interconnect panel flows 55

IO subsystem panel 57

O

operating system drilldown 40

Oracle architecture 7-8

Oracle Enterprise Manager 28

Oracle RAC architecture 48

Oracle user wizard 17, 68

Spotlight on Oracle Getting Started GuideIndex

72

P

parse activity 39

permissions 12-13, 15-16

predictive diagnostics 10, 18, 30, 36, 41,45, 68

open 45

preview mode 16

privileges 12-13

Spotlight user 17

project lucy 15

R

result cache 40

REXEC 22, 52

RSA 23

S

server processes panel 28

server processes panel flows 27-28

sessions drilldown 39

sessions panel 27

sessions panel flows 27

Set timed statistics now 20

sga panel 29

sga panel flows 28-29

sort activity 39

space manager 31, 43

Spotlight on Windows

drilldowns 43

Spotlight users 17

SQL & application workload drilldown 39

SQL optimizer 43, 46

SSH 22, 52

start Spotlight 15

StealthCollect 24

T

Toad DBA Suite for Oracle 13-14, 25

top sessions drilldown 9, 67

top SQL drilldown 39, 46

trial key 16

tuning drilldown 40, 46

U

uninstall 14

user, Spotlight 17

V

virtual system 23, 28, 42

W

wait activity 9

waiting events 46

Windows 22, 52