IBM® Tivoli®

IBM Tivoli Composite Application Manager for Microsoft® SQL Server Agent

KB Notes and HOW TOs

Version 1.0

CONTENTS

Contents .................................................................................................................................. 2

1. Overview ........................................................................................................................... 3

1.1 Introduction .................................................................................................................. 3

1.2 Terms and abbreviations .............................................................................................. 3 1.3 User Guides ................................................................................................................. 3

2. HOW TOs........................................................................................................................... 4

2.1 How to configure Agent ................................................................................................ 4 2.2 How to Configure Agent in Cluster ............................................................................... 6 2.3 Domain user account configuration steps & windows authentication issue. ................ 6

2.4 How to enable / reload Perfmon counters? .................................................................. 6 2.5 Verify if the MSSQL performance counters have been enabled on the system running the SQL server. ..................................................................................................................... 7

3. Environment Variables ..................................................................................................... 8

4. Problems, Symptoms, Solutions .................................................................................. 10

4.1 Agent failed to connect SQL Server because database is not connected .................. 10

4.2 Blank Configuration window issue ............................................................................. 10

4.3 SQL Server 2000 Support and ITM Frameworks ....................................................... 10 4.4 Running two different versions of MSSQL Agent/ SQL Servers on same machine .... 11

4.5 Table details collection Taking Time ........................................................................... 11 4.6 Crash or Hang issues due to Event Errorlog ............................................................. 12 4.7 Unable To see Configuration Panel ........................................................................... 12

4.8 Database Detail and Device Detail Calculation Steps ................................................ 12 4.9 koqos.exe crash issue ............................................................................................... 16

4.10 Upgrade 64 bit v6.2 Agent from previous version 32 bit ......................................... 17 4.11 Database Summary, Database Detail and Device Detail workspaces shows incorrect value in specific scenario ............................................. 17

4.12 Upgrade SQL agent to v6.3.0 or v6.3.0.1 ............................................................... 17

4.13 Minimum SQL Server Permission required ............................................................. 18

1. Overview

1.1 Introduction This document lists and provides solutions to common problems faced by customers of IBM Tivoli MS SQL server Monitoring Agent The intended audience of the document is Level 2 personnel for IBM Tivoli MS SQL Server Monitoring Agent and the customers of the product. Various sections of the document refer to configuration of agent, setting environment variable and troubleshoot ways to solve various problems occurring in customer environment. 1.2 Terms and abbreviations

1.3 User Guides Please refer following user guides. sqlagent63_user.pdf.

Term / Abbreviation Description

IBM International Business Machines

ITM IBM Tivoli Monitoring

MS Microsoft

L3 Level three support

PDF Adobe Postscript Data Format

TEP Tivoli Enterprise Portal

2. HOW TOs

2.1 How to configure Agent You can install and configure the Microsoft SQL Server agent locally or remotely using a GUI or command line.

Note: the ―Table Detail Continuous Collection‖ option is available from v6.3.0 onwards. Following is the list of configuration settings.

Sr. No.

Configuration Attribute

Description / Configuration value

1 Server Name Name of the SQL Server instance that is to be monitored. If monitoring the SQL Server Default Instance, use host name, else use SQL Server Instance name

2 Login Login is required only when windows authentication parameter is set to False.

3 Password Password for SQL User, in case it is SQL Authentication. Password is required only when Windows Authentication parameter is set to False.

4 Home Directory

Home Directory path of the SQL Server Instance being monitored.

5 Error Log File

Fully qualified location and name of the SQL Server Error Log.

6 Windows Authentication

Enables or disables Windows Authentication

7 Support Long Lived Database Connections

Enables or disables long lived database connections The following attribute groups do not use Long Lived Database Connections: 1 MS SQL Text 2 MS SQL Filegroup Detail 3 MS SQL Server Summary

8 Extended Parms

Disables capture of Table Detail attributes

9 Include Use this feature to filter databases that you want to monitor.

Please refer User Guide for Details.

2.2 How to Configure Agent in Cluster 2.2.1: Configure Agent in Cluster manually Whenever we try to add agent & collector services in cluster, then make sure first these services has been stopped before adding the services in cluster. After that make the service type as Manual. For Detail steps to configure Agent in cluster please refer 1. Latest SQL Server Agent User Guide. 2. Demo video for Configuring SQL Agent in Cluster

http://www.ibm.com/developerworks/wikis/display/tivolimediagallery/Microsoft+SQL+Server+agent+in+a+cluster+environment

2.2.2: Configure Agent in Cluster using Utility User can use the Cluster Configuration Utility for configuring Agent in Cluster Automatically. http://www-01.ibm.com/support/docview.wss?uid=swg21620558

2.3 Domain user account configuration steps & windows authentication issue. User must check 'Windows Authentication' checkbox in agent configuration, if the Microsoft SQL Server Agent service is configured to use Windows User. This user account will be used to access the Microsoft SQL Server, and hence this account needs to have rights on the SQL Server. Same is the case for Collector Service.

2.4 How to enable / reload Perfmon counters? The agent collects most of the data from Perfmon objects provided by Microsoft. In certain cases, the agent doesn‘t display data on the TEP for certain counters. The reason behind this could be the counters haven‘t been loaded / enabled on the machine where the agent is running. Enabling / reloading the concerned counters could solve the problem of data not being displayed on the TEP.

If you see the below message in the collector log file :- UEX2492E KDDNTREG(4874) write_title_lst(): SQL Server Instance performance data is not enabled! or …. Failure during creation of perflib counter index file; rc = 8 Below are the steps to unload & load the perfmon counters :- For Default instance:- 1)To unload the perfmon counter :- execute below command on windows command prompt. unlodctr MSSQLSERVER 2) To Load perfmon counter :- execute below command on windows command prompt.

lodctr <SQL Server Path>\binn\sqlctr.ini For Named instance :- --------------------------- 1) To Unload the perfmon counters, execute below command on windows command prompt. unlodctr MSSQL$<instance_name> Note:- <instance_name> is SQL server instance name. 2) To Load the perfmon counters, execute below command on windows command prompt. lodctr <SQL Server path>\binn\MSSQL$InstanceName.ini 2.5 Verify if the MSSQL performance counters have been enabled on the system

running the SQL server. This can be done using the perfmon utility as follows: 1. Start the perfmon utility. 2. Click the '+' symbol 3. Check whether SQLServer:Databases, SQLServer:Buffer Manager, etc. are available on the 'Performance object' dropdown list.

3. Environment Variables

Note: Some of the Environment variables may not be available in older SQL Agent versions.

COLL_ALERT_SEV:- This environment variable is added to manage output of problem details and problem summary attribute groups. By default in the problem detail workspace we see the error messages whose severity is 17 and greater. If we want to see the error message below 17 then need to use this variable with severity above which error message will be displayed on portal. E.g. if the value for this variable is set as 10, then the severity 10 and greater will be displayed on portal.

WAIT_TIMEOUT:- If any of the cursors (attribute groups) takes more than 45 seconds to collect data then in that case agent might hang or situations will be fired incorrectly. Using this variable we can increase the wait time between agent exe and collector exe. We can check cursors which may be taking more than 45 seconds.

QUERY_TIMEOUT:- Whenever any of the database objects (like views/tables/stored procedures) is in locked state then none of the application will be allowed to fire query on SQL server. Query fired by the agent will be in wait state till SQL server object becomes free to execute next query. So in this case agent might hang & data will not be displayed on portal. So using the above variable we can limit the time to which agent will wait for response from SQL server to the fired query, if response is not received in the specified time then that query will be killed by SQL server & agent will go ahead & display the other data. Note:- For huge number of tables/databases/jobs etc. this will not work as whole collection will not completed in the specified 45 seconds because query timeout for all queries will be cumulative I.e. for KOQDBD collection there are let's say 10 queries required to fetch data for 1 databases. If we have set query timeout of 10 sec, then for one database it requires 10*10=100 seconds, so for 4 databases it goes to 400 seconds so in this case query timeout will not work.

COLL_DBCC_NO_LOCK:- If some of the queries from database detail & device detail workspaces take more time to execute then in that case we can use this variable to fire query with option ‗WITH (NOLOCK)‘ so that query fired on the SQL server will not wait if object on which query fired is locked.

COLL_WRAPLINES:- Maximum number of lines in one ―col.out‖ file is defined in this variable. By default it is set to 30,000 lines (about 2 MB). Note: from the release v6.3.0 onwards, the default value is 90,000 lines

COLL_NUMOUTBAK:- By default only one backup of col.out files are created. In case user wises to make more backup copies, set this variable, like as follows, If number of files to be backed up is, 1. less than 10, set the number directly, e.g. 1 or 3 or 9 2. less than 1000, and greater than 9, set number starting hyphan symbol (-) e.g. -10 or -352 or -500 etc.

COLL_DEBUG:- If we want to enable full tracing of collector then set it to 'dddddddddd' (10 times ‗d‘)

COLL_CLUSTERSERVER:- Whenever SQL server is in a cluster then this variables value must be the name of the virtual server (i.e. name of the cluster)

KOQ_NAMED_INSTANCE:- For default instance this variables value must be 0 & for named instance it must be 1.

COLL_PORT:- If SQL server instance is configured on non-default port (e.g. port other than 1433), user needs to set this variable to specify the port number to monitor the SQL Server Instance.

COLL_JOB_DISABLED:- A new environment variable has been added named 'COLL_JOB_DISABLED', which can be set by user. The variable has been introduced for not displaying the information of the disabled jobs in the job details workspace. Set this variable to ―1‖, so disabled jobs will not be displayed, set it to ―0‖ otherwise. By default all jobs will be displayed. Note: this variable is available from v6.2.3.1 onwards.

COLL_KOQDBD_INTERVAL:- A new environment variable has been added named 'COLL_KOQDBD_INTERVAL‘ which can be set by user. If user has large number of databases, then the KOQDBD collection takes longer time, in this case if user does not want frequent collection, he can set the interval time between two consecutive collections by setting the above environment variable. Note: this variable is available in v6.2.3.1 IF01 onwards COLL_KOQDEVD_INTERVAL:- A new environment variable has been added named 'COLL_KOQDEVD_INTERVAL'; it is used to set interval time between two consecutive collections. The KOQDEVD collection takes longer time, if users have large number of devices. In this case user may not wish to have frequent collection. He can set interval time between two consecutive collections using above environment variable. Note: This variable is available in v6.3.0 Fixpack01 onwards.

4. Problems, Symptoms, Solutions

4.1 Agent failed to connect SQL Server because database is not connected If we see problem in connection that is agent fail to connect to SQL server then we need to check following

check if SQL User and password is correct.

Check from control panel if you can connect to SQL Server from ODBC

Use Microsoft utilities like osql.exe for SQL Server 2005 and sqlcmd.exe for SQL Server 2008 - Open Command Prompt - Change directory to <SQL Server Path>\90\tools\binn - execute the following commands, 1. Test connection to SQL Server 2005 command with windows authentication:- For Default instance osql.exe -E -S <SQL Server Name> Named instance :- osql.exe -E -S <Computer Name\Instance Name> 2. Test connection to SQL Server 2008 with windows authentication:- Default instance sqlcmd.exe -S <SQL Server Name> Named instance :- sqlcmd.exe -S <Computer Name\Instance Name> 4.2 Blank Configuration window issue Sometimes user may see no values in the configuration window. This happens if agent does not find all required registry entries for SQL Server instance under HKEY_LOCAL_MACHINE\SOFTWARE\Candle tree. In this case click on 'New' button in configuration window and provide the required parameters. - Verify if registry value for KOQ_NAMED_INSTANCE is set, zero (0) for default instance and one (1) for named instance. -Verify if you are having 32 Bit SQL server instance and Agent is 64 Bit.

4.3 SQL Server 2000 Support and ITM Frameworks Support of SQL server 2000 has been deprecated from SQL Server Agent version 621 (IBM Tivoli Monitoring for Microsoft Applications v620) and later.

Please refer the following table for the SQL server 2000 and ITM support.

4.4 Running two different versions of MSSQL Agent/ SQL Servers on same machine -It is not possible to install two versions of SQL server monitoring agent on same machine. - As support of SQL Server 2000 is removed from SQL agent version 621 onward, it is not possible to monitor SQL Server 2000 and SQL server 2005 and above on same machine.

4.5 Table details collection Taking Time Whenever there is large number of tables in customer environment, the agent hangs or no data is displayed on portal. Following is solution for said problem, For v620 IF15, 622, and 623, user needs to enable the 'Table Detail Collection Settings' through MTEMS windows. New Changes In v630 FixPack01 for table details collection settings: - The Table Detail Continuous Collection option is set in the agent configuration panel by default (the Checkbox is in ―Checked‖ state). If you have already configured the settings related to Table Detail Collection (like daily, weekly or Monthly), then those settings will be used, instead of ―Table Detail Continuous Collection option‖. - The default interval between the two consecutive collections is set to 3 minutes. - Specifying a value for the DBSETTINGS.db_tbldconcoll parameter is now optional. You can specify a value for this parameter either by using the command line or the agent configuration panel. - While configuring the agent remotely by using the command line or by using the GUI, the setting ‗Continuous Table Collection' takes precedence over the setting 'Table Detail Scheduling', i.e. if both the settings are set, the setting ‗Continuous Table collection‘ will be in effect, and not the 'Table Detail Scheduling'.

SQL Agent version MSSQL 2000 ITM versions

MSSQL 6.2.0 y ITM 6.2.0 or higher

MSSQL 6.1.2 y ITM 6.1.0 Fix Pack7 or higher

MSSQL 6.2.1 n ITM 6.2.0 Fix Pack1 or higher

MSSQL 6.2.1.1 n ITM 6.2.0 Fix Pack1 or higher

MSSQL 6.2.2 n ITM 6.2.1 Fix Pack1 or higher

MSSQL 6.2.3 n ITM 6.2.2 Fix Pack2 or higher

MSSQL 6.3.0 n ITM 6.2.2 Fix Pack2 or higher

MSSQL 6.3.0 FixPack01

n ITM 6.2.2 Fix Pack2 or higher

For rest of the agent, user needs set the value ―KOQTBLD‖ in the edit box ―Extended Parms‖ available in Agent Configuration window to Disable the table details Collection. Note: Use this setting only if the KOQTBLD (Table details) takes more than 45 seconds. For details refer the user guide of the latest release of SQL Server Agent. 4.6 Crash or Hang issues due to Event Errorlog MS SQL Error Event Details – New attribute group is added in SQL agent version 6.2.3. Whenever Errorlog file gets updated, and if that message has severity greater than default severity (i.e. 17) then this message will be immediately displayed on portal. If user wish to see the previous messages, then user needs to set following environment variables COLL_ERRORLOG_STARTUP_MAX_TIME COLL_ERRORLOG_STARTUP_MAX_EVENT_ROW COLL_ERRORLOG_MAX_EVENT_ROW Note: for details about these variables, please refer user guide. As this newly added attribute group is not available in default query or any workspace, user need to create new query and workspace. 4.7 Unable To see Configuration Panel If we are not able to see configuration panel by right clicking on 'Configuration using defaults' on the Template for SQL server in MTEMS window then this issue is related with framework & not the agent side issue. 4.8 Database Detail and Device Detail Calculation Steps Database Details: 1. Data Size (MB) 2. Data Free Space (MB): 3. Database Max Growth Size (MB) : 4. Database Growth Percentage (MB) A] Data Size (MB): This value is getting from performance monitor.

DB size (MB) = {Value getting from Perfmon * 100} / 1024. Tip: Multiply by 100 for ODI/Portal precision E.g. in above case it is 4 MB B] Data Free Space (MB): Case 1: a. Enable Auto Growth = on b. File Growth = in Percentage c. Maximum File Size = Unrestricted File Growth.

DB free (MB) = Drive Free Space. E.g. if data base file drive is c: and it has 13.5 GB free space then Data Freespace (MB) is 13824 MB. Case 2: a. Enable Auto Growth = on b. File Growth = in Percentage c. Maximum File Size = 100 MB.

DB free (MB) = Data base maximum size – Data Size (Calculated in previous step). E.g. In this case 100 MB - 4MB = 96 MB Case 3: a. Enable Auto Growth = off

To calculate this we have to calculate the free number of pages available in the file and we are getting this by using below queries. USE ‘Databasename’ select TotalPages = sum(convert(dec(15),size)) from sysfiles where (status & 64 = 0)

select Reservedpages = sum(a.total_pages) from sys.allocation_units a

DB free (MB) = ((TotalPages – Reservedpages)*8)/1024

C] Database Max Growth Size (MB):

Data_Max_Growth_Size = DB size (MB) + DB free (MB).

DB size (MB): Calculated in [A]. DB free (MB): Calculated in [B].

D] Database Growth Percentage (MB): Data_Max_Growth_Pecentage = (DB size (MB) / (DB size (MB) + DB free (MB))) * 100;

E] Data File Free Spaces [MB] USE ‘Databasename’ select TotalPages = sum(convert(dec(15),size)) from sysfiles where (status & 64 = 0)

select Reservedpages = sum(a.total_pages) from sys.allocation_units a

Data File Free Spaces (MB) = ((TotalPages – Reservedpages)*8)/1024

E] Aggregate Free Space [MB] Aggregate Free Space = Data File Free Spaces [MB] + DB free (MB).

Device Details: Device Size (MB) and Device Free Space (MB): Case 1: Enable Auto growth = on Maximum file size = Unrestricted file Growth.

Device Size (MB): Drive Size. Device Free Space (MB): Free Drive Size Case 2: Enable Auto growth = on Maximum file size = Restricted file Growth (100 MB).

Device Size (MB): Maximum File Size (in above case it is 100 MB). Device Free Space (MB): Maximum File Size (MB) – Data Size (MB) [Calculated in step [A]) Case 3: Enable Auto growth = off

Device Size (MB): Data Size (MB) (Calculated in step [A]) For calculating the Device Free space, use the following query. USE NameOfDataBase select fileproperty('NameOfDataBase','spaceused') As 'allocated'

Allocation = (allocated *8 )/1024

Device Free Space (MB): Device Size (MB) – Allocation 4.9 koqos.exe crash issue koqos.exe gets crashed whenever the OS related performance counter are disabled. (i.e. Counter from PerfOS.dll, PerProc.dll ).We can use the 'exctrlst.exe' to enable/disable the counters.

4.10 Upgrade 64 bit v6.2 Agent from previous version 32 bit

User can upgrade any lower SQL agent version to the latest one however upgrading to 64 bit v6.2.3 SQL Server Agent from all earlier 32 bit SQL Server Agent is not possible. Following are the steps for upgrading 623 64 Bit SQL Agent. 1. Uninstall existing 32 Bit agent. 2. Install SQL 623 64 bit. Note: Once the agent is upgraded to upper versions like SQL Agent v623, user has to reconfigure the agent. 4.11 Database Summary, Database Detail and Device

Detail workspaces shows incorrect value in specific scenario Database Summary, Database Detail and Device Detail workspaces shows incorrect value in specific scenario, where we have changed database state to 'read only' for last created database. To Solve this problem: -Recycle SQL Server Agent. -Disable Long Live connection in SQL Agent configuration 4.12 Upgrade SQL agent to v6.3.0 or v6.3.0.1 In v6.3.0 the Agent and Collector service names are changed, if user does a fresh install the new service names can be seen, however when you upgrade SQL agent to v6.3.0 or v6.3.0.1 from any previous version, agent (koqagent.exe) and collector (koqcoll.exe) service names remains same. This does not have any functional impact on agent behavior.

4.13 Minimum SQL Server Permission required

Required minimum Server role / database Roles

MS SQL Database Detail Sysadmin or db_owner

MS SQL Job Details Database Role (msdb database): SQLAgentOperatorRole

MS SQL Job Summary AND

Database Role (msdb database): db_datareader

MS SQL Log shipping job detail Database Role (msdb database): db_datareader

MS SQL Log shipping DB detail

MS SQL Log shipping errors

MS SQL Table Detail Database Role: db_owner

(for each database)

—OR—

Server Role: System

Administrator

—OR—

VIEW DATABASE STATE and VIEW SERVER STATE

permissions are required on the SQL Server

MS SQL Server Details Database Role:db_owner OR Server Role: System administrator

MS SQL Server Summary VIEW SERVER STATE permissions are required on the SQL Server

MS SQL Statistics Summary

MS SQL Lock Details

MS SQL Lock Summary