db2 autonomics - autopilot for the dba
TRANSCRIPT
May 21, 2008 • 4:00 p.m. – 5:00 p.m.Platform: DB2 for Linux, UNIX, Windows
Jayesh ThakrarIntersect Solutions, LLC
Session: D12
DB2 Autonomics - Autopilot for the DBA
2
Jayesh ThakrarIntersect Solutions, LLC
Session D12
DB2 Autonomics -Autopilot for the DBA
3
DB2 Autonomics –Autopilot for the DBA
• Describe DB2 LUW Autonomic features
• Show how to implement Autonomic features using GUI as well as DB2 commands / SQL
• Present some of the best practices to implement DB2 Autonomics for recoverability, availability and performance
Make deploying, managing and supportingDB2 LUW a pleasant & scalable experience
Session Objectives –
4
DB2 Autonomics –Autopilot for the DBA
• Introduction
• Discuss DB2 Autonomics FeaturesStorage Management
DB2 Configuration Management
Routine Maintenance
Health Monitoring and Notification
Workload Management
• Discuss Implementation Best Practices
• Questions & Answers
Agenda
5
Autonomics – What and Why?
6
What is Autonomics ?Autonomic Computing is an initiative started by IBM in 2001. Its ultimate aim is to create computer systems capable of self-management, to overcome the rapidly growing complexity of computing systems management, and to reduce the barrier that that complexity poses to further growth.From : Wikipediahttp://en.wikipedia.org/wiki/Autonomic_Computing
Autonomic computing architecture is a range of software technologies that enable you to build an information infrastructure that can, to lesser and greater degrees, manage itself, saving countless hours (and dollars) in human management. And all this without giving up control of the system. From : Hello Autonomic Computing!http://www.ibm.com/developerworks/edu/ac-dw-ac-intro-i.html
In a nutshell, administrators spend too much time doing repetitive tasks that the system should be able to do for itself.From : An autonomic Computing Roadmaphttp://www.ibm.com/developerworks/autonomic/library/ac-roadmap/
7
Why Autonomics and Why Now?
• High database to DBA ratio• Large number of databases on a server• Large, rapidly growing databases• Automation - reduce human intervention• Progressing maturity• Competitive advantage
Bottom line – reduce TCO !
8
Autonomics in DB2 LUW
Storage Management
Configuration Management
Health Monitoring & Actions
Workload Management
Routine Maintenance
9
Autonomics in DB2 LUW
Storage Management
10
Traditional Storage Management
Size database and storage needs upfrontSelectively place tablespace containersMonitor tablespace free space Monitor storage free space
11
Automatic Storage Management
Size database and storage needsSpecify storage locations and create databaseCreate tablespacesSit back and relax !
Tablespaces grow as needed and alert you when filesystem or directory out of space !
12
Automatic Storage Management
• CREATE DATABASE MYDB• CREATE DATABASE MYDB AUTOMATIC STORAGE YES• CREATE DATABASE MYDB AUTOMATIC STORAGE YESON /db2/data01, /db2/data02,
/db2/data03, /db2/data04DBPATH ON /db2/mydb
Examples – Database Creation
13
Automatic Storage Management
<storage_path>/<instance_name>/<node_no>/<tablespace_no>/<container_no>.<ext>
Where
Automatic Storage Container Paths
<storage_path> = Implicitly or explicitly defined storage path
<instance_name> = Name of instance under which database is created
<node_no> = The database partition number
<tablespace_no> = Tablespace id (as in SYSCAT.TABLESPACES)
<container_no> = Container id
<ext> = CAT (catalog) / TMP (temporary) / UTM (user temporary) / USR (user or regular) LRG (large)
14
Automatic Storage Management
• CREATE TABLESPACE Staging_Data• CREATE TABLESPACE Staging_Data
MANAGED BY AUTOMATIC STORAGE• CREATE TABLESPACE Staging_Data
PAGESIZE 32 K MANAGED BY AUTOMATIC STORAGE
• CREATE TEMPORARY TABLESPACE Temp_32K PAGESIZE 32 K
REGULAR / LARGE TABLESPACE = DMS TYPE WITHAUTOMATIC STORAGE
TEMPORARY TABLESPACE = SMS TYPE
Examples – Table Space Creation
15
Automatic Storage Management
• Additional storage paths can be added after database creation• Automatic storage can only be enabled at database creation• Databases with automatic storage allow SMS & DMS tablespaces• Automatic storage also works with partitioned databases (v9.0+)• Database partition expressions can be used e.g.
CREATE DATABASE MYDBON "/data01_ $N", "/data02_ $N"
"/data03_ $N", "/data04_ $N"
• On database restore, storage paths can be preserved or redefined• DB2 v9.5 – can resize automatic tablespace to reduce size• DB2 v9.5 – automatic compression for large tables
Additional Info
16
Autonomics in DB2 LUW
Configuration Management
17
Configuration Management
sqllib/SQLnnnnn/SQLDBCON
sqllib/db2systm
Configuration Parameters – where do they go
18
Configuration ManagementTraditionally - Exasperating and Overwhelming Task
DB2 Registry Level Parameters = 100+
Instance Level Parameters = 50+
Database Level Parameters = 50+
Memory Capacity Management
Agent Management
I/O, Storage and Logging Management
Locking Management
Query Processing & Compiler Management
Communication Management
Security Management & Misc.
Effective at next statement / transaction
Effective at next application connect
Effective on database re-activation
Effective on instance restart
DBA
19
Configuration Management
• Dynamic configuration parameters• Better “default values”
RESET DATABASE MANAGER CONFIGURATIONRESET DATABASE CONFIGURATION FOR MYDB
• STMM - Self-Tuning Memory ManagerUPDATE DBM CFG USING <parameter> AUTOMATICUPDATE DB CFG FOR MYDB USING <parameter> AUTOMATIC
• “Autoconfigure” all dynamic parameters using db2 clp command or Wizard
• GUI Wizard-based interface to autoconfigure & STMM
Autonomic Features – Sit Back and Relax !
20
Configuration ManagementDefault Database Manager Parameters
21
Configuration ManagementAutoconfigure• Autoconfgure at database creation
(via command-line or using database creation wizard)CREATE DATABASE MYDB AUTOCONFIGURE
APPLY DB AND DBM
• Autoconfigure anytime – command-line or Configuration Advisor
22
Configuration ManagementAutoconfigure Keywords
Keyword Valid Values Explanationmem_percent 1–100 Percentage of memory to dedicate. If other applications (other than the
operating system) are running on this server, set this to less than 100
workload_type simple, mixed, complex
Simple workloads tend to be I/O intensive and mostly transactions, whereas complex workloads tend to be CPU intensive and mostly queries.
num_stmts 1–1 000 000 Number of statements per unit of work
tpm 1–200 000 Transactions per minute
admin_priority formance, recovery, both
Optimize for better performance (more transactions per minute) or better recovery time
is_populated yes, no Is the database populated with data?
num_local_apps 0–5 000 Number of connected local applications
num_remote_apps 0–5 000 Number of connected remote applications
isolation RR, RS, CS, UR It is only used to determine values of other configuration parameters. Nothing is set to restrict the applications to a particular isolation level and it is safe to use the default value.
bp_resizeable yes, no Are buffer pools resizeable?
23
Configuration ManagementSTMM - Self-Tuning Memory Manager• STMM = DB2 feature to dynamically manage database
memory among 2 or more of the following “consumers” –BUFFERPOOLS PCKCACHESZ (Package cache)LOCKLIST (Lock list – total memory for locks within the database)MAXLOCKS (max. % of locks per application)SORTHEAP (sort list heap – max. memory allocated per sort)SHEAPTHRES_SHR (sort heap threshold for shared sorts)STMTHEAP (SQL Statement heap)APPLHEAPSZ (Application heap size)APPL_MEMORY (Application memory size)STAT_HEAP_SZ (Statistics heap size – used during runstats)
• STMM – takes over configuring / tuning of database parameters(adjustments done dynamically)
24
Configuration ManagementSTMM – How to Enable And Other Information• Set SELF_TUNING_MEM database parameter to ON• Set 2 or more database parameters (consumers) to AUTOMATIC• Set DATABASE_MEMORY parameter to
AUTOMATIC (Windows and AIX only)COMPUTED (total memory requirements computed at database activation time, with some room for growth/overhead)Explicit value (all consumers “bound” by this upper limit)
• STMM adopts to all of the following –Changing database workload (e.g. OLTP during business hours and batch processing at night)Multiple DB2 instances and database on the database serverNon-DB2 and OS memory needs
• Dynamic adjustments at 30 seconds to 10 minute intervals• Note – STMM works only for “active databases”
25
Configuration ManagementSTMM – How to Monitor STMM Behavior• Detailed STMM activity log files in stmmlog directory
(located in instance/sqllib)Multiple stmm.<NNN>.log files (each about 10 MB)Contains detailed record of “Cost Benefit Analysis” for each consumer
• To examine current values of database memory consumers –Use database snapshot commandget snapshot for database on MYDB
Use db2pd command(many options – see DB2 Command Reference Manual)
• Note - following DB2 registry entries affect STMM’s operations –DB2_LARGE_PAGE_MEMDB2MEMDISCLAIMDB2MEMMAXFREEDB2_MEM_TUNING_RANGEDB2_PINNED_BP
26
Configuration ManagementSTMM – Partitioned Databases• STMM can work in partitioned databases also• STMM started only if database activated explicitly• All STMM monitoring and adjustments based on a single partition• STMM changes propagated to all partitions in database• Individual partitions can be disabled for STMM• Database parameters can be set manually at a partition• Check current database partition used for STMM using –
CALL SYSPROC.ADMIN_CMD( ’get stmm tuning dbpartitionnum’ )
• Change current STMM database partition using –CALL SYSPROC.ADMIN_CMD( ’update stmm tuning dbpartitionnum <db_partition_num>’ )
• STMM advisable if servers with similar configuration & workload• Disable STMM on partitions with dissimilar configuration/workload
(e.g. catalog partition or dedicated co-ordinator partition)
27
Configuration ManagementOther GUI Wizards
28
Autonomics in DB2 LUW
Routine Maintenance
29
Database MaintenanceFeature Overview• Runs DB2 utilities only on “as needed basis” –
BACKUPREORGSRUNSTATSSTATEMENT STATISTICSSTATISTICS PROFILING
• Alternatively can notify to run the following utilities –BACKUPREORGSRUNSTATS
• Enable / disable via database configuration parameters• Can define “online” and “offline” maintenance windows• GUI wizard for database maintenance – more features
Maintenance windows only through WizardSpecify automatic maintenance for subset of tables and indexes Specify backup destinationEnable offline / online maintenance individuallySpecify recipients of notification messages (available as CLP for health monitoring)
30
Database MaintenanceDatabase Configuration Parameters
31
Database MaintenanceGUI Wizard – Introduction
32
Database MaintenanceGUI Wizard – Enable / Disable Maintenance
33
Database MaintenanceGUI Wizard – Define Online / Offline Windows
34
Database MaintenanceGUI Wizard – Define Notification List
35
Database MaintenanceGUI Wizard – Configure Type of Maintenance
36
Database MaintenanceGUI Wizard – Define subset of tables for reorg
37
Database MaintenanceGUI Wizard – Reorg Options
38
Database MaintenanceGUI Wizard – Maintenance Specs Review
39
Database MaintenanceAdditional Information• Automatic Backup
Backup to disk, tape, TSM or vendor DLL / plug-inOnline (if archive logging enabled) or offline backupsFor partitioned databases, backup happens at each partitionDisk-based backup can delete previous/obsolete backupsNo compressionIncremental backups only in DB2 LUW V9.5
• ReorgsOffline reorgs onlyFor partitioned databases, reorg evaluation done on catalog parttion onlyReorg Options include rebuild or retain compression dictionary
• Runstats, Statement Statistics & Statistics ProfilingOnline operationFor partitioned databases, evaluation done at catalog partition onlyEnable runstats only or all of themProfiling – updates individual table runstats profilesStatement statistics – collects SQL details for input to profiling and runstats
40
Autonomics in DB2 LUW
Health Monitoring & Actions
41
Health MonitoringFeature Overview• Monitors “health” of instances and “active” databases• Instance level parameter to enable/disable health monitoring• Pre-defined 29 Key Health Indicators (KHI) monitored continuously
Instance level and database level key health indicators Tablespace level and tablespace container level key health indicators
• User-configurable thresholds for each KHI (where value-based)• Different KHI inspected at different, pre-defined intervals • On threshold – notification and optional user-defined action• DB2 DAS used for notification to email or pager• KHIs can be enabled/disabled individually• Command-line / GUI interface to enable, disable and configure KHIs• Command-line / SQL functions to retrieve monitoring data• “Action” for alerts can be SQL or OS scripts/command• Default and user-defined parameters passed to action scripts
42
Health MonitoringKey Health Indicator Listingdb2.db2_op_status db.lock_escal_rate
db2.sort_privmem_util db.apps_waiting_locks
db2.mon_heap_util db.catcache_hitratio
db.db_op_status db.pkgcache_hitratio
db.db_backup_req db.shrworkspace_hitratio
db.hadr_op_status db.db_heap_util
db.hadr_delay db.auto_storage_util
db.log_util db.sort_shrmem_util
db.log_fs_util ts.ts_auto_resize_status
db.deadlock_rate ts.ts_util_auto_resize
db.locklist_util ts.ts_util
db.spilled_sorts ts.ts_op_status
db.tb_reorg_req tsc.tscont_util
db.tb_runstats_req tsc.tscont_op_status
db.max_sort_shrmem_util
43
Health MonitoringDB2 CLP Commands for Health Monitor• GET / UPDATE ALERT CONFIGURATION• GET / UPDATE CONTACTGROUPS• GET / UPDATE CONTACTS• GET / UPDATE HEALTH NOTIFICATION CONTACT LIST• GET HEALTH SNAPSHOT• GET RECOMMENDATIONS FOR HEALTH INDICATOR
44
Health MonitoringUPDATE ALERT CONFIGURATION - Syntax
Any of the KHI listed earlier
Parameter name can be ALARM, WARNING, SENSITIVITY ACTIONSENABLED, THRESHOLDSCHECKED
See next slide for Script Details
45
Health MonitoringUPDATE ALERT CONFIGURATION - Syntax
46
Health MonitoringUPDATE ALERT CONFIGURATION - Examples• Disable the checking of an individual KHI
UPDATE ALERT CONFIGURATION FOR DATABASE MANAGER USING db2.mon_heap_util SET THRESHOLDSCHECKED NO
• Get & update the threshold for automatic storage path(s) free spaceGET ALERT CONFIGURATION FOR DATABASE ON MYDBUSING db.auto_storage_util UPDATE ALERT CONFIGURATION FOR DATABASE ON MYDBUSING db.auto_storage_util SET WARNING 88, ALARM 95
• Get & update action for tablespace stateGET ALERT CONFIGURATION FOR TABLESPACES ON MYDBUSING ts.ts_op_statusUPDATE ALERT CONFIGURATION FOR TABLESPACES ON MYDBUSING ts.ts_op_status ADD ACTION SCRIPT <full_path_for_script> TYPE OSWORKING DIRECTORY <full_path_for_directory> ON WARNING USER <username> USING <password>
• Delete action script for database stateGET ALERT CONFIGURATION FOR DATABASE ON MYDBUSING db.log_util DELETE ACTION SCRIPT <full_path_for_script> ONWARNING
47
Health MonitoringUPDATE CONTACTS / CONTACTGROUPS - Syntax
Keyword can be ADDRESS (email address), TYPE (email or pager) MAXPAGELEN (max. length for pager message) or DESCRIPTION
48
Health MonitoringGET HEALTH SNAPSHOT - Syntax
Show all history data (usually upto 9 previous data collections). Also helpful in determining “monitoring period”
Shows all objects being examined (e.g. table statistics)
49
Health MonitoringGET RECOMMENDATIONS FOR HEALTH INDICATOR - Syntax
50
Health MonitoringAdditional Information• Disable certain KHI if STTM enabled and relevant
parameters set to AUTOMATICdb.sort_shrmem_utildb.spilled_sortsdb.max_sort_shrmem_utildb.locklist_utildb.lock_escal_ratedb.pkgcache_hitratio
• For operating system (OS) action scripts, following information passed as parameters –
List of user-supplied parameters (if any)Health indicator short nameFully qualified object nameHealth Indicator valueAlert state
51
Autonomics in DB2 LUW
Workload Management (Version 9.5 onwards)
52
Workload ManagementFeature Overview (v9.5 onwards)• Allows classifying and grouping user requests,
database activity and DB2 resources and thresholds / limits • Allows granular management of system load,
system resources and concurrency for better SLA • Allows better response to load spikes and runaway queries• Allows batch processing and OLTP to co-exist• SQL based access to historical and real-time activity
and workload• Activity monitors on service class, workload, etc
(similar to event monitors)• Ability to profile and capture workload
53
Workload ManagementNew Concepts• Service Class
User-definable logical execution environment that has allocated resources(CPU priority and prefetch I/O priority) and thresholdsAll DB2 activity (work) carried out within a service class2-tier hierarchy – service class and subclassDefault service classes present in a new database
• WorkloadWorkload = all “database activity” from user sessions, database maintenance activities, etc.Includes QUERY (SELECT or XQuery) , DDL, DML, LOAD, CALLWorkload can be grouped on various database connection attributesWorkload assigned to specific service class (i.e. resources and thresholds)
• Work class & Work Class SetWork class = individual set of database activity – e.g DDL, DMLWork class set = collection of one or more work classes
54
Workload Management
DBA
SELECT ….
FROM Account_Header
WHERE …….
INSERT INTO Contact(…..)
VALUES (…..)
LOAD FROM Product_Data.CSV OF DEL INSERT INTO …..
Finance Manager
Sales Manager
Different users connecting to database and generating database requests & activities
DB2 LUW
55
Workload Management
DBA
ERP_APP_WORKLOAD
CRM_APP_WORKLOAD
SELECT ….
FROM Account_Header
WHERE …….
INSERT INTO Contact(…..)
VALUES (…..)
LOAD FROM Product_Data.CSV OF DEL INSERT INTO …..
Finance Manager
Sales Manager
DBA_WORKLOAD
Based on (certain) connection and user attributes, the work request is considered part of a specific workload
DB2 LUW
56
Workload Management
DBA
ERP_APP_WORKLOAD
CRM_APP_WORKLOAD
SELECT ….
FROM Account_Header
WHERE …….
INSERT INTO Contact(…..)
VALUES (…..)
LOAD FROM Product_Data.CSV OF DEL INSERT INTO …..
Finance Manager
Sales Manager
DBA_WORKLOAD
ERP_SERVICE_CLASS
CRM_SERVICE_CLASS
UTIL_SERVICE_CLASS
Each workload is assigned to a DB2 Service Class –which has associated DB2 resources and thresholds
DB2 LUW
57
Workload ManagementSQL - CLP, Views, Functions and Procedures - 1• CLP Commands
CREATE / ALTER / DROP WORKLOADCREATE / ALTER / DROP SERVICE CLASSCREATE / ALTER / DROP WORK CLASS SETCREATE / ALTER / DROP THRESHOLDCREATE / ALTER / DROP HISTOGRAM TEMPLATEGRANT USAGE ON WORKLOAD …. TO …..SET WORKLOAD TO ….
• ViewsWORKACTIONSWORKACTIONSETSWORKCLASSESWORKCLASSSETSWORKLOADAUTHWORKLOADCONNATTRWORKLOADS
58
Workload ManagementSQL - CLP, Views, Functions and Procedures - 2• Table Functions
WLM_GET_ACTIVITY_DETAILSWLM_GET_QUEUE_STATSWLM_GET_SERVICE_CLASS_AGENTSWLM_GET_SERVICE_CLASS_WORKLOAD_OCCURRENCESWLM_GET_SERVICE_SUBCLASS_STATSWLM_GET_SERVICE_SUPERCLASS_STATSWLM_GET_WORKLOAD_OCCURRENCE_ACTIVITIESWLM_GET_WORKLOAD_STATSWLM_GET_WORK_ACTION_SET_STATS
• ProceduresWLM_CANCEL_ACTIVITYWLM_CAPTURE_ACTIVITY_IN_PROGRESSWLM_COLLECT_STATS
59
DB2 Autonomics Best Practices
60
DB2 Autonomics – Best PracticesDatabase Setup and Configuration• Avoid micro-managing tablespace containers –
instead, create pools of storage for different content typesStorage pool for data (tables and indexes)Use multiple storage paths to enable I/O parallelism (even with single data storage pool)Storage pool for database directory (optional – can be part of storage pool for data)Storage pool for logs (optional – can be part of storage pool for data)Storage pool for archived logsStorage pool for backup
• Avoid manual configuration - use autoconfigure and STMM• Use / configure DAS – DB2 Administration Server• Setup contacts and contact groups• Enable health monitoring • Disable specific health indicators (KHI) if using STMM• Minor annoyances – tablespace state during backup and
monitor heap size warning at first database activation / startup
61
DB2 Autonomics – Best PracticesDatabase Monitoring and Maintenance• Enable following automatic maintenance parameters
Automatic runstatsAutomatic statement statisticsAutomatic statistics profiling (only non-partitioned & non intra-parallel enabled systems)Automatic profile updates (only non-partitioned & non intra-parallel enabled systems)For profiling and statement statistics create query feedback warehouse –call SYSINSTALLOBJECTS ('ASP', 'C', '', '')ASP = Automatic Stats Profiling
• Collect system state, configuration and workload baseline dataCreate separate schema / database for the dataCollect data on a periodic basis (e.g. once a day or every 4 hours)Use snapshot functions to collect “state” informationUse DB2 catalog views to collect object information (e.g. table and index sizes)Use WLM (Workload Manager) views to collect workload dataIn health monitor action scripts collect configuration and activity snapshots when triggeredIn health monitor action scripts collect recommendations when triggered
• Consider integrating DB2 health monitor with enterprise monitoring
62
DB2 Autonomics – ReferencesInformation Sources• DB2 Online and PDF Documentation
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsphttp://publib.boulder.ibm.com/infocenter/db2luw/v9/index.jsphttp://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp
• IBM Developerworks Articleshttp://www-128.ibm.com/developerworks/db2/zones/dba/
• IBM Chat-with-the-Labs Presentationshttp://www-306.ibm.com/software/data/db2/9/labchats.html
• IBM Redbookshttp://www.redbooks.ibm.com/
• IBM Research Publicationshttp://www.research.ibm.com/
• DB2 Newsgroup (available via many providers – e.g.)groups.google.com
63
Questions?
Session D12
DB2 Autonomics -Autopilot for the DBA
64
Jayesh ThakrarIntersect Solutions, LLC
Session D12
DB2 Autonomics -Autopilot for the DBA
Contact Information