db2 accounting reporting
TRANSCRIPT
Understanding DB2 for z/OS Accounting Information
John Campbell & Florence DuboisIBM DB2 for z/OS Development
© 2012 IBM Corporation
[email protected]@uk.ibm.com
Acknowledgements
• Many thanks to all people who contributed to this presentation, in particular:
– Namik Hrle
– Akira Shibamiya
– John Tobler
– Miao Zheng
© 2012 IBM Corporation
Agenda
• DB2 traces: sources of key performance indicators
• Where is a DB2 transaction spending its time?
• CPU vs. suspension time
• Most common reasons for large suspension times
• SQL / Locking / BP / GBP accounting counters
• Tuning hints and tips as we go along
© 2012 IBM Corporation
•
DB2 Accounting vs. Statistics Data
• DB2 statistics trace records
– Contain information about the activity of the entire DB2 subsystem (including the DB2 system address spaces, such as db2MSTR, db2DBM1, db2IRLM, db2DIST
• DB2 accounting trace records
– Contain information about the activity performed by an individual DB2 thread or ‘transaction’ (or x threads in case of roll-up accounting)
© 2012 IBM Corporation
(or x threads in case of roll-up accounting)
– The focus of this presentations is here
Synchronous/ Asynchronous Agents
• Main synchronous agent
– Used to execute the SQL statement
– Usually a TCB that originates in the user’s address space
– In case of DDF, it is an (independent enclave) SRB instead of a TCB, which originates in the DDF address space
– In case of parallelism, the TCB (SRB for DDF) is helped by synchronous SRBs
• Asynchronous agent
© 2012 IBM Corporation
• Asynchronous agent
– Normally performs system or ancillary functions
– Usually an SRB, but it can also be a TCB
– The asynchronous CPU load is typically 10% of the synchronous load in most systems
• Can be higher in DDF and/or data sharing
• More important to see if the ratio changes than looking at absolute numbers
• Can indicate a change in workload -Access path change - Maintenance causing regression
TSO attachCall attachUtilities attach (*)
RRS attach (incl. JCC T2)IMS attachCICS attach
Use TCB
Synchronous Agent
Different DB2 Attachments
• Different attachments used by DB2
© 2012 IBM Corporation
CICS attach
DDF “attach”
All use XMEM services
Uses SRB
(*) Except with zIIP offload during IX build phase and RUNSTATS (V10)
Synchronous Agent
TCB
SQL
How Work Enters DB2 – Allied Thread
© 2012 IBM Corporation
USER (allied) DBM1 MSTR IRLM
SRB
SQL
How Work Enters DB2 – Distributed Work
© 2012 IBM Corporation
DIST (distributed)
DBM1 MSTR IRLM
TCBAsync I/O Physical
loggerDeadlock detection
SQL
Asynchronous ’Agents’ in DB2
© 2012 IBM Corporation
USER (allied)
DBM1 MSTR IRLM
Systemcheckpoint Lock
resumeOpen/close
DIST (distributed)
SRB
SQL
Auth
Synchronous/ Asynchronous Agents CPU times
• The synchronous CPU load is recorded in accounting
• The asynchronous CPU load is recorded in statistics
– If a thread has to wait for the async. agent to complete its work, this elapsed time is typically recorded in an accounting class 3 wait time counter
• Important note
© 2012 IBM Corporation
• Important note
– Since DDF work doesn’t have an agent outside the DB2 address spaces to charge for the work, the CPU time used by synchronous agents for DDF work also shows up in DB2 statistics (as DDF enclave SRB time). So be aware of double accounting!
CPU TIMES (from Statistics ) v10 TCB TIME PREEMPT SRB NONPREEMPT SRB TOTAL TIME PREEMPT IIP SRB
------------------------------- -------------- ------ --------- --------------- --------------- ------------ ---
SYSTEM SERVICES ADDRESS SPACE 8:05.013248 21:20.445149 12.782106 29:38.240503 N/A
DATABASE SERVICES ADDRESS SPACE 21:51.008289 13:35.088803 16.280806 35:42.377897 3 :19:20.488131
IRLM 0.320224 0.000000 22.664271 22.984495 N/A
DDF ADDRESS SPACE 2:12.462002 2:50:01.556367 2:15.574735 2:54:29.593103 2:37:50.897412
TOTAL 32:08.803762 3 :24:57.090319 3:07.301917 4:00:13.195998 5 :57:11.385543
TCBAsync I/O Physical
loggerDeadlock detection
(From Accounting )
Sync and Async Work in DB2 Stats and Accounting
© 2012 IBM Corporation
USER (allied)
DBM1 MSTR IRLM
logger
Systemcheckpoint
detection
Lock resumeOpen/
close
DIST
SRB
SQL
SQL
Auth
AVERAGE APPL(CL.1) DB2 (CL.2)
------------ ---------- ----------
ELAPSED TIME 0.225604 0.099221
NONNESTED 0.220121 0.097390
STORED PROC 0.005483 0.001831
CP CPU TIME 0.002241 0.002066
AGENT 0.002241 0.002066
NONNESTED 0.001787 0.001699
STORED PRC 0.000454 0.000366
IIP CPU TIME 0.002082 0.001987
STORED PROC N/A N/A
How CPU Time is Reported (Acctg vs. Stats)
Application DBM1 MSTR IRLM
Accounting Statistics
TSQL processing Scans
Synchronous I/O Lock requests
Open, Close
Extend
ArchivingError checkingAllocate archive dsn
© 2012 IBM Corporation
TCB
SRB
Synchronous I/O
Global lock requests*
Buffer updates
Lock requests
Logical logging
GBP reads*
The same as in the TCB case, but only in enclave pre-emptible SRB mode.
Reported in TCB instrumentation.
Extend
Preformat
Asynchronous I/OMemory managementReal-time StatisticsCastout*P-lock negotiation*Prefetch in CF*SYSLGRNX updates*GBP checkpoints*
BSDS processing
Physical loggingCheckpointsBackoutsDeallocationUpdate commit
GBP writes at commit*Global unlock at commit*
Error checking
Management
Deadlock detection
Lock resume
Global lock conflictresolution*
(*) data sharing specific
Allocate archive dsn
Response Time Factors
presentation network application logic network database processing
© 2012 IBM Corporation
General transaction flow
Response Time Scope
TRANSACTION
PROGRAM A
STATEMENT 1STATEMENT 2
.
.
PLAN
DBRM A (*)
STATEMENT 1STATEMENT 2
.
.
© 2012 IBM Corporation
PROGRAM B
.
STATEMENT 1STATEMENT 2
.
.
PACKAGE B
.
STATEMENT 1STATEMENT 2
.
.
Application space DB2 (*) No longer possible in V10
Response Time: DB2 View
thread allocation
1st SQL
2nd SQL
DB2 can collect data during the entire thread life: activity time
It includes time spent in:
• Presentation layer (GUI)• Application logic• Network
© 2012 IBM Corporation
nth SQL
thread deallocation
Out of DB2 In DB2
For all practical purposes, this is the DB2 Accounting Class 1 elapsed time
• Network• DB2 processing
activitytime
DB2 Accounting Data
• DB2 Accounting trace
– DB2 command -STA TRA(ACCTG) …. Or at DB2 start-up SMFACCT DSNZPARM
– Typically written to SMF (as SMF type 101 record)
• IFCIDs
– IFCID 3 – plan level info
– IFCID 239 – package level info
• Since V8, IFCID 239 is written as a separate record, but cut at the same time
© 2012 IBM Corporation
• Since V8, IFCID 239 is written as a separate record, but cut at the same time
• Accounting Trace classes
– Class 1 – Total time (elapsed and CPU) – IFCID 3
– Class 2 – Time in DB2 (elapsed and CPU) – adds info to IFCID3
– Class 3 – Suspension time in DB2 – adds info to IFCID 3
– Class 7 – Package/DBRM info (similar to class 2)
– Class 8 – Package/DBRM info (similar to class 3)
– Class 10 – SQL/Locking/BP info at package/DBRM level
A DB2 Accounting Data is Written at:
• Thread deallocation (including abends)
• (re)Signon in case of thread reuse by RRS, IMS, CICS
– (many options in CICS)
• Commit for RRS threads using accounting-interval COMMIT on signon / authsignon / context signon
• Commit when CMTSTAT=INACTIVE and provided connection can go inactive
– Things that prevent a connection from going inactive:
© 2012 IBM Corporation
– Things that prevent a connection from going inactive:
• Touched a package that uses KEEPDYNAMIC(YES)
• Active DGTT (not explicitly or implicitly dropped)
• Open cursor with hold
• Held LOB locator
• If only reason for not going inactive is the use of KEEPDYNAMIC(YES), still cut accounting record (and reset enclave)
– To please our SAP friends
DB2 Accounting Data is Written at: -2
• When using rollup accounting for DRDA and RRS
– Every ACCUMACC times you would have normally written an accounting record
– If gets stale (not enough occurrences of the rollup “group”)
– If rollup blocks are using too much storage
– QWACRINV indicates reason why acctg record was written
• Rollup accounting blessing and curse
– Reduces the number of accounting records
© 2012 IBM Corporation
– Reduces the number of accounting records
– Can do rollup based on different “group by” criteria
– Some accounting information not available in rollup records(some enhancements in V10)
– Makes problem shooting more complicated (see later)
When is an accounting record written?
• When the thread de-allocates – normal case
1st
SQLPgm end
1st
SQLPgm end
PgmStart
PgmStart
© 2012 IBM Corporation
Time
Thread create
StatsAcctg
Thread de-allocate
Thread create
Thread de-allocate
StatsAcctg
When is an accounting record written? - 2
• Commit does not create an accounting record, except
– For inactive connections in DRDA (coming up)
– RRS using accounting-interval commit (and no open held cursors)
1st
SQL CommitPgm end
PgmStart CommitCommit
© 2012 IBM Corporation
Time
Thread create
Thread deallocate
StatsAcctg
When is an accounting record written? - 3
• When a thread is reused
– Applies to IMS and CICS
– (There are additional options in CICS – see CICS Transaction Server - DB2 Guide)
1st
SQL CommitPgm end
PgmStart
New tran1st SQL Commit
New tran1st SQL
© 2012 IBM Corporation
Time
Create Thread + Signon
Thread de-allocate
Signon Signon
StatsAcctg StatsAcctg StatsAcctg
Accounting pitfall
• When a thread is reused but there is a long time between transaction end and the next transaction doing signon
1st
SQL CommitPgm end
PgmStart
New tran1st SQLRegion idle
CLASS 1 time reported
© 2012 IBM Corporation
Time
Create Thread + Signon
Thread de-allocate
Signon
CLASS 1 time reported
StatsAcctg StatsAcctg
When is an accounting record written? - 4
• When the connection goes inactive
– Distributed thread
– CMTSTAT=INACTIVE
1st
SQL Commit
Pgm endPgm
Start
New tran1st
SQL Commit
New tran1st
SQL
© 2012 IBM Corporation
Time
AcquireDBAT
Conn.ends
Conn.inactive
Conn.inactive
AcquireDBAT
AcquireDBAT
StatsAcctg StatsAcctg StatsAcctg
When is an accounting record written? - 5
• When using rollup accounting (DDF and RRS)
– E.g. ACCUMACC=3, ACCUMUID=2 (‘group by’ transaction name)
– Values will be the totals of all 3 transactions
1st
SQL Commit
Pgm endPgm
Start
New tran1st
SQL Commit
New tran1st
SQLCommit
© 2012 IBM Corporation
Time
AcquireDBAT
Conn.ends
Conn.inactive
Conn.inactive
AcquireDBAT
AcquireDBAT
Conn.inactive
AcquireDBAT
AcctgAcctg AcctgStatsAcctg
Accounting Class 1 Data
thread allocation
1st SQL
2nd SQL
For local applications, DB2 provides an accumulative timer that includes both CPU spent in application and CPU spent in DB2.
activity
Class 1elapsedand CPU
© 2012 IBM Corporation
thread deallocation
Out of DB2 In DB2
DB2.
Activity time is very close to Class 1 elapsed time.
For local applications, Class 1 (elapsed – CPU) time can point to application inefficiencies.
activitytime
Accounting Class 2 Data
thread allocation
1st SQL
2nd SQL
Class 2 timers are accumulative
Time spent out of DB2Class 1 elapsed – Class 2 elapsed
Class 1elapsedand CPU
Class 2elapsedand CPU
© 2012 IBM Corporation
2nd SQL
thread deallocation
Time spent waiting in DB2Class 2 elapsed – Class 2 CPU
Class 7 is used for package/DBRM scope reporting. It is equivalent to Class 2
Out of DB2 In DB2
Accounting Class 3 Data
thread allocation
1st SQL
2nd SQL
Class 8 is used for package level reporting. It is equivalent to Class
Class 1elapsedand CPU
Class 2elapsedand CPU
Class 3Suspensions
© 2012 IBM Corporation
2 SQL
thread deallocation
It is equivalent to Class 3.
Out of DB2 In DB2
in Appl in DB2 IRLM
Class 1 Class 2 (in DB2) 1st SQL..
SQL ..
SQL ..
...Wait for lock
(Creating Thread)
DB2 Times Terminology - Accounting Class 1,2,3
© 2012 IBM Corporation
AgentAgent, non-nested ET
(Terminating Thread)
...Wait for I/O
Agent, non-nested CPU
Class 3(susp time)
...Commit
DB2 Times Terminology –Class 1&2 Elapsed Time with Nested Activity
in Appl in DB2 in UDFElapsed Time
Class 1 Class 2 (in DB2)
Trigger1
1st SQL ..
SQL ..
UDF
SQL ..
...UDF
(Creating Thread)
© 2012 IBM Corporation
AgentTrigger
UDF
Agent, non nestedTrigger
UDF
Trigger2 SQL ..
(Terminating Thread)
DB2 Accounting Report Class 1-2-3 SectionsTIMES/EVENTS APPL(CL.1) DB2 (CL.2) IFI (CL.5) CLASS 3 SUSPENSIONS ELAPSED TIME EVENTS------------ ---------- ---------- ---------- --------- ----------- ------------ --------ELAPSED TIME 1:42.494131:42.49413 0.111129 N/P LOCK/LATCH(DB2+IRLM) 0 .000000 0
NONNESTED 0.208342 0.0571680.057168 N/A SYNCHRON. I/O 0.000000 0STORED PROC 0.000000 0.000000 N/A DATABASE I/O 0.000000 0UDF 1:42.238001:42.23800 0.0061650.006165 N/A LOG WRITE I/O 0.000000 0TRIGGER 0.0477970.047797 0.0477970.047797 N/A OTHER READ I/O 0.000000 0
OTHER WRTE I/O 0.000000 0CP CPU TIME 28.858601 0.029797 N/P SER.TASK SWTCH 0.020768 1
AGENT 28.858601 0.029797 N/A UPDATE COMMIT 0.020768 1NONNESTED 0.016157 0.005022 N/P OPEN/CLOSE 0.000000 0STORED PRC 0.000000 0.000000 N/A SYSLGRNG REC 0.000000 0UDF 28.822345 0.004677 N/A EXT/DEL/DEF 0.000000 0TRIGGER 0.020098 0.020098 N/A OTHER SERVICE 0.000000 0
PAR.TASKS 0.000000 0.000000 N/A ARC.LOG(QUIES) 0.000000 0LOG READ 0.000000 0
SECP CPU 0.000000 N/A N/A DRAIN LOCK 0.000000 0
© 2012 IBM Corporation
SECP CPU 0.000000 N/A N/A DRAIN LOCK 0.000000 0CLAIM RELEASE 0.000000 0
SE CPU TIME 0.000000 0.000000 N/A PAGE LATCH 0.000000 0NONNESTED 0.000000 0.000000 N/A NOTIFY MSGS 0.000000 0STORED PROC 0.000000 0.000000 N/A GLOBAL CONTENTION 0.000000 0UDF 0.000000 0.000000 N/A COMMIT PH1 WRITE I/O 0.000000 0TRIGGER 0.000000 0.000000 N/A ASYNCH CF REQUESTS 0.000000 0
TCP/IP LOB 0.000000 0PAR.TASKS 0.000000 0.000000 N/A TOTAL CLASS 3 0.020768 1
SUSPEND TIME 27.525561 0.020768 N/A AGENT N/A 0.020768 N/A PAR.TASKS N/A 0.000000 N/A STORED PROC 0.000000 N/A N/A UDF 27.52556127.525561 N/A N/A
NOT ACCOUNT. N/A 0.060564 N/A DB2 ENT/EXIT N/A 104 N/A EN/EX-STPROC N/A 0 N/A EN/EX-UDF N/A 0 N/A DCAPT.DESCR. N/A N/A N/P LOG EXTRACT. N/A N/A N/P
DB2 Accounting Report Class 1-2 Sections
TIMES/EVENTS APPL(CL.1) DB2 (CL.2) TIMES/EVENTS APPL(CL.1) DB2 (CL.2) ------------ ---------- ---------- ------------ ------- --- ----------ELAPSED TIME 1:42.494131:42.49413 0.111129 ... ... ...
NONNESTED 0.208342 0.0571680.057168STORED PROC 0.000000 0.000000 SUSPEND TIME 27.525561 0.020768 UDF 1:42.238001:42.23800 0.0061650.006165 AGENT N/A 0.020768 TRIGGER 0.0477970.047797 0.0477970.047797 PAR.TASKS N/A 0.000000
STORED PROC 0.000000 N/A CP CPU TIME 28.858601 0.029797 UDF 27.52556127.525561 N/A
AGENT 28.858601 0.029797
© 2012 IBM Corporation
AGENT 28.858601 0.029797 NONNESTED 0.016157 0.005022 NOT ACCOUNT. N/A 0.060564 STORED PRC 0.000000 0.000000 UDF 28.822345 0.004677 TRIGGER 0.020098 0.020098
PAR.TASKS 0.000000 0.000000 ... ... ...
Reporting of Nested Activity
• In DB2 accounting data, you can distinguish between
– Non-nested activity
– Nested activity (cannot distinguish between the level of nesting – that requires performance traces)
• Trigger
• Stored Procedure (SP)
• User Defined Function (UDF)
•
© 2012 IBM Corporation
• For SP and UDF that run in a WLM-managed AS, DB2 can distinguish between Class 1 (program logic - COBOL code) and class 2 (executing SQL statements)
• For triggers, SQL PL SP and non-fenced UDFs, CL1=CL2
Reporting of Nested Activity - 2
• Only actual work done by SP is reported as nested activity at the plan level
– That is work done by the CALL sp
– When using result set cursors (WITH RETURN), typically only the OPEN cursor is done by the SP and is reported as nested activity
– The FETCHing is done by the calling application and is reported as non-nested time at the plan level
© 2012 IBM Corporation
• Package level data shows the FETCHes (as well as CPU/ET time incurred) as if they were done by the SP package (that performed the OPEN cursor)
in Appl in DB2 In SPElapsed Time
Class 1 Class 2 (in DB2) 1st SQL ..
CALL sp SP
OPEN curs WITH RETURN
SQL ..
(Creating Thread)
Nested Activity for WITH RETURN processing
SP end
© 2012 IBM Corporation
Agent
SP
Agent, non nested
SP
(TerminatingThread)
SP endFETCH
FETCH
FETCH
DB2 Accounting Class 1-2 (Plan Level) Sections
TIMES/EVENTS APPL(CL.1) DB2 (CL.2)------------ ---------- ----------ELAPSED TIME 0.941479 0.608886
NONNESTED 0.731353 0.4021540.402154STORED PROC 0.2101260.210126 0.2067320.206732UDF 0.000000 0.000000TRIGGER 0.000000 0.000000
CP CPU TIME 0.249935 0.192192AGENT 0.249935 0.192192
NONNESTED 0.248412 0.191915
SQL DML TOTAL-------- --------SELECT 0INSERT 0
ROWS 0UPDATE 0
ROWS 0MERGE 0
© 2012 IBM Corporation
NONNESTED 0.248412 0.191915STORED PRC 0.001523 0.000277UDF 0.000000 0.000000TRIGGER 0.000000 0.000000
PAR.TASKS 0.000000 0.000000... ... ...
SUSPEND TIME 0.000323 0.000000AGENT N/A 0.000000PAR.TASKS N/A 0.000000STORED PROC 0.000323 N/AUDF 0.000000 N/A
NOT ACCOUNT. N/A 0.416693 ... ... ...
MERGE 0DELETE 0
ROWS 0
DESCRIBE 0DESC.TBL 0PREPARE 0OPEN 1FETCH 5204
ROWS 5203CLOSE 1
DML-ALL 5206
DB2 Accounting Report Class 7-8-10 (Package Level)
• Package level data is being reported as if the FETCHes are executed by the SP
MRSBMCBM (main) TOTAL MRSBMS (SP) TOTAL------------------ -------- ------------------ ------- -SELECT 0 SELECT 0INSERT 0 INSERT 0UPDATE 0 UPDATE 0DELETE 0 DELETE 0
© 2012 IBM Corporation
DESCRIBE 0 DESCRIBE 0PREPARE 0 PREPARE 0OPEN 0 OPEN 1FETCH 0 FETCH 5204CLOSE 0 CLOSE 1
LOCK TABLE 0 LOCK TABLE 0CALL 1 CALL 0
DB2 Accounting Report Class 7-8 (Package Level)MRSBMCBM (main) VALUE MRSBMCBM TIMES ------------------ ------------------ --------------- --- ------------TYPE PACKAGE ELAPSED T IME - CL7 0.151778 LOCATION STLEC1 CP CPU TI ME 0.083859 COLLECTION ID BARTCOB AGENT 0.083859 PROGRAM NAME MRSBMCBM PAR.TAS KS 0.000000 CONSISTENCY TOKEN 1906C50F153EE340 SE CPU TI ME 0.000000 ACTIVITY TYPE NONNESTED SUSPENSION-CL8 0.000000 ACTIVITY NAME 'BLANK' AGENT 0.000000 SCHEMA NAME 'BLANK' PAR.TAS KS 0.000000
MRSBMS (SP) VALUE MRSBMS TIMES ------------------ ------------------ ------------------ ------------
© 2012 IBM Corporation
------------------ ------------------ ------------------ ------------TYPE PACKAGE ELAPSED T IME - CL7 0.4960330.496033LOCATION STLEC1 CP CPU TI ME 0.139170 COLLECTION ID BARTCOB AGENT 0.139170 PROGRAM NAME MRSBMS PAR.TAS KS 0.000000 CONSISTENCY TOKEN 1906C50417D039C3 SE CPU TI ME 0.000000 ACTIVITY TYPE STORED PROC SUSPENSION-CL8 0.000000 ACTIVITY NAME MRSBMS AGENT 0.000000 SCHEMA NAME SYSPROC PAR.TAS KS 0.000000 SUCC AUTH CHECK NO NOT ACCOU NTED 0.356863
Greater than Nested time at plan level - includes time for FETCH operations
Analyzing Acctg Data: Top-Down Analysis
• Accounting report (not trace) by connection type most useful for initial analysis
• For OMPE users:
ACCOUNTING REPORT LAYOUT(LONG) ORDER(CONNTYPE)
EXCLUDE(PACKAGE(*))
© 2012 IBM Corporation
• To group by thread connection type such as TSO, CICS, DB2CALL, RRS, IMS, DRDA, UTILITY, etc. for the period of interest
• Also combine with STATISTICS REPORT LAYOUT(LONG) for the corresponding period
Analyzing Thread Activity Time
Thread activity time = Class 1 elapsed
Elapsed time spent out of DB2 = Class 1 elapsed – Class 2 elapsed(in application, network, idle..)
Elapsed time spent in DB2 = Class 2 elapsed
Processing time = Class 2 CPU (GCP and zIIP)
© 2012 IBM Corporation
Waiting time = Class 2 elapsed – Class 2 CPU
Suspended time = Class 3 suspension time
Not accounted time = Waiting time – Suspended time
What if Activity Time is Split as …
• Application logic inefficiency
– Often combined with Class 2 CPU << Class 1 CPU
• Network problems
• User think (coffee?) time
• Class 2 not active all the time
In DB2
Out of DB2
© 2012 IBM Corporation
• Class 2 not active all the time
• Potential pitfalls
– Thread reuse and long time between consecutive transactions (allied threads)
– Long running DDF threads when CMTSTAT=ACTIVE
What if Activity Time is Split as …
• What’s the actual number (not just the %)
• Is this a DB2 intensive process anyway?
•
In DB2
Out of DB2
© 2012 IBM Corporation
• Need to analyze time distribution in DB2 : Where is the time really spent inside DB2?
Time in DB2 (Class 2 Time)
• Burning CPU cycles (class 2 cpu time)
– Non-nested or nested
– On a general CP or zIIP
• Suspended in DB2 for a known event (class 3 suspension time)
– Many different types of suspensions (21 in V10)
• In DB2 but don’t know what it is we are doing
© 2012 IBM Corporation
• In DB2 but don’t know what it is we are doing
– Mostly things that happen outside of DB2’s control, such as
• Lack of CPU or paging
– Should be a small %
Accounting CPU time reporting
• CP CPU time
– CPU time used on general purpose engine
• SE CPU time
– Any CPU used on a specialty engine (zIIP or zAAP)
– Not included in GP CPU time
• Sub-capacity machines
– SE CPU time is reported as if had run on general CP
© 2012 IBM Corporation
– SE CPU time is reported as if had run on general CP
– Can result in total CPU= (CP + SE CPU) > ET for CPU intensive work
• SECP CPU
– Could have run on specialty engine but didn’t
– No longer reported in V10
Accounting CPU time reportingCONNTYPE: DRDA
AVERAGE APPL(CL.1) DB2 (CL.2)------------ ---------- ----------ELAPSED TIME 2.840103 1.913457
NONNESTED 2.839901 1.913339STORED PROC 0.000202 0.000118UDF 0.000000 0.000000TRIGGER 0.000000 0.000000
CP CPU TIME 0.260398 0.251655AGENT 0.260398 0.251655
NONNESTED 0.260326 0.251584
© 2012 IBM Corporation
NONNESTED 0.260326 0.251584STORED PRC 0.000072 0.000071UDF 0.000000 0.000000TRIGGER 0.000000 0.000000
PAR.TASKS 0.000000 0.000000
SECP CPU 0.000726 N/A
SE CPU TIME 0.261583 0.246227NONNESTED 0.261583 0.246227STORED PROC 0.000000 0.000000UDF 0.000000 0.000000TRIGGER 0.000000 0.000000
PAR.TASKS 0.000000 0.000000
What if In DB2 Time is Split as …
• Turn off expensive traces
• Inefficient access paths
− Run EXPLAIN
− Check transaction profile− #SQL stmts− #locks
ProcessingWaiting
© 2012 IBM Corporation
• What is the largest contributor
− Class 3 and 8 analysis
− #locks− #getpages
Processing
Waiting
Time in DB2 (Class 2 time)
• Burning CPU cycles (class 2 cpu time)
– Non-nested or nested
– On a general CP or zIIP
• Suspended in DB2 for a known event (class 3 suspension time)
– Many different types of suspensions (21 in V10)
• In DB2 but don’t know what it is we are doing
© 2012 IBM Corporation
• In DB2 but don’t know what it is we are doing
– Mostly things that happen outside of DB2’s control, like
– Lack of CPU or paging
– Should be a small %
Accounting Class 3 Data
• Class 3 accounting strongly recommended
• Provides information about “known” DB2 suspensions. For suspensions, DB2 reports:
– The number of suspensions
– The total time it was suspended for those
• Always look at both #suspensions and suspension time
• Double accounting is avoided
© 2012 IBM Corporation
• Double accounting is avoided
• Negligible overhead except when high internal DB2 latch contention, e.g. over 10,000/sec
Sync.EU Switch
Locking
I/O C
lass
3 S
uspe
nsio
n T
ypes
Synchronous read/write & log write ����
Synchronous read/write ���� ���� ���� ����
Log write ���� ���� ���� ����
Other agents’ read ���� ���� ���� ���� ����
Other agents’ write ���� ���� ���� ���� ����
Force-at-commit database writes (LOG NO LOBs only) ���� ���� ���� ����
TCP/IP LOB XML ���� ����
IRLM lock/latch & DB2 internal latch ���� ���� ���� ����
IRLM lock/latch ����
DB2 internal latch ����
Page latch ���� ���� ���� ���� ����
Drain lock ���� ���� ���� ���� ����
Claim release ���� ���� ���� ���� ����
Synchronous Execution Unit Switch total ����
Open/Close ���� ���� ���� ����
Define/Extend/Delete ���� ���� ���� ����
V5V5 V6V6 V7,8V7,8 V9V9 V10V10
© 2012 IBM Corporation
Archiving
Scheduling
Data Sharing
Cla
ss 3
Sus
pens
ion
Typ
es
SYSLGRNX recording ���� ���� ���� ����
Commit ���� ���� ���� ����
Other services ���� ���� ���� ����
Archive Log command ���� ���� ���� ���� ����
[Archive] log read ���� ���� ���� ���� ����
Stored procedures ���� ���� ���� ���� ����
UDFs ���� ���� ���� ����
Global locks total ���� ����
Parent L-locks ���� ���� ����
Child L-locks ���� ���� ����
Other L-locks ���� ���� ����
Page set/Partition P-locks ���� ���� ����
Page P-locks ���� ���� ����
Other P-locks ���� ���� ����
Sending Notify messages ���� ���� ���� ���� ����
Asynchronous coupling facility request completion ���� ���� ���� ����
Lock/Latch Suspensions
• Thread can be suspended for:
– Lock contention
• includes preformatting (X’09’)
– IRLM latch contention
– DB2 internal latch contention
• Page latch contentions are reported in a separate timer
•wa
UPDATE
applicationaddress space
DB2address space
lock suspend
lock unavailable
© 2012 IBM Corporation
• DB2 reports:
– Accumulated time thread was suspended
– Total number of suspensions
• Time/events is an average wait due to a lock/latch conflict. The number of true lock suspensions (from the Locking block) is often a better divisor.
ait
lock resume
buffer update
Large Lock/Latch Suspensions – Large number?
Number of IRLM lock suspensions high?− Application problems− Incompatible workload mix− Page/row level locking− Inefficient preformatting
�Use PAV�Check ICF catalog performance
Number of DB2 internal latch suspensions high?− LC06: Index tree P-lock latch contention caused most
likely by splits of GBP-dependent index pages− LC11: Generating Identity Column− LC14: BP LRU chain – Isolate small highly accessed
tables into separate pools− LC19: Log – Speed up log I/O− LC24: EDM LRU or BM latch− LC25: EDM Pool hash chain
© 2012 IBM Corporation
Waits forlock/latch suspensions
In DB2 Time
Number of IRLM latch suspensions high?− If more than 10% of all IRLM requests− IRLM Trace on− Low IRLM dispatching priority− Frequent IRLM Query requests (e.g. -DIS DATABASE
LOCKS, or MODIFY irlmproc, STATUS)− Low deadlock detection cycle and many lock
suspensions
− LC25: EDM Pool hash chain− LC32 Storage Manager Pool Header
Large Lock/Latch Suspensions – High average time?
Time per suspension large? • Lock suspensions
– Non-committing transactions
– Concurrent DDL
• RELEASE(DEALLOCATE)
– For detail analysis, use performance trace classes 6 and 7
Waits forlock/latch suspensions
In DB2 Time
© 2012 IBM Corporation
classes 6 and 7
• DB2 latch suspensions
– Designed to be short duration
– Can be a symptom of another problem
– For detail analysis, use performance trace class 11
suspensions
Notes
• Lock/Latch wait = Lock wait + IRLM latch wait + Internal DB2 latch wait
– In the rare case of over 10,000 per second, disabling class 3 may significantly bring down class 1 and 2 cpu time
– Look at IRLM lock suspension and IRLM latch suspension counter to get a better idea which one of the 3 is likely to be the culprit
© 2012 IBM Corporation
which one of the 3 is likely to be the culprit
• DB2 10
– Separate out into IRLM lock/latch and DB2 latch
– More efficient DB2 latch resume logic
• Less chance to get resumed and suspended again without ever obtaining the latch in between
Synchronous I/O Suspensions Large?
Number of suspensions high− Excessive Getpages− Buffer pools contention− Disorganized data/indexes
SynchronousI/O Waits
In DB2 Time
© 2012 IBM Corporation
Time per suspension large − Make sure there are enough I/O resources− Use RMF to verify− For detailed analysis in DB2,
use performance class 4
Synchronous Database I/O Suspensions
• A common timer for:
– Synchronous database reads
– Synchronous database writes
• A separate timer for synchronous log writes
• Use BP statistics to find out how
getpage missUPDATE
applicationaddress space
DB2address space
read begin
© 2012 IBM Corporation
• Use BP statistics to find out how many of the suspensions should be attributed to synchronous reads vs. writes
• Avg I/O wait time in Acctg rpt =0.028539 sec/12.41 events = 0.002 sec / sync DB IO
wait
read end
buffer update
Tuning the # Sync I/O suspensions
• If many database I/O waits, check
– Excessive Getpages because of a bad access path
• Use Explain to tune access path
– Buffer pools contention
• Other applications ruining our hit ratio
• BP too small
• (V)DWQT setting
© 2012 IBM Corporation
•• Look at BP statistics
– Disorganized data/indexes
• Check statistics
– Minimize index splits
– Zero freespace for data for minimum read and write I/O in heavy Insert or APPEND YES
Tuning I/O Wait Time
• If high database I/O wait, check
– Enough I/O resources ? (Use RMF to verify)
– DASD contention
– High IOSQ time – use PAV
– Faster I/O devices (use SSD)
– Control Unit cache misses
– CPU contention
© 2012 IBM Corporation
– CPU contention
– Inappropriate I/O priority
• Use I/O striping where appropriate
– Less effective with newer I/O devices
– Still beneficial for non-saturated I/O configuration with infrequent commit and large amount of data to be read or written in one I/O, e.g. list prefetch, heavy write
Synchronous Log write I/O Suspensions
Force log buffers to diskCommit
applicationaddress space
DB2address space
• “Normally” only forced write to active log at commit time
• All log buffers that have not been written are written at commit
• Most common cases are listed on next foil
write begin
© 2012 IBM Corporation
wait
write begin
write end
No more log buffers to write
What triggers sync log write I/Os ..
• For transactions that use 2PC, Phase 1 of 2-phase commit
• Synchronous write (to disk or GBP)
– Log write ahead protocol
• Running out of log output buffers
• Identity columns/sequences that are using NO CACHE (or low cache value) or ORDER is used (in data sharing)
– Hidden SPRMNLGF zparm to avoid especially if identity columns are used
© 2012 IBM Corporation
– Hidden SPRMNLGF zparm to avoid especially if identity columns are used
• V10 up to 40% reduction in (dual) log I/O wait because of parallel log write I/O’s
What triggers sync log write I/Os ..
• Index split in data sharing
– Reduce index page splits by
• Non-zero freespace (FREEPAGE, PCTFREE) for random insert
• PCTFREE 0 for sequential insert
• Larger index page size in V9
– Trade-off with possibly higher index page latch contention and P-lock contention
• Asymmetric index page split in V9
•
© 2012 IBM Corporation
• P-lock negotiation
• IMMEDWRITE(YES) for GBP-dependent objects
• If GBP-dependent, when taking (a completely empty) index page off the IX chain
• If GBP-dependent, new (reuse after mass delete or brand-new) page used in segmented + UTS
– PK83735/PK94122 (V9) and V8 PM17813/17817 (V8) should address this
Wait for read i/o by another application agent or prefetch engine
Suspensions for Other Agents’ Reads
wa
getpage miss
FETCH
applicationaddress space
DB2address space
prefetch begin
Number of suspensions high?− Excessive GetPages
�Use Explain to tune access path− Buffer pools contention
�Look at BP statistics
In DB2 Time
Wait forotheragents’reads
What if …
© 2012 IBM Corporation
ait
first block of pagesFETCH
getpage hit
getpage hit
�Look at BP statistics− Disorganized data/indexes
�Check statistics
Time per suspension large?− Use RMF and, for detailed analysis
performance class 4 to check on: �DASD contention�Control Unit cache misses
�Check SEQCACHE=YES�Consider parallelism/partitioning�Inappropriate I/O priority
Suspensions for Other Agents’ Writes
UPDATE
applicationaddress space
DB2address space
Number of suspensions high?− Too frequent checkpoints
�Check DB statistics, LOGLOAD− Too low (vertical) deferred write
In DB2 Time
Wait forotheragents’ writesWhat if …
buffer update
UPDATE
© 2012 IBM Corporation
wait
Page being written
page written
− Too low (vertical) deferred write threshold for high re-reference access pattern�Check BP statistics
Time per suspension large?− DASD contention− CU cache misses− Inappropriate I/O priority
buffer update deferredwrite start
UPDATE
Notes
• Other write I/O wait = wait for write i/o by another application agent or write engine, may include some time waiting for log write-ahead
• When updating a buffer being written, first agent waits for OTHER WRITE I/O. All others wait for PAGE LATCH contention.
© 2012 IBM Corporation
Synchronous EU Switch Suspensions
w
SQL
applicationaddress space
DB2address space
switch to a TCB/SRB
any of these tasks
• ‘Update’ commits− Single-phase commit− Phase 2 of two-phase commit− Abort− Deallocation
• Dataspace Manager tasks
A separate counter for:
© 2012 IBM Corporation
wait
task completed
− Delete/Define/Extend data set− Reset data set− VSAM catalog access
• OPEN/CLOSE data set− Not include HSM RECALL of data
set
• SYSLGRNX recording− And down-level detection
• Other services tasks− DDF application requester
considerations
Large Suspensions for Synchronous EU Switches?
In DB2 Time
Waits forSynchronous
• Check log efficiency− DB statistics, RMF
• Check VSAM catalog performance
© 2012 IBM Corporation
SynchronousExecution UnitSwitches
• PCLOSET or PCLOSEN too low
• Check if DSMAX is too low
Service Task Wait - Commit Related Timers
• Main components of commit processing
– Synchronous log writes
– Cursors, locking, etc. clean-up
– Force-at-commit synchronous database writes, or GBP writes in data sharing
Class 3 Timer Read-only commit Single-phase commit Two-phase commit
© 2012 IBM Corporation
1 Included in wait for synchronous EU switch for 'update' commit.2 These are log writes that happen in phase 1 of two-phase commit.
Log writes that happen in phase 2 are included in wait for synchronous EU switch for 'update' commit.3 Applies to LOBs with LOG(NO) only. These database writes happen in phase 1 of two-phase commit.
Wait for log writes no no1 yes2
Wait for synchronous EUswitch for update commit
no yes yes
Wait for force-at-commitdatabase writes
no no1 yes3
Service Task Wait - 2
• Dataset [Extend]/Define/Delete
• Aka Dataspace manager work
– Extend for preformatting, takes 0.02 to 1sec each time depending on device type and alloc unit/size
– x'09' lock wait in async preformat since V7 so no longer shows up here
– Counting pieces of a segmented TS (at open time)
– Dataset Delete/Define when Stogroup is used
© 2012 IBM Corporation
– Dataset Delete/Define when Stogroup is used
– IFCID 258 contains DBID/OBID for Dataset Extend
– IFCID 92 contains Define Cluster statement for Define/Delete
Service Task Wait - 3
• Dataset Open/Close
– Up to 20 tasks for parallel Open in V8, 40 in V9
– Typically no class3 acctg wait for dataset close as it is asynchronous
– V10 Access DB Mode(Open) – parallel data set open
– PM19528 V9/10 introduces IFCID370-371
• Others
© 2012 IBM Corporation
• Others
– Time waiting for server in TCP/IP requestor accounting
– VSAM catalog update
– Parallel query cleanup
– And many others which are typically infrequent
66
Commonly Observed OTHER SERVICE TASKS
• Parallel query cleanup x’14’/x’77’ or x’84’
– Shows up in repeated execution of short-running queries in parallelism
– Use higher SPRMPTH ZPARM value to replace this (default=120(ms))
– Of course, for a long-running query, parallel query with degree ANY can result in many times reduction in elapsed time
• VSAM catalog update x’0A’/x’94’
© 2012 IBM Corporation
• VSAM catalog update x’0A’/x’94’
• DB2 for z/OS application requestor using TCP/IP waiting for server response x’1B’/x’8F’
67
How to find which service task invoked
• Performance trace of IFCID 46 thru 50
– IFCID 46 for service task switch
• 47 and 48 for Begin and End of SRB service task
• 49 and 50 for Begin and End of TCB service task
– Shows all nested service tasks
• Or use IFCID 170 and 171 for less output
© 2012 IBM Corporation
• Or use IFCID 170 and 171 for less output
– Shows one (outermost) service task for each event in class 3 accounting
Notes
• Correlation name of system agent (SYSOPR planname) in performance trace:
– eg. 021.OPNL for SYSLGRNX update in pageset open
• Chapter 9 of DB2 Diagnosis Guide and Reference contains a list of system agent correlation identifiers as well as a list of Resource Manager id's displayed in IFCID 46, 47, 49, and 170.
© 2012 IBM Corporation
Commonly observed service tasks
• Commonly observed service tasks and RMID/FC identifiers as shown in IFCID 46, 47, 49, and 170
– Update Commit x’03’/x’49’
– Dataset Open x’0A’/x’59’
– Dataset Close x’0A’/x’4F’
© 2012 IBM Corporation
– SYSLGRNX Update x’15’/x’44’
– Dataset Extend x’12’/x’65’
– Dataset Delete x’12’/x’6A’
– Dataset Define x’12’/x’68’
– Dataset Reset x’12’/x’6C’
– Dataset Count x’12’/x’88’
Notes
• FC = Function Code
• RMID = Resource Manager ID
• If a long time gap between IFCID 46 and 47 (begin SRB task) or 49 (begin TCB task), then it is possible that we are waiting for a service task to become available.
© 2012 IBM Corporation
Log Read Suspensions
w
ROLLBACK
applicationaddress space
DB2address space
read active or archive log
Wait forotheragents’ writes
data not in log buffer
What if … Wait forlog reads
Number of suspensions high?− Find out why the transactions rollback
(most rollbacks are done under EU-switch - those don’t show up here)
− Keep to a minimum
© 2012 IBM Corporation
wait
archive log
data retrieved
backout
− Keep to a minimum
Time per suspension high?− Increase the number and size of
active log data sets
• In V8 only archive log read wait was tracked• Starting in V9 active log prefetch read wait time also included• Vnext will also include sync. active log read
(currently NOT ACCOUNT time)
Archive Log Mode (Quiesce) Suspensions
UPDATE
applicationaddress space
DB2address space
Wait forotheragents’ writesWhat if …
Waits forquiesce
commit
buffer update
COMMITARCHIVE LOGMODE(QUIESCE)issued
© 2012 IBM Corporation
wait
suspend
resume
Number of suspensions high?− Too low QUIESCE system parameter
or command TIME option
Time per suspension high?− Long units of recovery− Inappropriate timing for quiesce process
buffer update
commit
UPDATE
issued
archivebegins
Page Latch Suspensions
w
FETCH
applicationaddress space
DB2address space
page latch suspend
Wait forotheragents’ writes
getpage hit,but page latched Most often a symptom of something else
Often during high insert activity−Space map page
What if …
Wait forpage latchsuspensions
© 2012 IBM Corporation
ait
page latch resume
−Space map page�Partitioning�MEMBER CLUSTER
−Data page�Randomize inserts
−Use IFCID 226, 227 for detail analysis
Page Latch Contention
• Also consider
– Higher VDWT to possibly reduce write frequency
– Faster active log and database write I/O device
– If spacemap page,
• TRACKMOD NO instead of default YES
– However, no ‘real’ incremental image copy if TRACKMOD NO
• Concurrent inserts to different partitions
© 2012 IBM Corporation
• Concurrent inserts to different partitions
• Classic partitioned rather than segmented or UTS in V9
– If index page,
• V9 RANDOM instead of ASC/DESC
– Trade-off with higher data Getpage, I/O, lock requests
• V9 index page: smaller if random key insert, bigger if sequential key insert
Global Lock Suspensions
wa
UPDATE
applicationaddress space
DB2address space
lock suspend
Wait forotheragents’ writes
lock unavailable – held by another system
• Thread can be suspended for:− Global L-parent lock− Global L-child (row/page) lock− Other global L-lock− Global pageset P-lock− Global page P-lock
− Data− IX leaf pages− SM pages
© 2012 IBM Corporation
ait
lock resume
buffer update
− SM pages− Other P-locks
• DB2 reports:− Accumulated time spent suspended− Total number of suspensions− Sum of all IRLM, XES, and False
contentions in Data Sharing Locking section
Wait forotheragents’ writes
What if …Wait forglobal locksuspensions
High Global Lock Contention Events/Time
© 2012 IBM Corporation
• Keep CF utilization below 50% for availability, response time, CPU time• If most time or most events are “OTHER” P-lock or L-lock
– -STA TRA (P) C(6) … for a short time to get detailed info
High Global lock contention events/time
• To minimize P-lock contention/negotiation
– First find out from GBP statistics what is contributing to P-lock contention
– If spacemap page P-lock and/or data page P-lock, consider using MEMBER CLUSTER table space
– V8/9/10 Partitioned table space
–
© 2012 IBM Corporation
– V10 UTS
– High spacemap page latch contention possible when the latch is held waiting for data spacemap page P-lock contention to resolve.
High Global lock contention events/time
• P-lock contention/negotiation - continued
– Consider setting VDWQT=0 to free page P-locks as soon as possible
– If index tree P-lock (high index splits), consider
– Freespace tuning (PCTFREE FREEPAGE)
– Minimize index key size especially for unique or semi-unique multi-column index key
– V8 non-padded index if large VARCHAR(s) in index key
– V9 bigger index page size and asymmetric page split
© 2012 IBM Corporation
–
– If data page P-lock contention on small tables with LOCKSIZE ROW, consider MAXROWS=1 and LOCKSIZE PAGE to ‘simulate’ row locking and/or reduce SM page freespace update
High Global lock contention events/time
• Can also affect other counters
– Update Commit service task wait P-lock unlock for each updated page
– Lock/Latch wait can increase because of
– Possibly higher IRLM latch contention
– “Mixing” local/global lock waits (next foil)
© 2012 IBM Corporation
“Mixing” Global/Local Lock Suspensions
Acquire lock
Lock
TRAN AX-Lock page 5
TRAN BX-Lock page 5
lock suspend
TRAN CX-Lock page 5
Release lock
TRAN ACommit
TRAN BX-Lock page 5
TRAN BCommit
Lock
TRAN CX-Lock page 5
Time Member 2
© 2012 IBM Corporation
Lockresume
Time Member 1
lock suspend(Wait behind TRAN B)Lock suspend
LockresumeRelease lock
Global lock wait
Local lock wait
• Even though part of the wait time for tran C it is waiting for a global lock held by tran A, all suspension time is reported as local because at resume time, the lock that was released was local to this member
Time in DB2 (Class 2 time)
• Burning CPU cycles (class 2 cpu time)
– Non-nested or nested
– On a general CP or zIIP
• Suspended in DB2 for a known event (class 3 suspension time)
– Many different types of suspensions ( 21 in V10)
• In DB2 but don’t know what it is we are doing
© 2012 IBM Corporation
• In DB2 but don’t know what it is we are doing
– Mostly things that happen outside of DB2’s control, like
– Lack of CPU or paging
– Should be a small %
What Is Left?
• Processor wait
– Wrong dispatching priorities
• NO DB2 work should run in discretionary
– Overloaded processor
• Use RMF for details
• General tuning
• Processor upgrade
•
Suspensions
In DB2 Time
Processing
© 2012 IBM Corporation
• Paging
– Overcommitted real storage
• Use RMF for details
• MAXKEEPD too large
• Check Buffer pools, EDM pool, Sort, number of threads, …
• Real storage upgrade
Not Accounted
What Is Left? - 2
• OEM monitors
• Wait time for completion of parallel tasks
• System-managed duplexing lock requests
• HSM dataset recall (since PQ67298)
• Waiting for requests to be returned from
Suspensions
In DB2 Time
Processing
© 2012 IBM Corporation
• Waiting for requests to be returned from SNA DB2 Server
• Portion of OPEN processing
– Accompanied with the appropriate synchronous EU switch suspensions
Not Accounted
Notes
• Most common are:
– CPU wait under high CPU utilization, especially with lower dispatching priority
– Excessive detailed online tracing with vendor tools
• Other causes are much less frequent and widely variedSome events not being captured by DB2, but more events are being captured in newer DB2 versions
– See online support document:
© 2012 IBM Corporation
– See online support document: http://www.ibm.com/support/docview.wss?rs=64&context=SSEPEK&uid=swg21045823
Effect of lack of CPU resources on CL3/8 timers
wa
getpage missUPDATE
applicationaddress space
DB2address space
• “Clock” on the wait time can only be stopped after the thread is dispatched again
• If you have to wait for the CPU the timer keeps “ticking”
Start I/O
Start clock
© 2012 IBM Corporation
ait
tI
me
Stop clock
buffer update
suspend
I/O interrupt
resume
I/O time
CPU wait
Accounting for DDF Work
• Processing "in DB2" (Class 2) should (almost) be the same regardless of connection type
• Time in DB2 server is:
– Class 2 nonnested elapsed time + Class 1 stored procedure, UDF, and trigger ET+Nonnested (Class 1 CPU - Class 2 CPU) +Nonnested (class 1 SE CPU - class2 SE CPU)
•
© 2012 IBM Corporation
• Time outside of DB2 server is total Class 1 elapsed less previous calculation
• Active thread accounting records are created at thread deallocation (disconnect)
• Inactive thread accounting records are created at DBAT inactive (look for DBAT inactive)
DDF Accounting at Plan Level
TIMES/EVENTS APPL(CL.1) DB2 (CL.2)------------ ---------- ----------ELAPSED TIME 3:10.24012 3.387696
NONNESTED 3:10.24012 3.387696STORED PROC 0.000000 0.000000UDF 0.000000 0.000000TRIGGER 0.000000 0.000000
CP CPU TIME 1.020178 0.980084AGENT 0.979296 0.939458
NONNESTED 0.979296 0.939458STORED PRC 0.000000 0.000000UDF 0.000000 0.000000
• Time in DB2 = class 2 nonnested ET + class 1 nested activity ET
[stored procedure + UDF + Trigger]+ nonnested (class 1 CP CPU - class 2 CP CPU) + nonnested (class 1 SE CPU - class2 SE CPU)
• Time in DB2 = 3.387696 + (0.000000 + 0.000000 + 0.000000)+ (0.979296 - 0.939458 )
© 2012 IBM Corporation
UDF 0.000000 0.000000TRIGGER 0.000000 0.000000
PAR.TASKS 0.040882 0.040626
SECP CPU 0.000000 N/A
SE CPU TIME 0.008714 0.008714NONNESTED 0.008714 0.008714STORED PROC 0.000000 0.000000UDF 0.000000 0.000000TRIGGER 0.000000 0.000000
PAR.TASKS 0.000000 0.000000
SUSPEND TIME 0.000000 1.440549.. .. ..
+ (0.979296 - 0.939458 )+ (0.008714 - 0.008714)= 3.427534
• Time outside DB2 = c lass 1 ET - Time in DB2
• Time outside of DB2 =3:10.24012 – 3.427534 = 3:06.8126
• Note: Be careful with SE CPU time when running on a ‘knee-capped’ machine – using normalized CPU times
DDF Accounting at Plan Level -2
SQL DML TOTAL
---- DISTRIBUTED ACTIVITY -------------------------- ---------------------------------------------------REQUESTER : DDF1 ROLLBCK(1) RECEIVED: 0 THREADS INDOUBT : 0 PRODUCT ID : DB2 SQL RECEIV ED : 601 ROWS SENT : 46515 PRODUCT VERSION : V9 R1 M5 MESSAGES SENT : 603 BLOCKS SENT : 340 METHOD : DRDA PROTOCOL MESSAGES R ECEIVED : 604 CONVERSAT.INITIATED: 1 COMMITS(1) RECEIVED: 0 BYTES SENT : 2087111 NBR RLUP THREADS : N/ A
BYTES RECEIVED : 809497
COMMIT(2) RECEIVED : 1 COMMIT(2) RESP.SENT: 1 PREPARE RECEIVED : 1 BACKOUT(2) RECEIVED: 0 BACKOUT(2) RESP.SENT: 0 LAST AGENT RECV. : 0 COMMIT(2) PERFORMED: 0 BACKOUT(2) PERFORMED: 0 MESSAGES IN BUFFER : 4651 5 TRANSACTIONS RECV. : 1 FORGET SENT : 0
© 2012 IBM Corporation
SQL DML TOTAL-------- --------SELECT 0INSERT 226
ROWS 282UPDATE 30
ROWS 84MERGE 0DELETE 0
ROWS 0
DESCRIBE 0DESC.TBL 0PREPARE 144OPEN 142FETCH 199
ROWS 46515CLOSE 1
DML-ALL 742
• Blocking is crucial for performance: Rows areput into blocks which are then sent out in messages
• Note # rows in DML operations for multi-row operations (implicit or explicit)
Acctg Reporting when (CPU) Parallelism is used
• In DB2 accounting data you can distinguish between
– Agent (parent)
– Parallel task
• # acctg records depends on ZPARM PTASKROL (Opaque ZPARM)
– Yes (Default)
• One acctg record for the parent task, and one acctg record for all // activity
• When using sysplex //, coordinating member is same as above, and each assisting member
© 2012 IBM Corporation
• When using sysplex //, coordinating member is same as above, and each assisting member cuts one acctg record per // group (so can be more than one per query/tran)
– No
• Each parallel task cuts a separate acctg record (regardless of type of CPU //)
– Normally your friendly monitor takes care of ‘rolling up’ this information
OMPE Accounting Report/Trace for a Parallel Query
TIMES/EVENTS APPL(CL.1) DB2 (CL.2) IFI (CL.5) CLASS 3 SUSPENSIONS ELAPSED TIME EVENTS------------ ---------- ---------- ---------- --------- ----------- ------------ --------ELAPSED TIME 1:27.10913 1:27.10502 N/P LOCK/LATCH(DB2+IRLM) 0.067109 634
NONNESTED 1:27.10913 1:27.10502 N/A IRLM LOCK+LATCH N/A N/ASTORED PROC 0.000000 0.000000 N/A DB2 LATCH N/A N/AUDF 0.000000 0.000000 N/A SYNCHRON. I/O 0.016820 47TRIGGER 0.000000 0.000000 N/A DATABASE I/O 0.016820 47
LOG WRITE I/O 0.000000 0CP CPU TIME 24.253183 24.252075 N/P OTHER READ I/O 10:38.762530 98546
AGENT 0.002148 0.001053 N/A OTHER WRTE I/O 0.000000 0NONNESTED 0.002148 0.001053 N/P S ER.TASK SWTCH 0.319902 22
Susp. time of parent and ALL child tasks
© 2012 IBM Corporation
NONNESTED 0.002148 0.001053 N/P S ER.TASK SWTCH 0.319902 22STORED PRC 0.000000 0.000000 N/A UPDATE COMMIT 0.000000 0UDF 0.000000 0.000000 N/A OPEN/CLOSE 0.318962 10TRIGGER 0.000000 0.000000 N/A SYSLGRNG REC 0.000823 10
PAR.TASKS 24.251035 24.251022 N/A EXT/DEL/DEF 0.000000 0OTHER SERVICE 0.000117 2
.. .. .. .. .. .. .. ..
.. .. .. .. .. .. .. ..
SUSPEND TIME 0.000000 10:39.1664 AGENT N/A 0.000117 PAR.TASKS N/A 10:39.1662 STORED PROC 0.000000 N/A UDF 0.000000 N/A
NOT ACCOUNT. N/A 1:27.10385
NOT ACCT time calculated based on parent task only
Sum of CPU time of all // tasks
CPU time of parent task
Note: CPU time can exceed ET when using // for CPU intensive queries
Understanding the Transaction ‘Profile’
• To get a better feel for type of work performed by a transaction/job, always at least check:
– # and type of SQL statements (S/I/U/D)
– The # and type of lock request (local and global (when data sharing)
© 2012 IBM Corporation
– BP activity and GBP activity(when data sharing)
SQL Info in DB2 Accounting at Plan level
• PK62161 (V8/9) Counters added for multi-row operations
• PM24812 (V8/V9/V10)Counters will include only rows that are updated, deleted or inserted by direct user action
SQL DML AVERAGE TOTAL ------------- ---------- ----------SELECT 5.02 1217636 INSERT 0.84 203878
ROWS 1.69 409044 UPDATE 0.87 211103
ROWS 1.01 244029 MERGE 0.00 0 DELETE 0.10 24085
ROWS 0.10 24085
© 2012 IBM Corporation
• DB2 10 ‘DRDA bundled fetch’ may reduce #FETCH counter, but #ROWS-FETCH should be accurate
ROWS 0.10 24085
DESCRIBE 0.00 0 DESC.TBL 0.00 0 PREPARE 0.00 0 OPEN 3.14 762136 FETCH 29.10 7053106
ROWS 26.13 6333609 CLOSE 2.94 711718
DML-ALL 42.02 10183662
Locking Info in Accounting at Plan Level
• Extra info about MAX PG/ROW LOCKS HELD compared to stats
LOCKING AVERAGE TOTAL--------------------- -------- --------TIMEOUTS 0.00 0DEADLOCKS 0.00 0ESCAL.(SHARED) 0.00 0ESCAL.(EXCLUS) 1.00 4MAX PG/ROW LOCKS HELD 5004.00 5004LOCK REQUEST 5042.00 20168
© 2012 IBM Corporation
LOCK REQUEST 5042.00 20168UNLOCK REQUEST 34.00 136QUERY REQUEST 0.00 0CHANGE REQUEST 3.00 12OTHER REQUEST 0.00 0TOTAL SUSPENSIONS 1.75 7
LOCK SUSPENSIONS 0.00 0IRLM LATCH SUSPENS. 1.75 7OTHER SUSPENS. 0.00 0
Locking Info in Accounting at Plan Level - 2
• TIMEOUTS
– The number of times lock suspension ultimately resulted in a timeout
• DEADLOCKS
– The number of times lock suspension ultimately resulted in a deadlock. This happens when two or more application processes each hold locks on resources that the others need and without which they cannot proceed
• ESCAL.(SHARED/EXCLUSIVE)
© 2012 IBM Corporation
• ESCAL.(SHARED/EXCLUSIVE)
– Number of times the LOCKS PER TABLE(SPACE) parameter on the panel DSNTIPJ was exceeded and the lock size was promoted from a page lock to a table space lock for this thread. For example, an application updates most pages in a table without issuing commits
• Further investigation
– Typically start with statistics trace class 3 info + syslog + db2MSTR log
– Zoom in with Locking traces – perf. trace class 6,7 and SQL trace – perf. trace class 3 –STA TRA(P) C(1,2,3,6,7) DEST(xxx) should do
Locking Info in Accounting at Plan Level - 3
• MAX PG/ROW LOCKS HELD from Accounting trace is a useful indicator of commit frequency
– Page or row locks only
– AVERAGE is for average of MAX, TOTAL is for max of MAX (of Accounting records)
• So if transaction A had max. locks of 10 and transaction B had 20, then
– AVERAGE (avg. of max.) = 15
– TOTAL (max. of max.) = 20
© 2012 IBM Corporation
– In general, try to issue Commit to keep max. locks held below 100
LOCKING AVERAGE TOTAL--------------------- -------- --------...MAX PG/ROW LOCKS HELD 5004.00 5004
Locking Info in Accounting at Plan Level - 4
• Good Lock avoidance can significantly improve performance
– Quick check in acctg data can provide some clues whether or not lock avoidance is ok
– Lock avoidance may not be working effectively if Unlock request/commit is high
• >5/commit and > 1/3 of Lock request
• E.g. 50 Lock and 40 Unlock requests per commit
– Check table on “Lock avoidance factors” in PMTG to see under which conditions DB2 is able to use lock avoidance techniques
© 2012 IBM Corporation
• Using ISO(CS) CURRENTDATA(NO) – default now – gives DB2 most opportunities to avoid having to take locks
• Check the IRLM lock and IRLM latch suspensions in the locking section against the L/L CL3 suspensions
Data Sharing Locking Section
DATA SHARING AVERAGE TOTAL------------------- -------- --------GLOBAL CONT RATE(%) 0.48 N/AFALSE CONT RATE(%) 0.00 N/AP/L-LOCKS XES(%) 43.83 N/ALOCK REQ - PLOCKS 0.67 270412UNLOCK REQ - PLOCKS 0.63 252966CHANGE REQ - PLOCKS 0.01 2550
• Global Contention should be less than 3-5% of XES IRLM Requests– (IRLM + XES + False SUSP) /
((IRLM + XES + False SUSP) + (LOCK+CHANGE+UNLOCK +CONV/ XES))
© 2012 IBM Corporation
LOCK REQ - XES 5.23 2101826UNLOCK REQ - XES 0.89 358098CHANGE REQ - XES 0.04 16744SUSPENDS - IRLM 0.03 11906SUSPENDS - XES 0.00 0CONVERSIONS- XES 0.02 7204FALSE CONTENTIONS 0.00 0INCOMPATIBLE LOCKS 0.00 0NOTIFY MSGS SENT 0.01 2278
• False contention less than 1-3% of sum of all XES IRLM Requests
• P/L LOCKS XES% – Gives an idea how many of the locks
requests are propagated to XES (“degree” of sharing)
Local vs. Data Sharing locks
LOCKING TOTAL DATA SHARING TOT AL------------------- -------- ------------ --------TIMEOUTS 0 P/L-LOCKS(%) 5DEADLOCKS 0 P-LOCK REQ 193024ESCAL.(SHAR) 0 P-UNLOCK REQ 1916 24ESCAL.(EXCL) 0 P-CHANGE REQ 1 01MAX PG/ROW LCK HELD 91 LOCK - XES 739773LOCK REQUEST 14175179 UNLOCK-XES 193684UNLOCK REQST 138996 CHANGE - XES 101
© 2012 IBM Corporation
UNLOCK REQST 138996 CHANGE - XES 101QUERY REQST 0 SUSP - IRLM 19 6CHANGE REQST 0 SUSP - XES 5 7OTHER REQST 0 CONV - XES 31 7TOTAL SUSPENSIONS 12337 FALSE CONT 7 89
LOCK SUSPENS 3 INCOMP.LOCK 0IRLM LATCH SUSPENS 12334 NOTIFY SENT 0OTHER SUSPENS 0
At last – Finally – RejoiceRequires PK85543 (DB2) PK85159(IRLM) OA12164(XES)
Local vs. Data Sharing locks - 2
• LOCK REQUESTS are L-lock requests
• P-LOCK REQ are the number of P-lock requests
– P-locks are ALWAYS sent to XES so they are included in the LOCK - XES counter
• LOCK – XES are the number of lock requests propagated to XES. This includes
– L-locks from LOCK REQUEST that need to be propagated (because of the interest) and
– P-LOCK REQ that are always propagated to XES
© 2012 IBM Corporation
– P-LOCK REQ that are always propagated to XES
– Only those that are Synchronously propagated to XES.
• With current maintenance, we can now distinguish between suspended because of false contention vs. suspended because of XES heuristic conversions
BP Info in Accounting at Plan Level
• Not as many BP counters in DB2 accounting compared to DB2 statistics
BP0 BPOOL ACTIVITY AVERAGE TOTAL--------------------- -------- --------BPOOL HIT RATIO (%) 92.22 N/AGETPAGES 711.64 12605326BUFFER UPDATES 226.48 4011708SYNCHRONOUS WRITE 0.00 0
© 2012 IBM Corporation
SYNCHRONOUS WRITE 0.00 0SYNCHRONOUS READ 49.71 880538SEQ. PREFETCH REQS 0.03 597LIST PREFETCH REQS 0.19 3313DYN. PREFETCH REQS 1.11 19595PAGES READ ASYNCHR. 5.67 100352
BP Info in Accounting at Plan Level - 2
• #Getpages is a good indicator of the amount of work that needs to be done by the transaction
– Likely to change if access path changes, or object gets more disorganized
• #SYNC READs are important as you wait for those (check against the CL3 data)
• A lot of prefetch activity can be sign of access path problem
© 2012 IBM Corporation
• A lot of prefetch activity can be sign of access path problem
• BP info
– At the plan level – info for each BP
– At the package level – all BP together and only if accounting class 10 is active
GBP info
• GBP tuning is mostly done via stats info (not acctg)
• Check GBP hit/miss ratio for application
• Check application page P-lock activity
– # Page P-lock requests
GROUP BP0 AVERAGE TOTAL--------------------- -------- --------GBP-DEPEND GETPAGES 14707.80 2191462READ(XI)-DATA RETUR 0.32 47READ(XI)-NO DATA RT 0.00 0READ(NF)-DATA RETUR 101.45 15116READ(NF)-NO DATA RT 313.67 46737PREFETCH PAGES READ 9.48 1413CLEAN PAGES WRITTEN 0.00 0UNREGISTER PAGE 19.46 2899ASYNCH GBP REQUESTS 9.31 1387EXPLICIT X-INVALID 0.00 0
© 2012 IBM Corporation
– # Page P-lock requests
– # Page P-lock suspensions
– Type of page P-lock SM, data index leaf
• There is no GBP information at the package level
ASYNCH SEC-GBP REQ 0.00 0PG P-LOCK LOCK REQ 497.81 74174
SPACE MAP PAGES 17.58 2620DATA PAGES 0.80 119INDEX LEAF PAGES 479.43 71435
PG P-LOCK UNLOCK REQ 146.15 21776PG P-LOCK LOCK SUSP 1.17 175
SPACE MAP PAGES 0.58 87DATA PAGES 0.05 8INDEX LEAF PAGES 0.54 80
WRITE AND REGISTER 176.83 26347WRITE & REGISTER MULT 17.73 2642CHANGED PAGES WRITTEN 304.57 45381
GBP Read Tuning
– Local BP search -> GBP search -> DASD I/O
GROUP BP14 QUANTITY /SECOND /THREAD /COMMIT
----------------------------- -------- ------- ------- -------
...
SYN.READ(XI)-DATA RETURNED 1932.00 0.09 0.01 0.00
SYN.READ(XI)-NO DATA RETURN 39281.6K 1823.66 236.31 79.22
SYN.READ(NF)-DATA RETURNED 22837.00 1.06 0.14 0.05
SYN.READ(NF)-NO DATA RETURN 6955.8K 322.93 41.85 14.03
© 2012 IBM Corporation
– Local BP search -> GBP search -> DASD I/O
– SYN.READ(NF) = Local Buffer Pool miss
– SYN.READ(XI) = Local Buffer Pool hit but cross-invalidated buffer
– Most data should be found in GBP >> if not, GBP may be too small or pages have been removed because of directory entry reclaims
Sync.Read(XI) miss ratio should be < 10%ROTROT
TOTAL SYN.READ(XI) = SYN.READ(XI)-DATA RETURNED + SYN.READ(XI)-NO DATA RETURNSync.Read(XI) miss ratio = SYN.READ(XI)-NO DATA RETURN / TOTAL SYN.READ(XI)
DSQGESQM VALUE DSQGESQM TIMES DSQGESQM AVER AGE TIME AVG.EV TIME/EVENT
------------------ ------------------ --------------- --- ------------ ------------------ ------------ ------ ------------
TYPE PACKAGE ELAP-CL7 TIME-AVG 2:37.547703 LOCK/LATCH 0.017786 42.47 0.000419
CP CPU TIME 2:21.224699 IRLM LOCK+LATC H 0.000335 0.20 0.001641
LOCATION USTSYIPT0D AGENT 2:21.224699 DB2 LATCH 0.017451 42.27 0.000413
COLLECTION ID Q PAR.TAS KS 0.000000 SYNCHRONOUS I/O 0.634727 698.49 0.000909
PROGRAM NAME DSQGESQM SE CPU TIME 0.000000 OTHER READ I/O 0.494961 404.7 5 0.001223
SUSPENSION-CL8 1.278264 OTHER WRITE I/O 0.026058 9.56 0.002726
ACTIVITY TYPE NONNESTED AGENT 1.278 264 SERV.TASK SWITCH 0.104705 16.77 0.006242
ACTIVITY NAME 'BLANK' PAR.TAS KS 0.000000 ARCH.LOG(QUIESCE) 0.000000 0.00 N/C
SCHEMA NAME 'BLANK' NOT ACCOU NTED 15.044740 ARCHIVE LOG READ 0.000000 0.00 N/C
SUCC AUTH CHECK 0 AVG.DB2 E NTRY/EXIT 112.91 DRAIN LOCK 0.000000 0.00 N/C
OCCURRENCES 93 DB2 ENTRY/EXIT 10501 CLAIM REL EASE 0.000000 0.00 N/C
NBR OF ALLOCATIONS 649 PAGE LAT CH 0.000027 0.05 0.000505
SQL STMT - AVERAGE 13.94 CP CPU SU 5066356.40 NOTIFY MESSAGES 0.000000 0.00 N/C
SQL STMT - TOTAL 1296 AGENT 5066356.40 GLOBAL CONTENTION 0 .000000 0.00 N/C
NBR RLUP THREADS 93 PAR.TASKS 0.00 TCP/IP LO B XML 0.000000 0.00 N/C
SE CPU SU 0.00 TOTAL CL8 SUSPE NS. 1.278264 1.2K 0.0 01091
Package Level accounting information
© 2012 IBM Corporation
SE CPU SU 0.00 TOTAL CL8 SUSPE NS. 1.278264 1.2K 0.0 01091
DSQGESQM AVERAGE TIME AV.EVENT DSQGESQM AVERAGE TIME AV.EVENT
------------------------- ------------ -------- ------ ------------------- ------------ --------
GLOBAL CONTENTION L-LOCKS 0.000000 0.00 GLOBAL CONTENTION P-LOCKS 0.000000 0.00
PARENT (DB,TS,TAB,PART) 0.000000 0.00 PAGESET/PARTITION 0.000000 0.00
CHILD (PAGE,ROW) 0.000000 0.00 PAGE 0.000000 0.00
OTHER 0.000000 0.00 OTHER 0.000000 0.00 CL8 info
OCCURRENCES - # IFCID 239 entries for this package NBR OF ALLOCATIONS - # times switched to the packageNBR RLUP THREADS (new in V10)package level detail info with rollup (not class10) - no longer *ROLLUP*[In OMPE averages are based on # NBR RLUP THREADS (= #OCC with no rollup) ]
• Also SE CPU time at package level
DSQGESQM AVERAGE TOTAL DSQGESQM AVERAGE TOTAL DSQGESQM AVERAGE TOTAL
------------------ -------- -------- ------------------- -------- -------- --------------------- -------- --------
SELECT 0.00 0 BPOOL HIT RATIO (%) 99.25 N/A TIMEOUTS 0.00 0
INSERT 0.12 11 GETPAGES 4185.6K 389262K DEADLOCKS 0.00 0
UPDATE 0.04 4 BUFFER UPDATES 890.5K 82816786 ESCAL.(SHARED) 0.00 0
DELETE 0.06 6 SYNCHRONOUS WRITE 0.05 5 ESCAL.(EXCLUS) 0.04 4
SYNCHRONOUS READ 698.41 64952 MAX PG/ROW LOCKS HELD 477.02 44363
DESCRIBE 0.00 0 SEQ. PREFETCH REQS 2036.97 189438 LOCK REQUEST 1904.25 177095
PREPARE 0.00 0 LIST PREFETCH REQS 4.84 450 UNLOCK REQUEST 636.86 59228
OPEN 0.00 0 DYN. PREFETCH REQS 1976.46 183811 QUERY REQUEST 0.00 0
FETCH 0.00 0 PAGES READ ASYNCHR. 30603.12 2846090 CHANGE REQUEST 224.92 20918
CLOSE 0.00 0 OTHER REQUEST 0.00 0
SQL BP totals Locking
Package level Accounting - acctg CL10
© 2012 IBM Corporation
CLOSE 0.00 0 OTHER REQUEST 0.00 0
TOTAL SUSPENSIONS 0.20 19
LOCK TABLE 0.00 0 LOCK SUSPENSIONS 0.00 0
CALL 0.00 0 IRLM LATCH SUSPENS. 0.20 19
OTHER SUSPENS. 0.00 0
• Only available when accounting trace class 10 is active • Not available when rollup accounting is in effect (in plan for Vnext)• BP info is not at BP level, but all BP together• Note that sum SQL here is not same as SQL STMT TOTAL
(CL10 section does not include things like SET stmts, so expect <= )
DB2 V8 Enhancements
• Better control over basic accounting intervals
– KEEPDYNAMIC YES will not prevent completing the accounting interval at commit boundary
– At the same time change of WLM enclave is enabled for the new transaction
• Accounting roll-up (for RRS and DDF work)
– Addressing the issue about the large volume of accounting data
– Aggregating accounting data for the same values of the client-side identifiers (end
© 2012 IBM Corporation
– Aggregating accounting data for the same values of the client-side identifiers (end user ID, end user transaction/application name, end user workstation name)
• zparm ACCUMUID controls which combination of IDs is used
• zparm ACCUMACC controls when the aggregated accounting records are externalized
DB2 V10 Enhancements
• Better accounting rollup (for RRS and DDF work)
– Still don’t like it
– Packages not all lumped together (up to 24 packages will be rolled up separately – if more rest is rolled up into a single package)
– DDF info (up to 4 unique QLACs will be rolled up separately – if more the rest is rolled up into a single one)
© 2012 IBM Corporation
• Compression of SMF records (SMFCOMP=YES ZPARM)
– Should be enough to avoid a need to use ACCUMACC
– Small overhead (1%), great compression ratio (60-80%)
• Separating DB2 internal latch and IRLM lock/latch suspensions (see before)
DB2 V10 Enhancements
• Aggregate accounting statistics
– Introduced by APAR PM62797 (8/12)
– (OMPE APAR PM72949 to support this in stats reports)
– High level info accounting information per connection type
– New statistics class 9 will write IFCID369 (if acctg trace class 1 is also active)
– Written at 1 min (stats) interval
– Basically class 1,2,3 time info per connection type
© 2012 IBM Corporation
– Basically class 1,2,3 time info per connection type
• Fields from DSNDQWAC and DSNDQWAX
– ‘Draft’ sample OMPE report
CONNTYPE CL1 ELAPSED CL1 CPU CL1 SE CPU CL 2 ELAPSED CL2 CPU CL2 SE CPU CL3 SUSP CL 2 NOT ACC THREADS-------- ------------ ------------ ------------ ------- ----- ------------ ------------ ------------ ---------- -- --------BATCH 1:14.326062 8.882753 0.000000 1 :14.234073 8.879979 0.000000 1:05.997069 0.000000 3.00CICS N/P N/P N/P N/P N/P N/P N/P N/P 0.00DDF 3:20.927303 9.873839 8.240478 55.917303 8.697064 6.742272 39.852088 7.368151 15895.00IMS N/P N/P N/P N/P N/P N/P N/P N/P 0.00RRSAF 1:04.091521 0.143129 0.000000 0.050749 0.044200 0.000000 0.000062 0.006487 10.00UTILITY 21.317271 0.712988 0.031434 1.224435 0.189245 0.031434 0.987360 0.047829 72.00
© Copyright IBM Corporation 2012. All rights reser ved.U.S. Government Users Restricted Rights - Use, dupli cation or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
THE INFORMATION CONTAINED IN THIS PRESENTATION IS P ROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND AC CURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION,
Disclaimer
© 2012 IBM Corporation110
PRESENTATION, IT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RES PONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICEN SORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM P RODUCTS AND/OR SOFTWARE
IBM, the IBM logo, ibm.com, and DB2 are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml