peoplesoft instrumentation for oracle rdbms david kurtz go-faster consultancy ltd....

PeopleSoft Instrumentation for Oracle RDBMS

David KurtzGo-Faster Consultancy Ltd.

[email protected]

www.go-faster.co.uk

Who Am I?

• Oracle Database Specialist– Independent consultant

• Performance tuning– PeopleSoft ERP– Oracle RDBMS

• Book– www.psftdba.com

• OakTable•

UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle

©2013 www.go-faster.co.uk 2

http://www.psftdba.com/

Instrumentation

Data! Data! Data! I can't make bricks without clay.

The Adventures of Sherlock Holmes, Arthur Conan-Doyle



Instrumentation

• PeopleTools– SQL Trace– Batch Timings– Performance Monitor

• Oracle– Dynamic Performance

Views (V$)– SQL Trace– Automatic Workload

Repository (AWR)– Active Session History

(ASH)




ASH: A Brief Overview

• Samples active sessions every second• Circular buffer in memory

– v$active_Session_history– It should hold about 1 hour of data

• 1 in 10 samples stored in database– DBA_HIST_ACTIVE_SESS_HISTORY– Flushed out during AWR snapshot



What does ASH retain?

• Most of the columns are on v$session– Session

• Session ID and serial, query coordinator– Wait

• event id, name and parameters– SQL

• SQL_ID, plan hash, opcode, plan line– Object

• object, file, block and row numbers– Application

• module, action …



ASH –v- SQL*Trace

• ASH– Licensed (Diagnostics)– Always there– Real Time– No marginal overhead– Who is blocking me?– Statistical data– SQL ID– Plan if captured by AWR– Estimate of duration

• Per wait event• Per Plan Operation 11g

• SQL*Trace– Free– Enable, File, Profile– Reactive– Run-time overhead– Being Blocked– Every SQL & event– SQL HASH– Actual execution plan– Exact duration

• Operations in Plan



ASH –v- SQL*Trace

• ASH can be used to resolve many of your performance issues.

• Sometimes, you will still need SQL*Trace



Application Instrumentation

• It is essential to be able to match – database sessions– application processes

• DBMS_APPLICATION_INFO– set_module, set_action– Calls in application

• Introduced in PeopleTools 8.50-8.52.– Appears in SQL*Trace and ASH


CLIENT_INFO

– PeopleTools 7.53– Every application server service calls

• DBMS_APPLICATION_INFO.SET_CLIENT_INFO• V$SESSION.CLIENT_INFO• Operator ID, …

– Inserts into Audit tables can be done by database trigger.• Trigger uses READ_CLIENT_INFO to obtain PeopleSoft

OPRID

– Not copied to ASHUKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle


CLIENT_ID

• PeopleTools 8.52• Also set to PeopleSoft OPRID.• Client_ID is copied to ASH

– So you can identify the PS operator



PIA Instrumentation

• PeopleTools 8.50

• Module = Component• Action = Page

– Or XYZZY for search dialogue



IB Instrumentation


• Module = Service• Action = Queue



Application Engine


• Module = ‘PSAE’• Action = Program

– This is not good– See http://

blog.psftdba.com/2009/03/using-oracle-enterprise-manager-grid.html



http://blog.psftdba.com/2009/03/using-oracle-enterprise-manager-grid.html



Application Engine


• Module = ‘PSAE.’||AE program name||session id– session id is stored in

PSPRCSQUE.SESSIONID_NUM

• Action = Program.Section.Step.Actions



Cobol / SQR

• No instrumentation• DIY with trigger

– on Process Scheduler request table– http://

www.go-faster.co.uk/scripts.htm#psftapi.sql



http://www.go-faster.co.uk/scripts.htm#psftapi.sql

http://www.go-faster.co.uk/scripts.htm#psftapi.sql

OEM



©2013 www.go-faster.co.uk 19UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle

ASH Report in OEM

• You can run ASH reports via EM– This example is for a specific module

Enable Trace by Module/Action

• Like setting a watchpoint in a debugger.• See http://

blog.psftdba.com/2012/09/oracle-sql-tracing-by-module-action.html

• Can set SQL trace for a specific component/page.

• But you can’t predict the value of module for AE so may need to go back to the trigger based solution.UKOUG Apps 2013 - PeopleSoft

Instrumentation for Oracle©2013 www.go-faster.co.uk 20

http://blog.psftdba.com/2012/09/oracle-sql-tracing-by-module-action.html



Ability to Profile SQL duration by Process/Process Instance

PROGRAM MODULE ASH_SECS-------------------- -------------------------------- ----------PSQRYSRV PSQRYSRV 633520oracle oracle 328630rman backup archivelog 236050PSAESRV GL_JEDIT 203120rman backup incr datafile 132840FSPCCURR FSPCCURR 119350PSAPPSRV VCHR_EXPRESS 111520... ----------sum 2982340



Profile of Whole System

WITH x AS (SELECT /*+leading(x h) use_nl(h)*/ CASE WHEN h.module IS NULL THEN REGEXP_SUBSTR(h.program, '[^@]+',1,1) WHEN h.module LIKE 'PSAE.%' THEN REGEXP_SUBSTR(h.module, '[^.]+',1,2) WHEN h.module LIKE '%.%' THEN REGEXP_SUBSTR(h.module, '[^.]+',1,1) WHEN h.module LIKE '%@%' THEN REGEXP_SUBSTR(h.module, '[^@]+',1,1) ELSE h.module END AS module, REGEXP_SUBSTR(h.program, '[^.@]+',1,1) program, h.eventFROM dba_hist_snapshot x, dba_hist_active_sess_history hWHERE h.SNAP_id = X.SNAP_idAND h.dbid = x.dbidAND h.instance_number = x.instance_numberAND x.end_interval_time >= TO_DATE('20130730', 'YYYYMMDD')AND x.begin_interval_time <= TO_DATE('20130807', 'YYYYMMDD')AND h.sample_time BETWEEN TO_DATE('20130730','YYYYMMDD') AND TO_DATE('20130807','YYYYMMDD'))SELECT program, module, sum(10) ash_SecsFROM x GROUP BY program, moduleORDER BY ash_secs DESC



ASH Data for period of interest

Snapshots for Period of interest

Logic to trim module name at @ or .

Batch Timings by Exec Time



ASH Profile a Single Process

Process SQL Plan Exec Instance SQL_ID Hash Value ACTION secs ASH_SECS--------- ------------- ---------- -------------------------------- ----- ---------- 51159995 fpj0dmac7rw20 207497280 AR_POSTING.HSS_PYMT.ITEMS.S 2797 410 51159995 2qnfa4afzvj91 416825695 AR_POSTING.HS_PYMNT.ITEMS2.S 2797 320 51159995 1u37a5hzzc2z6 768682620 AR_POSTING.HS_PYMNT.ITEMS.S 2797 270 51159995 grhj2f0dcbd70 2964999827 AR_POSTING.HSS_DSO.BACK_OUT.S 2797 190 51159995 a6nrbg7sbx401 721249526 AR_POSTING.HSS_PYMT.UPD_STAT.S 2797 170 51159995 49yzw3xa50r1t 2665694782 AR_POSTING.HS_USER.INSERTS.S 2797 160 51159995 0wcx4syn7tjk2 1678621177 AR_POSTING.HS_USER.UPDATES.S 2797 150 51159995 44sjax92d6kqa 3752318859 AR_POSTING.HS_DSO.ADJ_BAL.S 2797 130 51159995 cm5xjajkfjvkg 3814124734 AR_POSTING.HSS_USR1.INSERTS.S 2797 130 51159995 djur79tkd6uum 2504040148 AR_POSTING.HS_USER.UPD_ACTV.S 2797 130 51159995 cb1cankq47ky9 362952569 AR_POSTING.HSS_USR1.UPDATES.S 2797 130 51159995 1awa6cr95ug3t 3137593480 AR_POSTING.HS_PYMNT.UPD_STAT.S 2797 110 51159995 03p8ga7kazdg4 2504040148 AR_POSTING.HSS_USER.UPD_ACTV.S 2797 100 51159995 5yrur2nv3nn2p 2663443920 AR_POSTING.HSS_DSO.NO_ACTIV.S 2797 80



How profile a process (i)

WITH x AS ( SELECT /*+LEADING(r q x h)*/ h.sql_id, h.sql_plan_hash_value, h.action , SUM(10) ash_secs , 10*count(t.sql_id) awr_secs FROM sysadm.psprcsrqst r , sysadm.psprcsque q , dbA_hist_snapshot x , dba_hist_active_Sess_history h LEFT OUTER JOIN dba_hist_sqltext t ON t.sql_id = h.sql_id WHERE x.END_INTERVAL_TIME >= r.begindttm AND x.begin_interval_time <= r.enddttm AND h.sample_time BETWEEN r.begindttm AND r.enddttm AND h.SNAP_id = X.SNAP_id AND h.dbid = x.dbid AND h.instance_number = x.instance_number AND q.prcsinstance = r.prcsinstance AND h.module = 'PSAE.'||r.prcsname||'.'||q.sessionidnum AND r.prcsname = 'AR_UPDATE2' AND r.prcsinstance = 51159995 GROUP BY h.sql_id, h.sql_plan_hash_value, h.action) …



Process Request TableOne Row per Process

Process Request Queue TableOne Row per Process

Specific Process Request

ASH data for period for which process was running

Hint to guarantee sensible join order, and partition

elimination

Filter ASH data by module

How profile a process (ii)

…), y AS ( SELECT ROW_NUMBER() OVER (PARTITION BY x.sql_plan_hash_value order by x.awr_secs desc) as ranking , x.sql_id, x.sql_plan_hash_value, x.action , SUM(x.ash_secs) OVER (PARTITION BY x.sql_plan_hash_value) tot_ash_secs , SUM(x.awr_secs) OVER (PARTITION BY x.sql_plan_hash_value) tot_awr_secs , COUNT(distinct sql_id) OVER (PARTITION BY x.sql_plan_hash_value) sql_ids FROM x)Select * from ywhere y.ranking = 1AND tot_ash_secs > 30ORDER BY tot_ash_secs DESC/



Ranking by SQL Plan Hash Value

DB Time for Execution Plan

AWR secs 0 if SQL not captured in ASH repository

Profile a Single Process

Process SQL Plan Exec Instance SQL_ID Hash Value ACTION secs ASH_SECS--------- ------------- ---------- -------------------------------- ----- ---------- 51159995 fpj0dmac7rw20 207497280 AR_POSTING.HSS_PYMT.ITEMS.S 2797 410 51159995 2qnfa4afzvj91 416825695 AR_POSTING.HS_PYMNT.ITEMS2.S 2797 320 51159995 1u37a5hzzc2z6 768682620 AR_POSTING.HS_PYMNT.ITEMS.S 2797 270 51159995 grhj2f0dcbd70 2964999827 AR_POSTING.HSS_DSO.BACK_OUT.S 2797 190 51159995 a6nrbg7sbx401 721249526 AR_POSTING.HSS_PYMT.UPD_STAT.S 2797 170 51159995 49yzw3xa50r1t 2665694782 AR_POSTING.HS_USER.INSERTS.S 2797 160 51159995 0wcx4syn7tjk2 1678621177 AR_POSTING.HS_USER.UPDATES.S 2797 150 51159995 44sjax92d6kqa 3752318859 AR_POSTING.HS_DSO.ADJ_BAL.S 2797 130 51159995 cm5xjajkfjvkg 3814124734 AR_POSTING.HSS_USR1.INSERTS.S 2797 130 51159995 djur79tkd6uum 2504040148 AR_POSTING.HS_USER.UPD_ACTV.S 2797 130 51159995 cb1cankq47ky9 362952569 AR_POSTING.HSS_USR1.UPDATES.S 2797 130 51159995 1awa6cr95ug3t 3137593480 AR_POSTING.HS_PYMNT.UPD_STAT.S 2797 110 51159995 03p8ga7kazdg4 2504040148 AR_POSTING.HSS_USER.UPD_ACTV.S 2797 100 51159995 5yrur2nv3nn2p 2663443920 AR_POSTING.HSS_DSO.NO_ACTIV.S 2797 80



Now investigate the SQL

• Extract SQL from AWR repository with the dbms_xplan package– Obtaining and Interpreting Execution Plans usi

ng DBMS_XPLAN • http://

www.go-faster.co.uk/ukougpres.htm#Intro_DBMS_XPLAN



http://www.go-faster.co.uk/pres/Intro_DBMS_XPLAN.ppt

http://www.go-faster.co.uk/pres/Intro_DBMS_XPLAN.ppt

http://www.go-faster.co.uk/ukougpres.htm#Intro_DBMS_XPLAN




SELECT * FROM table( dbms_xplan.display_awr( 'duu7n4uzsthxz',

768682620,NULL,'ADVANCED'));

SQL_ID duu7n4uzsthxz

--------------------

INSERT INTO PS_HS_CLSITM_TMP4 (…)

SELECT 51164249 , I.BUSINESS_UNIT ,…

FROM PS_ITEM I

,PS_ITEM_ACTIVITY A

,PS_POST_REQ_TMP4 R

WHERE R.PROCESS_INSTANCE = 51164249

AND R.PROCESS_P_HIST ='Y'

--

AND I.BUSINESS_UNIT = R.BUSINESS_UNIT

AND I.ITEM_STATUS = 'C'

AND I.HIST_STATUS = 'N'

--

AND A.BUSINESS_UNIT = I.BUSINESS_UNIT

AND A.CUST_ID = I.CUST_ID

AND A.ITEM = I.ITEM

AND A.ITEM_LINE = I.ITEM_LINE

AND A.ENTRY_TYPE = I.ENTRY_TYPE

AND A.ENTRY_REASON = I.ENTRY_REASON

AND A.ITEM_SEQ_NUM = (

SELECT MIN(ITEM_SEQ_NUM)

FROM PS_ITEM_ACTIVITY X

WHERE X.BUSINESS_UNIT = I.BUSINESS_UNIT

AND X.CUST_ID = I.CUST_ID

AND X.ITEM = I.ITEM

AND X.ITEM_LINE = I.ITEM_LINE

AND X.ENTRY_TYPE = I.ENTRY_TYPE

AND X.ENTRY_REASON = I.ENTRY_REASON)

AND A.ENTRY_AMT_BASE > 0



Execution Plan

Plan hash value: 768682620 -----------------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |-----------------------------------------------------------------------------------------------------------| 0 | INSERT STATEMENT | | | | | 222K(100)| || 1 | FILTER | | | | | | || 2 | HASH JOIN | | 23719 | 3613K| | 111K (1)| 00:22:19 || 3 | TABLE ACCESS BY INDEX ROWID | PS_POST_REQ_TMP4 | 8 | 112 | | 2 (0)| 00:00:01 || 4 | INDEX RANGE SCAN | PS_POST_REQ_TMP4 | 8 | | | 1 (0)| 00:00:01 || 5 | HASH JOIN | | 23719 | 3289K| 11M| 111K (1)| 00:22:19 || 6 | TABLE ACCESS BY INDEX ROWID| PS_ITEM | 118K| 9944K| | 30295 (1)| 00:06:04 || 7 | INDEX RANGE SCAN | PSKITEM | 280K| | | 812 (1)| 00:00:10 || 8 | TABLE ACCESS FULL | PS_ITEM_ACTIVITY | 3480K| 185M| | 67704 (1)| 00:13:33 || 9 | SORT AGGREGATE | | 1 | 42 | | | || 10 | TABLE ACCESS BY INDEX ROWID | PS_ITEM_ACTIVITY | 1 | 42 | | 5 (0)| 00:00:01 || 11 | INDEX RANGE SCAN | PS_ITEM_ACTIVITY | 1 | | | 4 (0)| 00:00:01 |-----------------------------------------------------------------------------------------------------------

Query Block Name / Object Alias (identified by operation id):-------------------------------------------------------------

1 - SEL$1 3 - SEL$1 / R@SEL$1 4 - SEL$1 / R@SEL$1 6 - SEL$1 / I@SEL$1 7 - SEL$1 / I@SEL$1 8 - SEL$1 / A@SEL$1 9 - SEL$2 10 - SEL$2 / X@SEL$2 11 - SEL$2 / X@SEL$2

Outline Data-------------

/*+ BEGIN_OUTLINE_DATA IGNORE_OPTIM_EMBEDDED_HINTS OPTIMIZER_FEATURES_ENABLE('10.2.0.4') OPT_PARAM('_unnest_subquery' 'false') OPT_PARAM('optimizer_dynamic_sampling' 4) OPT_PARAM('_optimizer_cost_based_transformation' 'off') ALL_ROWS OUTLINE_LEAF(@"SEL$2") OUTLINE_LEAF(@"SEL$1") OUTLINE_LEAF(@"INS$1") FULL(@"INS$1" "PS_HS_CLSITM_TMP4"@"INS$1") INDEX_RS_ASC(@"SEL$1" "I"@"SEL$1" ("PS_ITEM"."HIST_STATUS")) FULL(@"SEL$1" "A"@"SEL$1") INDEX_RS_ASC(@"SEL$1" "R"@"SEL$1" ("PS_POST_REQ_TMP4"."PROCESS_INSTANCE" "PS_POST_REQ_TMP4"."BUSINESS_UNIT")) LEADING(@"SEL$1" "I"@"SEL$1" "A"@"SEL$1" "R"@"SEL$1") USE_HASH(@"SEL$1" "A"@"SEL$1") USE_HASH(@"SEL$1" "R"@"SEL$1") SWAP_JOIN_INPUTS(@"SEL$1" "R"@"SEL$1") INDEX_RS_ASC(@"SEL$2" "X"@"SEL$2" ("PS_ITEM_ACTIVITY"."BUSINESS_UNIT" "PS_ITEM_ACTIVITY"."CUST_ID" "PS_ITEM_ACTIVITY"."ITEM" "PS_ITEM_ACTIVITY"."ITEM_LINE" "PS_ITEM_ACTIVITY"."ITEM_SEQ_NUM")) END_OUTLINE_DATA */

Note----- - dynamic sampling used for this statement



Limitations

• SQL may not be captured by AWR– SQL inside a loop– With bind variables – Without ReUseStatement

• (binds become literals)

• Increase AWR snapshot frequency– 15 minutes instead of an hour– Top N SQL threshold



Conclusion

• Instrumentation permits you to profile DB performance with ASH

• Proactively address performance issues



QUESTIONS?Nullius in verba



http://en.wikipedia.org/wiki/Royal_Society

You know my methods,

apply them.A Study in Scarlet, Arthur Conan-Doyle



peoplesoft instrumentation for oracle rdbms david kurtz go-faster consultancy ltd....

Documents