a new approach to managing oracle e-business suite piper-rx the hidden cost of workflow gary piper...
TRANSCRIPT
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
The Hidden Cost of WorkflowThe Hidden Cost of Workflow
Gary PiperGary PiperAUSOUGAUSOUGSydneySydney
August 2005August 2005
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Agenda…
Two issues Workflow load balancing (Cost) Purging obsolete workflow items
Symptoms: Workflow background process runs every 5 min 24*7 Poor performance from workflow and workflow &
background engine performance degrading over time Actions:
Establish Cause Purging - What and How
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
The Theory…
“Real-time” Workflow load is managed by identifying costly activities / processes, so that when they are encountered by the workflow engine they are immediately given a “deferred” status and the workflow engine continues processing the next activity. The deferred processes are then processed by a Workflow Background Process which can be run at an appropriate time.
X X
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
What Actually Happens…
The background process is run more frequently than is “required” Every 3 – 5 min 24 * 7 Every 15 min 24 * 7
What is actually being deferred? Should the activity be deferred ( Business decision ) If an activity is required to be processed immediately
Assess why the activity is so urgent Don’t run a regular background process to clear the
item ( fixes the symptom ) Don’t allow the items to become deferred in the first
place ( fix the cause )
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
What's In a Cost…
Each workflow activity has an associated cost. The “Cost” value represents the number of seconds it should take
to execute the activity. Generally when workflows are created, the cost is estimated or
ignored. Cost is both entered and displayed in seconds. The value stored in
the database wf_activities.cost is in hundredths of a second The default threshold for the Workflow Engine is 50 hundredths of a
second. Activities with a cost higher than this are deferred and are run by the Workflow Background process
In a perfect world when sufficient runtime information is available, a Cost Vs Actual review should be completed
This rarely happensThis rarely happens
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
How can this occur…
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
How can this occur…
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
The Basic Cause…
WorkflowDefinition
(Clone)
WorkflowBackground
ProcessDefinition
Over the wall
Cost is rarelydefined and set
Activity must be process immediately
Functional TeamFunctional Team Technical TeamTechnical Team
Cost ReviewsCost Reviews
Functional Changes
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Assess the Damage…
Frequency of execution Purging Obsolete Workflow Runtime Data
Is it running into itself Frequency of execution Vs run time Restart from beginning of prior run?
Workflow Background Process Example: (15 min 24 * 7)
0
5
10
15
20
25
30
19-0
4-20
04 :
0:6
19-0
4-20
04 :
3:2
4
19-0
4-20
04 :
6:4
0
19-0
4-20
04 :
12:
49
19-0
4-20
04 :
17:
41
19-0
4-20
04 :
21:
4
20-0
4-20
04 :
0:2
4
20-0
4-20
04 :
3:2
5
20-0
4-20
04 :
6:4
4
20-0
4-20
04 :
12:
58
20-0
4-20
04 :
18:
40
20-0
4-20
04 :
22:
6
21-0
4-20
04 :
1:2
6
21-0
4-20
04 :
4:4
5
21-0
4-20
04 :
7:5
1
21-0
4-20
04 :
12:
31
21-0
4-20
04 :
16:
53
21-0
4-20
04 :
20:
16
21-0
4-20
04 :
23:
46
22-0
4-20
04 :
2:4
2
22-0
4-20
04 :
6:1
22-0
4-20
04 :
9:5
5
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
So What is Being Processed…
How much is being processed? State change is overwritten, so there for little audibility Monitor by “Select count” of Deferred items type every n
minutes Look at the runtime of the BG process to identify peaks
Toad E-Business suite Plug-in (Beta)
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Review Actual Vs Execution Times…
SELECT wa.item_type "Item Type", substr(witt.display_name,1,40) "Display Name", wa.name, wa.version, count(*) "Executions", wa.cost "Intenal", wa.cost / 100 "Cost (Sec)", round(min(( wias.end_date - wias.begin_date )*86400),0) "Min (Sec)", round(avg(( wias.end_date - wias.begin_date )*86400),0) "Avg (Sec)", round(max(( wias.end_date - wias.begin_date )*86400),0) "Max (Sec)" FROM wf_activities wa, wf_item_types_tl witt, wf_item_activity_statuses wias WHERE wa.item_type = wias.item_type(+) and wa.item_type = witt.name and witt.language = userenv('LANG') and wa.cost > 50 and sysdate between wa.begin_date and nvl(wa.end_date, sysdate + 1) GROUP BY wa.item_type, witt.display_name, wa.name, wa.version, wa.cost ORDER BY wa.cost DESC;
Item Typ Display Name NAME VERSION Executions Intenal Cost (Sec) Min (Sec) Avg (Sec) Max(Sec)-------- ------------------------------ ------------------------------ ---------- ---------- ---------- ---------- ---------- ---------- ---------MSCEXPWF MSC: ASCP Exception Messages PROCESS RESPONSE 10 5028 10000 100 0 0 1POAPPRV PO Approval MASS_UPDATE_RELEASES 1 232013 10000 100 0 144265 8204232POAPPRV PO Approval NOOP 24 232013 10000 100 0 144265 8204232POAPPRV PO Approval PLACE_SOURCING_INFO_ON_REQ 1 232013 10000 100 0 144265 8204232REQAPPRV PO Requisition Approval NOOP 29 51560 10000 100 0 450 3130878
Note: Paper attendance required to interpret this outputNote: Paper attendance required to interpret this output
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Case Study - Workflow…
Issue: Client site in a constant state of performance issues for past
year Technical DBA constantly monitoring CPU and I/O usage and
looking for resource hogs Current Activity:
Technical DBA reviewing tools to identify CPU and I/O bottlenecks
A Capex is in place for hardware, fewer faster CPU’s (4 – 2) Background:
Technical DBA’s has limited E-Business Suite experience running the application
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Case Study - Workflow…
00
:00
- 0
1:0
0
01
:00
- 0
2:0
0
02
:00
- 0
3:0
0
03
:00
- 0
4:0
0
04
:00
- 0
5:0
0
05
:00
- 0
6:0
0
06
:00
- 0
7:0
0
07
:00
- 0
8:0
0
08
:00
- 0
9:0
0
09
:00
- 1
0:0
0
10
:00
- 1
1:0
0
11
:00
- 1
2:0
0
12
:00
- 1
3:0
0
13
:00
- 1
4:0
0
14
:00
- 1
5:0
0
15
:00
- 1
6:0
0
16
:00
- 1
7:0
0
17
:00
- 1
8:0
0
18
:00
- 1
9:0
0
19
:00
- 2
0:0
0
20
:00
- 2
1:0
0
21
:00
- 2
2:0
0
22
:00
- 2
3:0
0
23
:00
- 2
4:0
004-Feb-05 (Thu)05-Feb-05 (Fri)
06-Feb-05 (Sun)07-Feb-05 (Mon)
08-Feb-05 (Tue)09-Feb-05 (wed)
10-Feb-05 (Thu)11-Feb-05 (Fri)
12-Feb-05 (Sat)
13-Feb-05 (Sun)14-Feb-05 (Mon)
15-Feb-05 (Tue)16-Feb-05 (wed)
17-Feb-05 (Thu)18-Feb-05 (Fri)
19-Feb-05 (Sat)20-Feb-05 (Sun)
21-Feb-05 (Mon)22-Feb-05 (Tue)
23-Feb-05 (Wed)
24-Feb-05 (Thu)25-Feb-05 (Fri)
26-Feb-05 (Sat)27-Feb-05 (Sun)
28-Feb-05 (Mon)01-Mar-05 (Tue)
02-Mar-05 (Wed)03-Mar-05 (Thu)
900-1000
800-900
700-800
600-700
500-600
400-500
300-400
200-300
100-200
0-100
Concurrent Manager Activity Profile
Charts reproduced with permission of PIPER-Rx
15,000 Requests per day15,000 Requests per day
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Case Study - Workflow…
Concurrent ProgramConcurrent Program TotalTotal Per dayPer day Per HrPer Hr Per MinPer Min
Workflow Background Process 210,091 10,004.3 416.8 6.9
Process transaction interface 23,940 1,140.0 47.5 0.8
Cost Manager ( BOM ) 41,539 1,978.0 82.4 1.4
Event Log 4,687 223.2 9.3 0.2
Data Load 3,530 168.1 7.0 0.1
Interface Cleanup ( Custom ) 2,954 140.7 5.9 0.1
Interface trip Stop 4,022 191.5 8.0 0.1
WIP Move Transaction Manager 4,387 208.9 8.7 0.1
Import Items 3,012 143.4 6.0 0.1
Set 2,095 99.8 4.2 0.1
Set Stage 4,506 214.6 8.9 0.1
Receiving Transaction Processor 1,055 50.2 2.1 0.0
Material cost transaction worker 904 43.0 1.8 0.0
15,000 Requests per day15,000 Requests per day93.4% of all concurrent manager activity93.4% of all concurrent manager activity
Concurrent Request Activity (21 Days)
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Case Study - Workflow…
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
18,000
04-F
eb-0
5 (T
hu
)
05-F
eb-0
5 (F
ri)
06-F
eb-0
5 (S
un
)
07-F
eb-0
5 (M
on
)
08-F
eb-0
5 (T
ue)
09-F
eb-0
5 (w
ed)
10-F
eb-0
5 (T
hu
)
11-F
eb-0
5 (F
ri)
12-F
eb-0
5 (S
at)
13-F
eb-0
5 (S
un
)
14-F
eb-0
5 (M
on
)
15-F
eb-0
5 (T
ue)
16-F
eb-0
5 (w
ed)
17-F
eb-0
5 (T
hu
)
18-F
eb-0
5 (F
ri)
19-F
eb-0
5 (S
at)
20-F
eb-0
5 (S
un
)
21-F
eb-0
5 (M
on
)
22-F
eb-0
5 (T
ue)
23-F
eb-0
5 (W
ed)
24-F
eb-0
5 (T
hu
)
25-F
eb-0
5 (F
ri)
26-F
eb-0
5 (S
at)
27-F
eb-0
5 (S
un
)
28-F
eb-0
5 (M
on
)
01-M
ar-0
5 (T
ue)
02-M
ar-0
5 (W
ed)
03-M
ar-0
5 (T
hu
)
Actual Vs User Activity (estimated)
Graphs reproduced with permission of PIPER-Rx
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Case Study - Workflow…
0
500
1,000
1,500
2,000
2,500
3,000
00:0
0 -
01:0
0
01:0
0 -
02:0
0
02:0
0 -
03:0
0
03:0
0 -
04:0
0
04:0
0 -
05:0
0
05:0
0 -
06:0
0
06:0
0 -
07:0
0
07:0
0 -
08:0
0
08:0
0 -
09:0
0
09:0
0 -
10:0
0
10:0
0 -
11:0
0
11:0
0 -
12:0
0
12:0
0 -
13:0
0
13:0
0 -
14:0
0
14:0
0 -
15:0
0
15:0
0 -
16:0
0
16:0
0 -
17:0
0
17:0
0 -
18:0
0
18:0
0 -
19:0
0
19:0
0 -
20:0
0
20:0
0 -
21:0
0
21:0
0 -
22:0
0
22:0
0 -
23:0
0
23:0
0 -
24:0
0
Actual User Activity by Hour of Day
Graphs reproduced with permission of PIPER-Rx
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Case Study - Workflow…
00
:00
- 0
1:0
0
01
:00
- 0
2:0
0
02
:00
- 0
3:0
0
03
:00
- 0
4:0
0
04
:00
- 0
5:0
0
05
:00
- 0
6:0
0
06
:00
- 0
7:0
0
07
:00
- 0
8:0
0
08
:00
- 0
9:0
0
09
:00
- 1
0:0
0
10
:00
- 1
1:0
0
11
:00
- 1
2:0
0
12
:00
- 1
3:0
0
13
:00
- 1
4:0
0
14
:00
- 1
5:0
0
15
:00
- 1
6:0
0
16
:00
- 1
7:0
0
17
:00
- 1
8:0
0
18
:00
- 1
9:0
0
19
:00
- 2
0:0
0
20
:00
- 2
1:0
0
21
:00
- 2
2:0
0
22
:00
- 2
3:0
0
23
:00
- 2
4:0
004-Feb-05 (Thu)
05-Feb-05 (Fri)06-Feb-05 (Sun)
07-Feb-05 (Mon)08-Feb-05 (Tue)
09-Feb-05 (wed)
10-Feb-05 (Thu)11-Feb-05 (Fri)
12-Feb-05 (Sat)13-Feb-05 (Sun)
14-Feb-05 (Mon)
15-Feb-05 (Tue)16-Feb-05 (wed)
17-Feb-05 (Thu)18-Feb-05 (Fri)
19-Feb-05 (Sat)20-Feb-05 (Sun)
21-Feb-05 (Mon)
22-Feb-05 (Tue)23-Feb-05 (Wed)
24-Feb-05 (Thu)25-Feb-05 (Fri)
26-Feb-05 (Sat)
27-Feb-05 (Sun)28-Feb-05 (Mon)
01-Mar-05 (Tue)02-Mar-05 (Wed)
03-Mar-05 (Thu)
180-200
160-180
140-160
120-140
100-120
80-100
60-80
40-60
20-40
0-20
Concurrent Manager Activity (Adjusted)
Graphs reproduced with permission of PIPER-Rx
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Case Study - Workflow…0
0:0
0 -
01
:00
01
:00
- 0
2:0
0
02
:00
- 0
3:0
0
03
:00
- 0
4:0
0
04
:00
- 0
5:0
0
05
:00
- 0
6:0
0
06
:00
- 0
7:0
0
07
:00
- 0
8:0
0
08
:00
- 0
9:0
0
09
:00
- 1
0:0
0
10
:00
- 1
1:0
0
11
:00
- 1
2:0
0
12
:00
- 1
3:0
0
13
:00
- 1
4:0
0
14
:00
- 1
5:0
0
15
:00
- 1
6:0
0
16
:00
- 1
7:0
0
17
:00
- 1
8:0
0
18
:00
- 1
9:0
0
19
:00
- 2
0:0
0
20
:00
- 2
1:0
0
21
:00
- 2
2:0
0
22
:00
- 2
3:0
0
23
:00
- 2
4:0
0
04-Feb-05 (Thu)05-Feb-05 (Fri)
06-Feb-05 (Sun)07-Feb-05 (Mon)
08-Feb-05 (Tue)09-Feb-05 (wed)
10-Feb-05 (Thu)11-Feb-05 (Fri)
12-Feb-05 (Sat)
13-Feb-05 (Sun)14-Feb-05 (Mon)
15-Feb-05 (Tue)16-Feb-05 (wed)
17-Feb-05 (Thu)18-Feb-05 (Fri)
19-Feb-05 (Sat)20-Feb-05 (Sun)
21-Feb-05 (Mon)22-Feb-05 (Tue)
23-Feb-05 (Wed)
24-Feb-05 (Thu)25-Feb-05 (Fri)
26-Feb-05 (Sat)27-Feb-05 (Sun)
28-Feb-05 (Mon)01-Mar-05 (Tue)
02-Mar-05 (Wed)03-Mar-05 (Thu)
900-1000
800-900
700-800
600-700
500-600
400-500
300-400
200-300
100-200
0-100
00
:00
- 0
1:0
0
01
:00
- 0
2:0
0
02
:00
- 0
3:0
0
03
:00
- 0
4:0
0
04
:00
- 0
5:0
0
05
:00
- 0
6:0
0
06
:00
- 0
7:0
0
07
:00
- 0
8:0
0
08
:00
- 0
9:0
0
09
:00
- 1
0:0
0
10
:00
- 1
1:0
0
11
:00
- 1
2:0
0
12
:00
- 1
3:0
0
13
:00
- 1
4:0
0
14
:00
- 1
5:0
0
15
:00
- 1
6:0
0
16
:00
- 1
7:0
0
17
:00
- 1
8:0
0
18
:00
- 1
9:0
0
19
:00
- 2
0:0
0
20
:00
- 2
1:0
0
21
:00
- 2
2:0
0
22
:00
- 2
3:0
0
23
:00
- 2
4:0
0
04-Feb-05 (Thu)
05-Feb-05 (Fri)06-Feb-05 (Sun)
07-Feb-05 (Mon)08-Feb-05 (Tue)
09-Feb-05 (wed)
10-Feb-05 (Thu)11-Feb-05 (Fri)
12-Feb-05 (Sat)13-Feb-05 (Sun)
14-Feb-05 (Mon)
15-Feb-05 (Tue)16-Feb-05 (wed)
17-Feb-05 (Thu)18-Feb-05 (Fri)
19-Feb-05 (Sat)20-Feb-05 (Sun)
21-Feb-05 (Mon)
22-Feb-05 (Tue)23-Feb-05 (Wed)
24-Feb-05 (Thu)25-Feb-05 (Fri)
26-Feb-05 (Sat)
27-Feb-05 (Sun)28-Feb-05 (Mon)
01-Mar-05 (Tue)02-Mar-05 (Wed)
03-Mar-05 (Thu)
180-200
160-180
140-160
120-140
100-120
80-100
60-80
40-60
20-40
0-20
Revealing the “real” Activity profile
Graphs reproduced with permission of PIPER-Rx
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Purging…
PURGINGPURGING
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Purging Obsolete Workflow Runtime Data…
Argument set: Item Type - leave this value blank to purge all item types Beware: Insufficient resources to purge all
Warning: Workflow Purge will only purge items that have been defined with a
persistence of TEMPORARY and the number of persistence days have expired. A workflow with a persistence of Permanent will never be purged by the default purge settings
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
So What Is Purgable…
Persistence Persistence Work Flow Work FlowDisplay Name WF Item Type Type Days Count Purgable---------------------------------------- -------------------- ----------- ----------- ---------- ----------CSM Type 3 CSMTYPE3 TEMP 0 19415 0FA Account Generator FAFLEXWF PERM 8651 8651PA: HR Related Updates Workflow PAXWFHRU TEMP 0 8591 8480AR: Substitute Balancing Segment ARSBALSG PERM 4823 4823OM Order Line OEOL TEMP 0 4802 3547PO Approval POAPPRV TEMP 20 3324 2321Planning Exception Messages MRPEXPWF TEMP 0 3124 0System: Error WFERROR TEMP 0 2287 49……………………..
SELECT witt.display_name A, wit.name B, wit.persistence_type C, wit.persistence_days D, count(wi.item_key) E,
apps.wf_purge.getpurgeablecount(wit.name) F Secret weapon FROM wf_item_types wit, wf_item_types_tl witt, wf_items wi WHERE wit.name = witt.name and witt.language = userenv('LANG') and wit.name = wi.item_type GROUP BY witt.display_name, wit.name, wit.persistence_type, wit.persistence_days ORDER BY count(wi.item_key) DESC;
Note: Column formatting not shown
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
So What Is Purgable…
Toad E-Business suite Plug-in (Beta)
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Purging…
Warning: wf_item_attribute_values (up to 5:1) wf_notifications Gather Schema Statistics Index Rebuilds
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
State of the Indexes…
Index Name index Row Count Table Row Count Last Anal last Analyzed (days) PAR------------------------------ --------------- --------------- --------- -------------------- ---WF_ITEM_ACTIVITY_STATUSES_N1 708324 740035 01-APR-04 375 NOWF_ITEM_ACTIVITY_STATUSES_N2 13823 740035 17-JUN-03 663 NOWF_ITEM_ACTIVITY_STATUSES_N3 725979 740035 17-JUN-03 663 NOWF_ITEM_ACTIVITY_STATUSES_N4 725979 740035 17-JUN-03 663 NOWF_ITEM_ACTIVITY_STATUSES_PK 725979 740035 17-JUN-03 663 YES WF_ITEM_ATTRIBUTE_VALUES_PK 4194640 4178092 17-JUN-03 663 NO
SELECT index_name "Index Name", num_rows "index Row Count", decode(table_name, 'WF_ITEM_ACTIVITY_STATUSES',count_statuses.count, 'WF_ITEM_ATTRIBUTE_VALUES', count_values.count, null) "Table Row Count", last_analyzed "Last Analysed", trunc(sysdate - last_analyzed) "last Analyzed (days)", partitioned FROM dba_indexes, (SELECT count(*) count FROM WF_ITEM_ACTIVITY_STATUSES ) count_statuses, (SELECT count(*) count FROM WF_ITEM_ATTRIBUTE_VALUES ) count_values WHERE table_owner = 'APPLSYS' and table_name in ('WF_ITEM_ATTRIBUTE_VALUES', 'WF_ACTIVITY_ATTR_VALUES', 'WF_ACTIVITY_TRANSITIONS', 'WF_ITEM_ACTIVITY_STATUSES', 'WF_ITEM_ACTIVITY_STATUSES_H', 'WF_PROCESS_ACTIVITIES', 'WF_NOTIFICATIONS') and index_type = 'NORMAL' ORDER BY table_name, index_name;
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
Questions…
Questions?Questions?
Disclaimer: All material contained in this document is provided by the author "as is" and any Disclaimer: All material contained in this document is provided by the author "as is" and any express or implied warranties, including, but not limited to, any implied warranties of express or implied warranties, including, but not limited to, any implied warranties of merchantability and fitness for a particular purpose are disclaimed. In no event shall the author be merchantability and fitness for a particular purpose are disclaimed. In no event shall the author be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, loss of use, data, or profits; or business interruption) however caused and on any but not limited to, loss of use, data, or profits; or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use of any content or information, even if advised of the possibility of arising in any way out of the use of any content or information, even if advised of the possibility of such damage. It is always recommended that you seek independent, professional advice before such damage. It is always recommended that you seek independent, professional advice before implementing any ideas or changes to ensure that they are appropriateimplementing any ideas or changes to ensure that they are appropriate
A New Approach to Managing Oracle E-Business Suite
PIPER-Rx
The Hidden Cost of WorkflowThe Hidden Cost of Workflow
Gary PiperGary PiperAUSOUGAUSOUGSydneySydney
August 2005August 2005