Concurrent Managers 101 & 202

Gary Piper

UKOUG 2004 Birmingham

1 - 3 November 2004

Case 1: The Case of Too Many Managers…

• Standard and Custom Managers• As a general rule: No more than

1.8 to 2 * the number of CPU’s• Common misconceptions:

– More managers are better • 35 on 2 CPU’s ( 17.5 )• 70 on 8 CPU’s ( 8.8 )• 48 on 12 CPU’s ( 4 )

– Double the number of managers overnight

Resist the urge to add managersResist the urge to add managers

Financial Statement Generator…

• Assume 1 FSG will consume 1 CPU• Limit the number of FSGs that can run to less than the

number of CPUs• Create a custom FSG manager and setting the number

of manager processes to less than the number of CPUs• This reasoning should hold true for any resource

intensive process

Case 1: Example…

Manager Name Max Proc Run Proc Sleep Cache

FNDICM 1 1 30 0

STANDARD 70 70 30 100

FNDCRM 1 1 30 0

FNDSCH 1 1 30 0

INVMGR 1 1 60 5

PODAMGR 3 3 60 0

RCVOLTM 3 3 60 0

Advantages Disadvantages Application Impact Business Impact Corrective Action

Configuration: 8 CPU & High FSG usage

Beware the Silver Bullet…

• Site was heavily customised• Several custom programs have an average run time of

over 10 hours.• Account hierarchy reports take over 14 hours to run• The site has a seven segment chart of accounts with

over 200,000 code combinations and no indexes.• Other Sessions:

– 103 Discoverer sessions– 30 Forms sessions– 1 ADI session – 5 Toad sessions In Production– 1 SQL*PLUS In Production

Case 2: Sleep and Cache…

• Configuration– 4 CPUs

– Max 8 standard managers

• No custom managers• Site experienced exceptional growth • In this example:

– The concurrent program runs in less than 6 seconds.

– Items are scanned at a rate of 10 per minute

Single Manager…

Managers: 1Sleep: 60 secondsCache: 2Execution rate: 2 per min

Scan rate: 10 per minInput rate: 600 per hourRun rate: 120 per hourDeficit: 480 runs per hourApproximately 4 hours backlog per hour

4880%

Multiple Managers…

Managers: 3Sleep: 60 secondsCache: 2Execution rate: 6 per min

Scan rate: 10 per minInput rate: 600 per hourRun rate: 360 per hourDeficit: 240 runs per hourApproximately 40 min backlog per hour

48

48

48

144/18080%

Single Manager…

Managers: 1Sleep: 60 secondsCache: 10Execution rate: 10 per min

Scan rate: 10 per minInput rate: 600 per hourRun rate: 600 per hourDeficit: 0 runs per hour ( theoretical )

A More Practical Solution…

Managers: 2Sleep: 60 SecondsCache: 10Execution rate: 20 per min

Scan rate: 10 per minInput rate: 600 per hourRun rate: Up to 1200 per hourHead room: 600 per hour

Case 2: Sleep and Cache Example…

Manager Name Run Proc Sleep Cache Hits

FNDICM 1 20 0 3

STANDARD 16 2 1 480

FNDCRM 1 5 0 12

IMMEDIATE 2 10 1 6

QUICK 2 5 1 24

SLOW 3 5 2 36

SPECIAL 3 30 1 6

Advantages Disadvantages Application Impact Business Impact Corrective Action

23 additional managers ( INV,OE, PO, RCV) @ 5 second sleep 567

Configuration: 12 CPU & 3,500 request per day

Hidden Resource Usage…

Slow QSlow Q

Lookup Frequency…

Total Managers

Total

Processes

Accesses

Per Min

Accesses

Per Hour

Accesses

Per Day

2 3 5 300 7,200

Manager

Name

Running

Processes

Sleep

(Seconds)

Accesses

Per Min

% of

Total

Slow_Queue 1 60 1 20

Standard 2 30 4 80

Manager Configuration

Manager Impact

Additional I/O…

• Per Request (5,000)– Insert row (7) Indexes = (35,000)– Update to running (2) indexes = (10,000)– Update to complete (3) Indexes = (15,000)– 5,000 requests generate 60,000 index activities

• Manager reads @100 per Min 144,000 per day plus 144,000 index lookups

• User accesses ( has it finished yet? )• Conflict Resolution Manager

– Higher overhead– fnd_crm_history

Review Request Activity…

00:0

0 -

00:3

0

01:0

0 -

01:3

0

02:0

0 -

02:3

0

03:0

0 -

03:3

0

04:0

0 -

04:3

0

05:0

0 -

05:3

0

06:0

0 -

06:3

0

07:0

0 -

07:3

0

08:0

0 -

08:3

0

09:0

0 -

09:3

0

10:0

0 -

10:3

0

11:0

0 -

11:3

0

12:0

0 -

12:3

0

13:0

0 -

13:3

0

14:0

0 -

14:3

0

15:0

0 -

15:3

0

16:0

0 -

16:3

0

17:0

0 -

17:3

0

18:0

0 -

18:3

0

19:0

0 -

19:3

0

20:0

0 -

20:3

0

21:0

0 -

21:3

0

22:0

0 -

22:3

0

23:0

0 -

23:3

0

20-APR-04 (Tue)

18-APR-04 (Sun)

16-APR-04 (Fri)

14-APR-04 (Wed)

12-APR-04 (Mon)

10-APR-04 (Sat)

08-APR-04 (Thu)

06-APR-04 (Tue)

04-APR-04 (Sun)

180-200

160-180

140-160

120-140

100-120

80-100

14 Standard Managers

Graphs reproduced with permission of PIPER-Rx

Review Running Requests… 0

0:00

1:00

2:00

3:00

4:00

5:00

6:00

7:00

8:00

9:00

10:0

0

11:0

0

12:0

0

13:0

0

14:0

0

15:0

0

16:0

0

17:0

0

18:0

0

19:0

0

20:0

0

21:0

0

22:0

0

23:0

0

20-APR-04 (Tue)

18-APR-04 (Sun)

16-APR-04 (Fri)

14-APR-04 (Wed)

12-APR-04 (Mon)

10-APR-04 (Sat)

08-APR-04 (Thu)

06-APR-04 (Tue)

04-APR-04 (Sun)

02-APR-04 (Fri)

15-18

12-15

9-12

6-9

3-6

0-3

14 Standard Managers

Graphs reproduced with permission of PIPER-Rx

Case 3: Specialisation Rules Gone Wrong…

• Specialisation rules can be defined at the following levels:– By Program– By user– By Oracle User– Complex– Request Type

• Once created are hard to find• Creating a specialization rule will:

– Rebuild the fnd_concurrent_worker_requests view– Restart the concurrent managers– Not a good idea during month end

• KIS principle

Spcialisation rules…

Systems AdministratorConcurrentManagerDefine

Select the ManagerSelect specialization

Preparation…

• Step 1 – Create a concurrent manager “slow”• Step 2 – Activate the manager ( multiple steps )• Step 3 – ReduceReduce the number of STD managers• Step 4 – Ensure the new manager is active• Step 5 – Set the specialisation rules

– “Include” the selected program in the Slow Manager– “Exclude” the selected program from the Standard

manager• All custom developed programs should be initially assigned to the

“slow” queue and earn the right to be moved to the “normal” queues

Example of Include Only…

Manager Name Max Proc Run Proc Sleep Cache Pending

Internal Manager 1 1 15

Conflict_Resolution_Manager 1 1 30 48

Financial Statement Generator 12 12 60

General Accounting 10 10 8 128

INV Remote Procedure Manager 10 10 30 4

CRP Inquiry Manager 2 0 60

Inventory Manager 1 1 60 1

MRP Manager 1 0 60 1

Material Transactions Manager 3 3 300 3

PA Streamline Manager 1 0 1 1

Shipping Transaction Manager 1 0 60

Standard ( Disabled ) 0 0 60 1 128

Review… Configuration 8 CPU

Manager Name Max Proc Run Proc Sleep Cache Pending

Internal Manager 1 1 15

Conflict_Resolution_Manager 1 1 30 48

Financial Statement Generator 12 12 60

General Accounting 10 10 8 128

INV Remote Procedure Manager

10 10 30 4

CRP Inquiry Manager 2 0 60

Inventory Manager 1 1 60 1

MRP Manager 1 0 60 1

Material Transactions Manager 3 3 300 3

PA Streamline Manager 1 0 1 1

Shipping Transaction Manager 1 0 60

Standard ( Disabled ) 0 0 60 1 128

Request Types…

Request Types

Request Types…

• Change the queue a program will run in• Cannot change the queue once the request has been

submitted• Can prevent a program from running

– Assign a program to run in a queue that has been deactivated - “Pending Error”

– Once activated, pending request will run

– Assign a program to a request type “Don’t Run” that has not been assigned a manager

Request Types (cont)…

• Create a Request Type– Concurrent > Program > Types

– No impact

• Create Specialisation Rules– Concurrent > Manager > Define – Specialisation Rules

– Disallow & Allow– FND_CONCURRENT_WORKER_REQUESTS view

• Assign a program a request type– Concurrent > Program > Define

Example: Overnight / Don’t Run…

• Create a manager that runs only overnight – (work shifts * 2)

• Create a request type ”overnight”• Disallow ”overnight” from standard / allow in the

overnight manager• Assign the program the request type ”overnight” • Request type “don’t run”

– Create a request type “don’t run”

– Do not assign to a manager

– Assign the program the request type “don’t run”

Questions…

Questions Questions oror

Visit us at the Quest standVisit us at the Quest stand

Disclaimer: All material contained in this document is provided by the author "as is" and any express Disclaimer: All material contained in this document is provided by the author "as is" and any express or implied warranties, including, but not limited to, any implied warranties of merchantability and or implied warranties, including, but not limited to, any implied warranties of merchantability and fitness for a particular purpose are disclaimed. In no event shall the author be liable for any direct, fitness for a particular purpose are disclaimed. In no event shall the author be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, loss indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, loss

of use, data, or profits; or business interruption) however caused and on any theory of liability, of use, data, or profits; or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use of any content or information, even if advised of the possibility of such damage. It is always of the use of any content or information, even if advised of the possibility of such damage. It is always

recommended that you seek independent, professional advice before implementing any ideas or recommended that you seek independent, professional advice before implementing any ideas or changes to ensure that they are appropriatechanges to ensure that they are appropriate

Concurrent Manager 101 & 202

Gary Piper

OUUG 2004 Birmingham

1 - 3 November 2004