bpel pm 11g performance tuning- appendies

2

Contents

APPENDIX A: WEBLOGIC SERVER OVERVIEW ......................................................................................6

1 WLS CONFIGURATION .......................................................................................................................6

2 DOMAIN ........................................................................................................................................6

3 SERVER ..........................................................................................................................................7

4 ADMINISTRATION SERVER ..................................................................................................................8

5 MANAGED SERVER ...........................................................................................................................9

6 ADMINISTRATION SERVER TO MANAGED SERVER INTERACTION ................................................................. 10

7 CLUSTER ....................................................................................................................................... 10

8 NODE MANAGER ............................................................................................................................ 12

9 MACHINE ..................................................................................................................................... 13

APPENDIX B: AUDITING IN BPEL PM ................................................................................................. 15

1 AUDIT LEVELS ................................................................................................................................ 15

2 ORDER OF PRECEDENCE FOR AUDIT LEVEL SETTINGS ................................................................................. 15

APPENDIX C: ANTI PATTERNS ........................................................................................................... 18

1 SYNCHRONOUS ASYNCHRONOUS ..................................................................................................... 18

2 OVER USE OF ASYNCHRONOUS PROCESSES ............................................................................................ 19

3 OVER USE OF DURABLE PROCESSES ...................................................................................................... 19

4 NO FAULT HANDLING ...................................................................................................................... 19

5 SYNCHRONOUS FAULT HANDLING ....................................................................................................... 19

6 TO MANY RETRIES ........................................................................................................................... 20

7 CHATTING BPEL PROCESS (CALL BACK) ................................................................................................ 20

8 OVER USE OF FLOWN ...................................................................................................................... 20

9 LOOPS AND MORE LOOPS.................................................................................................................. 20

10 SYNCHRONOUS AND ASYNCHRONOUS PROCESSES ON SAME MANAGED SERVER/CLUSTER ............................... 20

11 DURABLE AND TRANSIENT PROCESSES ON SAME MANAGED SERVER/CLUSTER ............................................... 21

12 STICKY LOAD BALANCER .................................................................................................................. 21

13 NOT KEEPING ASPECT RATIO ............................................................................................................ 21

APPENDIX D: SQL QUERIES ............................................................................................................... 22

1 EM CONSOLE SQL QUERIES .............................................................................................................. 22

1.1 RECOVERY CONSOLE QUERIES ................................................................................................................... 22

1.2 RECENT FAULT AND REJECTED MESSAGES QUERY .......................................................................................... 22

1.3 RECENT COMPOSITE INSTANCE QUERY ........................................................................................................ 23

1.4 INSTANCE TAB PAGE QUERY ...................................................................................................................... 23

3

1.5 INSTANCE TAB PAGE SEARCH QUERY BASED ON NAME VS TITLE QUERY ............................................................. 23

1.6 FAULT AND REJECTED MESSAGE TAB PAGE QUERIES ...................................................................................... 23

1.6.1 Parent query ................................................................................................................................... 23

1.6.2 Child query ...................................................................................................................................... 24

2 MISCELLANEOUS ............................................................................................................................ 24

2.1 STORED PROCEDURE TO CONVERT BLOB IN STRING ....................................................................................... 24

2.2 QUERY TO FIND PERCENTAGE OF FREE SPACE ............................................................................................... 25

2.3 QUERY TO FIND THE WAIT EVENTS FOR LGWR USING ITS SID ........................................................................ 25

2.4 QUERY TO MONITOR REDO BUFFER ALLOCATION RETRIES ............................................................................. 25

2.5 SQL STATEMENT TO RECLAIM SPACE AFTER PURGING.................................................................................... 25

2.6 QUERY TO FIND OUT TOTAL SESSIONS ON A DATABASE .................................................................................. 25

2.7 QUERY TO FIND OUT UTILIZATION OF PROCESSES AND SESSIONS IN A DATABASE ................................................ 25

2.8 FIND OUT THE PROCESS INSTANCE FROM A CONVERSATION ID WHEN THERE IS NO INSTANCE NUMBER SHOWING IN

THE LOG FILE (BPEL INSTANCE ID FOR A TIMES OUT ITEM) .................................................................................... 26

2.9 QUERY TO GET AUDIT DETAILS FROM AUDIT_DETAILS TABLE ........................................................................... 26

2.10 QUERY TO GET AUDIT DETAILS FROM AUDIT_TRAIL TABLE ............................................................................ 26

2.11 QUERY TO GET XML MESSAGE WITH THE GIVEN INSTANCE ID ...................................................................... 27

2.12 QUERY TO GET XML MESSAGE WITH A GIVEN INSTANCE NAME ..................................................................... 27

2.13 QUERY TO GET PAYLOAD SIZE OF MESSAGE ................................................................................................ 27

2.14 QUERY TO GET EXECUTION TIME OF BPEL INSTANCES ................................................................................. 27

2.15 QUERY TO GET THE EXECUTION TIME OF BPEL INSTANCES AND TO FIND THE PARENT THAT HAS INITIALIZED THE

COMPOSITE .................................................................................................................................................... 27

2.16 QUERY TO IDENTIFY ALL THE FAULTS FOR THE MESSAGES THAT WERE SITTING IN BPEL ENGINE LEVEL RECOVERY AS

UNDELIVERED INVOKES ..................................................................................................................................... 28

APPENDIX E: BIG OR LARGE OR HUGE PAGES .................................................................................... 29

1 LINUX .......................................................................................................................................... 29

2 WINDOWS .................................................................................................................................... 30

3 SOLARIS ....................................................................................................................................... 30

4 REFERENCE: .................................................................................................................................. 30

APPENDIX F: ORA-01438: VALUE LARGER THAN SPECIFIED PRECISION ALLOWED ............................... 31

5 WHAT IS THE ERROR IN LOGS? ........................................................................................................... 31

6 EFFECTS........................................................................................................................................ 31

7 CAUSE ......................................................................................................................................... 31

8 SOLUTION ..................................................................................................................................... 32

APPENDIX G: LOBS IN THE SOAINFRA SCHEMA ................................................................................. 33

APPENDIX H: AWR, ADDM, & ASH REPORTS ..................................................................................... 36

4

1 AWR REPORT ............................................................................................................................... 36

2 ADDM REPORT ............................................................................................................................. 36

3 ASH REPORT ................................................................................................................................. 36

4 AWR REPORT ANALYSIS .................................................................................................................. 37

4.1 SQL STATEMENTS ORDERED BY ELAPSED TIME ............................................................................................ 37

4.2 SQL STATEMENTS ORDERED BY CPU TIME .................................................................................................. 37

4.3 SQL STATEMENTS ORDERED BY GETS ......................................................................................................... 38

4.4 SQL STATEMENTS ORDERED BY READS ....................................................................................................... 38

4.5 SQL STATEMENTS ORDERED BY EXECUTIONS ............................................................................................... 38

4.6 SQL STATEMENTS ORDERED BY PARSE CALLS .............................................................................................. 38

5 REFERENCE ................................................................................................................................... 39

APPENDIX I: MONITORING SCRIPTS .................................................................................................. 40

1 DATABASE MONITORING .................................................................................................................. 40

2 JMS MONITORING ......................................................................................................................... 41

3 AQ MONITORING ........................................................................................................................... 44

APPENDIX J: HOW TO MONITOR SOA SERVER MEMORY USAGE ........................................................ 48

1 SETUP: JCONSOLE OR VISUALVM (INSTALLED LOCALLY) ............................................................................ 48

2 SETUP: JVISUALVM (INSTALLED AT REMOTE MACHINE) ............................................................................ 52

3 SETUP: JROCKIT MISSION CONTROL (INSTALLED AT REMOTE MACHINE) ....................................................... 54

4 REFERENCE ................................................................................................................................... 55

APPENDIX K: HEAP DUMP FILES ANALYSIS: JROCKIT AND HOTSPOT JVMS ......................................... 56

1 EXAMPLE ANALYSIS OF A HEAP DUMP FILE USING ECLIPSE MEMORY ANALYZER ............................................ 56

2 REFERENCE ................................................................................................................................... 62

APPENDIX L: CAPACITY PLANNING .................................................................................................... 63

1 CAPACITY PLANNING FOR BPEL PM ................................................................................................... 63

1.1 DETERMINING PERFORMANCE GOALS AND OBJECTIVES CURRENT & FUTURE ................................................. 64

1.2 MEASURING PERFORMANCE METRICS ....................................................................................................... 65

1.3 IDENTIFYING BOTTLENECKS ...................................................................................................................... 65

1.4 IMPLEMENTING A CAPACITY MANAGEMENT PLAN ....................................................................................... 65

2 REFERENCE ................................................................................................................................... 66

5

Exhibits

Exhibit 82: WebLogic Homes ........................................................................................................ 14

Exhibit 83: Synchronous Asynchronous - 1 ................................................................................ 18

Exhibit 84: Synchronous Asynchronous - 2 ................................................................................ 19

Exhibit 85: Database Monitoring .................................................................................................. 40

Exhibit 86: JMS Monitoring ........................................................................................................... 42

Exhibit 87: AQ Monitoring ............................................................................................................ 45

Exhibit 88: Setup jConsole or visualVM (installed locally) - 1 ....................................................... 49




Exhibit 92: Heap Dump Files analysis - 1 ...................................................................................... 57






6

AppendixA:WebLogicServerOverview

1 WLS Configuration

WLS configuration is segmented by domain

Each domain represents a logically related group of WLS instances that you

manage from a single set of configuration files.

Each domain has one Administration Server, and can have multiple managed

servers and clusters

Administration Server - Central configuration controller for the entire domain

Managed Server - A running instance that hosts applications and resources needed by

those applications - The real work horses in a WebLogic domain

Cluster - group of Managed Servers running simultaneously and working together to

provide increased scalability and reliability

Node Manager is a per-machine process used to start and stop WLS instances.

Independent of all domains

2 Domain

What is it?

A logically related group of WLS instances that you manage from a single set of

configuration artifacts.

Whats in a domain?

Servers

Clusters of servers

Rules:

All WLS instances within the same domain must be at the same major and minor

version.

Servers within a domain can be at different Maintenance Pack levels as long as the

Administration Server is at the same Maintenance Pack Level or higher than its Managed

Servers.

7

Domain Restrictions

Each domain requires its own Administration Server for performing management activities.

When you use the Administration Console to perform management and monitoring tasks, you

can switch back and forth between domains, but in doing so, you are connecting to different

Administration Servers.

All Managed Servers in a cluster must reside in the same domain; you cannot split a cluster over

multiple domains.

All Managed Servers in a domain must run the same version of the WLS software. The

Administration Server may run either the same version as the Managed Servers in the domain,

or a later service pack.

If multiple domains created, each domain must reference its own database schema. Domains

cannot share configured resources or subsystem. For example, if one creates a JDBC data

source in one domain, one cannot use it with a Managed Server or cluster in another domain.

Instead, one must create a similar data source in the second domain. Furthermore, two or more

system resources cannot have the same name.

3 Server

What is it?

A configured instance to host applications and resources

WebApps, Enterprise Apps, Web Services,

JMS, JDBC, Diagnostics,

What types of servers are there?

Administration Server

Managed Server

All instances of WLS use a root directory to store their working copy of the domain's

configuration files, to store runtime data, and to provide the context for any relative

pathnames in the server's configuration. An Administration Server always uses the domain

directory as its root directory. A Managed Server can use the domain directory but can also use

any other directory that you define.

Applications can use the following resources and services:

Security providers, which are modular components that handle specific aspects of

security, such as authentication and authorization.

8

Resource adapters, which are system libraries specific to Enterprise Information Systems

(EIS) and provide connectivity to an EIS.

Diagnostics and monitoring services.

JDBC data sources, which enable applications to connect to databases.

Mail sessions.

XML entity caches and registry of XML parsers and transformer factories

Messaging services such as JMS servers and store-and-forward services

Persistent store, which is a physical repository for storing data, such as persistent JMS

messages. It can be either a JDBC-accessible database or a disk-based file.

Startup classes, which are Java programs that you create to provide custom, system-

wide services for your applications.

Work Managers, which determine how an application prioritizes the execution of its

work based on rules you define and by monitoring actual runtime performance. You can

create Work Mangers for entire WebLogic Server domains or for specific application

components.

Work Contexts, which enable applications to pass properties to a remote context

without including the properties in a remote call.

4 Administration Server

What is it?

Central configuration controller for the entire domain

What else does it do?

Hosts the Administration Console

Enables you to start and stop servers from a central location

Enables you to migrate servers and services within the domain

Enables you to deploy applications within the domain

Guidelines:

There must be exactly one* Administration Server in domain

An Administration Server controls only one domain.

For production use, we recommend not hosting application logic or resources on

the Administration Server

9

The Administration Server does not need to run at all times, but is required for making

configuration and deployment changes to a running domain.

The Administration Server operates as the central control entity for the configuration of the

entire domain. It maintains the domain's configuration documents and distributes changes in

the configuration documents to Managed Servers. One can also use the Administration Server

as a central location from which to monitor all resources in a domain.

To interact with the Administration Server, one can use the Administration Console, WLST, or

custom JMX client.

Each WLS domain must have one server instance that acts as the Administration Server.

In each domain, one WLS instance acts as the Administration Serverthe server instance which

configures, manages, and monitors all other server instances and resources in the domain. Each

Administration Server manages one domain only. If a domain contains multiple clusters, each

cluster in the domain has the same Administration Server.

5 Managed Server

What is it?

A running instance that hosts applications and resources needed by those

applications - The real work horses in a WebLogic domain

Managed Servers host business applications, application components, Web

services, and their associated resources. To optimize performance, Managed

Servers maintain a read-only copy of the domain's configuration document.

Each Managed Server is independent of all other Managed Servers in the domain

(unless they are in a cluster, defined later)

You can have as many Managed Servers in a domain as you need

Individual Managed Servers are typically added for capacity and application

isolation

Managed Servers can use the following resources:

Machine definitions that identify a particular, physical piece of hardware. A machine

definition is used to associate a computer with the Managed Servers it hosts. This

information is used by Node Manager in restarting a failed Managed Server, and by a

clustered Managed Server in selecting the best location for storing replicated session

data.

10

Network channels that define default ports, protocols, and protocol settings that a

Managed Server uses to communicate with clients. After creating a network channel,

you can assign it to any number of Managed Servers and clusters in the domain.

Virtual hosting, which defines a set of host names to which WLS instances (servers) or

clusters respond. When you use virtual hosting, you use DNS to specify one or more

host names that map to the IP address of a server or cluster. One also specifies which

Web applications are served by each virtual host.

6 Administration Server to Managed Server Interaction

The Administration Server stores the master copy of the domain configuration, including

the configuration for all managed servers in the domain

Each Managed Server stores a local copy of its configuration.

When a Managed Server starts, it connects to the Administration Server to synchronize

the configuration

When configuration is changed, the Administration Server sends changed configuration

to Managed Servers

7 Cluster

What is it?

A cluster is a group of Managed Servers running simultaneously and working

together to provide increased scalability and reliability

Scalability: through parallelism

Reliability/Availability: through replication and redundancy

A cluster appears as a single instance to most clients.

Clusters enable some advanced features, such as Whole Server Migration,

Service Migration, and clustered JMS destinations.

The server instances that constitute a cluster can run on the same machine, or be located on

different machines. You can increase a cluster's capacity by adding additional server instances

to the cluster on an existing machine, or you can add machines to the cluster to host the

incremental server instances. Each server instance in a cluster must run the same version of

WLS Software.

How Does a Cluster Relate to a Domain?

11

A cluster is part of a particular WLS domain.

A domain is an interrelated set of WLS resources that are managed as a unit. A domain includes

one or more WLS instances, which can be clustered, non-clustered, or a combination of

clustered and non-clustered instances. A domain can include multiple clusters. A domain also

contains the application components deployed in the domain, and the resources and services

required by those application components and the server instances in the domain. Examples of

the resources and services used by applications and server instances include machine

definitions, optional network channels, connectors, and startup classes.

One can use a variety of criteria for organizing WLS instances into domains. For instance, one

might choose to allocate resources to multiple domains based on logical divisions of the hosted

application, geographical considerations, or the number or complexity of the resources under

management.

In each domain, one WLS instance acts as the Administration Serverthe server instance which

configures, manages, and monitors all other server instances and resources in the domain. Each

Administration Server manages one domain only. If a domain contains multiple clusters, each

cluster in the domain has the same Administration Server.

All server instances in a cluster must reside in the same domain; one cannot "split" a cluster

over multiple domains. Similarly, one cannot share a configured resource or subsystem

between domains. For example, if one creates a JDBC connection pool in one domain, one

cannot use it with a server instance or cluster in another domain. (Instead, one must create a

similar connection pool in the second domain.)

Clustered WLS instances behave similarly to non-clustered instances, except that they provide

failover and load balancing. The process and tools used to configure clustered WLS instances

are the same as those used to configure non-clustered instances. However, to achieve the load

balancing and failover benefits that clustering enables, one must adhere to certain guidelines

for cluster configuration.

Cluster Guidelines

All servers in a cluster must also be in the same domain.

All servers within a cluster must be at the same Maintenance Pack level.

Clustered servers can be on the same or different machines.

You can have multiple clusters in a domain.

Communication in a Cluster

Peer to Peer using Sockets - used for:

12

Accessing non-clustered objects deployed to another clustered server instance

on a different machine.

Replicating HTTP session states and stateful session EJB states between a

primary and secondary server instance.

Accessing clustered objects that reside on a remote server instance.

Peer to Peer using Unicast or Multicast - used for:

Cluster-wide JNDI updates

Heartbeats

Cluster-wide JNDI tree

Lists local resources and resources available throughout the cluster

List is maintained on all servers in the cluster

8 Node Manager

What is it?

Utility/process running on a physical server that enables you to start, stop,

suspend, and restart WebLogic Server instances remotely

Must run on each physical server that hosts WebLogic Server instances that you

want to control with Node Manager

Not associated with a domain. Can start any server instance that resides on the

same physical server.

Optional, but required to start/stop servers using the Administration Console

Required for Whole Server Migration and for some configurations of Automatic

Service Migration

Server instances in a WLS production environment are often distributed across multiple

domains, machines, and geographic locations. Node Manager is a WLS utility that enables one

to start, shut down, and restart Administration Server and Managed Server instances from a

remote location. Although Node Manager is optional, it is recommended for high availability

environment.

A Node Manager process is not associated with a specific WebLogic domain but with a machine.

One can use the same Node Manager process to control server instances in any WLS domain, as

13

long as the server instances reside on the same machine as the Node Manager process. Node

Manager must run on each computer that hosts WLS instances -- whether Administration

Server or Managed Server -- that one wants to control with Node Manager.

9 Machine

What is it?

A definition that identifies a particular, physical piece of hardware.

A machine definition is used to associate a computer with the Managed Servers it

hosts.

Used by Node Manager in restarting a failed Managed Server

Used by a clustered Managed Server in selecting the best location for storing

replicated session data

14

Exhibit 1: WebLogic Homes

15

AppendixB:AuditinginBPELPM

1 Audit levels

Although audit levels can be configured in various areas, they mostly (but not in every case) fall under one of these four levels:

Off: Absolutely no composite instance or payload information is collected. Although this

is the best in terms of performance, it severely limits the visibility as no information is

logged, rendering it an option that is not recommended in most cases. Instances are

created, but nothing is logged to the database or displayed on the console, which may

lead to difficulties in fault diagnosis and incident analysis.

Development: Both composite instance and payload information is collected. Though

this option provides the most detail, it is the worst performing of all the audit levels. It is

to be set in development environments for debugging purposes, but not in production

environments, except for transactions that specifically require that degree of auditing.

Audit levels set to development capture payloads throughout most activities, such as Assign and Transform

Production: Although composite instance information is collected, most payload

details are not. This is the typical setting for production environments and provides the

best balance between performance and visibility. If payload details need to be captured,

it is best to consider setting the audit level to development only for specific composites, components, or services, as we shall describe later.

Inherit: Audit levels are inherited from the parent level.

2 Order of precedence for audit level settings

Audit levels can be manipulated across each of these. It is possible to set the audit level at the

component level, the composite level, or even the service engine level. But if the audit level is

set at the composite level as development and at the SOA Infrastructure level as development, which one takes precedence?

At a high level, the order of precedence is as follows:

Component > Composite > Service Engine > SOA Infrastructure

For example, if the audit level is set to development at the composite level and production at the SOA Infrastructure level, setting at the composite level overrides that of the SOA

Infrastructure.

If the composite audit level is set to inherit, it will inherit the settings from the applicable service engine. If the service engine is also set to inherit, it will inherit the settings from the SOA Infrastructure.

16

As a best practice, set all audit level settings to inherit and controlling it at the SOA Infrastructure level. Then, as the need for different levels of auditing are required, start

manipulating the service engine, composite, and component audit levels as needed.

But as one start making changes in audit settings at different levels, simple level of order of

precedence become complex and murky.

Examples of Order of Precedence

Component Composite

Service

Engine

SOA

Infrastructure Who Takes Precedence?

No property Off Production Development Composite.

The audit level is set to Off. The service

engine and SOA Infrastructure audit

levels do not take effect.

No property Inherit Developme

nt

Production Service engine.

The audit level is set to Development.

The payload is shown in the assign

activity. The SOA Infrastructure audit

level does not take effect.

No property Inherit Inherit Production SOA Infrastructure.

The audit level is set to Production.

No property Inherit Production

/Developm

ent/Off/In

herit

Off The overall audit is not shown.

The composite inherits the audit level

from the SOA Infrastructure. The

payload is shown in the assign activity

based on the service engine audit level

setting.

Developmen

t

Off Production Development Composite.

Since the composite audit level is set to

Off, the overall audit is not shown. The

service engine audit level is shown, but

the Development setting for the

component takes precedence.

The payload is shown in the assign

activity based on the component audit

level setting of Development.

17

Component Composite

Service

Engine

SOA

Infrastructure Who Takes Precedence?

Inherit Off Production Development Composite.

Since the composite audit level is set to

Off, the overall audit is not shown. The

service engine audit level is not shown

because Off is inherited from the

composite.

Notes:

When the composite audit level is set to Off, there is no audit trail generated for this composite and all service engines used within the composite.

When the composite audit level is set to Inherit, it always inherits the settings of the

SOA Infrastructure.

When the composite audit level is set to Off, the component inherits the service engine settings.

18

AppendixC:AntiPatterns

While developing BPEL processes/composites, one must avoid few anti patterns from

performance perspective. List includes from design of BPEL Process/composite perspective as

well as other areas as well.

Synchronous Asynchronous

Over use of asynchronous processes

Over use of durable processes

No fault Handling

Synchronous Fault Handling

To many retries

Chatting BPEL Process (call back)

Over use of FlowN

Loops and more loops

Synchronous and Asynchronous processes on same managed server/cluster

Durable and transient processes on same managed server/cluster

Sticky load balancer

Not keeping aspect ratio: Increasing capacity for a component (say WLS) but not

increasing others (like database).

1 Synchronous Asynchronous

Never call an asynchronous process from a synchronous process directly (web service). It is a

very bad design decision. In this kind of design, calling process (synchronous one) will slow

down significantly.

Exhibit 2: Synchronous Asynchronous - 1

Situation become worse if calling process is synchronous and transient while called process is

asynchronous and durable.

19

Exhibit 3: Synchronous Asynchronous - 2

If synchronous to asynchronous call cannot be avoided, use some decoupling trick like

introducing JMS queue and Durable Asynchronous process in between.

2 Over use of asynchronous processes

By design, asynchronous processes interact with data base multiple times. This very nature of

asynchronous process makes them resource intensive.

In any deployment, excessive usages of asynchronous processes strain the available resources

and may affect performance of overall system.

3 Over use of durable processes

A durable process interacts with SOAINFRA schema multiple time which puts strain on WLS as

well as on database.

In any deployment, excessive usages of durable processes strain the available resources and

may affect performance of overall system.

4 No fault Handling

No fault handling in BPEL process affects the system at multiple levels. BPEL Engine consumes

resources to process fault, log files writing eats up I/O resources, increased size aand number of

log files may affect availability of disk space and increased interaction with SOAINFRA tables to

log increased number of audit information.

Do appropriate and sufficient fault handling.

5 Synchronous Fault Handling

One of the most common bad designs is synchronous fault handling. In case of fault,

synchronous fault handling will slow down the BPEL process.

Develop a separate set of BPEL processes to log faults and exceptions, also send notifications

and storage of fault data.

20

6 To many retries

Avoid large number of retries in case of exception/fault in BPEL processes. Most of the time retries does not help but loads the system excessively which even may lead to collapse of WLS.

While setting number of retries in BPEL process, always consider the consequences of downing of underlying WLS.

7 Chatting BPEL Process (call back)

As famous mathematician Pythagoras has said - Silence is better than unmeaning words. Any

chatting BPEL Process is burden on resources. Avoid call back if not required.

8 Over use of FlowN

Branches in flowN activity are executed serially in a single thread (that is, the Nth branch is

executed only after N-1 execution has completed). Execution is not completely parallel. This is

because the branches do not execute in concurrent threads in this mode. Instead, one thread

starts executing a flow branch until it reaches a blocking activity (for example, a synchronous

invoke). At this point, a new thread is created that starts executing the other branch, and the

process continues. This creates the impression that the flow branches are executing in parallel.

In this mode, however, if the flow branches do not define a blocking activity, the branches still

execute serially.

The very nature of FlowN slows down the BPEL PM and excessive use of FlowN slows down

whole system.

9 Loops and more loops

forEach activity executes similar to FlowN. While loop executes enclosed activities repeatedly.

Most of the time number of loops to be executed are unknown at design time which introduces

big unknown in estimating resources required for any deployment.

Avoid using loops and especially embedded loops inside loops.

10 Synchronous and Asynchronous processes on same managed

server/cluster

Avoid deployment of synchronous and asynchronous BPEL processes on same managed server.

Tuning of BPEL engine which is hosting synchronous processes is slightly different from BPEL

engine which is hosting asynchronous.

21

11 Durable and transient processes on same managed server/cluster

Avoid deployment of transient and durable BPEL processes on same managed server. Tuning of

BPEL engine which is hosting transient processes is slightly different from BPEL engine which is

hosting durable.

12 Sticky load balancer

In clustered environment, load balancer plays big role. If load balancer is sticky one (especially

IP sticky), it may fail whole premises of clustered environment. A sticky load balancer may load

few servers in cluster while leaving other with inadequate load. This imbalance will affect

overall performance of system.

13 Not keeping aspect ratio

Over the time period with change in load profile and due to non-technical reasons, one part of

infrastructures capacity will increase while other may cripple. For example increasing capacity

for a component (say WLS) but not increasing others (like database) will affect performance of

the system.

22

AppendixD:SQLQueries

1 EM Console SQL Queries

1.1 Recovery console queries

SELECT * FROM (SELECT /*+ FIRST_ROWS */ a.*, ROWNUM rnum FROM (SELECT MESSAGE_GUID AS a1, DLV_TYPE AS a2, CIKEY AS a3, CLUSTER_NODE_ID AS a4, CLUSTER_NODE_KEY AS a5, COMPONENT_NAME AS a6, COMPONENT_TYPE AS a7, COMPOSITE_LABEL AS a8, COMPOSITE_NAME AS a9, COMPOSITE_REVISION AS a10, CONV_ID AS a11, DOMAIN_NAME AS a12, ECID AS a13, EVENT_NAME AS a14, EXT_INT1 AS a15, EXT_STRING1 AS a16, EXT_STRING2 AS a17, HEADER_PROPERTIES_BIN_FORMAT AS a18, HEADERS_REF_ID AS a19, OPERATION_NAME AS a20, PARTNER_LINK AS a21, PRIORITY AS a22, PROPERTIES AS a23, RECEIVE_DATE AS a24, RECOVER_COUNT AS a25, STATE AS a26, TENANT_ID AS a27, CONV_TYPE AS a28, RES_SUBSCRIBER AS a29 FROM DLV_MESSAGE WHERE ((((COMPONENT_TYPE = ?) AND (STATE IN (?, ?))) AND (RECEIVE_DATE

23

1.3 Recent composite Instance query

SELECT /*+ FIRST_ROWS(40) */ ID, CONVERSATION_ID, HAS_ASSOC, PARENT_ID, UPDATED_TIME, CREATED_BY, ECID, TITLE, TEST_RUN_NAME, INDEX5, TEST_RUN_ID, INDEX3, TEST_SUITE, INDEX1, TEST_CASE, BATCH_INDEX, SOURCE_NAME, COMPOSITE_DN, SOURCE_TYPE, CREATED_TIME, SOURCE_ACTION_TYPE, INDEX6, SOURCE_ACTION_NAME, INDEX2, STATE, BATCH_ID, LIVE_INSTANCES, TAGS, STATE_COUNT, BUSINESS_STATUS, VERSION, INDEX4, PARTITION_DATE, UPDATED_BY, TENANT_ID FROM COMPOSITE_INSTANCE WHERE (CREATED_TIME >= :1 ) ORDER BY CREATED_TIME DESC

1.4 Instance tab page query

SELECT /*+ FIRST_ROWS(40) */ ID, CONVERSATION_ID, HAS_ASSOC, PARENT_ID, UPDATED_TIME, CREATED_BY, ECID, TITLE, TEST_RUN_NAME, INDEX5, TEST_RUN_ID, INDEX3, TEST_SUITE, INDEX1, TEST_CASE, BATCH_INDEX, SOURCE_NAME, COMPOSITE_DN, SOURCE_TYPE, CREATED_TIME, SOURCE_ACTION_TYPE, INDEX6, SOURCE_ACTION_NAME, INDEX2, STATE, BATCH_ID, LIVE_INSTANCES, TAGS, STATE_COUNT, BUSINESS_STATUS, VERSION, INDEX4, PARTITION_DATE, UPDATED_BY, TENANT_ID FROM COMPOSITE_INSTANCE WHERE (CREATED_TIME >= :1 ) ORDER BY CREATED_TIME DESC

1.5 Instance tab page search query based on name vs title query

SELECT /*+ FIRST_ROWS(50) */ ID, CONVERSATION_ID, HAS_ASSOC, PARENT_ID, UPDATED_TIME, CREATED_BY, ECID, TITLE, TEST_RUN_NAME, INDEX5, TEST_RUN_ID, INDEX3, TEST_SUITE, INDEX1, TEST_CASE, BATCH_INDEX, SOURCE_NAME, COMPOSITE_DN, SOURCE_TYPE, CREATED_TIME, SOURCE_ACTION_TYPE, INDEX6, SOURCE_ACTION_NAME, INDEX2, STATE, BATCH_ID, LIVE_INSTANCES, TAGS, STATE_COUNT, BUSINESS_STATUS, VERSION, INDEX4, PARTITION_DATE, UPDATED_BY, TENANT_ID FROM COMPOSITE_INSTANCE WHERE (TITLE LIKE :1 ) ORDER BY CREATED_TIME DESC

1.6 Fault and Rejected message tab page queries

There is one parent query fetches initial 40 records and then for each records its run another

child sql query.

1.6.1 Parent query

SELECT * FROM (SELECT /*+ FIRST_ROWS */ a.*, ROWNUM rnum from (SELECT f.CIKEY, f.NODE_ID, f.SCOPE_ID, f.COUNT_ID, f.FAULT_NAME, f.FAULT_TYPE, f.POLICY_NAME, f.POLICY_VERSION, f.POLICY_CATEGORY, f.POLICY_ECID, f.CREATION_DATE, f.MODIFY_DATE, f.MESSAGE, wi.LABEL, wi.CREATOR, wi.MODIFIER, wi.STATE, ci.ECID, ci.CMPST_ID, ci.DOMAIN_NAME, ci.COMPOSITE_NAME, ci.COMPONENT_NAME, ci.COMPOSITE_REVISION, ci.COMPOSITE_LABEL

FROM CUBE_INSTANCE ci, WI_FAULT f LEFT

JOIN WORK_ITEM wi ON f.CIKEY = wi.CIKEY AND f.NODE_ID = wi.NODE_ID AND f.SCOPE_ID = wi.SCOPE_ID AND f.COUNT_ID = wi.COUNT_ID WHERE ci.CIKEY = f.CIKEY AND ci.COMPONENTTYPE = ? AND ci.STATE != 9 AND (wi.STATE is null OR wi.STATE IN (9, 4, 13, 3)) ORDER BY f.CREATION_DATE DESC) a where ROWNUM < ? ) where rnum > ?

24

1.6.2 Child query

SELECT F.CIKEY, F.NODE_ID, F.SCOPE_ID, F.COUNT_ID, F.FAULT_NAME,

F.FAULT_TYPE,F.CREATION_DATE,F.MODIFY_DATE,

F.MESSAGE, WI.LABEL, WI.CREATOR,WI.MODIFIER,WI.STATE,CI.ECID,CI.CMPST_ID

FROM WI_FAULT F LEFT JOIN WORK_ITEM WI ON F.CIKEY = WI.CIKEY

AND F.NODE_ID = WI.NODE_ID AND F.SCOPE_ID = WI.SCOPE_ID AND F.COUNT_ID = WI.COUNT_ID

LEFT JOIN CUBE_INSTANCE CI ON F.CIKEY = CI.CIKEY WHERE F.CIKEY = ? AND F.NODE_ID = ?

AND F.SCOPE_ID = ? AND F.COUNT_ID = ?

bind => [2050981, BpInv1, BpSeq3.17, 2]

SELECT F.CIKEY, F.NODE_ID, F.SCOPE_ID, F.COUNT_ID, F.FAULT_NAME,

F.FAULT_TYPE,F.CREATION_DATE,F.MODIFY_DATE,

F.MESSAGE, WI.LABEL, WI.CREATOR,WI.MODIFIER,WI.STATE,CI.ECID,CI.CMPST_ID

FROM WI_FAULT F LEFT JOIN WORK_ITEM WI ON F.CIKEY = WI.CIKEY

AND F.NODE_ID = WI.NODE_ID AND F.SCOPE_ID = WI.SCOPE_ID AND F.COUNT_ID = WI.COUNT_ID

LEFT JOIN CUBE_INSTANCE CI ON F.CIKEY = CI.CIKEY WHERE F.CIKEY = ? AND F.NODE_ID = ?

AND F.SCOPE_ID = ? AND F.COUNT_ID = ?

2 Miscellaneous

2.1 Stored procedure to convert Blob in string create or replace function BLOBCONVERTER(B BLOB) return clob is c clob; n number; begin if (b is null) then return null; end if; if (length(b)=0) then return empty_clob(); end if; dbms_lob.createtemporary(c,true);

25

n:=1; while (n+32767

26

2.8 Find out the process instance from a conversation ID when there is no

instance number showing in the log file (BPEL Instance ID for a Times Out

Item)

Error

com.oracle.bpel.client.delivery.ReceiveTimeOutException: Waiting for response has timed out. The conversation id is LocalGUID:54s013a1a13963a8:-3e1457ad:279ee837833:-39ca. Please check the process instance for detail.

Solution

select ds.cikey from dlv_message dm, dlv_subscription ds where dm.properties like '%%' and dm.conv_id = ds.conv_id;

The cikey is the process instance ID.

2.9 Query to get audit details from audit_details table select UTL_COMPRESS.LZ_UNCOMPRESS(bin) from audit_details where cikey =

2.10 Query to get audit details from audit_trail table

Step 1: Ensure there is a function to bring together all blocks pertaining to an instance key and

then uncompress it (log column is RAW data type) and return as BLOB.

create or replace function get_Audit_Trail_Log(cikey in integer) return blob is

cursor cu_log(cikey_1 integer) is

select * from audit_trail atr where cikey = cikey_1 order by count_id;

bl blob;

begin

dbms_lob.createtemporary (bl, true);

for r1_log in cu_log(cikey)

loop

dbms_lob.append (bl,r1_log.log);

end loop;

return(bl);

end;

Step 2: use UTL_COMPRESS.LZ_UNCOMPRESS to uncompress the returned BLOB.

27

select UTL_COMPRESS.LZ_UNCOMPRESS(get_Audit_Trail_Log (cikey)) from cube_instance where cikey =

2.11 Query to get XML message with the given instance ID select xd.Document_ID, UTL_COMPRESS.LZ_UNCOMPRESS(xd.DOCUMENT) from XML_DOCUMENT_REF xdr left join XML_DOCUMENT xd on xdr.DOCUMENT_ID = xd.DOCUMENT_ID where xdr.COMPOSITE_INSTANCE_ID = ''

2.12 Query to get XML message with a given instance name select xd.Document_ID, UTL_COMPRESS.LZ_UNCOMPRESS(xd.DOCUMENT) from XML_DOCUMENT_REF xdr left join XML_DOCUMENT xd on xdr.DOCUMENT_ID = xd.DOCUMENT_ID where xdr.COMPOSITE_DN like '%%'

2.13 Query to get payload size of message select document_id , DOCUMENT_TYPE, DBMS_LOB.GETLENGTH(document) from XML_DOCUMENT

2.14 Query to get execution time of BPEL instances select composite_name COMPOSITENAME,A.CMPST_ID COMPOSITE_INSTANCE_ID,creation_date BEGIN_TIME,modify_date END_TIME,

(extract(day from modify_date - creation_date)*86400+ extract(hour from modify_date - creation_date)*3600+ extract(minute from modify_date - creation_date)*60+

extract(second from modify_date - creation_date)) duration_in_second,A.* from cube_instance A where state = 5 and creation_date

between TO_DATE('','YYYY-MM-DD HH24:MI:SS')and TO_DATE('','YYYY-MM-DD HH24:MI:SS')

and composite_name in (composite_name)

2.15 Query to get the execution time of BPEL instances and to find the parent

that has initialized the composite select qu.row_id,qu.ID,qu.composite_id,cu.STATUS,qu.created,qu.duration_in_second

from CUBE_INSTANCE cu,

(

select cmpinst.TITLE as row_id,cube.CIKEY as ID,cube.CMPST_ID as composite_id,

cube.CREATION_DATE as created,(extract(day from cube.modify_date - cube.creation_date)*86400 +

extract(hour from cube.modify_date - cube.creation_date)*3600 +

extract(minute from cube.modify_date - cube.creation_date)*60 +

extract(second from cube.modify_date - cube.creation_date)) duration_in_second

28

from CUBE_INSTANCE cube,COMPOSITE_INSTANCE cmpinst

where cube.CMPST_ID=cmpinst.ID and cube.COMPOSITE_NAME='COMPOSITE_NAME' and

cube.COMPONENT_NAME='COMPONENT_NAME' and cube.STATE>4 and

cube.CREATION_DATE between TO_DATE('', 'YYYY-MM-DD HH24:MI:SS')

and TO_DATE('', 'YYYY-MM-DD HH24:MI:SS')) qu

where

cu.CMPST_ID=qu.composite_id and cu.PARENT_ID=concat('bpel:',qu.ID)

2.16 Query to identify all the faults for the messages that were sitting in BPEL

Engine Level recovery as undelivered Invokes select wif.* from WI_FAULT WIF where wif.cikey IN

(select ci.cikey from Dlv_Message dlv, cube_instance ci where dlv.State = 0

and dlv.dlv_type = 1 and dlv.ecid= ci.ecid and dlv.conv_id = ci.conversation_id

and dlv.receive_date >TO_DATE(', 'YYYY-MM-DD HH24:MI:SS'))

29

AppendixE:BigorLargeorHugePages

A page, memory page, or virtual page is a fixed-length contiguous block of virtual memory, and

it is the smallest unit of data for the following:

memory allocation performed by the operating system for a program; and

transfer between main memory and any other auxiliary store, such as a hard disk drive.

Virtual memory allows a page that does not currently reside in main memory to be addressed

and used. If a program tries to access a location in such a page, an exception called a page

fault is generated. The hardware or operating system is notified and loads the required page

from the auxiliary store (hard disk) automatically. A program addressing the memory has no

knowledge of a page fault or a process following it. Thus a program can address more (virtual)

RAM than physically exists in the computer. Virtual memory is a scheme that gives users the

illusion of working with a large block of contiguous memory space (perhaps even larger than

real memory), when in actuality most of their work is on auxiliary storage (disk). Fixed-size

blocks (pages) or variable-size blocks of the job are read into main memory as needed.

A transfer of pages between main memory and an auxiliary store, such as a hard disk drive, is

referred to as paging or swapping.

It is not recommended to allocate too many Huge Pages. These pre-allocated pages can only

be used for shared memory. This means that unused Huge Pages will not be available for

other use than for shared memory allocations even if the system runs out of memory and

starts swapping.

Every Operating System has different way to configure Large Pages.

1 Linux

1. To enable Reserve memory to be used for large pages, executing following command:

echo n > /proc/sys/vm/nr_hugepages

Where n is the number of desired pages.

Execute this step as soon as possible after the machine has been started since

ongoing memory usage creates fragmentation and Linux might be unable to allocate the number of specified pages.

2. To enable the JRockit JVM to allocate large pages, make a hugetblfs file system available by using following command:

mount -t hugetlbfs nodev /mnt/hugepages

3. Grant the user executing the Java application read and write permission to the file

system. This can be achieved by either the mount command or with the chmod and chown commands.

30

2 Windows

1. As Administrator, give the user who will run the application the permission to lock

pages in memory by opening the Start menu and selecting:

Control Panel Administrative Tools Local Security Policy Local Policies User Rights Assignments Lock pages in memory.

2. Select Lock pages in memory.

3. Make enough free consecutive memory available by either logging off from computer

or rebooting it.

3 Solaris

Nothing has to be configured in the O/S to enable an application to use large pages.

4 Reference:

Page (computer memory) Retrieved July 26, 2013:

https://en.wikipedia.org/wiki/Page_(computer_memory)

31

AppendixF:ORA-01438:valuelarger

thanspecifiedprecisionallowed

Some time while running through the logs (SOA managed server out and diagnostic log file); one may encounter ORA-01438 error. This error is due to exiting bug in BPEL PM.

5 What is the error in logs?

Error while invoking bean "cube delivery": Exception not handled by the Collaxa Cube system.[[ an unhandled exception has been thrown in the Collaxa Cube systemr; exception reported is: "ORABPEL-00000

Exception not handled by the Collaxa Cube system. an unhandled exception has been thrown in the Collaxa Cube systemr; exception reported is: "java.sql.SQLDataException: ORA-01438: value larger than specified precision allowed for this column

at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:83) at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:135) at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:210)

6 Effects

Server spends additional processing cycles on logging errors, which results in degraded

performance. Apart from performance issue, lots of disk space will be utilized by log files.

7 Cause

This is an existing bug (12621337: ORA-01438 ON LIVE_INSTANCES COLUMN). The issue is with

the precision of the column COMPOSITE_INSTANCE.LIVE_INSTANCES. The currently defined

data type is NUMBER(3) which can at the most hold a value up to 999.

32

8 Solution

The solution is simple, increase precision of column to NUMBER(38).

Steps to apply solution:

1. Stop the SOA domain

2. Log in to the SOAINFRA schema as SYSDBA

3. Modify the COMPOSITE_INSTANCE table to increase precision

alter table soainfra.composite_instance modify (live_instances NUMBER(38))

33

AppendixG:LOBsintheSOAINFRA

schema

TABLE LOB COLUMN

BPM_CUBE_ACTIVITY_INSTANCE ERRORMESSAGE

BPM_CUBE_ACTIVITY SOURCECODE

BPM_CASE_DATA DATA

BPM_AUDIT_QUERY AUDIT_COMMENT

BPM_ACTIVITY_INSTANCE ERRORMESSAGE

BPM_ACTIVITY SOURCECODE

SOAQTZ_TRIGGERS JOB_DATA

BPM_CUBE_PROCESS DEPLOYMENTINFO

BPM_CUBE_AUDITINSTANCE BUSINESSINDICATORS

B2B_WIRE_MESSAGE ERROR_TEXT_CLOB

B2B_WIRE_MESSAGE ERROR_DESCRIPTION_CLOB

B2B_TRANSPORT_MANAGER STATS

B2B_EXT_BUSINESS_MESSAGE PROTOCOL_WORK_AREA

B2B_EXT_BUSINESS_MESSAGE ERROR_TEXT_CLOB

B2B_EXT_BUSINESS_MESSAGE ERROR_DESCRIPTION_CLOB

B2B_EXT_BUSINESS_MESSAGE ERROR_DETAIL_CLOB

MDS_LARGE_ATTRIBUTES ATT_LONG_VALUE

MDS_METADATA_DOCS MD_CONTENTS

MDS_STREAMED_DOCS SD_CONTENTS

WFRULEDICTIONARY DICTIONARY

WFROUTINGSLIP ROUTINGSLIP

WFMESSAGEATTRIBUTE BLOBVALUE

WFHEADERPROPS PROPERTIES

WFEVIDENCE SIGNATURE

WFEVIDENCE PLAINTEXT

WFCERTIFICATEREVOKED VALIDATIONDATA

WFCERTIFICATE CERTIFICATE

WFATTACHMENT CONTENT

BPELNOTIFICATION MESSAGE

PC_TASKPAYLOAD PAYLOAD

PC_TASKATTACHMENT CONTENT

XML_DOCUMENT DOCUMENT

TEST_INSTANCE RESULTS

REJECTED_MSG_NATIVE_PAYLOAD PAYLOAD

REJECTED_MESSAGE ERROR_MESSAGE

REJECTED_MESSAGE STACK_TRACE

REFERENCE_INSTANCE ERROR_MESSAGE

34

REFERENCE_INSTANCE STACK_TRACE

COMPOSITE_INSTANCE_FAULT ERROR_MESSAGE

COMPOSITE_INSTANCE_FAULT STACK_TRACE

ATTACHMENT ATTACHMENT

EDN_LOG_MESSAGES MSG

EDN_EVENT_ERROR_STORE PAYLOAD

BPM_PML_HS METADATA_CHANGE

BPM_PML_HS CHANGES

WLI_QS_REPORT_DATA DATA_VALUE

MEDIATOR_PAYLOAD BIN

MEDIATOR_CASE_INSTANCE FAULT_OBJ

MEDIATOR_CASE_DETAIL AUDIT_TRAIL

MEDIATOR_AUDIT_DOCUMENT DOCUMENT

WFUSERTASKVIEW DEFINITION

BPM_USERAPPLICATIONDATA APPLICATIONDATA

BPM_PRESENTATION DEFINITION

B2B_BAM_QTAB USER_DATA"."HEADER"."PROPERTIES

B2B_BAM_QTAB "USER_DATA"."TEXT_LOB"

B2B_BAM_QTAB USER_PROP

VARIABLE_SENSOR_VALUES BLOB_VALUE

VARIABLE_SENSOR_VALUES CLOB_VALUE

FAULT_SENSOR_VALUES MESSAGE

COMPOSITE_SENSOR_VALUE CLOB_VALUE

COMPOSITE_SENSOR_VALUE BLOB_VALUE

ACTIVITY_SENSOR_VALUES ERROR_MESSAGE

BRDECISIONUNITOFWORK UOW_DATA

BRDECISIONINSTANCE DECISION_TRACE

BRDECISIONFAULT MESSAGE

CUBE_SCOPE SCOPE_BIN

AUDIT_DETAILS BIN

EDN_OAOO_DELIVERY_TABLE SYS_NC00032$(*)

EDN_OAOO_DELIVERY_TABLE USER_PROP

EDN_EVENT_QUEUE_TABLE SYS_NC00032(*)

EDN_EVENT_QUEUE_TABLE USER_PROP

TASK_NOTIFICATION_Q_T USER_DATA."HEADER"."PROPERTIES

TASK_NOTIFICATION_Q_T USER_DATA."TEXT_LOB"

TASK_NOTIFICATION_Q_T USER_PROP

B2B_APP_MESSAGE APP_MESSAGE_PROPS

B2B_APP_MESSAGE ERROR_TEXT_CLOB

B2B_APP_MESSAGE ERROR_DESCRIPTION_CLOB

B2B_DATA_STORAGE CLOB_VALUE

B2B_DATA_STORAGE BLOB_VALUE

AG_INSTANCE MILESTONE_STATE

35

BPM_PROJECTSHAREDATA VALUEBLOB

SOAQTZ_JOB_DETAILS JOB_DATA

SOAQTZ_CALENDARS CALENDAR

SOAQTZ_BLOB_TRIGGERS BLOB_DATA

WI_FAULT MESSAGE

TEST_DETAILS TEST_RESULT

TEST_DEFINITIONS DEFINITION

36

AppendixH:AWR,ADDM,&ASHReports

Database has Automatic Workload Repository (AWR), Automatic Database Diagnostic Monitor

(ADDM) and Active Session History (ASH) to gather and analyze database performance statistics

which helps optimize database and SQL statements.

1 AWR Report

AWR report provides performance summary that include:

CPU bottlenecks

Undersized Memory Structures

I/O capacity issues

High load SQL statements

High load PL/SQL execution and compilation, and high-load Java usage

Oracle RAC specific issues

Sub-optimal use of Oracle Database by the application

Database configuration issues

Concurrency issues, Hot objects

2 ADDM Report

ADDM report provides information pertaining to areas of database which are consuming most

time:

Lock contention

Excessive parsing

I/O capacity

Incorrect sizing of SGA, PGA

ADDM report also provides:

Recommends solutions and quantifies expected performance benefits.

Recommends changes to hardware, database configuration, database schema, or

applications

Estimated impact on performance

3 ASH Report

ASH report provides information about:

37

Historical view of active sessions Sampled every second

Top User Events (frequent wait events)

Load Profile

Details to the wait events

Top Queries and PL/SQL

Top Sessions

Top Blocking Sessions

Top DB Objects (Objects/Files/Latches)

Activity Over Time

Enables after the fact analysis

Drilldown to a more granular period

P1, P2, P3 values

List details of only a session, SQL, Wait class service, module over a period of a time

Determines blocking session, hot segment

ASH is helpful when there's a sudden performance degradation of the database.

4 AWR Report Analysis

AWR Report has various sections but from performance perspective take a deep dive in

following sections:

SQL statements ordered by Elapsed Time

SQL statements ordered by CPU Time

SQL statements ordered by Gets

SQL statements ordered by Reads

SQL statements ordered by Executions

SQL statements ordered by Parse Calls

4.1 SQL statements ordered by Elapsed Time

This section lists top SQL statements which has large elapsed time (CPU time plus any other

wait times) in descending order of Total Elapsed time. Check out the SQL statements which are

taking large time to execute. Take help from DBA to find out the reason.

If the query execution is taking long time, the JDBC calls made from SOA will wait till it gets the

response back which could cause stuck threads, timeout issues etc.

4.2 SQL statements ordered by CPU Time

Most of the times the SQL statements from the Elapsed Time list also find their slot as top SQL

statements in the CPU Time list as well. But it's not in all cases. If the SQL exists in Elapsed Time

list but not in CPU Time list, then there is an issue with recursion associated with this SQL

statement.

38

Larger CPU time usually caused by excessive function calls, missing filter columns in the SQL

statement which results full table scan.

If SQL statement under consideration is select type then indexing might help to improve

performance.

4.3 SQL statements ordered by Gets

SQL statements listed in this section due to large logical reads from DB Block buffers. Usually

reading from logical read is desirable but excessive reading from logical also can cause

performance issues.

High logical read results due to missing indexes on the filter columns or full table scan. Setting

proper filter columns and/or indexes might help to improve performance by reducing the

logical read time.

4.4 SQL statements ordered by Reads

SQL statements listed in this section due to larger disk reads (physical read) rather than reading

from buffers. Disk reads are undesirable. Excessive disk reads can cause performance issues.

Usually larger disk reads is due to missing indexes.

To improve performance, increase the buffer cache (If the memory is not a constraint at

database server) and/or tune the query with additional filters and/or tune table by adding

additional indexes.

4.5 SQL statements ordered by Executions

SQL statements listed in this section are executed most number of time in the given time

period. Usually it is OK if application design or load pattern warrants repeated execution of few

SQL statements execution.

Ensure that SQL statements listed here must not be in above lists. SQL statements listed here

must be very efficient.

4.6 SQL statements ordered by Parse Calls

SQL statements listed in this section are parsed/re-parsed repeatedly. Whenever a SQL

statement contains unsafe bind variables, then it get parsed each time. It leads to larger

execution time.

Check for SQL statements listed here and in Elapsed Time list as well. If one is found in both lists

then investigate, why this statement is parsed multiple times and whether it's normal.

39

5 Reference

Using Automatic Workload Repository for Database Tuning: Tips for Expert DBAs:

http://www.oracle.com/technetwork/database/manageability/diag-pack-ow09-

133950.pdf

Oracle Database Performance Tuning Guide:

http://docs.oracle.com/cd/B28359_01/server.111/b28274/toc.htm

Beginning Performance Tuning: Active Session History:

http://www.oracle.com/technetwork/issue-archive/2013/13-jan/o13dba-1871177.html

Oracle Database 2 Day + Performance Tuning Guide:

http://docs.oracle.com/cd/B28359_01/server.111/b28275/toc.htm

40

AppendixI:MonitoringScripts

In a typical deployment, during LnP (even in production) test few of the resources need to be

monitored. One can take help of reference scripts to monitor few of the critical resources.

Before using these monitoring scripts, one should be aware that these scripts consume

resources which will affect performance of the system.

1 Database Monitoring

This script monitors database connections in use at a given time. One can schedule this script to

run continually (say with gap of 5 min) during LnP test.

Later, one can compile the report from Email and log files generated and perform trend analysis

with load pattern.

Exhibit 4: Database Monitoring

SQL files

SQL files lists statements which will be executed by sh file periodically.

dbmon.sh

This file does heavy lifting by invoking SQL files, sending emails and writing logfiles.

ORACLE_HOME: Set Oracle home in place of

41

LD_LIBRARY_PATH: Put down path of LD_LIBRARY in place of

PATH: Ensure that Oracle database client library is in path. Replace

with location of Oracle database client library

DBMON_HOME: This is home for this script. Replace

with location where monitoring scripts are

kept.

dbmon.properties

DBNAME: Database name. Replace with database name.

DBHOST: Database host name. Replace with database host name.

DBPORT: Database listening port. Replace with database listening port

number.

DBSERVICE: database Service name. Replace with database

service name.

DBUSER: Database user name. Replace with database user name.

DBPASSWORD: Database password. Replace with database password.

SOAENV: SOA environment, under consideration. Replace > with SOA

environment under consideration.

MAILFROM: From where e-mail has been sent. Replace [email protected]

with an e-mail ID who should send this mail.

MAILTO: Comma separated list of e-mail IDs to whom email should be sent. Replace

[email protected],[email protected] with comma separated list of e-mail

IDs.

MAILCC: Comma separated list of e-mail IDs to whom email should be sent as

carbon copy reciepient. Replace

[email protected],[email protected] with comma

separated list of e-mail IDs.

DBSESSIONCOUNTTHRESHOLD: Threshold for session count. Replace

with positive integer value.

output.html and dbmon.log

These files will store the log information HTML and text format in logs folder.

2 JMS Monitoring

This script monitors pending messages on JMS queue under consideration. One can schedule

this script to run continually (say with gap of 5 min) during LnP test.


with load pattern.

42

Exhibit 5: JMS Monitoring

There are couples of config files which need to be modified as required. But lets talk about

jmsmon.sh.

jmsmon.sh

43

In this file one need to replace with

location where monitoring scripts are kept.

Files in config and sub folder

jmsmon.properties

This file contains details of JMS server. Queues on this JMS server will be monitored.

hostname: Host name of computer where JMS server is hosted. Replace

with hostname.

portString: Port number where JMS server is listening. Replace with

port number.

Username: User name of JMS server. Replace with user name.

Password: Password of JMS server. Replace with password.

MailTo: Email ID where email should be sent in case of failure. Replace

with Email ID.

Servers: Semicolon separated list of servers, if JMS has cluster. Replace

= with semi-colon separated list of

servers. For example JMSServer1;JMSServer2;JMSServer3.

monitorconfig.csv

This file contains list of queues (comma separated) which need to be monitored. Here one

needs to list each queue with value for:

QueueName

WarningThreshold-LC

CriticalThreshold-LC

WarningThreshold-PendingMsgCount

CriticalThreshold-PendingMsgCount

WarningThreshold-CurrentMsgCount

CriticalThreshold-CurrentMsgCount

domains.csv

This file contains list of (comma separated):

Partition

Service Name

Weblogic Domain or System Name

Destination Name

destinations.emailcc

44

This file lists email IDs with respect to each domain under consideration.

mailcctmp

This file lists domains for which carbon copy (cc) mail is sent.

dntmailtmp

This file lists domains for which carbon copy (cc) mail is not sent.

bin

This folder contains java code. This java code is tested for three server cluster. One may need to

modify if there different number of servers in cluster.

lib

This folder contains java libraries required by java code.

logs

This folder will contain log file.

3 AQ Monitoring

This script monitors pending messages on JMS queue under consideration. One can schedule

this script to run continually (say with gap of 5 min) during LnP test.


with load pattern.

This script is consists of few file which are dedicated to do specific work. Let us go through each

piece one by one.

45

Exhibit 6: AQ Monitoring

DB_1_aq_monitor.sql and DB_2_aq_monitor.sql

These files contain SQL statements which query AQ tables. These SQL statements are executed

by commands listed in aqmon.sh file.

There are two sample files but one can add as much as many required. This count is determined

by number of databases to be connected for AQ monitoring.

In each sample file set SQL queries are listed as per the AQs to be monitored in this schema. In

DB_1_aq_monitor there are 3 SQL statements while in DB_2_aq_monitor has two. It also

means that first database has three AQs to be monitored while second database has two.

Add SQL files as required and also keep SQL statements in each files as required.

In the sample files replace with schema name where AQ tables are. Also

replace with AQ table.

aqmon.sh

This file contains script which picks up SQL statements from DB_1_aq_monitor.sql and

DB_2_aq_monitor.sql files, write log files and sends email.

This file should be called from scheduler (or cron job). In this file one need to modify certain

values as per deployment under consideration.

46

ORACLE_HOME: Set Oracle home in place of

LD_LIBRARY_PATH: Put down path of LD_LIBRARY in place of

PATH: Ensure that Oracle database client library is in path. Replace

with location of Oracle database client library

AQMON_HOME: This is home for this script. Replace

with location where monitoring scripts are

kept.

To add new databases, refer header comments in file.

aqmon.properties

This file contains details of database to be connected and e-mail details.

DBNAME_1: Database name. Replace with database name.

DBHOST_1: Database host name. Replace with database host name.

DBPORT_1: Database listening port. Replace with database listening port

number.

DBSERVICE_1: database Service name. Replace with database

service name.

DBUSER_1: Database user name. Replace with database user name.

DBPASSWORD_1: Database password. Replace with database

password.

On similar line update DBNAME_2, DBHOST_2, DBPORT_2, DBSERVICE_2,

DBUSER_2, and DBPASSWORD_2.

To add new database details add DBNAME_3, DBHOST_3, DBPORT_3, DBSERVICE_3,

DBUSER_3, and DBPASSWORD_3.

SOAENV: SOA environment, under consideration. Replace > with SOA

environment under consideration.

MAILFROM: From where e-mail has been sent. Replace [email protected]

with an e-mail ID who should send this mail.

MAILTO: Comma separated list of e-mail IDs to whom email should be sent. Replace

[email protected],[email protected] with comma separated list of e-mail

IDs.

MAILCC: Comma separated list of e-mail IDs to whom email should be sent as

carbon copy reciepient. Replace

[email protected],[email protected] with comma

separated list of e-mail IDs.

aq_monitor.html and aqmon.log

These files will store the log information HTML and text format in logs folder.

48

AppendixJ:HowtoMonitorSOAServer

MemoryUsage

During LnP test monitoring of SOA server for JVM memory usage, garbage collection, memory

related error is very important because of obvious reasons. After LnP analysis of memory usage,

heap analysis and thread dump analysis are required to take a deep dive to understand causes

and solution of issues.

For this purpose, there are several tools but following might be sufficient from WLS and BPEL

PM perspective.

jConsole

visaulVM or JvisualVM

JRockit Mission Control (JRMC)

1 Setup: jConsole or visualVM (installed locally)

Since most of the time BPEL PM and other servers might be running on servers which do not

have any window component, so running UI application is not possible. In this case one needs

to export display to a computer which has window component. To do this follow following

steps:

a. Ensure that jConsole or visualVM or/and JRockit Mission control is/are installed on the

computer which need to be monitored.

b. Ensure that target machine where one needs to open UI has putty and XMing installed

c. Start XMing on target machine

d. Configure Putty:

In the box below Host Name (or IP Address) key-in server name or IP address.

Make sure SSH is selected.

Key-in server name or IP address in the box below Saved Sessions.

49

Exhibit 7: Setup jConsole or visualVM (installed locally) - 1

Navigate till Connection SSH X11 and click on X11

50


Select Enable X11 forwarding check box.

Scroll the left hand window back to the top and click on the Session heading

51


Click the Save button. The host name entered should now appear below Default

Settings. In the future, one will be able to connect by simply double-clicking this host

name.

Click the Open button.

In case of PuTTy Security Alert window click on Yes

SSH window should appear. Provide credentials and navigate to folder where

jConsole or visualVM is present.

Start jConsole (./jconsole &) or visualVM (./visualvm &) as background process

Find out process ID (PID) of the server to be monitored

By this time jConsole or visualVMs UI should be available at target machine.

In jConsole or visualVM, using PID monitor Heap usage and garbage collection.

In ideal scenario, one should get saw tooth shaped memory usage and garbage collection

pattern.

52


2 Setup: JvisualVM (installed at remote machine)

JVisualVM comes with HotSpot JVM and can be found at $HotSpot_JDK_INSTALL/bin.

JVisualVM should be installed on separate machine to minimize the monitoring overhead. On

the runtime server, one needs to provide the JMX parameter to JVM.

To configure a JMX port, one should edit setSOADomainEnv.sh file in

$Middleware_Home/user_projects/domains//bin to add the -

Dcom.sun.management.jmxremote parameters.

Define this parameter for a specific server in the domain since reuse of the JMX port number

between servers is not allowed.

if [[ "${SERVER_NAME}" = "soa_server1" ]] then PORT_MEM_ARGS="-Xms1024m -Xmx2048m -XX:PermSize=256m -XX:MaxPermSize=512m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9004 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" echo "DEBUG! (soa_server1) USER_MEM_ARGS: ${USER_MEM_ARGS}" fi

Server needs restart, for this parameter to take into effect.

53

To start jVisualVM GUI, execute

./jvisualvm -J-XX:MaxPermSize=256m

from $HotSpot_JDK_INSTALL/bin.

To ensure that jVisualVM installation has required plugins, download all plugins.

Select Tools Plugins download and install all of the available plugins.

Restart JVisualVM.

Select the Add Remote Host icon from menu.

Key in the Host name and Display name where the SOA managed server is running and select

OK.

Right click the new Remote connection and select Add JMX Connection...

In the Connection field enter the defined JMX port of 9004, this is from -

Dcom.sun.management.jmxremote.port=9004, also check the Display name box.

Select OK, open the + to the left of the Remote connection and double click the JMX

connection, verify the following tabs are displayed.

Note: It is possible that the Visual GC tab will only show the message "Not supported for this

JVM." You will get this if the JDK running JVisualVM and the SOA server are not the same

version or if the operating systems do not match. One way to get this tab to display is to run

JVisualVM from the same JDK as the SOA server.

After double left clicking the SOA managed server , verify the following tabs are displayed.

Monitor Tab

It provides the graphs of the running system.

Also there is a Perform GC button to force a garbage collection and a Heap Dump button to

cause a heap dump. The resulting heap dump will be eventually loaded into JVisualVM in a new

tab where it can be analyzed. The loading may be slow and not detailed enough.

One can take heap dump from the command line from the JDK installation where the SOA

server is running:

$HotSpot_JDK_INSTALL/bin/jmap -dump:live,file=heap.dump.out,format=b

54

The resulting heap dump file *.out can be viewed from Eclipse Memory Analyzer (MAT).

Visual GC Tab

It provides a visual representation of the memory Spaces being used in real-time in the

PermGen, Old Gen, Eden Space, and Survivor Spaces (S0 & S1). This gives an idea of how how

full each partition is at any given time. One can use this to scale the defined memory for the

spaces based on actual load.

The Graphs section also provides information on the maximum and current sizes of the spaces

and their garbage collection statistics.

Tracer Tab

It allows to choose which metrics to keep track of in a running graph. After selecting the Start

button the graphing of each metric starts.

After desired time tracing can be stopped and can be exported as .csv file ( Select floppy disk

button "Export all data").

3 Setup: JRockit Mission Control (installed at remote machine)

JRMC is used to monitor JRockit JVM. JRMC should be installed on a separate machine from the

SOA runtime server to minimize the monitoring overhead. On the runtime server, one need to

setup the JMX monitoring port.

In 11g SOA server installation one can edit the setSOADomainEnv.sh file in $Middleware_Home/user_projects/domains//bin to add the -Xmanagement parameters. Add following to this file:

if [[ "${SERVER_NAME}" = "soa_server1" ]] then PORT_MEM_ARGS="-Xms2048m -Xmx2048m -Xmanagement:ssl=false,authenticate=false,port=7093" echo "DEBUG! (soa_server1) USER_MEM_ARGS: ${USER_MEM_ARGS}" fi

These parameters must be defined for a specific server in the domain since reuse of the JMX

port number between servers is not allowed.

The -Xmanagement parameters will setup the JVM for the JRMC connection. To come in effect

these parameters, soa_server1 need to be restarted.

55

Start JRMC. Right click the Connectors folder and select New Connection:

Fill out the Host of the SOA server and the Port as defined in -Xmanagement:port=7093,

selecting "Test Connection" should show OK

Select Finish and the new connection should appear under Connectors, highlight the new

connection and select to start a console which will monitor the JVM in real-time.

This will depict memory, CPU usage, threads, and methods are being used.

memleak depicts the type of java objects present, growth rate, percentage of heap, instances,

and total size from largest to smallest. This view helps in determining which objects are taking

up the most memory:

To keep record of for comparison purpose, use Flight recording option. Data is kept in .jfr file.

4 Reference visualVM: http://visualvm.java.net jConsole: http://docs.oracle.com/javase/6/docs/technotes/guides/management/jconsole.html Oracle Fusion Middleware Performance and Tuning for Oracle WebLogic Server:

http://docs.oracle.com/cd/E21764_01/web.1111/e13814/wldf.htm

Oracle JRockit Mission Control Introduction to Mission Control Client:

http://docs.oracle.com/cd/E15289_01/doc.40/e15067/toc.htm

Oracle JRockit Documentation: http://docs.oracle.com/cd/E15289_01/index.htm

Putty: http://www.putty.org XMing: http://sourceforge.net/projects/xming

56

AppendixK:HeapDumpFilesanalysis:

JRockitandHotSpotJVMs

Eclipse Memory Analyzer (MAT) and Java Heap Analysis Tool (jhat) can be used to analyze of

heap dump of JRockit and HotSpot.

Heap dump files from JRockit and HotSpot has following name format:

JRockit: jrockit_.hprof

HotSpot: java_pid.hprof

One can download MAT from eclipse web site and jhat can be found at

$JDK_INSTALL_LOCATION/bin/jhat.

A heap dump file usually is larger in size than the maximum heap size defined for the server in

the -Xmx parameter. The large heap dump file may exceed the memory capacity of MAT. In this

scenario MAT may run out of memory or MAT may truncate the file or corrupt the file.

In this scenario, configure MAT to run with more memory by editing the MemoryAnalyzer.ini

file. To configure MAT to run with 7GB of memory, just add following lines to the file:

-vmargs -Xmx7168m

Restart MAT.

Maximum memory setting for MAT depends upon:

Operating system (32 or 64 bit)

Physical RAM available (installed and sharing with other applications running)

1 Example Analysis of a Heap Dump File Using Eclipse Memory

Analyzer

Here is a practical example of a heap dump collected at the time of out-of-memory using a

HotSpot JVM when opened using Eclipse Memory Analyzer Version 1.0.1. This was collected

from an 11g SOA installation where Microsoft SQLServer Database was being used as the

57

dehydration store.

The problem in this case was that the Data Direct Microsoft SQLServer JDBC Drivers caused a

memory leak in the container. After some time the leak caused the java heap allocation to run

out of memory. The solution was to change from the Data Direct SQLServer JDBC Drivers to the

Microsoft JDBC Drivers.

Analysis of the heap dump in Eclipse Memory Analyzer led to the conclusion that there was a

problem with the Drivers and that a change in Drivers would most likely resolve the memory

leak problem.

These are some screen shots from the heap dump analysis done in Eclipse Memory Analyzer

(MAT) for this case.

A. Open the *.hprof file in MAT: File -> Open File -> java_pid8368.hprof:

Exhibit 11: Heap Dump Files analysis - 1

B. Select "Leak Suspects Report" and then click on Finish:

58


C. This will produce an Overview Tab of the Biggest Java Objects by Retained Size and lists of Actions and Reports:


D. The other tab is "default_report org.eclipse.mat.ami:suspects" which contains the Leak Suspects. This shows that MAT has detected two leak suspects:

59


If there are no leaks suspects are shown then there are three possibilities:

There are no leak suspects

Max. heap size is not large enough

Heap size is not tuned

To confirm one of the possibilities, one should try some alternative settings of JVM

E. Scrolling Down on the Same Tab Shows More Detail on the Leak Suspects:

60


These are two instances of the same Leak Suspect. The "weblogic.sqlserverutil.ddo" is a hint that the

problem is related to a MSSQL Server database connection.

F. The idea that the problem is with a MSSQL Server Database Connection is reinforced by selecting the details link and then opening the "Shortest Paths to the Accumulation Point":

61


This looks to point to a SQLServer data source. In a SOA installation a SQLServer datasource may be

allocated for use by a databa

bpel pm 11g performance tuning- appendies

Documents

administration server

managed server

server interaction

weblogic server overview

use of asynchronous

appendix b

contents appendix

appendix c