oracle ebs r12.2 - deployment and system administration

InfraOptimization

A Comprehensive Training on Oracle E-Business Suite R12.2.x: Deployment and System Administration

Presented by: Md Mozammel Hoque , CEO & BPO Consultant at InfraOptimization

Hosted by: PHP Family

Expected Training Outcome Minimizing EBS R12.2.n Maintenance Downtimes followed by Oracle & ITIL

IT Operation Excellence

InfraOptimization

Name Md Mozammel HoqueDesignation CEO & BPO Consultant at InfraOptimization

Career at a glance 17 years+ as Executive leader, Consultant, Researcher, Enterprise Architect, Program Manager, and Software Engineer in Telco, Banking, Manufacturing and Retail Industries. He focuses on customer experience, Business strategy and Process Optimization, innovation, leadership, and improved operational efficiency. He supported more than 70 Clients (on site and off-shore) with hundreds of projects Implementer and Designer on Oracle ERP, Telco Billing, DWH, Campaign Management, Fraud and Revenue Management, Security and Audit, covering North America, Europe, West Africa and Asia. He worked as (Assistant) Researcher on the field of Socio-Anthropology, and had teaching experience.

He has served as CEO and BPO Consultant at InfraOptimization, as Management & Strategy Consultant at E-Tec Ltd., as CTO at NTS, as Sr. Technical Consultant at Huawei (Base Zone: West Africa), as Oracle Trainer at IBCS-PRIMAX and as APPS DBA at DatabaseIntelligence (Russia) on behalf of DataIntensity (USA).

Publication A number of Technical and Non-Technical articles were published in 3MS/HUAWEI Case Center.Introduced 7 RIGHTS concept for better business output, which was disclosed (and organized by) at the Faculty of Business Administration, University of Chittagong, Bangladesh on May 25, 2016.

Public Speaker He has been the speaker on Optimization: Business and IT hosted by the University of Chittagong, Bangladesh and IBCS-Primax (Dhaka, Bangladesh).

Professional Certification Oracle Certified Expert (OCE) - Oracle RAC 11g Release 2 and Performance Tuning, Oracle Certified Specialist (OCS) - Oracle Database 11g Data Warehouse, Oracle Certified Professional (OCP) - R12, 11g, 10g and 9i.ITIL V3 Foundation Certified from EXIN. Passed Total 31 exams with all distinct result i.e. Database, Operating System, Network, Security, OCS/CBS, Storage, ITIL etc. in HUAWEI exam system.

Reward Honorable Performance Reward for the year of 2013, Best performance award for the year of 2012 for Etisalat Nigeria project from Huawei, and Two times best performance award (2010 and 2011) from Zain (now Airtel) Telco in Nigeria on behalf of International Turnkey System.

Education MSC and BSC in Applied Informatics in Economics from Rostov State University of Civil Engineering, Russia in 2006 MA in 1997 and BA in 1996 in History from the University of Dhaka, Bangladesh

Languages known Bengali, English, Russian, and French

Presenter Profile

Our Philosophy

Learn from Learners & Feel it.

- What to Know | Do

- Why to know | Do

- How to Know | Do

InfraOptimization

Process:Key Learning Principle

Procedure

InfraOptimization

Content of Training

Pathway to Enterprise Architecture: Business Architecture & IT Architecture EBS R12.2.x Technology Architecture & Roadmap Pathway to Design Label - Deployment, Implementation & Upgrade System Administration on - Install, Patch, Clone, Concurrent Manager, Autoconfig, Fusion Middleware , Weblogic & Maintenance Tools. Pathway to Performance Tuning & Troubleshooting IT Service Continuity Management: Beyond DR (Disaster Recovery)

Forms of Training

Lecture: Concept Lab: Practical Case Study: Live & Past

InfraOptimization

Applications Technology Release 12.2 Value Proposition

Oracle E-Business Suite Applications Technology provides the technology infrastructure to develop, manage, and integrate Oracle E-Business Suite applications.

The Applications Technology team incorporates and adapts products from Oracle Database and Oracle Fusion Middleware to suit the unique requirements of Oracle E-Business Suite. The breadth and depth of the technology adoption is extensive, including:

• System Management Tools, such as the Application Management Suite

• Developer Productivity Tools, such as Oracle Jdeveloper

• Business Intelligence Tools, such as KPIs, and Oracle BI Publisher

• Infrastructure Tools, such as Identity Management

• Integration Tools, such as Applications Integration Architecture

InfraOptimization

Applications Technology Release 12.2 Value Proposition Cont 1

The result is that Oracle E-Business Suite offers unparalleled productivity, scalability, performance, and availability - thereby reducing the total cost of ownership (TCO) for enterprises of all sizes.

Oracle E-Business Suite Applications Technology Release 12.2 is focused on the following strategic areas:

• Technology Update• Usability• Integration• Security• Manageability

Source: ORACLE E-BUSINESS SUITE RELEASE VALUE PROPOSITIONRelease 12.2 Applications TechnologyPrepared by E-Business Suite Product Management and StrategyLast Updated: September 18, 2013Version: 2.0

InfraOptimization

Reference Doc IDs on EBS R12.2.xOracle E-Business Suite Release 12.2 Information Center (Doc ID 1581299.1)Release Schedule of Current Database Releases (Doc ID 742060.1)https://blogs.oracle.com/stevenchan/resource/certifications.html

Oracle E-Business Suite Release 12.2 Technology Stack Documentation Roadmap (Doc ID 1934915.1)Oracle E-Business Suite Release Notes, Release 12.2 (Doc ID 1320300.1)

Oracle E-Business Suite Product Specific Release Notes, Release 12.2.3 (Doc ID 1605928.1)Oracle E-Business Suite Release 12.2.3 Readme (Doc ID 1586214.1)Oracle E-Business Suite Technology Stack and Applications DBA Release Notes for Release 12.2.3 (Doc ID 1606170.1)

Oracle E-Business Suite Product Specific Release Notes, Release 12.2.4 (Doc ID 1903217.1)Oracle E-Business Suite Release 12.2.4 Readme (Doc ID 1617458.1)



Oracle E-Business Suite Release 12.2: Technical Planning, Getting Started, and Go-Live Checklist (Doc ID 1585857.1)

Recommended Browsers for Oracle E-Business Suite Release 12 (Doc ID 389422.1)

InfraOptimization

EBS R12.2 Architecture

A Tier is a logical grouping of services, potentially spread across more than one physical machine. The three-tier architecture that comprises an Oracle E-Business Suite installation is made up of the database tier, which supports and manages the Oracle database; the application tier, which supports and manages the various Oracle E-Business Suite components, and is sometimes known as the middle tier; and the desktop tier, which provides the user interface via an add-on component to a standard web browser.

Application Tier Services Root services, Web Entry Point services, Web Application services, Batch Processing services, Other services

A Node is a logical set of processes running on one hardware machine. Sometimes a node is also referred to as a "server" or an "instance". In a single-node installation of Oracle E-Business Suite, all the Applications processes (including the database processes) run on one node, whereas in a multi-node installation, the processes are distributed across multiple nodes. For example, Node 1 : DB 11.2.0.3, Node 2: Web and forms, Node 3 : concurrent manager.

A multi-node installation of Release 12 supports both shared and non-shared application tier file systems.

A Service is a functional set of Oracle E-Business Suite application processes running on one or more nodes. Where applicable, the term 'service' is replacing the more traditional term of 'server'. For example, Web services process HTTP requests, and Forms services process requests for activities related to Oracle Forms. The Concurrent Processing server supports data-intensive programs that run in the background.

Concepts and Terminology: Tier, Node, Service/Server

Source: Configuring Applications Node Services in Oracle E-Business Suite Release 12.1 (Doc ID 406558.1)

Understanding Three tier Technical Architecture:‐EBS R12.2 Three-Tier Architecture Perspective

EBS R12.2 Technology Architecture

Desktop or Client Tier - provides the user interface using a web browser to access the applications.Application Tier - The application tier, sometimes referred to as the middle tier, hosts the various services and servers that implement and execute the business logic, and mediates communications between the desktop tier and the database tier.Database Tier - supports the Oracle RDBMS and is the main data storage area. The database server does not communicate directly with desktop clients

InfraOptimization

EBS R12.2 Application Tier StructureSource: Oracle E-Business Suite Concepts Release 12.2 Part Number E22949-12

EBS R12.2 Application Tier Services at-a-glance

Service Group Service(s) Service Control Script

Root Service Node Manager adnodemgrctl.sh Web Administration WebLogic Admin Server adadminsrvctl.sh Web Entry Point Services

Oracle HTTP Server adapcctl.sh Oracle Process Manager adopmnctl.sh

Web Application Services

oacore admanagedsrvctl.sh oafm

forms forms-c4ws

Batch Processing Services

Oracle TNS Listener adalnctl.sh Concurrent Manager adcmctl.sh Fulfillment Server jtffmctl.sh Oracle ICSM ieoicsm.sh

Other Services Forms Server adformsrvctl.sh

Oracle MWA Service mwactlwrpr.sh

Most notably, the Web Administration service group contains WebLogic Administration server, and - unlike other service groups - can be enabled only on one of the Application tier nodes. In other words, it is not supported to enable WebLogic Administration server on any other Application tier node except the node on which it was enabled during Rapid Install.

InfraOptimization

R12.2 Technical Configuration

Tables: FND_OAM_CONTEXT_FILES, FND_NODES etc.

Configuration Activity In Release R12.2 Configuration Management Tools

Oracle E-Business Suite Database, Concurrent Processing, Oracle Developer 10g, profile options, and other Oracle E-Business Suite components.

Oracle Applications Manager. OAM & AutoConfig

HTTP Configuration.

Most HTTP configuration is managed via native Oracle WebLogic Server tools, Fusion Middleware Control, or manually editing of the configuration files. Only a limited set of HTTP configuration files are maintained via AutoConfig.

Fusion Middleware Control

Configuration of oacore, oafm, forms and forms-c4ws services

Properties for the oacore, oafm, forms and forms-c4ws services, including the classpath and JVM arguments, need to be updated through native WebLogic tools such as WebLogic Administration Console. The context variable values are used only to set the initial values during managed server creation.

WebLogic Server Admin Console

Managing JVM instances of the oacore, oafm, forms and forms-c4ws services.

Each JVM instance of a service corresponds to a managed server of that service type. The number of instances needs to be controlled by explicitly creating or deleting managed servers for the service.

InfraOptimization

Understanding Client & Application Tier Components

HTML-Based Applications Architecture

InfraOptimization

EBS Release 12.2 Technology Stack

End User

FM 11g

InfraOptimization

EBS Release 12.2 ArchitectureFrom Weblogic Perspective: WHAT TO KNOW

Java code including OAF

All EBS Forms Functionality

Web Services, Secure Search etc.

InfraOptimization

EBS Release 12.2 ArchitectureFrom Weblogic Perspective: WHAT TO KNOW ON SCALING

InfraOptimization

EBS R12.2 Deployment Planning Consideration

1. Performance2. Scalability – Vertically | Horizontally3. Maximum Availability4. Ease of Management & Maintenance

InfraOptimization

EBS R12.2 Deployment Architecture

Deployment Design Decisions MOS Doc ID: 1375686.1

File system configuration: Distributed | Shared

Load balancing: DNS Load balancing: DNS| HTTP (using load balancer)

Web Entry Point: One | Multiple

DMZ Configuration (Doc ID 1375670.1)

InfraOptimization

EBS R12.2 Deployment Design Decisions

Distributed File SystemsMOS Doc ID 1383621.1

Shared File SystemMOS Doc ID 1375769.1

InfraOptimization

EBS R12.2 Deployment Design DecisionsMOS Doc ID 1375686.1

InfraOptimization

EBS R12.2 Deployment Design Decisions

MOS Doc ID 1570554.1

Oracle E‐Business Suite Setup Guide Release 12.2

Oracle Database Real Application Clusters (RAC)

PCP - Parallel Concurrent Processing

InfraOptimization

EBS Release 12.2 Technology Stack Certification

Source: ORACLE E-BUSINESS SUITE RELEASE CONTENT DOCUMENT Applications Technology Releases 12.1 and 12.2 (inclusive of 12.0.2 – 12.0.7) Prepared by ATG Documentation Last Updated: March 6, 2015 Version: 3.0

InfraOptimization

InfraOptimization

Pathway to Sizing for EBS R12.2

Considering Factor for Sizing for EBS R12.2

Oracle Applications is certified on operating systems, not specific models of hardware.Certified Platform: Linux x86,Linux x86-64,HP-UX Itanium,HP-UX PA-RISC (64-bit),IBM AIX on Power Systems (64-bit),Microsoft Windows Server (32-bit),Oracle Solaris SPARC (64-bit)

Number of concurrent E-Business Suite users, i.e. Oracle Application Testing Suite

Sizing information systems: SPC - Storage Performance Council & TPC - Transaction Processing Performance Council

InfraOptimization

EBS R12.2 Architecture & File System StructureSource: Sharing The Application Tier File System in Oracle E-Business Suite Release 12.2 (Doc ID 1375769.1)General Three Tier Architecture Application/Database Tier Components

InfraOptimization

EBS R12.2 File System StructureSource: https://docs.oracle.com/cd/E26401_01/doc.122/e22949/T120505T120509.htm (Part Number E22949-12)

https://docs.oracle.com/cd/E26401_01/doc.122/e22949/T120505T120509.htm

InfraOptimization

EBS R12.2 File System Structure Cont 1• 10.1.2 OH holds the AS

components for Forms and Reports

• Fusion Middleware Home contains WebLogic Server, HTTP configuration files

• INST_TOP contains only maintenance scripts

• Fusion middleware home contains the following directories– EBS_Domain: E-Business Suite weblogic domain, contains all domain-specific configuration and log files.– wlserver_10.3: Binaries and libraries required to support WebLogic Server– oracle_common: Java Required Files (JRFs) used by EBS.– webtier: HTTP configuration and log files used by EBS.– Oracle_EBS-app1: Contains configuration files related to oacore, forms, oafm, and forms-c4ws

• FS_NE contains files needed by both the file systems– Patch staging area, ADOP log files & ADOP utility

• Not all log files are not stored under $INST_TOP– HTTP log files are located under the native instance home ($FMW_HOME/webtier)– Oracle WebLogic Server log files are located under the domain home ($EBS_DOMAIN_HOME)– Concurrent processing log files are located on the none-editioned file system (fs_ne)

InfraOptimization

Deciding RAID Levels for EBS R12.2 File SystemSource: https://en.wikipedia.org/wiki/Standard_RAID_levels

InfraOptimization

Oracle Applications Technology LayerThe Oracle Applications technology layer is an integrated collection of components whose functionality is applicable to all Oracle Applications modules, i.e. CRM, HRMS, Financials, Manufacturing. Components in the Oracle Applications technology layer include:

Applications DBA (AD) Set of tools for administration Application Object Library (FND) Consists of a collection of reusable code, programs, and database objects that provides common functionality across all products.Applications Utilities (AU) To maintain the Oracle Applications system. Workflow (WF) To Model and maintain your business processes using a graphical workflow builder. Alert (ALR) To email system notifications to users when an exception or event occurs. OA Framework (OAF/FWK) A Java and XML-based development method to develop HTML-based applicationsOracle XML Publisher (XDO) To produce output reports in the desired format.

InfraOptimization

Oracle Applications Database Objects

• Data objects - Used for storing and accessing business data. These objects include tables, indexes, sequences, and index-organized tables. • Code objects - Used to process the data. These objects include triggers, PL/SQL packages, Java stored procedures, synonyms, and views.• Hybrid objects - Used to both store and process data. Materialized views are an example of hybrid objects.

InfraOptimization

Oracle Applications Schemas

Product Schema and its own objects

Code Objects

Schemas Used During SignonWhen you sign on to Oracle Applications, you initially connect to the public schema, APPLSYSPUB. Once your user name and password have been verified, Oracle Applications connects you to the APPS schema and allows you to choose a responsibility. You can then use the Forms interface or HTML interface to access data that resides in a product schema.

(Global Schema)

a better name for APPLSYS would or could be FND

InfraOptimization

Purpose of 2 applsyspub entries in context file

egrep -i APPLSYSPUB $CONTEXT_FILE

<username oa_var="s_gwyuid_user">APPLSYSPUB</username> This is used to generate the env variable $GWYUID at OS level

<GWYUID oa_var="s_gwyuid">APPLSYSPUB/PUB</GWYUID> This is used to populate the passoword in DBC file ($FND_SECURE/ <SID>.dbc).

EBS R12.2 Login ProcessReference Doc ID 1984710.1Oracle E-Business Suite Concepts Release 12.2 Part Number E22949-12

Doc ID: 1569732.1, 2072420.1, 1316142.1 http --> HS/mod_wl_ohs --> http --> WLS

s_wls_admin_user

Session TablesICX_SESSIONSFND_LOGINS

Oracle Application Framework Architecture

EBS R12.2 Login ProcessReference Doc ID 1984710.1

When a HTTP request is made for EBS, the request is received by the Oracle HTTP Server (OHS). ($FMW_Home/webtier/instances/EBS_web_<instance_name>_OHS1/config/OHS/EBS_web_<instance_name>/httpd.conf)

When the configuration of OHS is for a resource that needs to be processed by Java, such as logging into EBS, the OHS configuration will redirect the request to the Web Logic Server (WLS) Java process (OACore in this case).

WLS determines the J2EE application that should deal with the request, which is called "oacore".

This J2EE application needs to be deployed and available for processing requests in order for the request to succeed. The J2EE application needs to access a database and does this via a datasource which is configured within WLS.

All code must access the data model via the APPS synonym!

Doc ID: 1569732.1, 2072420.1, 1316142.1 http --> HS/mod_wl_ohs --> http --> WLS

s_wls_admin_user

Session TablesICX_SESSIONSICX_PARAMETERSFND_LOGINS

Differences between Forms Server and Forms ServletSource: Troubleshooting Forms Applet issues in E-Business Suite (Doc ID 1300317.1)Reference: Using Forms Socket Mode with Oracle E-Business Suite Release 12 (Doc ID 384241.1)Although Forms Server (or SOCKET mode) and Servlet implementation mainly affects the Application tier there is some important differences on how communication is done between the Forms applet and run-time process making it useful to provide some background information on this topic.

InfraOptimization

EBS R12.2.x Password managementSteps to change the APPS, APPLSYS, and APPS_NE password using FNDCPASS or AFPASSWD for EBS 12.2 (Doc ID 1674462.1)source EBSapps.env run ; env |grep -i base, grep BASE $CONTEXT_FILE

• FNDCPASS and AFPASSWD utilities can be used for setting Oracle E-Business Suite schema passwords• It is recommended that the FND_USER and FND_ORACLE_USERID tables are backed up before system passwords (APPLSYS, APPS,

APPS_NE) are changed• AFPASSWD is an enhanced version of FNDCPASS, only prompts for passwords required for the current operation. AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -f <FNDUSER> (Application Users) AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -o <ORAUSER> (Apps Product users) AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] –a (All Apps Product users) AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -l <ORAUSER> <TRUE|FALSE> (Locks Single User) AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -L <TRUE|FALSE> (Locks non-essential users) AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -s <APPLSYS> (Changes APPLSYS) AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -m <HASH-MODE> [MIGRATE] (Migrated to non-recoverable Hashing

scheme passwords)

AFPASSWD -f SYSADMINEnter the ORACLE password of Application Object Library ‘APPSUSER’:Connected successfully to APPS.Working…Enter new password for user [SYSADMIN]:Verify new password for user [SYSADMIN]:Password is changed successfully for user SYSADMIN.Password is changed successfully for user SYSADMIN.AFPASSWD completed successfully.

AFPASSWD -c apps@PROD -s APPLSYSEnter the ORACLE password of Application Object Library ‘APPSUSER’:Connected successfully to APPS.Enter the password for your ‘SYSTEM’ ORACLE schema:Connected successfully to SYSTEM.Working…Enter new password for user:Verify new password for user:Working…AFPASSWD completed successfully.

Note: For Changing APPLSYS/APPS/APPS_NE Password We Need To Shutdown Application Tier And Once We Finish Changing The Password We Need To Run Autoconfig On All Tiers

InfraOptimization

EBS R12.2.x APPS Password change• Shut down the application tier services• Change the APPLSYS password using FNDCPASS or AFPASSWD utility.• Start AdminServer using the adadminsrvctl.sh script from your RUN filesystem• Update the “apps” password in WLS Datasource as follows:

– Log in to WLS Administration Console.– Click Lock & Edit in Change Center.– In the Domain Structure tree, expand Services, then select Data Sources.– On the “Summary of JDBC Data Sources” page, select EBSDataSource.– On the “Settings for EBSDataSource” page, select the Connection Pool tab.– Enter the new password in the “Password” field.– Enter the new password in the “Confirm Password” field.– Click Save & Click Activate Changes in Change Center.

• Execute Autoconfig to update all application tier files • Start all the application tier services using the adstrtal.sh script.

• Log in to WLS Administration Console, Under Domain Structure tree select Data Sources.

• On the “Summary of JDBC Data Sources” page, select EBSDataSource.

• On the “Settings for EBSDataSource” page, select Monitoring > Testing.

• Select “oacore_server1” & click Test DataSource• Look for the message “Test of EBSDataSource on server

oacore_server1 was successful”.

Doc ID 1377400.1 - Install Sanity Check: How To Deploy A Basic Java Web Application To Confirm Your WebLogic Domain Is FunctionalDoc ID 1940996.1 - Oracle E-Business Suite 12.2 Data Source Connection Pool Diagnostics

InfraOptimization

OAM Generic Collection Service concurrent managerSource: How To Determine The Current Value and Set The Retention Time To Keep the Runtime Instance Data Of Table FND_OAM_FORMS_RTI In The Database? (Doc ID 1600075.1) & What Are .RTI And .FLI Files? (Doc ID 470850.1)How To Find The IP Address Of The Client Machine From Where A Particular Forms User Is Connected ? (Doc ID 879092.1)

env|grep FORMS_RTI_DIR

1. To determine the current retention time, run the following select:

select FCQ.CONCURRENT_QUEUE_NAME, FCQS.SERVICE_PARAMETERSfrom FND_CONCURRENT_QUEUES FCQ, FND_CONCURRENT_QUEUE_SIZE FCQSwhere FCQ.CONCURRENT_QUEUE_NAME like 'OAMGCS%'AND FCQ.APPLICATION_ID = FCQS.QUEUE_APPLICATION_IDAND FCQ.CONCURRENT_QUEUE_ID = FCQS.CONCURRENT_QUEUE_ID;

2. To Manually change the time:a. Navigate: (Responsibility) System Administrator > Concurrent > Manager > Define > Query out record 'OAM Generic Collection Service%'b. Click button 'Work Shifts' > Edit the value of 'Parameter' and save.

Notes:Data in table FND_OAM_FORMS_RTI is populated by the OAM Generic Collection Service concurrent manager and the information is cleaned up and repopulated by the same program. The OAM Generic Collection Service takes these parameters: NODE: the name of the node on which the service runs. LOADINTERVAL: the load interval for periodic runtime instance information uploading. ORACLE_HOME: the ORACLE_HOME in which the Forms Listener runs. RTI_KEEP_DAYS: the number of days to keep the runtime instance data in the database. FRD_KEEP_DAYS: the number of days to keep Forms Runtime Log files.

InfraOptimization

Oracle E-Business Suite SYS.DUAL PUBLIC Privileges Security Issue (CVE-2015-0393)

Source: Use of Common Vulnerability Scoring System version 2.0 (CVSS 2.0) by Oracle (Doc ID 394487.1)Oracle Critical Patch Update January 2015 Documentation Map (Doc ID 1935468.1)

SELECT * FROM sys.dba_tab_privs WHERE owner = 'SYS' AND table_name = 'DUAL';

SELECT 'revoke ' || privilege || ' ON sys.dual FROM public;' "SQL" FROM sys.dba_tab_privs WHERE owner = 'SYS' AND table_name = 'DUAL' AND grantee = 'PUBLIC' AND privilege != 'SELECT';

Note:

For R12 only, the CPU patch has manual instructions to run the adgrants.sql script. This script will create the AD_ZD_SYS package in the SYS schema. AD_ZD_SYS has a single procedure FIX_SYSUSER, which revokes unnecessary grants on SYS.DUAL.

select count(*) from sys.dual; Must return only one row.select count(*) from fnd_dual;

InfraOptimization

Monitoring & Auditing User ActivityE-Business Applications Performance Basic Skills Cookbook (Doc ID 1516355.1)Concurrent Request Data Gathering (Doc ID 740770.1)APXINWKB APP-10771 Could Not Reserve Records Due to Database Record Lock (Doc ID 140969.1)How To Kill Inactive Sessions Based on the Information in the Monitor User Form (Doc ID 103516.1)Oracle Critical Patch Update January 2015 Documentation Map (Doc ID 1935468.1)

Use of Common Vulnerability Scoring System version 2.0 (CVSS 2.0) by Oracle (Doc ID 394487.1)Oracle Database Security Assessment Tool (DBSAT) (Doc ID 2138254.1)

How to find or see the requests ran in a module or a responsibility or by a user or for a specific period (Doc ID 2098951.1)

Monitor Users: System Administrator Responsibility Security: Users > Monitor CTL + F11

Enable User Tracking – Setting the Sign-On: Audit Level profile option:

Log in to Oracle APPS with system administrator responsibilityNavigate to Profile > SystemMake sure the Site option is checked.Navigate to Profile and search for sign > Click FindSelect the “Sign-On: Audit Level” > Click FindChange its value to be FormFrom the Menu bar click File > Save

InfraOptimization

Technology Stack Registration

oraInst.loc inventory_loc=/etc/oraInventory

oraInst.locUnder /etc on Linux and AIXUnder /var/opt/oracle on Solaris and HP-UX

Global and Local Inventory explained (Doc ID 360079.1)FAQs on Central Inventory and Oracle Home Inventory (Local Inventory) in Oracle RDBMS (Doc ID 564192.1)How To Create, Update or Rebuild The Central Inventory For Oracle Applications E-Business Suite Release 12 ? (Doc ID 742477.1)

Oracle Universal Installer Inventory Essentials for Apps SysadminsSource: https://blogs.oracle.com/stevenChan/entry/ebs_oui_inventory_primer

InfraOptimization

Oracle Universal Installer Inventory Essentials for Apps SysadminsSource: https://blogs.oracle.com/stevenChan/entry/ebs_oui_inventory_primer

InfraOptimization

Cloning Cloning is the process used to create a copy of an existing E-Business Suite System

Source System Target System

Single Node to Single NodeSingle Node to Multi Node

Multi Node to Single NodeMulti Node to Multi Node

Type of Clone

InfraOptimization

Pre-Requisite Before CloningReference Doc ID: 1383621.1Before cloning to a new system, you must first prepare the Source System by following the below steps:

Verify disk space requirements on Source System Verify OS requirements on Target System Verify Inventory Requirements /* It is required and must be identified by the value defined in oraInst.loc file */ Verify Source and Target System software components and versions: Source and Target must be on the same Operating System (OS) and patch level Apply required patches Check for in-progress online patching cycle: On the Source system, there should not be any in-progress online patching cycle, i.e. cleanup and fs_clone should be run. Run AutoConfig on the application tier (Optional) Synchronize appsutil on the database tier nodes (Optional) Run AutoConfig on the database tier (Optional) Run EBS Technology Codelevel Checker (ETCC) (Doc ID 2008451.1) on DB and APPS tier (Optional) Maintain snapshot information of APPL_TOP (adadmin tool): Current view snapshots & Named snapshots You must run this task once for each APPL_TOP before you can apply any patch that contains a "compatible feature prereq" line on that APPL_TOP A complete snapshot is required for the automatic prerequisite patch checking feature to operate APPL_TOP snapshot information is stored in the AD_SNAPSHOTS, AD_SNAPSHOT_FILES, and AD_SNAPSHOT_BUGFIXES tables

InfraOptimization

Clean Global InventoryClean Temporary Directories Delete Database Technology Stack Delete Applications DatabaseDelete Applications Technology Stack Delete Application File System

Pre-Requisite Before CloningCleaning Environments

InfraOptimization

Cloning Techniques

InfraOptimization

Cloning Methods Off-Line On-Line/ Hot Cloning/AMP CloningNote: Apps Tier can be cloned on-line at any point in time. DB Tier, i.e. Database, can be cloned on-line by RMAN only.

InfraOptimization

Cloning TechniquesRapid Clone

Reference Doc ID• Cloning Oracle Applications Release 12 with Rapid Clone (Doc ID 406982.1)• Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone(Doc ID 1383621.1)• Setup Guide: Part No. E22953-10 (Chapter 17 Cloning Oracle E-Business Suite with Rapid Clone)• How To Collect Log Files from Cloning in 12.2 (Doc ID 1961471.1)• E-Business Suite Clone Log Parser Utility (Rapid Clone) (Doc ID 1447553.1)• Cloning Oracle Applications Release 11i with Rapid Clone (Doc ID 230672.1)• Troubleshooting RapidClone issues with Oracle Applications R12.0 & R12.1 (Doc ID 603104.1)• http://docs.oracle.com/cd/E11857_01/em.111/e18709/T508706T512065.htm

Cloning TechniquesRapid Clone at EBS Application Tier

Up To R12.2.x-4Critic on this Clone Architecture: Repetitive Task

1. Clone (Source RUN FS): Source to Target2. Clone (Target RUN FS): Target RUN FS to Target PATCH FS.

From R12.2.5-6

1. Clone (Source RUN FS): Source to Target

perl adcfgclone.pl appsTier dualfs RUN/appsprod/fs2

RUN/appstest/fs2

/appsprod/fs1 PATCH

/appstest/fs1 PATCH

Source Target

InfraOptimization

InfraOptimization

Cloning PhasesPrepare Source System by applmgr & oracle user, NOT root

$ cd $INST_TOP/admin/scripts$ perl adpreclone.pl appsTier

$ cd <RDBMS ORACLE_HOME>/appsutil/scripts/<CONTEXT_NAME>$ perl adpreclone.pl dbTier

InfraOptimization

Cloning PhasesCopy Bits by ‘application’ OS user, i.e. NOT root userNote:

Comparing the size of the file systems between Target and Source

du -sk EBSapps (source) du -sk EBSapps (target)

Cloning PhasesConfigure Target System

## As Oracle Usercd $ORACLE_HOME/appsutil/clone/binperl adcfgclone.pl dbTier /* perl adcfgclone.pl <component> [context] */

## As applmgr user

…/EBSapps/comn/clone/binperl adcfgclone.pl appsTierWHERE:

<component> on Apps Node : { appsTier | atTechStack | dev10gHome | fmwHome | wlsConfig | ohsConfig | appltop } <component> on DB Node : { dbTier | dbTechStack | database | dbconfig | addracnode } <context> : Target Context File

InfraOptimization

Cloning TechniquesConfigure Target System

Configclone appsTier

Configclone dbTier

InfraOptimization

Cloning TechniquesOther Configuration Commands

InfraOptimization

Cloning TechniquesCustom Cloning: Source-to-Target Refreshing

Target System

Source-Target direct Synchronization Ideal for refreshing environments Rsync use for faster copy

InfraOptimization

Cloning TechniquesMaximum Availability Techniques

Target System

AutoConfig:To Manage EBS System Configurations You must run AutoConfig for your system to make any context file updates to your system

Doing so generates the new configuration files for the associated technology stack

AutoConfig uses the parameters stored in the context file and system configuration templates to create new process control scripts and update system profiles

Running AutoConfig - Application Tier: $ADMIN_SCRIPTS_HOME/adautocfg.sh apps/<apps_passwd>Running AutoConfig - Database Tier: <RDBMS_ORACLE_HOME>/appsutil/scripts/<CONTEXT_NAME>/adautocfg.sh apps/<apps_passwd>

Rolling Back an AutoConfig Session:<INST_TOP>/admin/out/<MMDDhhmm>/restore.sh<RDBMS ORACLE_HOME>/appsutil/out/<CONTEXT_NAME>/<MMDDhhmm>/restore.sh

Script

Note (Doc ID 1582473.1):If the run file system data is missing from the database simply source the run file system environment and run autoconfig.If the patch file system data (managed servers or context file) is missing from the database then run the following after sourcing the run file system environment.

$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=<full path to patch context file> logfile=/tmp/patchctxupload.log

InfraOptimization

AutoConfig Instantiation Mechanism

AutoConfig Operation

• AutoConfig template files are used as the starting point for creating site-specific configuration files. AutoConfig evaluates the context variables in a template file, determines the actual values required, and creates a configuration file with these values substituted. This process, described in more detail later in this chapter, is called instantiation. There is one template file for each configuration file. Template files are located in the various <PROD>_TOP/admin/template directories on the application tier, and in the <RDBMS_ORACLE_HOME>/appsutil/template directory on the database tier.

• AutoConfig driver files are used to list the corresponding template files and locations, and specify the commands to be executed. For example, the commands might update profile options. Driver files are located in each <PROD>_TOP/admin/driver directory on the application tier, and in the <RDBMS_ORACLE_HOME>/appsutil/template directory on the database tier.

• AutoConfig configuration files, such as httpd.conf, are created as a result of AutoConfig instantiating the corresponding template files. Configuration files contain values corresponding to the settings specified for a particular site. After AutoConfig has been run, numerous configuration files will have been created in various directories.

• AutoConfig creates configuration files by reading several different types of file from various locations.

Reference: Oracle E-Business Suite Concepts R12.2

InfraOptimization

Reference Doc ID to Autocinfig

Note 387859.1 - Using AutoConfig to Manage System Configurations in Oracle E-Business Suite Release 12

Note 949160.1 - Interesting Documents Around Autoconfig in E-Business-SuiteNote 387859.1 - Using AutoConfig to Manage System Configurations in Oracle E-Business Suite Release 12Note 458282.1 - Context VariablesNote 461326.1 - Unable to Create a Custom Product TOPNote 845700.1 - How to troubleshoot autoconfig errorsNote 406558.1 - Configuring Applications Node Services in Oracle E-Business Suite Release 12.1Note 2072420.1 - E-Business Suite 12.2 Detailed Steps To Change The R12.2 Default Port To 80

Note 218089.1 - Autoconfig FAQ

InfraOptimization

Net Services Topology Data Model

The Net Services Topology Data Model stores the entire topological information about a single Oracle Application instance. The data model stores information about each node in the Oracle Applications instance which is then used to generate the Net Service configuration files (for example tnsnames.ora). AutoConfig seeds the data model with relevant data.The Net Services Topology Data Model stores the following information: On the database tier: Hostname, Database SID, Database Name, Instance Name, TNS Descriptors..... On the application tier: Hostname, FNDFS and FNDSM alias descriptors.........

The Net Services Topology Data Model is seeded every time you run AutoConfig on the respective tier. Every time you run AutoConfig on the database tier, the relevant data is seeded/updated for the database tier. Respectively, every time you run AutoConfig on the application tier, the relevant data is seeded/updated for the application tier.

To deregister the current application tier from the Net Services Topology Data Model, invoke the following command: $ perl <AD_TOP>/bin/adgentns.pl appspass=<APPSpwd> contextfile=<CONTEXT> -removeserver

To deregister the current database tier from the Net Services Topology Data Model, invoke the following command: $ perl <RDBMS ORACLE_HOME>/appsutil/bin/adgentns.pl appspass=< APPSpwd> contextfile=<CONTEXT> - removeserver

Context FilesXML based Context Files, uses by Autoconfig, is a Repository of all the installed Configuration Parameters with value: Application Context File and Database Context File, i.e. hundreds of config and env files present in EBS.

Run the following script to determine if all context files have been registered in the database. The script should return a run and patch context file for each application server and one for the database server.

select path,status,serial_number,version from fnd_oam_context_files where name not in ('TEMPLATE','METADATA') and (status is null or status !='VALID');

If the run file system data is missing from the database simply source the run file system environment and run autoconfig.If the patch file system data (managed servers or context file) is missing from the database then run the following after sourcing the run file system environment. This command is executed on the RUN file system to upload the PATCH context file:

$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=<full path to patch context file> logfile=/tmp/patchctxupload.log

Example (Doc ID 1582473.1):$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=/oraapps/applprod/fs2/inst/apps/PROD_phpdb02/appl/admin/PROD_phpdb02.xml logfile=/tmp/ctx_upload.log

Editing Context Files by OAM

By Autoconfig

Allows to sync multiple nodes

Doc ID : 455492.1Open the $APPL_TOP/admin/adovars.env file and ensure that the environment variables below are defined.$ADJVAPRG –versionecho $CLASSPATH $AFJVAPRG –versionecho $AF_CLASSPATH

InfraOptimization

Understanding AutoConfigReference Doc ID 387859.1

Using AutoConfig to Manage Oracle E-Business Suite ServicesService Group Service(s) Service Control Script

Root Service Node Manager adnodemgrctl.sh Web Administration WebLogic Admin Server adadminsrvctl.sh Web Entry Point Services Oracle HTTP Server adapcctl.sh

Oracle Process Manager adopmnctl.sh

Web Application Services oacore

admanagedsrvctl.sh oafm forms forms-c4ws

Batch Processing Services Oracle TNS Listener adalnctl.sh Concurrent Manager adcmctl.sh Fulfillment Server jtffmctl.sh Oracle ICSM ieoicsm.sh

Other Services Forms Server adformsrvctl.sh

Oracle MWA Service mwactlwrpr.shMost notably, the Web Administration service group contains WebLogic Administration server, and - unlike other service groups - can be enabled only on one of the Application tier nodes. In other words, it is not supported to enable WebLogic Administration server on any other Application tier node except the node on which it was enabled during Rapid Install.

AutoConfig ComponentsDoc ID 387859.1

APPS: <INST_TOP>/appl/admin/<CONTEXT_NAME>.xmlDB: <RDBMS_ORACLE_HOME>/appsutil/<CONTEXT_NAME>.xml

<PROD_TOP>/admin/template<RDBMS ORACLE_HOME>/appsutil/template

<AD_TOP>/admin/driver/ <FND_TOP>/admin/driver/

Context File

Temp late File

Driver

AutoConfig Tools: 1. <INST_TOP>/admin/scripts/adautocfg.sh 2. <AD_TOP>/bin/adchkcfg.sh | adtmplreport.sh | admkappsutil.pl 3. <FND_TOP/patch/115/bin/GenCtxInfRep.pl

InfraOptimization

AutoConfig Check Utility Tool & Performance Profiler

The check config utility is a great method for analyzing the impact of AutoConfig. Simply execute the utility and review the HTML output file.

Another utility, the Performance Profiler will allow you to review and resolve performance related issues that are experienced during the execution of autoconfig. Like the check config utility, you run the tool and analyze the HTML output file.

Both tools allow you to refine and tune autoconfig executions.

• Review the affects of executing AutoConfig

• HTML file is generated by the Check Config Utility

• Execute Check Config Tool:$AD_TOP/bin/adchkcfg.sh

Previewing AutoConfig Changes on All Tiers

• Review the timings of executing AutoConfig• HTML file is generated by the Performance

Profiler Utility• Execute Performance Profile Utility:

perl $AD_TOP/bin/adconfig.pl contextfile=<CtxFile> [product=<product_top>] –profile

Optimizing AutoConfig Performance Through Profiling: Identifying AutoConfig Performance Bottlenecks

InfraOptimization

AutoConfig Search Utilityhttps://blogs.oracle.com/stevenChan/entry/everything_you_ever_wanted_to_know_about_autoconfiUsing AutoConfig to Manage System Configurations in Oracle E-Business Suite Release 12 (Doc ID 387859.1)

The AutoConfig Search Utility (formally known as the TXK Context Variable Information Utility) scans your AutoConfig infrastructure for a requested string and generates a detailed report about the relevant AutoConfig variables and their usage. The report contains, among other things, the following information about relevant AutoConfig variables:

1. Description2. Current value3. Default value4. Relevant AutoConfig template files

perl $FND_TOP/bin/txkrun.pl -script=GenCtxInfRep -keyword="<keyword>"

perl $FND_TOP/bin/txkrun.pl -script=GenCtxInfRep -keyword=jdbc -outfile= txkContext_Apps.html

The utility takes the following arguments:

keyword (required): all or part of a context variable nameoutfile (required): the report file. If only the name and not the complete path for the report file is provided, the report will be generated in the <APPLTMP> directory.contextfile (optional): complete path to the context file.reportType (optional): the report type. Valid values are html (default) or text

InfraOptimization

Running AutoConfig in Parallel in EBS 12Source: https://blogs.oracle.com/stevenChan/entry/power_tools_autoconfig_parallel_runs_in_ebs12

In an R12 E-Business Suite instance, AutoConfig can now be run simultaneously across multiple nodes. This new feature significantly lowers maintenance downtime for multi-node installations.

Executing AutoConfig in Parallel ModeThe following command can be used to run AutoConfig in 'parallel mode'Application Tier: perl $AD_TOP/bin/adconfig.pl contextfile=<CtxFile> [product=<product_top>] –parallelDatabase Tier: perl $ORACLE_HOME/appssutil/bin/adconfig.pl contextfile=<CtxFile> –parallel

where<CtxFile> is the absolute path to the context file<product_top> is the Product short name

InfraOptimization

Configuring EBS R12 Database Tiers with AutoConfigSource: https://blogs.oracle.com/stevenChan/entry/autoconfig_build_context_for_ebs12

The AutoConfig Build Context utility (adbldxml.pl) is needed like in case of cross-platform migrations and database upgrades.

perl $ORACLE_HOME/appsutil/bin/adbldxml.pl

Note: In R12, this utility is supported only on the database tier.

The AutoConfig Build Context utility reads information from EBS DB and environment variables and builds the context file out of the database context template ($ORACLE_HOME/appsutil/template/adxdbctx.tmp).

InfraOptimization

Phases of Operation of AutoConfig

InfraOptimization

Oracle EBS R12.2: Concurrent Processing Server

Concurrent Processing Server

Most user interactions with Oracle Applications data are conducted via the HTML interface or the Forms interface. However, some reporting and updating programs may need to be run either periodically or on an ad hoc basis. As these programs may require a large number of computations, they are run in the background at a time, and with a priority, such that the work of interactive users is not impeded. Such programs are run on the Concurrent Processing server. You submit a request to run a concurrent program through an Oracle Applications form. The request inserts a row into a database table, specifying the program to be run. Concurrent managers read the applicable requests in the table and start the relevant concurrent programs.

Note: Log and output files from concurrent programs are generated on the Concurrent Processing server. These should be monitored for errors, and archived or deleted when no longer needed. Failure to do so will eventually result in a shortage of disk space.

InfraOptimization

Concurrent Manager:Introduction Batch Processing Sub-System Programs are run as operating system background processes. These programs may be written using variety of Oracle tools, programming languages for executables, or OS scripts.

The Internal Concurrent Manager (ICM), which functions as the "boss" of all the other managers, i.e. controls all other concurrent managers. It administers the startup and shutdown of managers as defined by their work shift, monitors for process failure, and cleans up if a failure occurs.Warning: You cannot alter the definition of the Internal Concurrent Manager.

The Conflict Resolution Manager (CRM) enforces rules designed to ensure that incompatible concurrent requests do not run in the same conflict domain.

The Standard Manager as shipped with Oracle Applications will accept and run any concurrent requests. It has no specialization. The Standard manager is active all the time; it works 365 days a year, 24 hours a day. Warning: You should not alter the definition of the Standard concurrent manager. If you do, and you have not defined additional managers to accept your requests, some programs may not run. Use the Standard manager as a safety net, a manager who is always available to run any request. Define additional managers to handle your installation site's specific needs.

Monitoring Concurrent Manager

Concurrent > Manager > Administer

Are the Managers up and running ?

Concurrent Request – LifecycleHigh Level

Phase Activity

Pending / Normal The request is waiting to be run

Pending / Standby The request is waiting at CRM

Running / Normal The request is runningCompleted / Normal The request has finishedInactive / No Manager The request cannot be run. No manager

available to process it.

Concurrent Request – LifecyclePhase Status Description

Pending

Normal Request is waiting for the next available manager.Standby Program to run request is incompatible with other program(s) currently runningScheduled Request is scheduled to start at a future time or dateWaiting A child request is waiting for its Parent request to mark it ready to run. For example, a report in a report

set that runs sequentially must wait for a prior report to complete.

Running

Normal Request is running normallyPaused Parent request pauses for all its child requests to complete. For example, a report set pauses for all

reports in the set to complete.Resuming All requests submitted by the same parent request have completed running. The Parent request is

waiting to be restarted.Terminating Running request is terminated, by selecting Terminate in the Status field of the Request Details zone.

Completed

Normal Request completes normally.Error Request failed to complete successfullyWarning Request completes with warnings. For example, a report is generated successfully but fails to print.Cancelled Pending or Inactive request is cancelled, by selecting Cancel in the Status field of the Request Details

zone.Terminated Running request is terminated, by selecting Terminate in the Status field of the Request Details zone.

InactiveDisabled Program to run request is not enabled. Contact your system administrator.On Hold Pending request is placed on hold, by selecting Hold in the Status field of the Request Details zone.No Manager No manager is defined to run the request. Check with your system administrator.

Concurrent Requests, Programs and Processes

Concurrent Manager:Tables

Table ContentFND_CONCURRENT_REQUESTS Details of user requests, including status,

start date, and completion date.

FND_CONCURRENT_PROGRAMS Details of concurrent programs, including execution method, whether the program is constrained, and whether it must be run alone.

FND_CONCURRENT_PROCESSES Cross-references between concurrent requests and queues, and a history of concurrent manager processes.

FND_CONCURRENT_QUEUES Information about each of the concurrent manager queues.

Select for Update

Select for Update

Select for Update

Select for Update

Select for Update

Select for Update

Job

Status

ABCDE

Job Run

Job Run

Job Run

Job Run

Select Runnable

Select Runnable

Select Runnable

Select Runnable

AB

AB

AB

AB

A B

A B

A B

A B

AB

AB

AB

AB

Lock?

Lock?

Lock?

Lock?

Running

X

X

X

Select for Update

Select for Update

Select for Update

Running

X

X

Update Set A Running

Update Set B Running

Running

Select Runnable

Select Runnable

CD

CD

CD

CD

C D

C D

Lock?

Lock?

Update Set C Running

X

Running

Select Runnable

Select for Update

Update Set A Complete

Running

Select for Update

Select RunnableComplete

Update Set D Running

E

EE Update Set E RunningLock?

Running

Update Set D Complete

Complete

None

Worker 1

Worker 2

Worker 3

Worker 4

FND_CONCURRENT_REQUESTS

Sleep

Concurrent Manager

Service Management

InfraOptimization

With Service Management, the Internal Concurrent Manager (ICM) manages the various service processes across multiple hosts. On each host, a Service Manager acts on behalf of the ICM, allowing the ICM to monitor and control service processes on that host. System administrators can then configure, monitor, and control services though a management console which communicates with the ICM.

Service Management provides a fault tolerant system. If a service process exits unexpectedly, the ICM will automatically attempt to restart the process. If a host fails, the ICM may start the affected service processes on a secondary host. The ICM itself is monitored and kept alive by Internal Monitor processes located on various hosts.

InfraOptimization

Concurrent Manager:Tuning Consideration

• 50%of performance tuning is in the business!– Review long-running auto resubmitted jobs– Review short/long-running jobs – selective parameters?– Don’t purge jobs that the users then have to run again

• Points to note– More managers/workers != more throughput (More workers = More

accesses)– Avoid enabling an excessive number of managers/workers

• New systems: a start point is 1-2 target process per CPU– Balance with resources required by online activity – managers/workers

• Carefully consider reducing the CRM sleep time to 5 ~ 10 secs• Set Concurrent: Force Local Output File Mode" to "Yes“ • Set CONC_PMON_METHOD=LOCK

InfraOptimization

Concurrent Manager:Enhancing Performance – Best Practice

Tuning Concurrent Queue Parameters

• Sleep Seconds is the number of seconds your Concurrent manager waits between checking the list of pending concurrent requests (concurrent requests waiting to be started).Tips: High number, e.g. 120s avoids constant polls to check for new requests. Low number , e.g. 20s is good enough when number of submitted requests is very high.Note: Concurrent Manager will not sleep at all if there are pending requests.

• Cache Size at least twice the number of target processes.• Process Increase it if the pending/normal requests are high based on physical server

capacity • Purging & Defragmentation associated tables• Gather Schema Statistics

Workload Management

• Specialized Concurrent managers

Output Post Processor – OPP

• Java Heap Size

Transaction Managers

Backend queue concurrent manager details

Select rpad(user_concurrent_queue_name ,50,'-') "Queue Name",TARGET_NODE, MAX_PROCESSES,running_processes, nvl(to_char(SLEEP_SECONDS), 'Not Set') Sleep_Second, nvl(to_char(cache_size),'Not Set')Cache_sizefrom FND_CONCURRENT_QUEUES_VL fcq order by 1;

InfraOptimization

Concurrent Manager:Tuning Generic Guideline

ThroughputTo maximize throughput consider reducing the sleep time of the Conflict Resolution Manager (CRM). The default value is 60 seconds. You can consider setting to 5 or 10 seconds.Avoid enabling an excessive number of standard or specialized managers. It can degrade the performance due polling on queue tables (FND_CONCURRENT_REQUESTS...). You need to create specialized managers only if there is a real need.

Avoid enabling an excessive number of standard or specialized managers. It can degrade the performance due polling on queue tables (FND_CONCURRENT_REQUESTS...). You need to create specialized managers only if there is a real need.

Purging Tables and Log Files; and Defragment the tables

Truncate the reports.log file in log directory. Refer Note.844976.1 for more details

$APPLCSF/$APPLLOG/ $APPLCSF/$APPLOUT

FND_CONCURRENT_REQUESTS, FND_CONCURRENT_PROCESSES,FND_CRM_HISTORY, FND_ENV_CONTEXT,FND_TEMP_FILES

Gather Schema Statistics Run cmclean.sql followed by Doc ID: 134007.1 and CCM.sql followed by Note: 171855.1

InfraOptimization

Concurrent Manager:Tuning Pending/Normal Requests

InfraOptimization

Concurrent Manager:Tuning Pending/Standby Requests

InfraOptimization

Concurrent Manager:Tuning Other Status Requests

Running / NormalIf the request is taking longer time, enable Database Trace and tkprof and

find out expensive DML.

Completed / Error Check the request log file and take action accordingly

Inactive / No ManagerIf “Standard Manager” has “Specialization Rules” defined which has rows with “Include”. You must only define exclude. If you add any program with “Include” options, other programs get excluded automatically.

InfraOptimization

Concurrent Manager:Tuning Inactive/No Manager Requests

Troubleshooting Concurrent Manager:Enable Tracing

Enable Tracing On Program Level System Administrator Responsibility

•

•

Navigation: Concurrent -> Program -> Define Click the “Enable Trace” button

Note:- It is not suggested as this generates trace for all the request submitted by all end- users. Hence this could generate lots of trace files which is an additional overhead and degrade performance.

Enable Tracing On Request Level (Recommended )Responsibility: System Administrator Navigate: Profiles > SystemQuery Profile Option “Concurrent: Allow Debugging”Set profile to YesLogon to the Responsibility that runs the Concurrent Program In the Submit Request Screen click on Debug Options (B)

Select the Checkbox for SQL Trace and select desired Trace level. Submit the Request.

InfraOptimization

Concurrent Manager:Tuning OptionsSleep Seconds• Sleep Seconds - is the number of seconds your Concurrent manager waits between checking the list

of pending concurrent requests (concurrent requests waiting to be started). A manager only sleeps if there are no runnable jobs in the queue.

Tip: During peak time, when the number of requests submitted is expected to be high, Set the sleep time to a reasonable wait time(e.g. 30 seconds) dependent on the average run time and to prevent backlog. Otherwise set the sleep time to a high number (e.g. 2 minutes) . This avoids constant polls to check for new requests.

• If you have multiple manager processes, sleep time is the max time to wait until the new request is picked up from the queue.

• e.g. 5 Standard managers, sleep time 60 saeconds = average time to pick up the new request from the queue if all managers are idle: 60/5 = ~12 seconds.

• Guideline for tuning: • Set «sleep seconds» = avg requirement * processes * (1 - utilization)• Example 1: Req = 20s, util =20%, proc = 3

– 20 * 3 * (1 - 0.2) = 20 * 3 * .8 = 48 seconds• Example 2: Req = 20s, util =90%, proc = 3

– 20 * 3 * (1 - 0.9) = 20 * 3 * .1 = 6 seconds , less then requirement? The problem is somewhere else!

InfraOptimization

Concurrent Manager:Tuning OptionsCache size• Cache size = The number of pending requests being fetched for execution from the requests queue• What cache size to choose?

• Small cache sizes are better for managers executing long running requests• Small cache sizes are better for managers with lots of processes• Larger cache size is good for managers with few processes executing quick (or both quick and

slow) requests• Larger cache size is good if you have a lot of new requests incoming all the time

• If at least one request was fetched, the queue will be checked again right after all the cached requests have been processed

• This reduces the significance of the cache size• Guideline for tuning:

• Increase the cache size (number of requests cached) to at least twice the number of target processes. For example, if a manager's work shift has 1 target process and a cache value of 3, it will read three requests, and try to run those three requests before reading any new requests.

Tip: Enter a value of 1 when defining a manager that runs long, time-consuming jobs, and a value of 3 or 4 for managers that run small, quick jobs.This is only guidance and a balance needs to struck in tuning the cache, so with fast jobs you need to cache to get enough work for a few minutes. With slow jobs, a small queue helps should you need to reprioritize requests.

InfraOptimization

Concurrent Requests

No Work

At least 1 job

RRA - Report Review Agent (FNDFS )

1. The Concurrent Processing server communicates with the data server using Oracle Net.

2. The resultant concurrent program log or output file from a request is passed back as a report to the Report Review Agent.

3. The Report Review Agent passes a file containing the entire report to the forms server.

4. The Forms Services component passes the report back to the user’s browser one page at time. Profile options can be used to control the size of the files and pages passed, to suit report volume and available network capacity.

InfraOptimization

Online viewing of report log and output files Viewing of reports one page at a time Storage of report output on the concurrent processing node

The log or output file that results from a concurrent request is sent to a utility known as the Report Review Agent, which passes a file containing the entire report to the Forms services, which passes the report back to the user’s browser one page at a time.

Output Post Processor - OPP

Increase Number of OPP processes

Increase the number of

Threads

OR

Concurrent Processing uses the Output Post Processor (OPP) to enforce Post Processing actions for concurrent requests.Tip :

It is suggested to increase the number of "threads" as far as you can until you start hitting the memory limit. If you still need more throughput, then you can increase "Processes". If you run lots of "Huge Reports" and running into OutOfMemoryError then you can increase the "Processes".The limiting factor with increasing "threads" is memory, if you set the max heap size of a JVM to 2GB, then all "threads" will share this memory.Concurrent -> Manager -> Define > "Output Post Processor"oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=5


Heap Size To prevent the error "java.lang.OutOfMemoryError: Java heap space“ check the current value :

SELECT service_id, service_handle, developer_parameters FROM fnd_cp_servicesWHERE service_id =(SELECT manager_type FROM fnd_concurrent_queues WHERE concurrent_queue_name = 'FNDCPOPP');

Increase the maximum Java heap size for the OPP to 1024MB ( Max 2GB):

UPDATE fnd_cp_services SET developer_parameters =

'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx1024m'

WHERE service_id = (SELECT manager_type FROM fnd_concurrent_queues

WHERE concurrent_queue_name = 'FNDCPOPP');


InfraOptimization

ErrorThe Output Post-processor is running but has not picked up this request. No furtherattempts will be made to post-process this request, and the request will be marked with Warning status.

ActionSetting the Profile Option ‘Concurrent: OPP Response Timeout’ to a higher value may be necessary.

If the value of the Profile Option ‘Concurrent:OPP Response Timeout’ is 120 then increase it to 240.

(Value * 2).

Increase the number of processes or threads (or both) .


InfraOptimization

ErrorThe Concurrent Manager has timed out waiting for the Output Post Processor to finish this Request.

Check that there are enough Output Post Processor Service Processes running. More information may be found in the Service Process logfile.

Action

Check the Profile Option "Concurrent:OPP Process Timeout". Increase Increase the setting to a higher value.

For example: Set the value to 10800 sec (3 hours).After this change, the Concurrent Request will wait maximum 3 hours, for the Output Post Processor to finish the Request.

InfraOptimization

Oracle EBS R12.2 Data Model

InfraOptimization

EBS Data ModelReference Doc

Oracle® E-Business Suite Concepts Release 12.2 Part No. E22949-12 September 2016Oracle Applications Tablespace Model FAQs (Doc ID 269293.1)

Oracle E-Business Suite Concepts Release 12.2 Part Number E22949-12Oracle E-Business Suite Setup Guide Release 12.2 Part Number E22953-15Useful E-Business Suite 12.2 Documents (Doc ID 1585889.1)

Oracle E-Business Suite Release 12.2 Information Center - Implement (Doc ID 1583153.1)

EBS File Comparison Report [VIDEO] (Doc ID 1446430.1)EBS ATG Seed Data Comparison Report (Doc ID 1327399.1)EBS Data Model Comparison Report Overview (Doc ID 1290886.1)

https://support.oracle.com/epmos/main/downloadattachmentprocessor?parent=DOCUMENT&sourceId=1327399.1&attachid=1327399.1:EBS_SEED_DATA_VID&width=100%&height=100%

https://support.oracle.com/epmos/main/downloadattachmentprocessor?parent=DOCUMENT&sourceId=1327399.1&attachid=1327399.1:EBS_SEED_DATA_VID&width=100%25&height=100%25



Database FeaturesSchemaproduct code objects (Packages, Procedures, Functions, Triggers, Views, Materialized views, Java classes, Queues) are stored in the APPS schema, whereas product data objects (Tables, Sequences, Indexes, Constraints, Queues) are stored in the relevant base product schemas, i.e. GL, INV, AP etc.

Relationship Between APPS Schema and Base Product Schemas

Schemas and Data Access

Some views access packages or functions, where the value returned by the package or function may depend on the environment having been set up properly. The environment is initialized automatically when accessing Oracle E-Business Suite through the Sign-On screen, or when using concurrent processing with Oracle Reports or SQL scripts.

Consequently, if you connect directly to a schema, the rows returned by the view may be different from those returned when you are running in an Oracle E-Business Suite environment. For example, a view may reference a profile option: when accessed from SQL*Plus, the site value of the profile option will be used, rather than the setting for a particular Oracle E-Business Suite user.

Space Management:Oracle Applications Tablespace Model (OATM)

Oracle E-Business Suite Release 12.2 utilizes as standard a modern infrastructure for tablespace management, the Oracle Applications Tablespace Model (OATM). The Oracle Applications Tablespace Model is similar to the traditional model in retaining the system, undo, and temporary tablespaces. The key difference is that Oracle E-Business Suite products in an OATM environment share a much smaller number of tablespaces, rather than having their own dedicated tablespaces.

Oracle E-Business Suite schema objects are allocated to the shared tablespaces based on two main factors: the type of data they contain, and I/O characteristics such as size, life span, access methods, and locking granularity. For example, tables that contain seed data are allocated to a different tablespace from the tables that contain transactional data. In addition, while most indexes are held in the same tablespace as the base table, indexes on transaction tables are held in a single tablespace dedicated to such indexes.

The Oracle Applications Tablespace Model provides a variety of benefits, which are summarized in the list below:

Simplifies maintenance and recovery by using far fewer tablespaces than the older model. Uses locally managed tablespaces, whose choice of extent management types (autoallocate or uniform) allows more precise control over unused space and hence can help reduce fragmentation. Takes advantage of automatic segment space management, eliminating the need for manual space management tasks. Increases block-packing compared to the older model, reducing the overall number of buffer gets and thereby improving runtime performance. Makes best use of the restricted number of raw devices available in Oracle Real Applications Cluster (Oracle RAC) and other environments, where every tablespace requires its own raw device. Maximizes usefulness of wide disk stripe configurations.

Oracle E-Business Suite Release 12.2 requires an Oracle database block size of 8K. No other block size may be used.

Oracle Applications Tablespace Model (OATM):Types & Contents

Tablespace Type Tablespace Name ContentsTransaction Tables APPS_TS_TX_DATA Tables that contain transaction data.Transaction Indexes APPS_TS_TX_IDX Indexes on transaction tables.

Reference APPS_TS_SEED Reference and set-up data and indexes.Interface APPS_TS_INTERFACE Interface and temporary data and indexes.Summary APPS_TS_SUMMARY Summary management objects, such as materialized views, and other objects that record

summary information.Nologging APPS_TS_NOLOGGING Materialized views not used for summary management and temporary objects.

Advanced Queuing/AQAPPS_TS_QUEUES Advanced Queuing and dependent tables and indexes.

Media APPS_TS_MEDIA Multimedia objects, such as text, video, sound, graphics, and spatial data.Archive APPS_TS_ARCHIVE Archive-purge-related objectsUndo UNDO Automatic Undo Management (AUM) tablespace. UNDO segments are identical to ROLLBACK

segments when AUM is enabled.Temp TEMP Temporary tablespace for global temporary table, sorts, and hash joins.System SYSTEM System tablespace.

Object type Tablespace TypeIndex Organized Tables

Transaction_tables

Global Temporary Tables

The database automatically always creates these objects in the TEMP tablespace.

Advanced Queuing tables

Advanced Queuing/AQ

Materialized View SummaryMaterialized View Logs

Transaction_tables

Domain Indexes Transaction_indexesIndexes With the exception of indexes on Tables classified as 'Transaction_Tables', all indexes will share the

same tablespace type as the base object (table/mv).

InfraOptimization

Online Patching:Pathway to Minimizing Planned Downtime

InfraOptimization

Using the Online Patching Readiness Report in Oracle E-Business Suite Release 12.2 (Doc ID 1531121.1)

Oracle E-Business Suite Release Notes, Release 12.2 (Doc ID 1320300.1)

Oracle E-Business Suite Developer's Guide Release 12.2 Part Number E22961-14

Online Patching with Oracle E-Business Suite 12.2Kevin Hudson Senior Director, Applications Technology, Oracle E-Business Suite Development September 22, 2016

Oracle E-Business Suite Release 12.2: Online Patching FAQ (Doc ID 1583902.1)

Online Patching:References

InfraOptimization

source EBSapps.env runadop -helpadop –examples

Oracle Patch Application Assistant (PAA):admsi.plperl $AD_TOP/bin/admsi.pl -patch_top=<patch-top-directory> -appspass=<apps-password>

• In 12.2 all patching operations are online by adop utility– EBS will remain available to users during patching operations. In another word, critical business

operations were not interrupted by maintenance operations– Downtime Redefined as "Cutover“, which is predictable: The time taken to bounce the Middle

Tiers• Online patching uses

– EBS Edition Based Redefinition (EBR)– Web Logic Server

Online Patching:Copy the Code, NOT the Data

• Copy the production code• Apply patches to the copy

• EBS uses both the file system and the database to store the code and data that make up the application– Code

• Stored both on the File system & In the Database• Any code object changed in a patch is copied

– Data• Stored both on the File system & In the Database• Application data is NOT copied by a patch

InfraOptimization

Edition-Based Redefinition:The key to Online Application Upgrade and Maintenance

The revolutionary new capability is edition-based redefinition:

Code changes are installed in the privacy of a new edition

Data changes are made safely by writing only to new columns or new tables not seen by the old edition. An editioning view exposes a different projection of a table into each edition to allow each to see just its own columns.

A cross-edition trigger propagates data changes made by the old edition into the new edition’s columns, or (in hot-rollover) vice-versa.

• Used by Online Users• Never changed by a Patch

• Used by the Patching Tools• Changes do no affect the running Application

InfraOptimization

AD Online Patching: ADOP Patching Procedures

Adop Phase Scope

Prepare

Gets patching filesystem ready to apply a patch.Checks that ETCC has been run against the database Oracle Home and that all required patches are applied.synchronize filesystems - If any configuration changes have been made, this will cause the prepare phase to initiate a fs_clone of the current run filesystem to the patch filesystem.run autoconfig if necessary.check tablespaces for sufficient free space before starting.check APPS_FNDFS listener to make sure concurrent requests will launch.launch concurrent request - "Online Patching In Progress" (ADZDPATCH).synchronize snapshots between run and patch filesystems.

Apply

merging patches - ADOP determines correct options automatically.patch repository location.ADOP log directories.how to run ADOP unattended by passing the required passwords into the ADOP command.how to use adctrl with ADOP.viewing the ADOP patch log file while ADOP is running.

Finalize

compiles invalids objects in the patch edition.determines DDL statements to execute at cutover.validates that the system is ready for a cutover.gathers statistics for performance.

Cutover

once started, the abort phase can not be issued.runs the finalize phase if it was missed.cancels concurrent request - "Online Patching In Progress" (ADZDPATCH).shuts down the internal concurrent manager.stops any processes that may have been started on the patch filesystem.stops the RUNNING applications tiers.executes the DDL statements prepared during the finalize phase.loads jar files into the database.synchronizes snapshots between run and patch filesystems.broadcasts a message to local users that a cutover is taking place and to re-source their working environment.

Cleanup

cleans up old editionsexplains the three methods of cleanup.QUICK cleanup performs removal of obsolete crossedition triggers and seed data.STANDARD cleanup performs removal of obsolete crossedition triggers and seed data, drops (removes) obsolete editioned code objects (covered objects). (DEFAULT MODE)FULL cleanup performs maximum cleanup, which drops all obsolete code and data from earlier editions

Online Patching Cycle

InfraOptimization

ReferencesOracle E-Business Suite Maintenance Guide Release 12.2 Part Number E22954-21Oracle E-Business Suite Concepts Guide Release 12.2OOW16 - Online Patching with Oracle E-Business Suite 12.2 [CON6710](http://www.slideshare.net/vasuballa/oow16-online-patching-with-oracle-ebusiness-suite-122-con6710)

https://blogs.oracle.com/stevenChan/entry/webcast_replay_available_technical_previewhttp://www.oracle.com/webfolder/technetwork/tutorials/obe/ebs/ebs121/atg/OnlinePatching.pdf

The Patch "19045166" (p19045166_R12_GENERIC.zip: Run the master perl script adzddiagnostic.pl with the command "perl adzddiagnostic.pl") is particularly useful in identifying the causes of issues with the adop patching utility and other AD-TXK utilities.

http://www.slideshare.net/vasuballa/oow16-online-patching-with-oracle-ebusiness-suite-122-con6710

http://www.slideshare.net/vasuballa/oow16-online-patching-with-oracle-ebusiness-suite-122-con6710

https://blogs.oracle.com/stevenChan/entry/webcast_replay_available_technical_preview

http://www.oracle.com/webfolder/technetwork/tutorials/obe/ebs/ebs121/atg/OnlinePatching.pdf

http://www.oracle.com/webfolder/technetwork/tutorials/obe/ebs/ebs121/atg/OnlinePatching.pdf

Monitoring: Online Patching Session

InfraOptimization

1. Way of knowing history of online patching

select adop_session_id,prepare_status,apply_status,finalize_status,cutover_status,cleanup_status,abort_status,status,abandon_flag,node_name from ad_adop_sessions order by adop_session_id;

Note: Y denotes that the phase is done N denotes that the phase has not been completed X denotes that the phase is not applicable R denotes that the phase is running (in progress) F denotes that the phase has failed P (is applicable only to APPLY phase) denotes at least one patch is already applied for the session id C denotes that the status of this ADOP session has completed

Note: Numerical statuses are only relevant for the cutover phase...

These status values are updated when a step has completed, and are as follows:

N denotes that the phase has not been completed0 denotes that cutover/force_shutdown has started1 denotes the "force_shutdown" step has successfully executed3 denotes the "db_cutover" step has successfully executed4 denotes the "fs_cutover" step has successfully executed6 denotes the "force_startup" step has successfully executed Y denotes that the phase is done

cutover statuses

cutover_status='Y' 'COMPLETED'cutover_status not in ('N','Y','X') and status='F' 'FAILED'cutover_status='0' 'CUTOVER STARTED'cutover_status='1' 'SERVICES SHUTDOWN COMPLETED'cutover_status='3' 'DB CUTOVER COMPLETED'cutover_status='D' 'FLIP SNAPSHOTS COMPLETED'cutover_status='4' 'FS CUTOVER COMPLETED'cutover_status='5' 'ADMIN STARTUP COMPLETED'cutover_status='6' 'SERVICES STARTUP COMPLETED'cutover_status='N' 'NOT STARTED'cutover_status='X' 'NOT APPLICABLE'

2. Check the current status of the adop cycle

adop –status

usageadop -status generates a summary reportadop -status <sessionID> generates a summary report for that session IDadop -status -detail generates a detailed report

3. Way of finding issues on different adop phases

sqlplus <apps_schema_name>/<apps_Schema_password> @$AD_TOP/sql/ADZDSHOWLOG.sql adzdshowlog.out file generated

select * from ad_zd_logs order by log_sequence desc;

4. Way of knowing the enabling problem

@$AD_TOP/sql/ADZDSHOWDDLS.sql

select count(1) from ad_zd_logs; If the count is increasing then enablement is progressing.

5. Way of knowing which patches were applied in each ADOP_SESSION_ID (patching cycle)

select * from ad_adop_session_patches order by end_date desc;

orset pagesize 200;set linesize 160;column adop_session_id format 999999999999;column bug_number format a15;column status format a15;column applied_file_system_base format a23;column patch_file_system_base format a23;column adpatch_options format a15;column node_name format a15;column end_date format a15;column clone_status format a15; select adop_session_id, bug_number, status, applied_file_system_base, patch_file_system_base, adpatch_options, node_name, end_date, clone_statusfrom ad_adop_session_patchesorder by end_date desc;

Note: STATUSN - Not Applied In the current node but applied in other nodesR - Patch Application is going on.H - Patch failed in the middle. (Hard Failure)F - Patch failed in the middle but user tried to skip some failures.S - Patch Application succeeded after skipping the failed jobs.Y - Patch Application succeeded.

C - Reserved for clone and config_clone. Indicates clone completed

select ADOP_SESSION_ID,BUG_NUMBER,CLONE_STATUS,STATUS,NODE_NAME from AD_ADOP_SESSION_PATCHES order by ADOP_SESSION_ID;

6. Log File for Errors

$COMMON_TOP/clone/FMW/logs/CLONE<timestamp>.error$COMMON_TOP/clone/FMW/logs/CLONE<timestamp>.log$INST_TOP/admin/log/clone directory$APPLRGF/TXK/ohsCloneLog ( this captures cloning failures for OHS )

7. Ensure the following RUN and PATCH file system ORACLE_HOME(s) are registered in the inventory.xml - 10.1.2 Oracle Home - Oracle_EBS-app1 Oracle Home - webtier Oracle Home - oracle_common Oracle Home more …/ContentsXML/inventory.xml |grep LOC

8. Confirm that the FND_OAM_CONTEXT_FILES table now holds the correct data

Run the following script to determine if all context files have been registered in the database. The script should return a run and patch context file for each application server and one for the database server.

select path,status,serial_number,version from fnd_oam_context_files where name not in ('TEMPLATE','METADATA') and (status is null or status !='VALID');

InfraOptimization

Run the following script to determine the number/types of managed servers registered in the run and patch file systems. The script should return an equal number of managed servers between the run and patch file systems

SELECT extractValue(XMLType(TEXT),'//host[@oa_var="s_hostname"]'),extractValue(XMLType(TEXT),'//oacore_server_ports'),extractValue(XMLType(TEXT),'//forms_server_ports'),extractValue(XMLType(TEXT),'//oafm_server_ports'),extractValue(XMLType(TEXT),'//forms-c4ws_server_ports'),extractValue(XMLType(TEXT),'//oaea_server_ports')from fnd_oam_context_fileswhere name not in ('TEMPLATE','METADATA')and (status is null or status !='H')and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type')='run'and CTX_TYPE = 'A';

SELECT extractValue(XMLType(TEXT),'//host[@oa_var="s_hostname"]'),extractValue(XMLType(TEXT),'//oacore_server_ports'),extractValue(XMLType(TEXT),'//forms_server_ports'),extractValue(XMLType(TEXT),'//oafm_server_ports'),extractValue(XMLType(TEXT),'//forms-c4ws_server_ports'),extractValue(XMLType(TEXT),'//oaea_server_ports')from fnd_oam_context_fileswhere name not in ('TEMPLATE','METADATA')and (status is null or status !='H')and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type')='patch'and CTX_TYPE = 'A';

InfraOptimization

InfraOptimization

Thank you

InfraOptimization