phelps12.1 technical upgrade best practices (3)

E-Business Suite Release 12.1 Upgrade Best Practices -Technical InsightElke Phelps Senior Principal Product ManagerE Business Applications TechnologyE-Business Applications Technology

Contributors:Isam Alyousfi


Isam AlyousfiSenior Director, Applications PerformanceUdayan ParvateDirector, EBS Release Engineering

Th f ll i i i t d d t tli lThe following is intended to outline our general product direction. It is intended for information purposes only and may not be incorporated into anypurposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be , , y,relied upon in making purchasing decisions. The development, release, and timing of any features or f ti lit d ib d f O l ' d t ifunctionality described for Oracle's products remains at the sole discretion of Oracle.

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.3 3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

R12 1 Technical Upgrade Overview

Program Agenda

R12.1 Technical Upgrade Overview

R12.1 Supported Upgrade PathsAgenda R12.1 Upgrade Resources

R12.1 Upgrade Best Practices to Minimize Downtime

Reference (Customer Upgrade ( pgSnapshots and more)

Q&A


Q&A

R12.1 Technical Upgrade OverviewOverview


R12.1 Rapid Install (RI)R12.1 Maintenance Pack (MP)( )

R12.1 was generally available (GA) in May 2009– Via Oracle Software Delivery Cloud (formerly known as Electronic y ( yProduct Delivery (EPD)) and Oracle Stores

Can be used by new and upgrading customers (11.5.9 d b ) t di tl t R12 1and above) to go directly to R12.1

– If you are on R11i, use R12.1 RI from the software delivery cloud. Follow instructions from the “Upgrade Guide: 11i to 12.1” and “12.1.1 pgRelease notes”– If you are on R12.0.X, use R12.1 MP (7303030) from My Oracle Support (MOS). Follow instructions from “R12.1 Maintenance Pack Install


pp ( )Instructions (752619.1)”

EBS R12.1 Release Update Pack (RUP) 3(12.1.3) Release( )

EBS 12.1.3 was released in Jul 2010 and delivers bugfixes and targeted functionality enhancementsbugfixes and targeted functionality enhancements Available from My Oracle Support (MOS) as a patchCan ONLY be applied after upgrade to R12 1Can ONLY be applied after upgrade to R12.1Currently, EBS 12.1.3 is the latest suite wide RUP available for R12 1available for R12.1EBS 12.1.3 Installation instructions : 1080973.1


R12.1 Technology Stack

TECHNOLOGY COMPONENT

VERSION INCLUDED

11i10CU2

VERSION INCLUDED

12.0.4 RI

VERSION INCLUDED

12.1 RI

VERSION CERTIFIED WITH

MINIMUM REQUIRED VERSION

• Apps Mid tier-Forms/Reports 6.0.8.25 10.1.2.2 10.1.2.3 - 10.1.2.3

• Apps Mid tier-Apps Mid tierJava Oracle Home/

• Apps Mid tier-JDK

1.0.2.2/1.4.2 10.1.3.0/1.5 10.1.3.4/1.6.0 10.1.3.5 10.1.3.4/1.6

• Database 9.2.0.6 10.2.0.3 11.1.0.710gr2: 10.2.0.5

11gr2: 11.2.0.1, 11.2.0.2, 11.2.0.3

10.2.0.5


For details “Oracle E-Business Suite Release 12.1 Maintenance Pack Installation Instructions” (752619.1)

11i10CU2 R12 0 4 RI R12 1 RI 12 1 3 RUP

R12.1 Key Facts11i10CU2 R12.0.4 RI R12.1 RI 12.1.3 RUP

#of Product Schemas

209 195 (25 removed,11 added)

201 (25 removed,17 added)

No changes

#file calls in DB portion of the U driver

104242 144940 156622 23408

PROD db size 31 GB 45 GB 50 GB NAPROD db size

File system size

31 GB26 GB

45 GB28 GB

50 GB28 GB

NA

#files shipped in RI / RUP

268359 357778 385755 52103RI / RUP

#of Jobs (Changed + New) in DB portion of the U d i

NA ~95488 ( Vs 11.5.10.2 ) ~104057 ( Vs 11.5.10.2 )~31843 ( Vs 12.0.4 )

~23474 ( Vs 12.1.1)


driver

R12.1 Supported Upgrade PathsPaths


R12.1 Upgrade PathsMinimum EBS suite level for direct upgrade to R12 1Minimum EBS suite level for direct upgrade to R12.1

– 11i9, 11i9CU1, 11i9CU2 or above– 11i10, 11i10CU1, 11i10CU2 or above– R12.0 and above

Minimum EBS suite level required for database versions 10 2 0 4 requires 11i9CU2– 10.2.0.4 requires 11i9CU2

– 10.2.0.5/11.1.0.7/11.2.0.1/11.2.0.2/11.2.0.3 require 11i10CU2/R12.0.4Certified upgrade path options (761570.1) can be categorized intopg p p ( ) g

A.Upgrade database and EBS level in a single downtimeB.Upgrade database and EBS level in separate downtimes


C.Apply upgrade interoperability DB patches and then upgrade EBS

R12.1 Upgrade Paths Continued..11.0/11i.X => 12.1

12 1SOURCE

TARGET

< = 11.5.812.1

11.2.0.1/11.2.0.211.2.0.311i10cu2 B ,C

B ,C

12.110.2.0.5

11.0B ,CB ,C

11i9cu2

11.5.9/cu111.5.10/cu1

12.111.1.0.7A

B ,C

12.110 2 0 4

A. Upgrade database and EBS level in a single downtimeB. Upgrade database and EBS level in separate downtimesC Apply upgrade interoperability DB patches and then upgrade EBS


10.2.0.4C. Apply upgrade interoperability DB patches and then upgrade EBS

R12.1 Upgrade Paths Continued..12.0.X => 12.1

12.1

TARGET

12.110.2.0.4

B ,CB ,C

SOURCE

12.111.2.0.1/11.2.0.2

11.2.0.3

< = 12.0.3 A

B C

A

> = 12.0.4 12.111.1.0.7

B ,C

B ,C

12.110 2 0 5

B ,CA. Upgrade database and EBS level in a single downtimeB. Upgrade database and EBS level in separate downtimesC Apply upgrade interoperability DB patches and then upgrade EBS


10.2.0.5C. Apply upgrade interoperability DB patches and then upgrade EBS

R12.1 Upgrade Resources


R12.1.3 EBS Data Model Comparison ReportPer product database object comparison between two releases for

the following object types (1290886.1)Regular tables Partitioned tables Index organized tables Global–Regular tables, Partitioned tables, Index organized tables, Global

temporary tables, Queued tables–Views, Materialized views, Materialized view logs–Indexes, Sequences, Advanced queues, Packages, Triggers

Available for R12.1.3 Vs (11i9,11i10cu2,12.0.4,12.0.6,12.1.1,12.1.2)BenefitsBenefits

–Customers can focus on what has changed–Easier to analyze impact on customizations, planned test coverage


–Differences viewable for all products in the same report via simple UI

R12.1.3 EBS ATG Seed Data Comparison ReportPer product EBS ATG Seed data type comparison between twoPer product EBS ATG Seed data type comparison between two releases (1327399.1)

– Supports most of the EBS delivered seed data types. e.g. Menus, Functions, Profiles and many more

Available for R12.1.3 Vs (11i10cu2,12.0.4,12.0.6, 12.1.1)BenefitsBenefits

– Meant for Advanced user with prior knowledge about EBS Seed data delivery

– Easier for developers/consultants/testing team to analyze impact on customizations, planned/desired test coverage

– Post Go-live, to answer end-user questions


, q

EBS File Comparison Report (1446430.1) EBS file system comparison per product for R12.1.3 Vs 11i10CU2

–Supports most of the file types found in EBS file system For ASCII files provides new removed and changed files compared to–For ASCII files, provides new, removed and changed files compared to

prior release–For binary files, provides new, removed files compared to prior release

Benefits–In upgrade planning phase, customers can get information about new, removed and changed filesremoved and changed files –Assistance in analyzing impact on customizations, personalization and planning functional flow testsDiff i bl f ll d t i th t i i l UI


–Differences viewable for all products in the same report via a simple UI

EBS “pre-install” Patches Report (1448102.1)

What is a pre-install patch ?– Any patch that fixes upgrade itself (critical upgrade failures, upgrade performance issues)performance issues)– Must be applied to the R12.1 RI file system using “adpatch preinstall=y” option

Report (revised ~monthly) logistics and usage– EBS wide Report (grouped by product family)

Patches categorized ( failure performance data fix upgrade integrity )– Patches categorized ( failure, performance, data fix, upgrade integrity )–Merge and apply R12.1 CUP1 with listed patches. Then apply merged upgrade driver


– Helps avoid time/effort chasing individual SRs for known upgrade fixes

R12.1 Upgrade Best Practices


How to minimize upgrade downtime ?

Reduce amount of “work” for production upgrade

U

Use proven best practices for the commissioned “work”

UPTI

Optimize upgrade for your hardware

IME

Tune individual long runners and use special techniques


Tune individual long runners and use special techniques

Upgrade Best PracticesIdentification of required patches

– Prepare a complete list of pre and post patches and recommended code levelscode levels

Keep the system current on AD/ATG/OAM code e.g. latest AD/ATG RUPs on 11i/R12.0 and once on R12.1Hi h i it t h f MOSHigh priority patches from MOSConsolidated Upgrade Patches (CUP) – EBS R12.1 CUP1 (7303029:12.1.0)( )– Latest pre-install patches (1448102.1)Review “Known-issues” sections from key “living” documents such as Release notes MP Install Instructions


as Release notes, MP Install Instructions

Upgrade Best Practices Continued..Patch merging, sequencing and adpatch options

– Use non-interactive patchingMerge patches (228779 1) Merge NLS patches per language– Merge patches (228779.1). Merge NLS patches per language.

– Perform uptime maintenance when possible Hot patching of iHelp, NLS patches, HRGLOBAL (1330470.1)g ( )Upload patch history

– Use adpatch options such as nomaintainmrc, phtofile, nolink, nogenform nogenrep nocompile jsp noautoconfig novalidatenogenform, nogenrep, nocompile jsp, noautoconfig, novalidate(1078973.1)


Upgrade Best Practices Continued..Downtime reduction procedures– Run autoconfig in parallel on a multi-node system (387859.1)


Upgrade Best Practices Continued..Downtime reduction procedures

– Use “Shared APPL_TOP” (384248.1) with “Distributed AD” for (236469.1) upgrades and regular maintenance for multi-node instances

No need to apply the same patch on multiple tiers

Distributed AD adds to the degree of parallelism by distributing AD workers across application tier nodes and improves timing for D/G portion of the patch driver.


Upgrade Best Practices Continued..Shared APPL_TOP and Distributed AD

adpatchworkers 1-10

Admin/CMServer

adpatchworkers 1-10

adpatchworkers 1-10

FormsServer

adpatchworkers 11-20

Sh d

adpatchworkers 1-10

WebServer

D b


Shared APPL_TOP

adpatchworkers 1-10Web

Server

DatabaseServer



Upgrade Best Practices Continued..Staged APPL_TOP (734025.1)

ProductionProduction

1 Cl1 Cl 4 S P t h Hi t3 S APPL TOP 1. Clone1. Clone 4. Sync Patch History3. Sync APPL_TOP

3. Update DB

Stage2. Upgrade to R12

2. Upgrade DB


Upgrade Best Practices Continued..Downtime reduction procedures continued..

– Use TUMS (“The Upgrade Manual Script”) To avoid running tasks not relevant to your systemTo avoid running tasks not relevant to your system

– Import statisticsExport statistics gathered during test runs and import during finalExport statistics gathered during test runs, and import during final run

– Automate as much as possible– Automate as much as possible

– Time every activity and deal with bottlenecks


Upgrade Best Practices Continued..Pre-upgrade activities

– Identify and execute tasks that could be completed in a separatedowntime period, prior to the production upgradedowntime period, prior to the production upgrade

Use applicable steps mentioned in the "Downtime reduction" and “Upgrade By Request” appendices E and G of the R12.1 upgrade guide

Upgrade RDBMS version to latest certified for the current APPS level ( 11.2.0.2 / 11.1.0.7 / 10.2.0.5 )

Convert to Oracle Applications Tablespace Model (OATM)Convert to Oracle Applications Tablespace Model (OATM)– Compacts data, optimizes storage settings and I/O

Other planned HW or OS upgrades


Other planned HW or OS upgrades

Upgrade Best Practices Continued..Pre-upgrade activities continued...

– Convert to Multiple Organization architecture (DocumentConvert to Multiple Organization architecture (Document 210193.1)

– Drop MRC schema ( 11.5.10 and above )p ( )

– Assign post upgrade jobs to specialized CM queue (by request type, see Document 399362.1)request_type, see Document 399362.1)



– Flush all the interfaces, such as Autoinvoice, Journal entry import order import etcimport, order import etc..

– Review and disable all debug, logging at site, responsibility, user levellevel

– Gather stats with GATHER_AUTO option for all schemas close to the start of downtimeto the start of downtime


P d ti iti ti d


– Pre-create new large indexes created by upgrade constrained to indexes on existing columnsconstrained to indexes on existing columns

– Minimize historical data to be upgraded as per business requirements –“Upgrade By Request”Upgrade By Request

Post-upgrade “hot-patch” of additional historical data outlined in 604893.1

– Purge old and/or transient data before upgradingOver 260 standard purge programs in R12


g gUse new “Purge Portal” in OAM to administrate

Upgrade Best Practices Continued..Purging

– Use OAM to configure, initiate and monitor purge programsSet the execution frequency and view program historySet the execution frequency and view program historyPrograms tagged with the “Purge” program type

System Administrator >Oracle Applications Manager >Purging/Critical Activities


Upgrade Best Practices Continued..

Key recommended fixes – Affecting Upgrade Performance

AD-Parallel improved scalability: 8917381 (Part of 9179588 AD CUP)Forms/reports generation regression with Database11g: 8557019p g g gOptimizer:Dynamic Sampling on indexes 12942119 (11.2.0.2)

– GeneralR i 244040 1 f l t t EBS d d f fiReview 244040.1 for latest EBS recommended performance fixes


Upgrade Best Practices Continued..Downtime reduction procedures continued..

– For the duration of the upgrade, consider…Disabling custom triggers and business eventsDisabling custom triggers and business eventsDisabling auditing if enabledIf possible run in noarchivelog mode

fDisabling flashback DBRemoving Transparent Data Encryption (TDE) from volume tables



Database tier configuration– Maximize SGA and PGA sizing

An upgrade only involves 10's of concurrent sessions; starting rules of thumb ...

– log buffer = 30 to 100 Mbg– shared pool = 1 to 4 Gb– pga target = 3 to 20 Gb

b ff h lti Gb b ith t i– buffer cache = multi Gb, be generous without causing excessive paging or swapping

Adjust with help from AWR pool advisories



Database tier configuration– Other upgrade specific init.ora changes

If specified, remove db_file_multiblock_read_count – Maximize multiblock I/O sizes

Set job_queue_processes = # of CPUS– adobjcmp.sql (Phases : plb+90 and last+63)

Set parallel_max_servers = 2 X CPUsHelps with large index creation stats gathering and some large upg+– Helps with large index creation, stats gathering and some large upg+ phase jobs

– Need to test with production-like DB server and I/O subsystemSh td th RAC i t


– Shutdown other RAC instances


Batch size and #workers– Batch size

10K is suitable for most installs, you can test other values from 1K up to 100K if time allows

– # WorkersStarting rule-of-thumb is between 1 and 1.5 x #COREs

– It is critical to do multiple rounds of testing, adjusting above settings to maximize server utilization, but constrained by factors such as , y

Memory utilization (no swapping/ excessive paging)CPU utilization (scale down if at 100%)I/O response times (scale down if averages > 20 ms)


I/O response times (scale down if averages > 20 ms)


Performance testing, monitoring and additional optimizations...– Analyze long runners via timing report, ad_task_timing analysis – Mine ad_task_timing to identify low worker utilization due to phasing

waits and review responsible culprits– Review targeted AWR Instance and SQL reportsg p

awrrpt.sql and awrsqrpt.sql

– Use My Oracle Support to check for known issues and workarounds for the longest running jobsfor the longest running jobs



Performance testing, monitoring and additional optimizations continued...

O t id th b ti i ti R i S t A l*– Outside-the-box optimizations – Requires Support Approval*Long running index creation and Statistics gathering

– As these tasks run with little or no concurrent tasks, id i lt t d t iconsider using alter system commands to increase

parallel_max_servers and pga_aggregate_target(Need to test so as to not exhaust machine resources)

– Large indexes: Pre-create, but do not alter column definitions

– For stats (adsstats.sql ; phase: last+63) : Consider stats i t RAC i ll d h th i t t


import or on RAC using all nodes when gathering stat


Performance testing, monitoring and additional optimizations continued...

O t id th b ti i ti R i S t A l*– Outside-the-box optimizations – Requires Support Approval*Long running mv related xdf’s (“en” phases)

– Cleanup/truncate large MV logs (requires MV complete refresh) Review long running jobs that you think are NA for your system

– Thousands of jobs and customer product/data/setup configurations

– Check with Oracle Support to validate– A fix maybe available or possible to optimize for your case


Customer Upgrade Snapshots


Customer Upgrade Snapshots Continued...11i to 12.1.3

• Toyota Motor Europe– Release: 11 5 9+ on IBM AIX to R12 1 3 on Oracle Linux 5 Appl tier– Release: 11.5.9+ on IBM AIX to R12.1.3 on Oracle Linux 5 Appl tier– DB size: 800 GB– #Workers: 32– #CPUs on DB server: 8 cores– Downtime reduction measures

• Online NLS patch application– #hrs for the 12.1.1 D driver: 21 hrs– #hrs for the 12.1.3 US upgrade: 4 hrs– #hrs for the 4 languages NLS patching 11 hrs


Customer Upgrade Snapshots11i to 12.111i to 12.1

CPS (Chicago Public Schools)R l 11 5 10 2 t 12 1– Release: 11.5.10.2 to 12.1

– DB size: 900GB– #Workers and batch size: 32, 10000– #CPUs on DB server: 2 node RAC, 8 CPUs per node– Downtime reduction measures

Distributed ADDistributed ADUpgrade RDBMS to 10.2.0.4 in a separate downtime # hrs for the D driver: 22 hrs

Customer snapshot


– Customer snapshothttp://www.oracle.com/customers/snapshots/chicago-public-schools-ebs-snapshot.pdf

Customer Upgrade Snapshots Continued...11i to 12.0.611i to 12.0.6

Cisco– Release: 11i to R12.0.6– DB size: 600GB– #Workers and batch size: 32, 20000– #CPUs on DB server: 16– Downtime reduction measuresDowntime reduction measures

Distributed AD– #hrs for the D driver: 5.5 hrs


Customer Upgrade Snapshots Continued...11i to 12.1.311i to 12.1.3

Dell – Release: 11i10 to R12.1.3 – DB size: 15TB , 16 node RAC Cluster– #Workers and batch size: 32 10000#Workers and batch size: 32, 10000– #CPUs on DB server: 8– Downtime reduction measures

Distributed ADPre-create large indexes

#hrs for the D driver: ~30 hrs


– #hrs for the D driver: ~30 hrs

Customer Upgrade Snapshots Continued...11i to 12.111i to 12.1

GE– Large EBS HRMS Implementation (~300K employees,10 Lang) g p ( p y , g)– Release:11.5.10.2 to R12.1.3, DB size: 838 G– Hardware: App Tier- 2 SUN T5240’s(64x64),

DB Tier - SUN M8000 (12 Dual Cores)– #Workers and batch size per App Server: 48, 10000– Downtime reduction measuresDowntime reduction measures

Distributed AD, Staged APPL_TOP

– #hrs for D driver:~10 hrs US, ~13 hrs NLS (11.5.10.2 -> R12.1.1)#h f DB P ti 2 5 h US 1 h NLS (R12 1 1 R12 1 3)


– #hrs for DB Portion:~2.5 hrs US, ~1 hr NLS (R12.1.1 -> R12.1.3)

Customer Upgrade Snapshots Continued...12.0 to 12.112.0 to 12.1

Zebra Technologies CorporationRelease: 12 0 6 to 12 1– Release: 12.0.6 to 12.1

– DB Size: 106GB– #Workers and batch size: 32, 10000– #CPUs on DB server: 8– Downtime reduction measures

Staged APPL TOPg _– #hrs for the U driver: 12 hrs– Customer snapshot

http://www oracle com/customers/snapshots/zebra-technologies-corporation-ebs-snapshot pdf


http://www.oracle.com/customers/snapshots/zebra technologies corporation ebs snapshot.pdf

Customer Upgrade Snapshots Continued...12 0 to 12 1

Oracle GSIRelease: 12 0 3+ to R12 1

12.0 to 12.1

– Release: 12.0.3+ to R12.1– DB size: 17TB– #Workers and batch size: 60, 10000

#CPUs on DB ser er 88 processors– #CPUs on DB server: 88 processors– Downtime reduction measures

Staged APPL_TOP for US and ten languages Ran data fixes for problems found in test upgrades prior to productionRan data fixes for problems found in test upgrades prior to production

upgrade to minimize stoppagesDistributed AD (4 servers,15 workers each)Pre-create large indexes (6 hrs savings)


– #hrs for the D driver: 14 hrs

Customer Upgrade Snapshots Continued...12 1 to 12 1 3

Oracle GSIRelease: 12 1+ to R12 1 3

12.1 to 12.1.3

– Release: 12.1+ to R12.1.3 – DB size: 17TB– #Workers and batch size: 200, 10000

#CPUs on DB ser er 150 processors– #CPUs on DB server: 150 processors– Downtime reduction measures

Staged APPL_TOP for US and ten languages Ran data fixes for problems found in test upgrades prior to productionRan data fixes for problems found in test upgrades prior to production

upgrade to minimize stoppagesDistributed AD (4 servers, 50 workers each)Pre-create large indexes



Customer Upgrade Snapshots Continued...12 0 to 12 1 2

AT&T Release: 12 0+ to R12 1 2

12.0 to 12.1.2

– Release: 12.0+ to R12.1.2– DB size: 10 TB– #Workers and batch size: 40, 10000

#CPU DB 32 P– #CPUs on DB server: 32 Processors– Downtime reduction measures

Staged APPL_TOP for US and ten languages Di t ib t d ADDistributed AD



References


ReferencesR12.1 documentation roadmap (790942.1)Oracle E-Business Suite Release 12.1 Info center (806593.1)D t b ti id li f R12 1 d (761570 1)Database preparation guidelines for R12.1 upgrade (761570.1)Patching FAQs (459156.1, 225165.1)Staged or shared APPL TOP and distributed AD (734025.1,Staged or shared APPL_TOP and distributed AD (734025.1, 384248.1, 236469.1)OAM “Patch Wizard” overview and FAQ (976188.1, 976688.1)AD Command Line Options for Release R12 (1078973 1)AD Command Line Options for Release R12 (1078973.1)Recommended Performance Fixes (244040.1)R12 Upgrade Sizing & Best Practices (399362.1)


pg g ( )

Additional Resources


Additional ResourcesEBS R12.1 Upgrade related Reportspg p

– EBS 12.1.3 Data Model Comparison Report (1290886.1)– EBS ATG Seed Data Comparison Report (1327399.1)

EBS File Comparison Report (1446430 1)– EBS File Comparison Report (1446430.1)– EBS pre-install patches Report (1448102.1)

White paper– Planning Your Oracle E-Business Suite Upgrade from Release 11i to

Release 12.1 (987516.1)– R12 Upgrade considerations by product: Financials (889733.1)– Oracle E-Business Suite Upgrades and Platform Migration

(1377213.1)Have Upgrade questions ? Please post on OTN R12 upgrade forum


http://forums.oracle.com/forums/forum.jspa?forumID=395&start=0

phelps12.1 technical upgrade best practices (3)

Documents