e-Business Suite on Linux RAC (Platform Transition through

Upgrade)

March 2005

Chakri Reddy Director of DeliveryZensar TechnologiesSan Jose, CA.

Ram LaxmananDirector, PMO OfficeZensar TechnologiesSan Jose, CA.

Agenda

• Who we are• Description of project Goals• Hardware Architecture & Scaling Plan• 11i Linux / RAC Architecture• Upgrade Process Overview • Upgrade Takeaways• Infrastructure Issues• Global Team Tactics• Questions & Answers

Who we are

• Zensar Technologies (www.zensar.com)– Global information technology organization– Decade of experience of providing IT and IT enabled service – Gained over 7500+ person years of experience and executed over 1250+

engagements– Leading software exporter in India, over 2000 skilled resources across six

delivery centers– 23 offices worldwide

• Zensar’s established Oracle Applications Practice– Enterprise Applications Group (EAG) is a focus area in packaged

solutions– Dedicated Oracle competency center, with over 600 skilled resources– Engagements include implementations, upgrades, customizations, 3rd

party integration and on-going support– Customers include Cisco, Logitech, Fujitsu, Fluke, PO Ned Lloyd and

Sprint

About Electronic Arts, our customer

• The #1 video game publisher in the US• Over a hundred popular titles such as NBA Street, Madden

NFL 2004, SimCity 4, Need for Speed, and Medal of Honor.

• Has also published extremely successful games based on Hollywood franchises, including The Lord of the Rings, Harry Potter and James Bond

• Total revenue for year 2003-04 : $3 billion• Has offices throughout the Asia-pacific, Europe and North

America – EU territories contributed to over $1 billion of revenue last year

Project Goals (Business)

• Move all business applications from RISC platform to Linux

• Plan for future growth and scalability• Cost savings in terms of licensing and hardware• Rapid deployment• Save $$$’s• OM BPR• Review and update customisations

Project Goals (Technical)

• Standardize the installation of Oracle Applications 11.5.9 across 4 global installations in Electronic Arts

• Upgrade to Oracle 11.5.9 e-business suite of products• Transition to e-business suite running on Linux RAC

– Along with the upgrade• Produce a common and extensible hardware environment

– That meets EA’s user and performance criteria, currently and into the future

• Introduce a hardware and software architecture that would permit both scaling of capacity and continuous operations

• The upgrade and platform transition, including user testing must be accomplished inside of 130 hours

Fact sheet

• Organizations, 1 in US, 21 in EU• Languages, 1 in US, 4 in EU• Database size, 200Gb in US, 350 in EU• Users, 175 in US, 350 in EU• Transaction volumes

– 9000 order lines booked per day– 30000 picking line details per day– 10000 concurrent requests per day

• Boundary systems, 15 in US, 25 in EU• Customizations, 1000 each in US and EU• Dual on-shore / off-shore delivery model

Hardware Architecture & Scaling Plan• Dell was selected as the vendor of choice at EA for world wide

deployment of Oracle 11.5.9• RedHat 2.1 was the operating system at all tiers of the architecture• “Multi-node” & “Multi-tier”• Combination of both local storage and SAN storage

– Intelligently deployed to get the highest performance and greatest economy from Oracle Applications

• Hardware load balanced Oracle Applications servers– Easily configured and robustly implemented.

• Dual concurrent process servers– Assuring rapid process completion – Confident & continuous operations, deployed with PCP (parallel concurrent

processing)

• Dual RAC nodes

11i Linux / RAC Architecture

IBM Compatible

IBM Compatible

IBM Compatible

IBM Compatible

User Tier

Load Balancer

Applications Server 01

Applications Server 02

SAN Storage Device

Application Tier Database Tier

Concurrent Server 01

Concurrent Server 02

DB Server 01

DB Server 02

Linux Production Hardware Diagram

EU Production Configuration

2 Applications Servers = Dell 2 processors 4 gig memory2 Concurrent Mgr Servers = Dell 2 processors 4 gig memory2 Database Servers = Dell 2 processors 4 gig memory

11i Linux / RAC Architecture

Storage

App 1

CM 1

DB Node1

App 2

CM 2

DB Node2

DatabaseDatafiles

L4 Switch

Load Balancer

RAC

Upgrade Process - Overview• Source

– Oracle Applications Release 11.03 with 8i DB running on HPUX– Single Node Configuration– 8 Processors at 575 MHz Speed

• Target– Oracle e-Business Suite Release 11.5.9– 2 Application Servers (H/W Load Balance)– 2 Concurrent Manager Servers with PCP Configuration– 2 RAC Load Balanced DB Nodes

• Upgrade Starts With– Single Node RISC Server Running 11.03– One Intel Linux Application Server– One Linux CM Server– One Linux DB Server

Upgrade Process - Overview

Fina

l Bac

kup

CA

T 1

& 2

Fini

shin

g St

eps

and

Test

ing

Rol

l Out

C

onne

ctiv

ity

Info

rmat

ion

Oracle 11.5.9 9.2.0.4

EA EU CRP-2 Linux Upgrade Model

Oracle 11.0.3 8.1.7.3 DB

eucr1db1

Rec

eive

and

Val

idat

e Li

nux

Har

dwar

e Pa

latfo

rm a

nd

HP

DB

ser

ver

Linu

x A

pp S

VR a

nd C

M

SVR

Con

figur

atio

n

Rec

eive

Sin

gle

Nod

e Su

rce

DB

& A

pp F

rom

EA

Linu

x R

AC

Sof

twar

e In

stal

l an

d C

onfig

urat

ion

Prep

are

for F

unct

iona

l and

D

BA

Cat

egor

y St

eps

RA

PID

Inst

all A

pps

CA

T 3

DB

Upg

rade

8.1

.7.4

Firs

t Bac

kup

Oracle 11.0.3 8.1.7.3 DB

eucr1db1 App SVR 01

App SVR 02

CM SVR 01

CM SVR 02

Pre

Aut

oUpg

rade

Valid

ate

Ora

cle

9.2.

0.4

& R

AC

Inte

rnal

Use

r Te

stin

g

Con

duct

H

ando

ver

Act

iviti

es

Gai

n A

ppro

val t

o R

elea

se th

e In

stan

ce

Aut

o - U

pgra

deS

econ

d B

acku

pA

uto

- Pat

ch

CA

T 4,

5, 6

Fun

ctio

nal

CA

T 4,

5, 6

DB

A +

NLS

Para

llel E

xpor

t/Im

port

DB

Inde

x C

reat

e

Oracle 11.0.3 8.1.7.4 DB

eucr1db1 App SVR 01

App SVR 02

CM SVR 01

CM SVR 02

Linux 11.5.9 APPS

Third

Bac

kup

Four

th B

acku

p

Oracle 11.0.3 8.1.7.4 DB

eucr1db1 App SVR 01

App SVR 02

CM SVR 01

CM SVR 02

Linux 11.5.9 APPS

Linux CM Servers

Linux 11.5.9 App SVR

Linux 9.2.0.4 RACDB

Phase 1

Phase 2

Phase 3

Linux 9.2.0.4 RACDB

Linux CM Servers

Linux 11.5.9 App SVR

Oracle 11.5.9 9.2.0.4

Downtime begins

4 hr

4 hr

5 hr

5 hr

*

13 h

r

23 h

r6

hr *

46 h

r

6 hr

*

18 h

r

6 hr

*

22 h

r 15 h

r

6 hr

*

4 hr ? hr

? hr

? hr

? hr

Downtimeends

11.5

.9.2

Cum

ulat

ive

Patc

h

cv

10 h

r

Mul

ti-no

de +

RA

C2

hr

Notes:

* Indicates actual backup time. Expectation is ~15 min backup time using mirror/split

Estimate times in italics e.g. Additional Setup

Upgrade = 188Backup = 29Total = 217

Tran

sfer

to V

irgin

ia1

hr

Post

Aut

opat

ch5

hr

Post

Impo

rt6

hr

Add

ition

al S

etup

8 hr

Pre

Expo

rt2

hr

Upgrade Process - Overview

HP ServerSingle-node Linux 11.5.9

App 1

Oracle 11.0.38.1.7.3

Oracle 11.5.99.2.0.4

Cat 1-3 Auto Upgrade

Auto Patch

11.5.9CM2 Cat 4-6

CM 1

Ora

cle

App

s 11

i – L

inux

App

s/C

M, H

P D

BExport & Import

Linux 11.5.9

App 1 App 2

CM 1 CM 2

DB Node1 DB Node2

ExportImport

IndexBuild

RAC (DB2)App2CM2

Upgrade Process - Overview• Customized Oracle Upgrade Approach

– Heterogeneous upgrade approach• Admin node running on Linux and DB node running on RISC server

– Pre upgrade steps• Rapid install (Linux application server & CM server)• 9i DB Installation (Linux DB server)

– Category 1, 2 and 3 steps• Perform all relevant category steps on RISC server running 11.03• Gather schema statistics (30%)• Backup applications

– Auto Upgrade• Admin node running on Linux and DB node running on RISC server• Batch size of 10000 & max workers of 20• Backup applications

– Auto Patch• Gather schema statistics (30%)• Backup applications

Upgrade Process - Overview• Customized Oracle Upgrade Approach

– Patch the applications• Apply consolidated update patch 2• Apply one off patches

– Category 4, 5 and 6 steps• Perform all relevant category steps• Apply customizations (if any)• Apply conversions (if any)

– e-Business suite functioning with• Linux application server• Linux CM server• RISC DB server

Upgrade Process - Overview• Customized Oracle Upgrade Approach

– Export source DB• Apply applications consolidated export import patch

(2981815 ) for 9i on SOURCE admin node. • Direct path full DB export

– Import DB• Apply patch # 2873456• Point your APPL_TOP to the 9i DB• Full system import or• Two phased import process where structure is imported first

and data is subsequently imported.• Index builds are always last with either of the import choice• Post import activities

– Gather schema statistics– Modify TNS names on Admin and CM nodes to point to 9i DB

Upgrade Process - Overview• Customized Oracle Upgrade Approach

– Converting to RAC DB• Modify ORACM configuration file• Follow Metalink documentation

– Adding Second Application Server• Change the application server name to load balancer• Review H/W load balancer setup• TAR and UNTAR the APPL_TOP onto second node

– Adding Second CM Server• Create second CM node by cloning from first CM• Setup Internal Monitor (ICM)• Define Primary & Secondary nodes for all Concurrent Managers• Define the transaction managers for all the instances in RAC cluster

Upgrade Takeaways

The Upgrade – Category 0The Upgrade – Category 0

• Total Production downtime of 98 hours• Perform as many iterations as possible

• 12 practice iterations were performed• Impact of hardware• Import process needs lot of attention• Impact of Orgs and Languages

• Be generous with your instance planning – you will need them!• Refine backup methodology, to provide total database backups as

rapidly as technically achievable• Perform Validate APPS Schema script on production regularly and fix

the issues• Make time for high quality testing of both standard and custom Oracle

Applications code– 20 One off patches applied on consolidated patch 2– Perform load test and regression testing

Infrastructure Issues

• Instance Planning– Differences from Traditional Unix to Linux platforms– Impact of server numbers– Don’t forget the disk space!

• Important considerations– Data Archiving– Getting it all to work! e.g. load balancing, DNS etc.– Log/out files, UTL, FTP– Individual server memory footprint

Global Team Tactics

• Global Project technical, functional and support teams• Managing communication• Coping with time zones

– Make it work to your advantage

• Sharing of knowledge– Client/Vendor sharing

• Examples of success

Agenda Revisited

• Who we are• Description of the Goals• Hardware Architecture & Scaling Plan• 11i Linux / RAC Architecture• Upgrade Process Overview • Upgrade Takeaways• Infrastructure Issues• Global Team Tactics• Questions & Answers

Q&A

Questions?

Zensar Technologies

Chakri Reddychakri@usa.zensar.com

Ram Laxmananr.laxmanan@usa.zensar.com