1 ©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice

MKS Instruments: validating the performance impact of migrating from Oracle E-Business to Oracle RAC and LinuxKevin Danehy

Director Global Applications

Dan Downing

VP & Principal Consultant

2

Objectives

• Convey business drivers and benefits of Oracle EBS Intel/Linux

migration

• Share best practices

– Modeling workload & designing a complex LoadRunner scenario

– Benchmarking

– Monitoring a distributed network traffic

– Interpreting results

Target Audience

• Oracle IT, PM planning and executing an effective

Oracle EBS load test

• Performance Tester tips and techniques

Objectives & Audience

Introductions

3

• Performance testing, tuning and managed hosting

• Specialize in ERP applications

• SaaS & Cloud load test delivery

• Application hosting in Top Tier data centers

• Mercury/HP Alliance Partner for 12 years

• Boston – Atlanta – DC - Dallas

for critical applications

3

MKS Instruments is the leading global provider of critical

technologies that enable advanced processes for

Semiconductor and other innovative, high growth

markets.

• #1 Critical Technology Supplier

• 2009, Semi – 51%; Other Markets – 49%

• Broadest technology portfolio

• Largest Critical supplier to Semi WFE market

• Global, Public Company

• Founded in 1961

• Strong financial and cash position

• 2200 employees worldwide

• Manufacturing in N. America, EU and China

• Support locations worldwide

Leading global provider of

technologies that

enable advanced

processes

for Semiconductor and

other innovative markets

3

4

MKS Products

• Measure

• Control

• Power

• Monitor

• Analyze

Critical process

parameters in

advanced

manufacturing

5

MKS Markets

6

MKS Around the Semiconductor Process

7

MKS IT Organization

8

MKS Outsourced Activities

Perot

Keane

Tricore

Solutions

9

Oracle Footprint at MKS

• Implemented Oracle eBusiness 11.5.9 in 2005– Order Mgt, AR, GL (Order-to-Cash)

– Purchasing, AP (Procure-to-Pay)

– Shop Floor Control, Inventory

– Depot Parts Repair / Install Base

• Interface Systems– Product Lifecycle Mgt (Omnify)

– Pack/Ship/Denied Party/Export (Kewill)

– Labels/Customer Facing Docs (Optio)

– Auto-invoicing (Oracle MWA)

• By 2008: 3 Plants rolled out– Andover MA, Rochester NY, China

– 650 users

– ~70% of worldwide business

• Original Production environment– IBM / AIX

– 3 Oracle App servers , Oracle9i DB

– 1.5 TB database

– Hosted at Perot in Plano TX

Oracle Infrastructure Transformation

• Problem Statement

– In 2005 MKS chose the Perot Systems Compute Utility Model to lease

its hardware and storage in support of Oracle E-Business Suite. Based

on current usage, future growth trends and changes to overall market

conditions, this solution is no longer cost feasible for MKS.

– The current Oracle environment is IBM hardware on IBM‟s proprietary

operating system. Is this right for MKS? Other platform choices?

• Approach:

– In May of „08 MKS IT began discussions with Perot to transform the

current model and infrastructure to a new architecture

– MKS IT consulted Perot, external subject matter experts and third party

consumers of data center services

– End state to determine a recommendation, investment and the cost

savings for MKS to transition from the Perot Compute Utility Model to

an owned and hosted infrastructure within the Perot data center

10

Why Linux? Decision Drivers

Drivers confirmed through discussions with other organizations that

have faced this decision

– Anchor Hocking: $300M; consumer glassware; Perot; deployed

– Clopay Doors: $1.2B; consumer building materials; deployed

– Cabot Microelectronics: $350M; semicon manufacture; deployed

– The Warranty Group: Private; consumer support services; deployed

– Harvard Pilgrim: $2B+; health services; Perot; migrating

– KLA Tencor; $2.5B+; semicon manufacture; actively considering

Drivers MKS

In-Source/out-source mix Out-source

Skill sets of supporting staffStrong skills within

Perot

Install base of business applications Small

Historical investment/sunk costs None

Cost to change Attractive ROI

Influence of relationship with IT vendors Open

11

MKS Oracle Infrastructure: What Fits at MKS?

Category Definition IBM/AIX Intel/LINUX

Functionality & Versatility

Features and function: solutions equal.

Upward scalability, with flexibility; “growth on

demand”? AIX

Vendor SupportWhich Solutions support model is best aligned

with dependant technologies (Oracle)

Technology AdoptionWhich technology has greater acceptance

within the industry over time?

CostsCost Comparison of options; see detailed

analysis

Technology and Skill

Sets

Which technology provides the better skill set

match? Which technology requires a higher

capability to support?

Innovation & Future

Capability

Which technology solution offers the higher

innovative “upside”?

Portability

Which solution is more portable across service

providers, if MKS chooses to change service

providers?

Strategic Fit

How does the solution fit into MKS‟ long term

technical strategy? Oracle shop versus

mainframe scalability

Slight Advantage

Distinct Advantage

Market Differentiator

12

Server Migration

13

Net change:

16 24 cpus

64 288 GB Memory

$$ SAVED:

Substantial

14

Business Drivers for Load Testing

• Cost reduction

– Migrating from IBM/AIX to Linux to reduce OS licensing $, HW maintenance $

– Implementing Oracle 10g RAC for scalability improvement

• Workload increase

– Adding 80 users (20%) at new plant (Wilmington MA)

• Will the new Intel/Linux architecture standup to the load?

15

Test Objectives

• Validate that Linux performance “equal to or better” than AIX 15 business workflow benchmark comparisons

• Validate that peak workload can be achieved 4200 mixed transactions per hour during peak processing time

Background Concurrent Managers triggered by interactive transactions

• Validate network capacities Rochester, Andover & Wilmington segments adequately sized for

traffic load

• System capacity meets criteria We/App & Oracle RAC servers show good load balancing and

growth headroom

Dedicated Concurrent manager server is a good solution

16

Testing Challenges

• Script updates Modify / re-develop script logic to match workflow changes

Modify script “init” sections to match changed Oracle Responsibilities

• Cascading data dependencies, single-use data Not new, but need to plan data flows and single-use data carefully

Plan, develop & execute “test data staging”

Schedule DB snapshots and restores

• Distributed load generation Prep, ship, install, test load generators and three locations

• Functional acceptance testing being done in parallel Schedule performance testing around this activity

On the Plus Side…

• Strong executive support CIO sponsorship, IT Director commitment

• Strong IT support Effective IT PM coordinating business lead support

Associated Perot PM to facilitate infrastructure support

Performance-knowledgeable Oracle Architect

• Benchmark from 2007 performance test for comparison

• Array of tools & tech team for system monitoring LoadRunner for Unix server monitoring

BMC Patrol for Oracle monitoring

Wireshark for network packet monitoring

17

18

Discover

Select processes

Define workflows

Model business workload

Approve Test Plan

Run/Fix

Run tests

Monitor system resources

Analyze results

Diagnose, fix

Re-run

Report

Interpret results

Develop conclusions

Write Performance Profile report

Present to stakeholders

Develop

Configure load server

Develop scripts

Configure resource monitors

Run shakedown tests

5 - Report1 - Discover 2 - Develop

Mentora’s 5 Steps of Performance Testing

19

MKS Load Test Environment

MKS Rochester NY

MKS Wilmington MA

MPLS 3 Mbps

MKS Andover MA

LR Injector

LR Controller

MPLS 6 Mbps

LR Injector

Oracle App

Servers

Oracle EBS

Perot Systems Plano TX

MPLS 15 Mbps

Oracle 3-node RAC

Dell EMC CX4-480

Oracle

DB

20

• Parameterize initialization for portability across environments Base url

Web server port number in login authentication

nca_connect Forms server port number

nca_connect “config”

nca_connect “module path”

• Example/*Oracle launch*/

web_browser("ebsr12.com:9097",

DESCRIPTION,

ACTION,

"Navigate=http://{app_srv}:{port_web}/", LAST);

/*Forms server connect*/nca_connect_server("{app_srv}", "{port_forms}",

"module=/ebstop/{module}/apps/apps_st/appl/fnd/12.0.0/forms/US/FNDSCSGN fndnam=APPS record=names config='{config}' icx_ticket='.{p_ICX_Ticket}..' resp='AR/CAN_CUSTOMER_MASTER_ADMIN' …");

Tips: Script Development

21

Tips: Workload Design - Users &Transactions

Modeled target

workload by

selecting the

functions with

the highest

transaction

volumes and

users

22

Tips: Plan Cascading Data Flows

• Map the data flow and dependencies

• ID the single-use data Design your data prep runs accordingly

Keep track of “used” data!

O2C-1

Enter Standard

Order

Customer

Name

Sales

Order #s

O2C-4

Run Pick Slips

O2C-5

Move Orders

OPS-5

Release Planned

Orders

OPS-11

Misc Inventory

Transaction

Released

Inv Items

Released

Orders

FIN-5

AP Voucher Entry

FIN-6

AP Inquiry

PO, Inv Date, Inv

Amount, Receipt#

Invoice

Lists

Paid

Invoices

Serialized Item,

Release Rule

Item#

OPS-7

Create Requisition

OPS-9

Create PO Line,

Release

OPS-10

Receive PO Line;

Std Inspection

PO# Serial#Requisition#

MKS

Inventory

Setup

DB Extract

Data

Process

Key:

Config-

uration

SQL Script

Item Ship

Orders

Data

Generation run

23

Tips: LoadRunner Scenario Design

Create an Excel model to develop the group schedule times: Start,

Launch, Duration, the iteration Pacing time, and injector load balancing

Finally, the LoadRunner Scenario!

24

Sure would be nice if there was an

Excel interface to Scenario Design!

Tips: Resource Monitoring

25

Manually monitor jvm heap memory

MKS Rochester NY

MKS Wilmington MA

MPLS 3 Mbps

MKS Andover MA

LR Injector

LR Controller

MPLS 6 Mbps

LR Injector

Oracle App

Servers

Oracle EBS

Perot Systems Plano TX

MPLS 15 Mbps

Oracle 3-node RAC

Dell EMC CX4-480

Oracle

DB

Solarwinds & Wireshark for Network traffic

LR Controller:

Unix Server

%cpu, load average,

cpu queue size

BMC Patrol:

Oracle contention

26

Tips: System Resource Graphs Against Load

Oracle

Server

resources

over load

Color Measurement Machine Min. Ave. Max.

% CPU APP20 – Web/Forms 0.1 13.4 22.8

% CPU APP21 – Web/Forms 0.1 11.7 20.8

% CPU APP22 – Conc Managers 0.2 1.1 1.8

% CPU DB10 – RAC 1 1.3 8.4 16.6

% CPU DB11 – RAC 2 0.5 8.2 13.8

% CPU DB12 – RAC 3 0.7 10.5 17.1

Load (users) 322

27

• Linux/RAC significantly more scalable than AIX/no RAC Increased jvms from 512 MB to 1GB

Dedicating one app server to Concurrent Managers reduces Batch contention on interactive processes

• Customer benefits 3x more capacity and $$$ saved!

Went live and met performance goals

Conclusions

Lessons Learned

Process

• Improve workflow documentation with screen shots so that can ID

workflow changes more quickly

• Improve advanced communication to stakeholders for better

resource & timeline planning

• Allocate more time for data scrubbing and shakedown testing

Technical

• IP Spoofing required to achieve effective load balancing given

Oracle‟s “sticky ip” app server assignment

• Upgrade to IE7 to overcome Oracle connectivity issues

• Enhance resource monitoring with time to implement LR-to-Oracle

direct monitoring

28

29

Resources

• Using LoadRunner with Oracle Apps R12 http://mentora.com/practical_solutions

• Oracle E-Business performance testing http://www.mentora.com/whitepapers

• Wireshark network sniffer– http://www.wireshark.org/download.html

• Oracle Metalink document on servlet vs socket mode http://metalink.oracle.com search for 384241.1

30 ©2010 Hewlett-Packard Development Company, L.P.

To learn more on this topic, and to connect with your peers after

the conference, visit the HP Software Solutions Community:

www.hp.com/go/swcommunity

31