darrell hiraoka –amgen middleware team lynn lu - · pdf filevcs db node 1 vcs db node 2...

24
Migrating Enterprise Manager 12c to a new data center with near zero downtime Darrell Hiraoka –AMGEN Middleware Team Lynn Lu - AMGEN DBA Team September 26, 2013

Upload: vuongnhu

Post on 30-Mar-2018

226 views

Category:

Documents


1 download

TRANSCRIPT

Mig

rati

ng

En

terp

rise M

an

ag

er

12c

to a

new

data

cen

ter

wit

h n

ear

zero

do

wn

tim

e

Da

rre

ll H

ira

oka

–A

MG

EN

Mid

dle

wa

re T

ea

m

Lyn

n L

u -

AM

GE

N D

BA

Te

am

Se

pte

mb

er

26

, 2

01

3

2

AG

EN

DA

•A

bout A

MG

EN

•B

ackgro

und

•W

hy O

EM

12c

•O

EM

12c H

A A

rchitectu

re

•D

R F

ailo

ver

Testing

•D

ata

cente

r M

igra

tion P

lan

•Im

ple

menta

tion C

halle

nges

•B

enefits

3

Ab

ou

t A

MG

EN

•W

orl

d’s

le

ad

ing

in

de

pe

nd

en

t

bio

tech

no

log

y c

om

pa

ny,

with

a

mis

sio

n to

se

rve

pa

tie

nts

–A

mgen m

edic

ines h

ave r

eached m

ore

than 2

5 m

illio

n p

atients

–P

resence in m

ore

than 5

0 c

ountr

ies

•M

ore

th

an

30

ye

ars

of p

ion

ee

rin

g

scie

nce

an

d v

ita

l m

ed

icin

es

•F

ocu

s s

ole

ly o

n d

isco

ve

rin

g,

de

ve

lop

ing

, a

nd

ma

kin

g h

um

an

the

rap

eu

tics

–S

pecia

lizin

g in innovative m

edic

ines f

or

serious illn

ess

–P

ioneer

and w

orld leader

in p

rote

in

thera

peutic m

anufa

ctu

ring

•B

roa

d a

nd

de

ep

pip

elin

e o

f n

ove

l

pro

du

ct ca

nd

ida

tes

•W

orl

d’s

le

ad

ing

in

de

pe

nd

en

t

bio

tech

no

log

y c

om

pa

ny,

with

a

mis

sio

n to

se

rve

pa

tie

nts

–A

mgen m

edic

ines h

ave r

eached m

ore

than 2

5 m

illio

n p

atients

–P

resence in m

ore

than 5

0 c

ountr

ies

•M

ore

th

an

30

ye

ars

of p

ion

ee

rin

g

scie

nce

an

d v

ita

l m

ed

icin

es

•F

ocu

s s

ole

ly o

n d

isco

ve

rin

g,

de

ve

lop

ing

, a

nd

ma

kin

g h

um

an

the

rap

eu

tics

–S

pecia

lizin

g in innovative m

edic

ines f

or

serious illn

ess

–P

ioneer

and w

orld leader

in p

rote

in

thera

peutic m

anufa

ctu

ring

•B

roa

d a

nd

de

ep

pip

elin

e o

f n

ove

l

pro

du

ct ca

nd

ida

tes

Mark

ete

d P

rod

ucts

Fo

r ad

dit

ion

al in

form

ati

on

ab

ou

t A

mg

en

pro

du

cts

, in

clu

din

g

imp

ort

an

t safe

ty i

nfo

rmati

on

, v

isit

ww

w.a

mg

en

.co

m

4

Backg

rou

nd

•D

ata

base v

ers

ions f

rom

8i to

11gR

2 (

tota

l ~

1100

insta

nces)

•W

ebLogic

vers

ions f

rom

8 t

o 1

2c (

tota

l ~

600 insta

nces)

5

Wh

y O

EM

12c

•N

ew

Featu

res in O

EM

12c

–N

FS

Sto

rag

e M

on

ito

rin

g

–Im

pro

ve

d R

ep

ort

ing

Fe

atu

res

–Im

pro

ve

d P

erf

orm

an

ce

Dia

gn

ostics

–P

rovis

ion

ing

an

d P

atc

hin

g

–M

etr

ic E

xte

nsio

ns

–R

ea

l T

ime

AD

DM

–Im

pro

ve

d M

on

ito

rin

g T

hre

sh

old

s

•N

eed a

n E

nte

rprise S

tandard

monitoring t

ool fo

r th

e

Data

base a

nd M

iddle

ware

team

s

–R

ep

lace

Cu

sto

m S

cri

pts

6

Wh

y O

EM

12c (

co

nti

nu

ed

)

–P

roa

ctive

ra

the

r th

an

re

active

mo

nito

rin

g

–S

tan

da

rd c

en

tra

lize

d to

ol fo

r m

on

ito

rin

g a

nd

re

po

rtin

g

•N

eed m

ore

capacity t

o s

upport

gro

wth

–L

eg

acy O

EM

11

g h

as lim

ite

d r

eso

urc

es

–B

oth

OM

S a

nd

DB

ru

nn

ing

on

sh

are

d S

ola

ris z

on

es

•N

eed t

o r

e-a

rchitect

OE

M e

nvironm

ent

to s

upport

HA

–R

em

ed

iate

sin

gle

po

int

of

failu

re

OE

M 1

2c

sim

pli

fie

d m

on

ito

rin

g o

f d

ata

ba

se

s a

nd

mid

dle

wa

re p

rov

idin

g a

glo

ba

l, c

en

tra

lize

d,

an

d h

igh

ly

av

ail

ab

le m

on

ito

rin

g t

oo

l.

7

OE

M 1

2c H

A A

rch

itectu

re

•O

racle

recom

mended L

evel 4 M

axim

um

Availa

bili

ty

Arc

hitectu

re (

MA

A),

achie

vin

g h

ighest

availa

bili

ty in t

he

most cost effective, sim

ple

desig

n.

•O

MS

on t

he p

rim

ary

site in A

ctive/A

ctive C

onfigura

tion.

Repository

runnin

g o

n V

ER

ITA

S t

wo-n

ode c

luste

r.

•D

uplic

ate

hard

ware

deplo

yed a

t th

e s

tandby s

ite.

•S

tandby/a

dditio

nal S

tandby O

MS

insta

lled a

t th

e

sta

ndby/D

R s

ite.

.

8

OE

M 1

2c H

A A

rch

itectu

re (

co

nti

nu

ed

)

•D

ata

base D

R u

sin

g O

racle

Data

Guard

with m

axim

um

perf

orm

ance

•Local lo

ad b

ala

ncer

imple

mente

d in b

oth

Prim

ary

and

DR

/sta

ndby s

ites

•S

oft

ware

Lib

rary

uses N

etA

pp

Sto

rage s

nap m

irro

r

replic

ation b

etw

een P

rim

ary

and S

tandby s

ites.

9

OE

M 1

2c

Arc

hit

ectu

re D

iag

ram

EM

CL

I

DN

S A

lias

Share

d s

oft

ware

libra

ry

Vip

Repository

data

base

share

d s

oft

ware

libra

ry

Vip

Repository

data

base

Data

Guard

Neta

pp

snap m

irro

r

OM

S1

OM

S2

VC

S D

B

node 1

VC

S D

B

node 2

VC

S D

B

node 1

VC

S D

B

node 2

OM

S1

OM

S2

Active

Site

Sta

ndby

Site

10

OE

M 1

2c H

ard

ware

/So

ftw

are

Deta

ils

Deta

ils

EM

12c R

ep

osit

ory

OM

S S

erv

er

Serv

er/

Hard

ware

16 c

ore

s,

132G

4 c

ore

s,

8G

Data

base/O

MS

vers

ion

11.2

.0.3

12.1

.0.2

Opera

ting

Syste

mR

HE

L5

RH

EL 5

Hig

h A

vaila

bili

ty2

node V

CS

(P

rim

ary

)

2node V

CS

(D

R)

2 s

erv

ers

(P

rim

ary

)

2 s

erv

ers

(D

R)

11

OE

M 1

2c M

on

ito

red

Targ

ets

Deta

ils

Data

base

Web

Lo

gic

Num

ber

of

targ

ets

(agent,

insta

nce,

liste

ner)

Data

bases:9

50

Agents

: 400

Lis

teners

:500

Applic

ations:

500

Agents

: 100

Insta

nces:5

50

Vers

ions

9.2

.0.8

to 1

1.2

.0.3

10g t

o 1

2c

Serv

er

pla

tform

/vers

ions

Win

dow

2008

, S

ola

ris

9,

Sola

ris 1

0,

RH

EL 4

,

RH

EL 5

, R

HE

L 6

Sola

ris 9

, S

ola

ris 1

0,

RH

EL

4,

RH

EL 5

, R

HE

L

6

12

DR

Failo

ver

Testi

ng

A form

al D

R e

xerc

ise w

as p

erf

orm

ed w

ith t

he h

igh level

ste

ps b

elo

w:

–S

witch

sta

nd

by d

ata

ba

se

to

be

pri

ma

ry

–S

witch

DN

S a

lias to

po

int

to D

R F

5 lo

ad

ba

lan

ce

r

–B

rea

k s

oft

wa

re lib

rary

sn

ap

mir

ror

–R

eco

nfig

ure

sta

nd

by O

MS

to

po

int

to n

ew

Pri

ma

ry D

B,

sta

rt

sta

nd

by O

MS

–R

e-s

yn

c t

he

re

po

sito

ry d

ata

ba

se

with

th

e m

an

ag

em

en

t

ag

en

ts

–V

eri

fy a

ll m

on

ito

red

ta

rge

ts a

nd

en

vir

on

me

nt

are

op

era

tio

na

l

13

DR

Failo

ver

Testi

ng

(co

nti

nu

ed

)

Hig

h level ta

sks p

erf

orm

ed t

o v

erify

DR

was o

pera

tional.

–In

sta

ll a

ne

w a

ge

nt

an

d c

on

fig

ure

ta

rge

ts

–Te

st

mo

nito

rin

g a

lert

s a

nd

esca

latio

n

–R

un

exis

tin

g jo

bs

–R

un

re

po

rts

Th

e D

R e

xe

rcis

e c

om

ple

ted

su

cc

es

sfu

lly

in t

he

pro

du

cti

on

pil

ot.

40

Da

tab

as

e s

erv

ers

(~

30

0 t

arg

ets

)

10

Mid

dle

wa

re s

erv

ers

(~

10

0 t

arg

ets

)

14

DR

Failo

ver

Testi

ng

Dia

gra

m

EM

CL

I

DN

S A

lias

Share

d s

oft

ware

libra

ry

Vip

Repository

data

base

share

d s

oft

ware

libra

ry

Vip

Repository

data

base

OM

S1

OM

S2

VC

S D

B

node 1

VC

S D

B

node 2

VC

S D

B

node 1

VC

S D

B

node 2

OM

S1

OM

S2

Active

Site

Dow

n

Site

15

Data

cen

ter

Mig

rati

on

Pla

n

•E

nte

rprise d

ata

cente

r consolid

ation d

rove t

he r

elo

cation

of t

he O

EM

12c infr

astr

uctu

re

•O

EM

12c r

elo

cation is t

he o

nly

option a

vaila

ble

that

would

meet th

e m

inim

um

dow

ntim

e r

equirem

ent

•T

he d

esig

n o

f th

e r

elo

cation is b

ased o

n o

ur

curr

ent H

A

imple

menta

tion a

nd D

R t

est re

sults

•T

he p

lan w

ill r

equire t

wo a

dditio

nal sta

ndby

environm

ents

whic

h w

ill b

e p

rovis

ioned in n

ew

data

cente

rs

16

Data

cen

ter

Mig

rati

on

Pla

n (

co

nti

nu

ed

)

Pla

nn

ing

PO

CD

ep

loym

en

tF

ina

l

Cu

tove

r

Th

e p

lan

ned

mig

rati

on

to

th

e t

wo

new

data

cen

ters

sh

ou

ld c

om

ple

te w

ith

in ~

30 m

in. T

he m

igra

tio

n t

ime

co

uld

be r

ed

uced

to

~10 m

in w

ith

fu

rth

er

au

tom

ati

on

an

d a

sm

all

er

DN

S T

TL

sett

ing

.

17

Data

cen

ter

Mig

rati

on

Pla

n -

Dep

loym

en

t

Share

d s

oft

ware

libra

ry

vip

Repository

data

base

OM

S1

OM

S2

VC

S D

B

node 1

VC

S D

B

node 2

Sta

ndby

Site

Share

d s

oft

ware

libra

ry

vip

Repository

data

base

OM

S1

OM

S2

VC

S D

B

node 1

VC

S D

B

node 2

Sta

ndby

Site

EM

CL

I

DN

S A

lias

Data

Guard

Neta

pp

snap

mirro

r

Share

d s

oft

ware

libra

ry

vip

Repository

data

base

OM

S1

OM

S2

VC

S D

B

node 1

VC

S D

B

node 2

Active S

ite

Share

d s

oft

ware

libra

ry

vip

Repository

data

base

OM

S1

OM

S2

VC

S D

B

node 1

VC

S D

B

node 2

Sta

ndby

Site

18

Data

cen

ter

Mig

rati

on

Pla

n –

Fin

al C

uto

ff

Share

d s

oft

ware

libra

ry

vip

Repository

data

base

OM

S1

OM

S2

DB

node 1

DB

node 2

Acti

ve S

ite

Share

d s

oft

ware

libra

ry

vip

Repository

data

base

OM

S1

OM

S2

DB

node 1

DB

node 2

Sta

ndby

Site

EM

CL

I

DN

S A

lias

Data

Guard

Neta

pp

snap

mirro

r

Share

d s

oft

ware

libra

ry

vip

Repository

data

base

OM

S1

OM

S2

VC

S D

B

node 1

VC

S D

B

node 2

OM

S

Dele

ted

Share

d s

oft

ware

libra

ry

vip

Repository

data

base

OM

S1

OM

S2

VC

S D

B

node 1

VC

S D

B

node 2

OM

S

Dele

ted

New

Pri

mary

DC

New

DR

DC

19

Imp

lem

en

tati

on

Ch

allen

ges

•S

oft

ware

Lib

rary

should

be s

hare

d a

cro

ss a

ll sites

during insta

llation (

NF

S m

ount)

, p

ost in

sta

llation w

ork

s

with local m

irro

red c

opy.

•12c a

gent

hig

hly

unsta

ble

in S

ola

ris 9

. F

ix w

as t

o

upgra

de t

he a

gent’s J

DK

vers

ion t

o 1

.6-u

43.

20

Imp

lem

en

tati

on

Ch

allen

ges (

co

nti

nu

ed

)

•F

als

e D

ata

base/M

iddle

ware

dow

n a

lert

s s

ent by O

EM

for

targ

ets

in S

ola

ris 9

. F

ix w

as w

ith t

he s

am

e J

DK

upgra

de.

•In

tegra

ting O

EM

12c a

gent

with V

CS

Clu

ste

r.

•In

corr

ect m

etr

ics a

s a

result o

f m

igra

ting t

em

pla

tes fro

m

earlie

r vers

ion o

f O

EM

.

•W

ebLogic

adm

in s

erv

er

dis

covery

issue –

targ

et

dis

covere

d b

ut att

ached t

o w

rong a

gent.

21

Imp

lem

en

tati

on

Ch

allen

ges (

co

nti

nu

ed

)

•12.1

.0.2

agent

sto

ps m

onitoring w

ith

TO

O_M

AN

Y_O

PE

N_F

ILE

S e

rror.

Patc

h fix

es t

he b

ug.

•N

eta

pp

plu

gin

did

not w

ork

as e

xpecte

d –

curr

ently

usin

g m

etr

ic e

xte

nsio

ns t

o m

onitor

Sto

rage.

•B

I publis

her

not in

tegra

ted in S

tandby O

MS

•Im

port

/Export

of te

mpla

tes -

monitoring t

em

pla

tes c

an

be e

xport

ed v

ia E

MC

LI,

but not

the a

ssocia

ted

notification r

ule

s.

22

EM

12c B

en

efi

ts

•R

eal tim

e s

tora

ge d

ata

colle

ction f

rom

monitore

d h

ost

targ

ets

.

•A

ll m

onitoring a

lert

s a

re g

oin

g t

hro

ugh t

he A

MG

EN

sta

ndard

escala

tion t

ool.

•Im

pro

ved b

lackout

–B

lackouts

were

not w

ork

ing a

s

expecte

d in 1

1g.

•E

nhanced a

vaila

bili

ty r

eport

-

Targ

ets

fre

quently in

unknow

n/p

endin

g s

tatu

s in O

EM

11g.

23

EM

12c B

en

efi

ts (

co

nti

nu

ed

)

•E

nhanced P

rocedure

Lib

rary

•In

frastr

uctu

re a

vaila

ble

for

team

to test self s

erv

ice

report

ing,

pro

vis

ionin

g,

auto

mation,

patc

hin

g,

and

pro

filin

g,

etc

.

OE

M 1

2c

pro

vid

es

Am

ge

n a

ce

ntr

ali

ze

d,

sc

ala

ble

, a

nd

hig

hly

av

ail

ab

le m

on

ito

rin

g t

oo

l fo

r th

e O

rac

le d

ata

ba

se

an

d W

eb

Lo

gic

mid

dle

wa

re p

latf

orm

s.

24

Th

an

k Y

ou

•O

racle

–P

err

en W

alk

er

& M

ark

McG

ill

•A

MG

EN

–D

BA

, M

iddle

ware

, N

etw

ork

, S

tora

ge,

UN

IX