- nloug · pdf file exadata planned maintenance rene kundersma oracle maa development june...

1

<Insert Picture Here>

Exadata Planned Maintenance

Rene Kundersma

Oracle MAA Development June 2012

2

Agenda

• Overview

• Patch Planning and Testing

• Patch Installation

2

3

Terminology

• MAA – Oracle Maximum

Availability Architecture

• GI – Grid Infrastructure

• RAC – Oracle Real Application

Clusters

• ASM – Oracle Automatic

Storage Management

• PSU – Patch set update

• CPU – Critical patch update

• BP – Bundle Patch

• MOS – My Oracle Support

• ULN – Unbreakable Linux

Network

• OEL – Oracle Enterprise Linux

• IB – InfiniBand

• OFED – OpenFabrics

Enterprise Distribution

4

Overview

3

5

Exadata ArchitectureReview

Database Grid Storage Grid

InfiniBand Network

Switches

Exadata Storage ServersDatabase Servers

•Firmware / OS

•Oracle GI / RDBMS

Ethernet Switch

Power Distribution PDU

Keyboard, Video, Mouse KVM

Additional Components

6

Planned MaintenanceSoftware and Hardware

Component Software

Maintenance

Hardware

Maintenance

Storage servers Yes Yes

Database servers Yes Yes

InfiniBand switches Yes No

Additional components Yes No

4

7

Exadata Storage Server

• Single patch download

• Same patch applies to all hardware

• Exadata software, OS, ILOM, firmware

• README and Support Note

• May include updates for database servers

• Minimal pack (< 11.2.3.1)

• Delivered via ULN starting in 11.2.3.1

# imageinfo

Active image version: 11.2.2.4.2.111221

8

Exadata Database ServerOperating System and Firmware

• Linux or Solaris x86

• Exadata-supplied updates

• Unbreakable Linux Network

• Minimal pack (<11.2.3.1)

• Solaris Support Repository

• Any software, any source (but)

• Maintain IB compatibility

• Maintain GI and DB requirements

• Shipped images/versions tested

# imageinfo

Kernel version: 2.6.18-238.12.2.0.2.el5

Image version: 11.2.2.4.2.111221

5

9

Exadata Database ServerOracle Database Server (RDBMS) and Grid Infrastructure

• Major, Maintenance, Patch Set versions (e.g. 11.2.0.2)

• Same as with non-Exadata storage

• May require minimum Exadata storage version

• Database 11.2.0.3 requires Exadata 11.2.2.4 or later

• Database Patch for Exadata (e.g. 11.2.0.3.4, aka BP)

• Tied to patch set

• Installed using OPatch

• Cumulative, and includes recent PSU / CPU

• Other patches (overlays, one-offs)

$ opatch lspatches

Quarterly Diskmon Patch for Exadata (JAN 2012 - 11.2.0.3.2) : (13513982)

Quarterly CRS Patch for Exadata (JAN 2012 - 11.2.0.3.2) : (13540563)

Quarterly Database Patch for Exadata (JAN 2012 - 11.2.0.3.2) : (13513783)

10

InfiniBand Switch

• Only update via Exadata-branded patch

# version

SUN DCS 36p version: 1.3.3-2

6

11

Patch PlanningLatest Releases (June 2012)

• MOS 888828.1• “The Patching Note”

• Critical issues and additional required patches

• Guidelines for related products (OEM, DBFS, EBS, RM, GG,

SAP)

Component Latest Software Recommended Minimum

Exadata Storage

Server

11.2.3.1.0 11.2.2.4.2

Database Server

11.2.0.3: 11.2.0.3.6

11.2.0.2: BP16

11.2.0.1: DB_BP12,GI_BP6

11.2.0.3: 11.2.0.3.4

11.2.0.2: BP13

11.2.0.1: DB_BP12,GI_BP6

InfiniBand Switch 1.3.3-2 1.3.3-2

12

Patch Planning and Testing

7

13

Patching Improvements Timeline

20092009

Auto Cell firmware update

20102010

Exadata bundle patch (BP)

Out-of-place patching step-by-step

Minimal Pack firmware update

Cell rolling patch apply

BP EM install

DB+GI BP merge

BP OPatch Auto install

20112011

BP DG Standby-First install

Minimal Pack Linux kernel update

BP RAC rolling install

BP OPlan

IB Switch update w/ ILOM

Linux upgrade step-by-step

11.2.0.2 upgrade step-by-step

Oracle Advisor webcasts

IOUG Exadata SIG webcast

• Reduce Downtime

• Simplify Install

• Reduce Risk

• Transfer Knowledge

Patching Improvements

20122012

11.2.0.3 upgrade step-by-step

11.2.0.3 Exachk readiness

11.2.0.3 QDPE

11.2.0.3 QFSDP

EM 12c Out-of-place patching

Linux dbserver ULN updates

14

Patch PlanningRelease Frequency

Component Patch Release Frequency

Exadata Storage Server Patch set: 6-12 months

Patch set update: 1-6 months

Database Server Patch set: 12-18 months

Database Patch for Exadata

11.2.0.3: Quarterly (w/ CPU)

Monthly interim for reactive patching

11.2.0.2: Quarterly (w/ CPU)

InfiniBand Switch Semi-annually to annually

Subject to change without notice

8

15

Planned MaintenanceSuggested Schedule

Frequency Planned Maintenance Action

3-6 monthsUpdate Exadata Storage Server software

Update Database Quarterly Database Patch for Exadata (QDPE)

1-2 years

Update Database patch set

Update InfiniBand switch software

Update Additional components software (if necessary)

2 years Replace disk controller batteries in storage and database servers

2-4 yearsUpdate Database major or maintenance release (e.g. 11.1 to 11.2, or 11 to

12)

3-4 yearsReplace flash card Energy Storage Modules (ESMs) in V2/X2 storage

servers

16

Patch PlanningNew Release Adoption

• Planning

• README and referenced Support Notes

• Exachk before and after (MOS 1070954.1)

• Required overlays for critical issues and one-off conflicts

• Bug fix list

e.g. 11.2.0.3 - MOS 1393410.1 11.2.0.2 - MOS 1314319.1

Stage General Recommendation

Production and late pre-production Every patch not mandatory

Wait for field experience to grow

Early pre-production or evaluation Keep current with latest release

9

17

Patch Testing Reduces Risk

• Output of testing

• Application functionality

• Application performance

• Installation method

• Timings

• Rehearsal, documentation, automation

• HA stages during patching

• Failure contingency

Patching Guidelines MOS 1262380.1

18

Test Environment

Test DBM Env Comments

Identical Equivalent to production environmentPrimary database

Standby database

Middle tier

Full data set w/ identical schema stats

Workload framework to mimic production

Smaller or

Older

No production scale performance test

No firmware patching test

None DANGER

RISK

+

10

19

Exadata PatchingPlanning Recommendations

• Current knowledge, current software

• The Patching Note (888828.1)

• Don’t fall far behind

• Mitigate your risks

• Plan to test

• Test your plan

20

Patch Installation

11

21

Patch DeliveryQuarterly Full Stack Download Patch (QFSDP)

• Download convenience

• Contains latest releases

• Infrastructure

• Storage

• InfiniBand switch

• PDU

• Database

• Systems Management

• Release aligned on QDPE (and CPU) schedule

|- Infrastructure/

| --ExadataStorageServer/

| -11.2.2.4.2/

| --InfiniBandSwitchSoftware/

| -1.3.3-2/

| --SunRackIIPDUMeteringUnitFirmware/

| -1.04/

|- Database/

| --11.2.0.3.2/

| --OPatch/

| --OPlan/

|- SystemsManagement/

| --GridControl11/

| --Agent/

| --EMPlugins/

| --OMS/

22

Patch MethodsPatching Tools

• System readiness

• Exachk (MOS 1070954.1)

Component Software Patch Method

Exadata Storage Server All patchmgr

Database Server

OS and firmware Linux – YUM (11.2.3.1)

Solaris – PKG

GI/DB BP OPatch

GI/DB Patch set OUI and DBUA

InfiniBand Switch All ILOM

12

23

Patch MethodsRolling versus Non-Rolling

• Patch non-rolling if maintenance window allows

• Rolling patching considerations

• Patch during low workload

• Cell Rolling Patching – Reduce disk failure risk w/ high redundancy

• DB Rolling Patching - Use Automatic Workload Management

RollingNon-RollingPatch Install Time +-

Complexity +-

Database Downtime+ -

24

• Patch contains all cell software and firmware

• Install nothing but this patch on cells

• Unsupported configuration changes

• Will be lost

• May cause future patching FAILURE

• patchmgr controls install and rollback

• One invocation patches all cells

• Rollback to most recent successfully installed version

Exadata Storage Server Patching

Method Downtime Patch install time

Rolling None Up to 2 hours per cell

Non-Rolling Up to 2 hours total Up to 2 hours total

13

25

Exadata Storage Server PatchingNon-Rolling Patch Apply

• Benefits

• Lowest total patching time

• No risk to single disk failure

• Consider

• Full outage up to 2 hours total

• Failed patch install on multiple

cellspatchmgr

Patch cell1

Patch celln

Patch cell3

Patch cell2

# ./patchmgr -cells cell_group –patch

26

Exadata Storage Server PatchingRolling Patch Apply

• Benefits

• No database downtime

• Not exposed to multiple cell

patch failures

• Consider

• ASM high redundancy to

reduce disk failure exposure

• Up to 2 hours per cell

• BP requirements

patchmgr

ASM offline

Patch

ASM onlinecell1

ASM offline

Patch

ASM onlinecell2

ASM offline

Patch

ASM onlinecell3

ASM offline

Patch

ASM onlinecelln

Prepare cell*

# ./patchmgr -cells cell_group –patch -rolling

14

27

Exadata Storage Server PatchingPatch Rollback

• patchmgr rollback

• To most recent previous (inactive) version

• Timing similar to installation

# imageinfo

Active image version: 11.2.2.2.0.101206.2

…

Inactive image version: 11.2.2.1.0.100927

# ./patchmgr -cells cell_group –rollback [-rolling]

28

Database Server PatchingLinux Operating System

• Oracle Linux

• >= 11.2.3.1

• YUM and ULN

• New image equivalent

• <= 11.2.2.4.2

• Minimal Pack install.sh

• Kernel (Linux), OFED, Firmware, OSW, minor configuration

• Not new image equivalent

• 5.3 to 5.5 – MOS 1284070.1

StopInstances and CRS Patch

StartInstances and CRS

15

29

Database Server PatchingLinux Operating System

• Keep in step with Exadata release

• Oracle Linux

• User-managed backups necessary for rollback

• Avoid oracle-validated rpm

• <11.2.3.1 Use ULN with care

• ofa and megaraid_sas packages require matching kernel

• Responses to common Exadata security scan findings

• MOS 1405320.1

30

Database Server PatchingSolaris Operating System

• Solaris 11

• Support Repository Update (SRU)

• updates all installed packages

• Image Packaging System (IPS)

“pkg update”

• Firmware manual

• Solaris 11 Express upgrade to

Solaris 11 (11.2.3.1)

• Database 11.2.0.3 required

Patch new BE Reboot

OS SRU Index

Solaris 11 1372094.1

# pkg list entire

NAME (PUBLISHER) VERSION IFO

entire 0.5.11-0.151.0.1.10 i--

16

31

Database Server PatchingDatabase and Grid Infrastructure Patches

• Cumulative

• Bug fix list 11.2.0.3 - MOS 1393410.1 11.2.0.2 - MOS 1314319.1

• OPatch installed and rolled back

• Post install SQL required*

• Rollback to base release (same as PSUs)

$ opatch napply <PATCH_LOCATION>

SQL> @rdbms/admin/catbundle.sql exa apply

$ opatch rollback –id <PATCH>

SQL> @rdbms/admin/catbundle_EXA_<SID>_ROLLBACK.sql

32

Database Server PatchingDatabase and Grid Infrastructure Patches

• Reduce downtime and risk

• RAC Rolling Installable

• OPatch Automation

• Out-of-Place patching

• OPlan

• Enterprise Manager

• Data Guard Standby-First

17

33

RAC Rolling Installable

• Benefit

• No database-wide downtime

• Consider

• Use Automatic Workload

Management

• Configure services and

connection failover

• Available 11.2.0.1 BP9*

• Most one-offs

Stop instance

OPatch DB

Stop CRS

OPatch GI

Start CRS

Start instance

node1

nodenP

Stop instance

OPatch DB

Stop CRS

OPatch GI

Start CRS

Start instance

node2

34

OPatch Automation (opatch auto)

• Benefit


• Single command per node

• Consider

• Mixed 11.2.0.2 / 11.2.0.1

homes (README)

• Available 11.2.0.2 BP2

OPatch

Stop instance

OPatch DB

Stop CRS

OPatch GI

Start CRS

Start instance

node1

OPatch

Stop instance

OPatch DB

Stop CRS

OPatch GI

Start CRS

Start instance

node2

OPatch

nodenP

18

35

Out-of-Place (OOP) Patching

• Benefit

Patch inactive home to reduce risk

1. Clone home (GI or DB)

2. Patch cloned home

3. Switch current home to cloned home (rolling or non-rolling)

• MOS 1136544.1 and OPlan

• EM 12c supports OOP

Performed during normal business

hours, ahead of “patching window”

36

OPlan

• Benefit

• Step-by-step patching instructions for Exadata BPs specific to your

environment

• In-place Non-rolling

• In-place RAC rolling

• Out-of-place (Recommended)


• MOS 1306814.1 Patch 11846294

19

37

Enterprise Manager Installable

• Benefit


• EM managed using

Provisioning Pack


• MOS 1265998.1

• 12c supports OOP

Enterprise Manager

Stop CRS

OPatch GI

OPatch DB

Start CRS

node1

Stop CRS

OPatch GI

OPatch DB

Start CRS

node2

P noden

38

Database Server PatchingDatabase and Grid Infrastructure Bundle Patches

• Use OPlan to generate patching plan

• Patch using Out-of-Place plan

• Rolling or Non-rolling switch to patched home

• Do not hard code ORACLE_HOME in user scripts

20

39

Standby System

• Patch replicated copy and switch

• Reduce risk and downtime

• Patching on standby without impact to primary

• Exadata Storage Server

• Database Server

• Operating System and Firmware (Minimal pack)

• Grid Infrastructure home

• InfiniBand switch

• Data Guard Standby-First Installable qualified BPs only

• Database server - Database home (MOS 1265700.1)

Data Guard

40

Database Server PatchingDatabase and Grid Infrastructure Patch Set Upgrade

• Wait until Exadata validated

• 11.2.0.3

• MOS 1373255.1

• 11.2.0.2

• MOS 1315926.1

• Storage Server compatibility

Database Required Storage

11.2.0.3 11.2.2.4 or later

11.2.0.2 11.2.2 or later

11.2.0.1 11.2.1 or later

• GI upgrade is rolling

• Database upgrade requires

downtime

• Reduce downtime and risk

• (Transient) Logical standby

• Golden Gate

21

41

InfiniBand SwitchFirmware

• Only update via Exadata-branded patch

• 1.0.1 => 1.1.3 => 1.3.3

• Rolling update

• Spine => Leaf => Leaf

• Installed (and rolled back) via ILOM

# ibswitchcheck.sh pre

-> load -source /tmp/sundcs_36p_repository_1.3.3_2.pkg

# ibswitchcheck.sh post

42

Consolidation Environments

• GI version >= DB version

• Required to 4th digit, recommended to 5th digit

• ASM compatible.rdbms

• Once advanced cannot be reset

• Maintain as few shared database homes as needed

• Patch out of place

22

43

Exadata PatchingPatching Recommendations

• Only patch healthy system

• Run Exachk before (and after)

• Patch copy, switch to copy

• Out-of-place

• Standby-First

• Patch non-rolling (if possible)

• Test and rehearse lowers downtime

• Patch rolling (when necessary)

• During low workload

• Reduce disk failure risk w/ high redundancy

• Use Automatic Workload Management

44

Exadata Patching & Upgrades Resources and References

• Latest Exadata software (MOS 888828.1)

• Exadata Critical Issues (MOS 1270094.1)

• Exadata Testing and Patching Practices (MOS 1262380.1)

• OPlan (MOS 1306814.1)

• Data Guard Standby-First Patch Apply (MOS 1265700.1)

• Enterprise Manager Patch Apply (MOS 1265998.1)

• 11.2.0.3 Database Upgrade on Exadata (MOS 1373255.1)

• 11.2.0.2 Database Upgrade on Exadata (MOS 1315926.1)

• Exadata Healthcheck (MOS 1070954.1)

• Patch READMEs and associated MOS

• Database Machine Owner’s Guide (/opt/oracle/cell/doc)

23

45

46

- nloug · pdf file exadata planned maintenance rene kundersma oracle maa development june...

Documents