CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

David Webster

HOT OR NOT?HOW TO RELEASE CHANGES WITHOUT IMPACTING CUSTOMERS

April 2014

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

WHAT IS THIS SESSION NOT ABOUT?

2

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

WHAT THIS SESSION IS ABOUT?

3

HOW DO WE MAKE CODE AND TABLE CHANGES TO AN ORACLE DATABASE WITHOUT IMPACTING ANY OF THE USERS?

HOW DO WE DO THAT EVEN ON A REALLY COMPLEX DATABASE WITH 1000S OF USERS/SESSIONS?

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

WHAT IS THIS SESSION ABOUT, SPECIFICALLY?

4

• Why put code in the DB?• How to change that code while the DB

is active.• Under what circumstances is that

challenging? (24x7, high executions, daemons, DB Links, dependencies)

• What can we do about it?

• ELEPHANT IN THE ROOM : EDITIONS

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

JUST BEFORE I START….

5

ALL TESTS ARE ON 11.2.0.3

HOT OR NOT….

HOT DOESN’T MEAN HOT, COLD DOESN’T MEAN COLD

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

ABOUT BETFAIR

6

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

ABOUT BETFAIR

7

PIONEERED ONLINE PERSON-TO-PERSON BETTING. BETFAIR IS THE BROKER, THE EBAY OF BETTING, NOT THE “HOUSE”

COMPANY STARTED IN 1999 WITH 2 FOUNDERS

OVER 1500 EMPLOYEES ACROSS THE WORLD

DATA CENTRES ACROSS EUROPE, IN AUSTRALIA AND US

FULL OFFERING OF BETTING PRODUCTS, INCLUDING SPORTSBOOK, POKER, CASINO AND ARCADE.

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

ABOUT DAVID WEBSTER

8

15 YEARS WORKING WITH ORACLE

SITE RELIABILITY ENGINEER – ORACLE PERFORMANCE

BLOG : HTTP://ANOTHERDAVEWEBSTER.WORDPRESS.COM/

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

WHY PUT CODE IN THE DATABASE?

9

• CENTRALISED LOGIC AND VERSIONING (I KNOW…)

• PL/SQL APIS. ABSTRACTION AND ENCAPSULATION

• PERFORMANCE

• DEPENDENCY TRACKING

• OTHERS……

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

WHY SHOULD I NOT PUT CODE IN THE DATABASE?

10

• PERFORMANCE…..DIGRESSION ALERT!!!PREFETCH & READ CONSISTENCY

• MAKES RELEASING CODE MORE DIFFICULT!

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

WHAT MAKES OUR EXCHANGE A COMPLEX DB?

11

1. IT’S 4 DATABASES

2. OVER 100K EXECUTIONS A SECOND

3. 1000S OF SESSIONS, 100S OF CLIENTS

4. LOTS AND LOTS OF CODE

5. BACKGROUND JOBS

6. 24X7 OPERATION

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

(RE)COMPILING PACKAGES

12

Q : WHAT DOES ORACLE NEED TO COMPILE A PACKAGE BODY?A : ORACLE REQUIRES AN EXCLUSIVE LOCK ON THE OBJECT

Q : DOES ANYTHING GET INVALIDATED WHEN I COMPILE A BODY?A : NO

Q : WHAT ABOUT A PACKAGE SPEC?A : COMPILING A PACKAGE SPEC INVALIDATES CALLING BODIES*

Q : IS MY (RE)COMPILATION LIMITED TO THE CODE THAT I CHANGE?A : NO…..

COMPILE ONLY WHAT YOU NEED

AVOID INVALIDATING PACKAGES

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

GETTING THE LOCK

13

• STOP BACKGROUND JOBS

• NEED A WINDOW

10 200 30 5040 60

Minute

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

PACKAGE STATE

14

WHAT IS PACKAGE STATE?

ORA-04068: existing state of

packages has been discarded

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

PACKAGE STATE

15

CONSTANTS

CONSTANT WORKAROUND

ORA-04068: existing state of

packages has been discarded

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

UNNECESSARY INVALIDATIONS : FGDT

16

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

UNNECESSARY INVALIDATIONS : FGDT

17

FINE GRAINED DEPENDENCY TRACKING

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

UNNECESSARY INVALIDATIONS : FGDT

18

NEEDED NOT NEEDED

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

CHANGING PACKAGE SPECIFICATIONS

19

CHANGING THE PACKAGE SPECIFICATION INVALIDATES ALL CALLING PACKAGES

EXCEPT THAT’S NOT TRUE

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

CHANGING PACKAGE SPECIFICATIONS

20

END MEANS AFTER THE LAST REFERENCED PROCEDURE!

TO BE SAFE, ADD TO THE VERY END

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

CHANGING PACKAGE SPECIFICATIONS

21

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

CHANGING PACKAGE SPECIFICATIONS

22

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

LEFT WITH A MESS?

23

1. WHEN OVERLOADING, REFACTOR INTERNALLY

2. IF YOU CAN’T DELETE, THROW AN ERROR

3. DO A COLD RELEASE

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

DATABASE LINKS

24

1. DON’T USE THEM

2. SESSION-TASTIC

3. CONFUSE THE POOR CBO

4. DON’T USE THEM

5. EDGE CASES AND COMPLEXITY

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

DATABASE LINKS

25

FIXED USER DATABASE LINK

CONNECTED USER DATABASE LINK

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

DATABASE LINKS

26

RED_TIGER

RED_SCOTT BLUE_SCOTT

RED_SCOTT

TABLE

RED_TIGER

Package PACKAGE reads BLUE_SCOTT.TABLE@BLUE

GRANT EXECUTE ON PACKAGE TO RED_TIGER

GRANT SELECT ON TABLE TO RED_SCOTT

GRANT SELECT ON TABLE TO RED_TIGER

Bug 13415892 - UNNECESSARY INVALIDATION OF REMOTE STUBS

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

SUMMARY

27

1. TO COMPILE (OR RECOMPILE) A PACKAGE, YOU NEED AN EXCLUSIVE LOCK

2. PACKAGE STATE LIMITS OPTIONS

3. MINIMISE UNECESSARY COMPILATIONa. SCOPE PL/SQL VARIABLESb. MODIFY SPECIFICATION CAREFULLY

4. BEWARE DATABASE LINKS

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

THANK YOUQ&A

David.Webster@betfair.com

28

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

29