1

Something Old, Something New:

Leveraging Oracle 12c’s Information Lifecycle Management (ILM)

Features for Improved Database Performance

Jim CzuprynskiZero Defect Computing, Inc.

April 4, 2014Session P-35

2

My Credentials

30+ years of database-centric IT experience Oracle DBA since 2001 Oracle 9i, 10g, 11g OCP > 100 articles on databasejournal.com and ioug.org Teach core Oracle DBA courses (G/I+RAC,

Exadata, Performance Tuning, Data Guard) 2009: Oracle Education Partner Trainer of the Year Speaker at Oracle OpenWorld, IOUG

COLLABORATE, OUG Norway, and Hotsos Oracle-centric blog (Generally, It Depends) Oracle ACE Director since February 2014

3

Our Agenda

The Persistence of Data Through Time Information Lifecycle Management (ILM) In-Database Archiving (IDA) Concepts Temporal Validity (TV) Concepts Temporal History: Old Feature, New Name Leveraging ILM ADO Features Q+A

4

The Persistence of Data Through Time

Data needs to be retained for inordinately long periods of time Data mining / warehousing / trend analysis Regulatory and legal requirements

Once data has been purged … it’s often needed again (usually immediately)

Therefore, data purging doesn’t happen nearly as often as it should

Handling schema changes after archiving is a nightmare …

… so, what we never purged data at all?

5

Information Lifecycle Management (ILM)

ILM encompasses three key feature sets: In-Database Archiving (IDA) stores active

and inactive data within same table Temporal Validity (TV) stores past, current,

and future data within same table Automatic Data Optimization (ADO) tiers

data according to its age and usage as well as how much space it consumes

6

In-Database Archiving (IDA): The Cloaking Device

7

In-Database Archiving (IDA) Avoids unnecessary deletion of rows when they

no longer contain valid data Activated via new ROW ARCHIVAL attribute of

data segment During initial INSERT, each row’s state is set to active

by placing a zero (0) in the ORA_ARCHIVE_STATE hidden column

A row can be marked as inactive by setting ORA_ARCHIVE_STATE to any non-zero value

Unless the ORA_ ARCHIVE_STATE column is mentioned in query, a row’s IDA status is invisible

Inactive rows can be compressed via ILM

8

Activating IDA

CREATE table oe.currency_conversions ( currency_from VARCHAR2(3) NOT NULL ,currency_to VARCHAR2(3) NOT NULL ,effect_dtm TIMESTAMP NOT NULL ,conversion_fctr NUMBER(10,6) DEFAULT 0) TABLESPACE example;

ALTER TABLE oe.currency_conversions ROW ARCHIVAL;

Activates In-Database

Archiving for an existing table

CREATE table oe.currency_conversions ( currency_from VARCHAR2(3) NOT NULL ,currency_to VARCHAR2(3) NOT NULL ,effect_dtm TIMESTAMP NOT NULL ,conversion_fctr NUMBER(10,6) DEFAULT 0) ROW ARCHIVAL TABLESPACE example;

Activates In-Database Archiving

immediately upon table

creation

9

COL column_name FORMAT A30 HEADING "Column Name"COL segment_column_id FORMAT 9999 HEADING "Seg|Col|#"COL internal_column_id FORMAT 9999 HEADING "Int|Col|#"COL hidden_column FORMAT A08 HEADING "Hidden?"COL virtual_column FORMAT A08 HEADING "Virtual?"COL user_generated FORMAT A08 HEADING "User|Gen'd?"TTITLE "Metadata for Table OE.CURRENCY_CONVERSIONS|(from DBA_TAB_COLS)"SELECT column_name ,segment_column_id ,internal_column_id ,hidden_column ,virtual_column ,user_generated FROM dba_tab_colsWHERE owner = 'OE' AND table_name = 'CURRCONV_PARTED'ORDER BY internal_column_id;TTITLE OFF

Viewing IDA Metadata

Metadata for Table OE.CURRENCY_CONVERSIONS (from DBA_TAB_COLS)

Seg Int Col Col UserColumn Name # # Hidden? Virtual? Gen'd?------------------------------ ----- ----- -------- -------- --------CURRENCY_FROM 1 1 NO NO YESCURRENCY_TO 2 2 NO NO YESEFFECT_DTM 3 3 NO NO YESCONVERSION_FCTR 4 4 NO NO YESSYS_NC00005$ 5 5 YES NO NOORA_ARCHIVE_STATE 6 6 YES NO NO

Note the addition of two new hidden

columns exclusively

for IDA

10

Populating IDA Test DataDECLARE ctr NUMBER := 0; curfr CHAR(3); curto CHAR(3); effdt DATE;BEGIN FOR ctr IN 1..36000 LOOP effdt := (TO_DATE('12/31/2015','mm/dd/yyyy') - (ctr/10)); INSERT INTO oe.currconv_parted VALUES( DECODE(MOD(ctr, 6), 0,'USD', 1,'JPY', 2,'EUR', 3,'AUD', 4,'CHF', 5, 'GBP') ,DECODE(MOD(ctr,36), 0,'JPY', 1,'EUR', 2,'AUD', 3,'CHF', 4,'GBP', 5,'CDN', 6,'EUR', 7,'AUD', 8,'CHF', 9,'GBP', 10,'CDN', 11,'USD', 12,'AUD', 13,'CHF', 14,'GBP', 15,'CDN', 16,'USD', 17,'JPY', 18,'CHF', 19,'GBP', 20,'CDN', 21,'USD', 22,'JPY', 23,'EUR', 24,'GBP', 25,'CDN', 26,'USD', 27,'JPY', 28,'EUR', 29,'AUD', 30,'CDN', 31,'USD', 32,'JPY', 33,'EUR', 34,'AUD', 35,'CHF') ,effdt ,ROUND(DBMS_RANDOM.VALUE(1,5),6) ); IF MOD(ctr, 5000) = 0 THEN COMMIT; END IF; END LOOP; COMMIT;EXCEPTION WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE('Fatal unexpected error has occurred: ' || SQLERRM );END;/

Generates 36,000 rows of historical test data for conversions between six key

currencies

SQL> UPDATE oe.currency_conversions SET ora_archive_state = 10 WHERE (currency_from = 'GBP') OR (currency_to = 'GBP'); COMMIT;

Activates IDA for any rows that

convert currencies from or to British

Pounds

11

Activating IDA “Cloaking”SQL> ALTER SESSION SET row archival visibility = ACTIVE;

COL currency_from FORMAT A08 HEADING "From|Currency"COL currency_to FORMAT A08 HEADING "To|Currency"COL effect_cdtm FORMAT A19 HEADING "Effective As Of:"COL conversion_fctr FORMAT 9999.999999 HEADING "Currency|Conversion|Factor"COL min_dtm FORMAT A19 HEADING "Minimum|Effective|Date"COL max_dtm FORMAT A19 HEADING "Maximum|Effective|Date"COL min_cvf FORMAT 9999.999999 HEADING "Minimum|Currency|Conversion|Factor"COL max_cvf FORMAT 9999.999999 HEADING "Maximum|Currency|Conversion|Factor"

TTITLE "Active Currency Conversion Factors|(from OE.CURRENCY_CONVERSIONS)"SELECT currency_from ,currency_to ,TO_CHAR(MIN(effect_dtm), 'yyyy-mm-dd hh24:mi') min_dtm ,TO_CHAR(MAX(effect_dtm), 'yyyy-mm-dd hh24:mi') max_dtm ,MIN(conversion_fctr) min_cvf ,MAX(conversion_fctr) max_cvf FROM oe.currconv_parted GROUP BY currency_from, currency_to ORDER BY currency_from, currency_to;TTITLE OFF

s

Active Currency Conversion Factors (from OE.CURRENCY_CONVERSIONS)

Minimum Maximum Minimum Maximum Currency CurrencyFrom To Effective Effective Conversion ConversionCurrency Currency Date Date Factor Factor-------- -------- ------------------- ------------------- ------------ ------------AUD CDN 2006-02-23 02:24 2015-12-29 12:00 1.004594 4.996108AUD CHF 2006-02-24 07:12 2015-12-30 16:48 1.000100 4.999312AUD EUR 2006-02-21 07:12 2015-12-27 16:48 1.005547 4.998198AUD JPY 2006-02-21 21:36 2015-12-28 07:12 1.000219 4.994470AUD USD 2006-02-22 12:00 2015-12-28 21:36 1.001565 4.999302CHF AUD 2006-02-21 04:48 2015-12-27 14:24 1.001286 4.995967CHF CDN 2006-02-23 14:24 2015-12-30 00:00 1.005144 4.989977. . .JPY EUR 2006-02-24 12:00 2015-12-30 21:36 1.012854 4.992390JPY USD 2006-02-21 12:00 2015-12-27 21:36 1.000304 4.995134USD AUD 2006-02-23 09:36 2015-12-29 19:12 1.010373 4.999434USD CDN 2006-02-21 14:24 2015-12-28 00:00 1.011684 4.996535USD CHF 2006-02-22 19:12 2015-12-29 04:48 1.004463 4.997040USD EUR 2006-02-24 00:00 2015-12-30 09:36 1.000201 4.998452USD JPY 2006-02-21 00:00 2015-12-27 09:36 1.010545 4.987463

25 rows selected. No results for British Pound appear …

SQL> SELECT currency_from, currency_to, TO_CHAR(effect_dtm, 'yyyy-mm-dd hh24:mi'), conversion_fctr FROM oe.currency_conversions WHERE (currency_from = 'GBP') OR (currency_to = 'GBP') ORDER BY 1,2,3;

no rows selected… and all British Pound entries are “cloaked”!

12

Deactivating IDA “Cloaking”SQL> ALTER SESSION SET row archival visibility = ALL;

Active Currency Conversion Factors (from OE.CURRENCY_CONVERSIONS)

Minimum Maximum Minimum Maximum Currency CurrencyFrom To Effective Effective Conversion ConversionCurrency Currency Date Date Factor Factor-------- -------- ------------------- ------------------- ------------ ------------AUD CDN 2006-02-23 02:24 2015-12-29 12:00 1.004594 4.996108AUD CHF 2006-02-24 07:12 2015-12-30 16:48 1.000100 4.999312AUD EUR 2006-02-21 07:12 2015-12-27 16:48 1.005547 4.998198AUD GBP 2006-02-23 16:48 2015-12-30 02:24 18.002189 21.999171. . .EUR GBP 2006-02-23 04:48 2015-12-29 14:24 18.009543 21.997833EUR JPY 2006-02-21 09:36 2015-12-27 19:12 1.003705 4.998172EUR USD 2006-02-22 00:00 2015-12-28 09:36 1.000780 4.997188GBP AUD 2006-02-21 16:48 2015-12-28 02:24 18.006136 21.977187GBP CDN 2006-02-24 02:24 2015-12-30 12:00 18.003076 21.997785GBP CHF 2006-02-21 02:24 2015-12-27 12:00 18.001375 21.995166GBP EUR 2006-02-22 07:12 2015-12-28 16:48 18.018042 21.999391GBP JPY 2006-02-22 21:36 2015-12-29 07:12 18.003179 21.995334GBP USD 2006-02-23 12:00 2015-12-29 21:36 18.006870 21.998615. . .USD GBP 2006-02-22 04:48 2015-12-28 14:24 18.001940 21.999533USD JPY 2006-02-21 00:00 2015-12-27 09:36 1.010545 4.987463

36 rows selected.

Inactive Currency Conversion FactorsFrom To ConversionCurrency Currency Effective As Of: Factor-------- -------- ------------------- ------------AUD GBP 2014-04-01 00:00 .466400CDN GBP 2014-04-01 00:00 .505900CHF GBP 2014-04-01 00:00 .489300EUR GBP 2014-04-01 00:00 .788300GBP AUD 2014-04-01 00:00 2.144300. . .GBP USD 2014-04-01 00:00 1.868200JPY GBP 2014-04-01 00:00 .004850USD GBP 2014-04-01 00:00 .53530012 rows selected.Cloaking device … off.

13

Temporal Validity (TV) and

Temporal History (TH)

14

Temporal Validity (TV)

TV allows specification of time periods when data stored within a table will be actually considered “valid” Implemented through new PERIOD FOR

table attribute Specified TV dimension can then be

populated as desired with both time-valid and non-time-valid data

Non-time-valid data can be compressed until it’s no longer needed

15

Implementing TV: Examples

DROP TABLE oe.product_pricing PURGE;CREATE TABLE oe.product_pricing ( product_id NUMBER(6) NOT NULL ,unit_price NUMBER(10,6) DEFAULT 0 ,beg_dtm TIMESTAMP NOT NULL ,end_dtm TIMESTAMP NOT NULL ,PERIOD FOR effective (beg_dtm, end_dtm)) TABLESPACE example;

… or, if columns are specified

during the table’s creation, they’ll be used as the new dimension

columns

DROP TABLE oe.product_pricing PURGE;CREATE TABLE oe.product_pricing ( product_id NUMBER(6) NOT NULL ,unit_price NUMBER(10,6) DEFAULT 0 ,beg_dtm TIMESTAMP NOT NULL ,end_dtm TIMESTAMP NOT NULL ,PERIOD FOR effective) TABLESPACE example;

The new dimension can be specified

without any predefined columns,

and they will be created automatically

…

16

TV: Populating Sample DataDECLARE ctr NUMBER := 0; begdt DATE := TO_DATE('01/01/2000','mm/dd/yyyy'); enddt DATE;BEGIN FOR ctr IN 1..180 LOOP enddt := ADD_MONTHS(begdt, 1); FOR pidx IN 1..10 LOOP INSERT INTO oe.pricing_parted VALUES( DECODE(MOD(pidx,10) , 0,1726, 1,1729, 2,1733, 3,1734, 4,1737 , 5,1801, 6,2056, 7,2058, 8,3399, 9,3400, 3515) ,ROUND(DBMS_RANDOM.VALUE(1,100),4) ,begdt ,enddt ); END LOOP; begdt := enddt; END LOOP; COMMIT;EXCEPTION WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE('Fatal unexpected error has occurred: ' || SQLERRM );END;/

Adds 1,800 rows of randomized data for Temporal Validity testing

17

Leveraging TV Session-Level FilteringCONNECT / AS SYSDBAGRANT EXECUTE ON DBMS_FLASHBACK_ARCHIVE TO oe;

GRANTs access to users to request a

valid time window

-- Summary of all product prices:SELECT product_id ,TO_CHAR(MIN(beg_dtm), 'yyyy-mm-dd.hh24:mi:ss') start_dtm ,TO_CHAR(MAX(end_dtm), 'yyyy-mm-dd.hh24:mi:ss') stop_dtm ,MIN(unit_price) min_price ,MAX(unit_price) max_price FROM oe.pricing_parted GROUP BY product_id ORDER BY product_id;

-- Detail for a single product:SELECT product_id ,unit_price ,TO_CHAR(beg_dtm,'YYYY-MM-DD') beg_dte ,TO_CHAR(end_dtm,'YYYY-MM-DD') end_dte FROM oe.pricing_parted WHERE product_id = 1726 ORDER BY product_id, beg_dtm;

EXEC DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME(level => ‘ALL');

Summary of Unit Pricing Statistics Within Product ID (from OE.PRICING_PARTED) Starting EndingProduct Time Time Minimum Maximum # Entries Period Period Unit Price Unit Price------- ---------- ------------------- ------------------- ----------- ----------- 1726 180 2000-01-01.00:00:00 2015-01-01.00:00:00 1.18 129.08 1729 180 2000-01-01.00:00:00 2015-01-01.00:00:00 1.08 99.29. . . 3399 180 2000-01-01.00:00:00 2015-01-01.00:00:00 1.79 98.31 3400 180 2000-01-01.00:00:00 2015-01-01.00:00:00 1.68 99.99

1800 rows selected.

Detailed Product Pricing for 1726 (from OE.PRICING_PARTED) Starting EndingProduct Unit Time Time # Price Period Period------- ----------- ------------------- ------------------- 1726 85.26 2000-01-01.00:00:00 2000-02-01.00:00:00 1726 40.55 2000-02-01.00:00:00 2000-03-01.00:00:00. . . 1726 34.48 2014-10-01.00:00:00 2014-11-01.00:00:00 1726 34.24 2014-11-01.00:00:00 2014-12-01.00:00:00

180 rows selected.

EXEC DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME(level => ‘ASOF‘, query_time => (TO_TIMESTAMP('2009-01-31','yyyy-mm-dd)));

Summary of Unit Pricing Statistics Within Product ID (from OE.PRICING_PARTED)

Starting EndingProduct Time Time Minimum Maximum # Entries Period Period Unit Price Unit Price------- ---------- ------------------- ------------------- ----------- ----------- 1726 1 2009-01-01.00:00:00 2009-02-01.00:00:00 53.32 53.32 1729 1 2009-01-01.00:00:00 2009-02-01.00:00:00 40.75 40.75 1733 1 2009-01-01.00:00:00 2009-02-01.00:00:00 10.11 10.11 1734 1 2009-01-01.00:00:00 2009-02-01.00:00:00 81.29 81.29 1737 1 2009-01-01.00:00:00 2009-02-01.00:00:00 77.01 77.01 1801 1 2009-01-01.00:00:00 2009-02-01.00:00:00 77.18 77.18 2056 1 2009-01-01.00:00:00 2009-02-01.00:00:00 63.37 63.37 2058 1 2009-01-01.00:00:00 2009-02-01.00:00:00 5.86 5.86 3399 1 2009-01-01.00:00:00 2009-02-01.00:00:00 30.52 30.52 3400 1 2009-01-01.00:00:00 2009-02-01.00:00:00 99.64 99.64

Detailed Product Pricing for 1726 (from OE.PRICING_PARTED)

Starting EndingProduct Unit Time Time # Price Period Period------- ----------- ------------------- ------------------- 1726 53.32 2009-01-01.00:00:00 2009-02-01.00:00:00

EXEC DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME(level => ‘CURRENT');

Summary of Unit Pricing Statistics Within Product ID (from OE.PRICING_PARTED) Starting EndingProduct Time Time Minimum Maximum # Entries Period Period Unit Price Unit Price------- ---------- ------------------- ------------------- ----------- ----------- 1726 1 2014-03-01.00:00:00 2014-04-01.00:00:00 77.32 77.32 1729 1 2014-03-01.00:00:00 2014-04-01.00:00:00 42.89 42.89 1733 1 2014-03-01.00:00:00 2014-04-01.00:00:00 88.22 88.22 1734 1 2014-03-01.00:00:00 2014-04-01.00:00:00 51.49 51.49 1737 1 2014-03-01.00:00:00 2014-04-01.00:00:00 2.73 2.73 1801 1 2014-03-01.00:00:00 2014-04-01.00:00:00 1.88 1.88 2056 1 2014-03-01.00:00:00 2014-04-01.00:00:00 99.57 99.57 2058 1 2014-03-01.00:00:00 2014-04-01.00:00:00 78.32 78.32 3399 1 2014-03-01.00:00:00 2014-04-01.00:00:00 67.21 67.21 3400 1 2014-03-01.00:00:00 2014-04-01.00:00:00 3.16 3.16

Detailed Product Pricing for 1726 (from OE.PRICING_PARTED) Starting EndingProduct Unit Time Time # Price Period Period------- ----------- ------------------- ------------------- 1726 77.32 2014-03-01.00:00:00 2014-04-01.00:00:00

18

Temporal History (TH)

Original name: Oracle Total Recall Later releases: Flashback Data Archive

Intrinsic to Oracle 12cR1 Permits bi-temporal queries:

Flashback Query uses AS OF clause Flashback Versions Query uses VERSIONS

BETWEEN clause

19

Leveraging Bi-Temporal Queries

108.85 108.00 94.21 119.11 98.90 51.59 15.07

Changed on 04-05-2013 107.25 98.90

Changed on 04-18-2013 95.23 119.11

Changed on 04-30-2013 98.90

SCN

01-13

02-13

03-13

04-13

05-13

06-13

07-13

08-13

Time

EXEC DBMS_FLASHBACK_ARCHIVE(‘ASOF’,’2013-04-01’);SELECT product_id ,unit_price ,TO_CHAR(beg_dtm,'YYYY-MM-DD') beg_dte ,TO_CHAR(end_dtm,'YYYY-MM-DD') end_dte FROM oe.pricing_parted AS OF PERIOD FOR effective TO_DATE('2013-04-15','yyyy-mm-dd') WHERE product_id = 1726 ORDER BY product_id, beg_dtm;

Flashback Query

EXEC DBMS_FLASHBACK_ARCHIVE(‘ASOF’,’2013-04-01’);SELECT product_id ,unit_price ,TO_CHAR(beg_dtm,'YYYY-MM-DD') beg_dte ,TO_CHAR(end_dtm,'YYYY-MM-DD') end_dte FROM oe.pricing_parted VERSIONS PERIOD FOR effective BETWEEN TO_DATE('2013-04-08','yyyy-mm-dd') AND TO_DATE('2013-04-30','yyyy-mm-dd') WHERE product_id = 1726 ORDER BY product_id, beg_dtm;

Flashback Versions Query

20

Leveraging ILM ADO Features

21

Automatic Data Optimization (ADO)

Moves or compresses data based on observed usage patterns Leverages heat maps to determine how

often data has been accessed Tracks exactly how data has been utilized

DML versus query Random access versus table scan

Usage patterns can be tracked at tablespace, segment, and even row level

22

Covering the Space-Time Continuum

An ADO policy can be based on either space or time Space-based:

Moves segment between tablespaces in different storage tiers

Movement is based on “fullness” of initial tablespace Time-based:

Compresses data more tightly within same object Data compressible at three levels:

ROW STORAGE (row level within a segment) SEGMENT (one database object at segment level) GROUP (multiple database objects)

23

IDA: Implementing ILM Storage TieringCREATE table oe.currconv_parted ( currency_from VARCHAR2(3) NOT NULL ,currency_to VARCHAR2(3) NOT NULL ,effect_dtm TIMESTAMP NOT NULL ,conversion_fctr NUMBER(10,6) DEFAULT 0) PARTITION BY LIST(ora_archive_state) ( PARTITION currconv_history VALUES (10,20,30,40,50) TABLESPACE ado_cold_data ILM ADD POLICY COMPRESS FOR QUERY HIGH SEGMENT AFTER 180 DAYS OF NO MODIFICATION ,PARTITION currconv_current VALUES (0) TABLESPACE ado_hot_data ) ROW ARCHIVAL NOLOGGING PARALLEL 4;

After 180 days of no further DML activity, the segment of the

partition containing all inactive rows will be compressed using HCC QUERY HIGH

compression

24

TV: Implementing ILM Storage TieringCREATE TABLE oe.pricing_parted ( product_id NUMBER(6) NOT NULL ,unit_price NUMBER(10,6) DEFAULT 0 ,beg_dtm TIMESTAMP NOT NULL ,end_dtm TIMESTAMP NOT NULL ,PERIOD FOR EFFECTIVE (beg_dtm, end_dtm)) PARTITION BY RANGE(end_dtm) ( PARTITION opp_history VALUES LESS THAN (TO_DATE('2014-01-01','yyyy-mm-dd')) TABLESPACE ado_cold_data ILM ADD POLICY COMPRESS FOR QUERY HIGH SEGMENT AFTER 180 DAYS OF NO ACCESS ,PARTITION opp_current VALUES LESS THAN (MAXVALUE) TABLESPACE ado_warm_data ) NOLOGGING PARALLEL 4;

25

Questions? Comments?

26

Please feel free to evaluate this session:

Session P35How Hot Is My Data? Leveraging Automatic Data

Optimization (ADO) Features in Oracle 12c Database For Dramatic Performance Improvements

If you have any questions or comments, feel free to:E-mail me at jczuprynski@zerodefectcomputing.comFollow my blog (Generally, It Depends):

http://jimczuprynski.wordpress.comFollow me on Twitter (@jczuprynski)Connect with me on LinkedIn (Jim Czuprynski)

Thank You For Your Kind Attention