real-time data warehousing using oracle database and obiee - collaborate'11

T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com

Stewart Bryson, Managing Director, Rittman Mead AmericaCollaborate, Orlando, April 2011

Real-Time DW with OBIEE and Oracle Database

1Thursday, 21 April 2011

Who am I?

• Stewart Bryson, Managing Director, Rittman Mead America

• Oracle BI/DW Architect and Delivery Specialist

• Oracle Database and BI Stack since 1996

• Regular Community Speaker

• Writer for Rittman Mead Blog:http://www.rittmanmead.com/blog

• Developer of Transcend Framework

• Email : stewart.bryson@rittmanmead.com

• Twitter : @stewartbryson

Who is Rittman Mead?

• Oracle BI&DW Experts

• Consulting, Training and Support

• Global Company, Offices in the US, UK and India

• Clients in the UK, USA, Europe, Middle-East

• Voted UKOUG BI Partner of the Year 2008

• 16 Sessions at Oracle Open World 2010

• Twitter : @rittmanmead

Rittman Mead BI Forum, May 2011

Registrations open for the Rittman Mead BI Forum, May 2011

• Brighton, UK, May 18-20th 2011• Atlanta, GA, May 25-27th 2011• Covers OBIEE, DW, EPM• Guest speakers including:‣ Jeff McQuigg‣ Kevin McGinley‣ Jean-Pierre Dijcks‣ Mike Durran‣ Shyam Varan Nath‣ Philippe Lions‣ Emiel van Bockel

• OBIEE 11g Masterclass withMark Rittman and Tony Heljula

• Full details at http://www.rittmanmead.com/biforum2011

What will this session answer?

• What do I mean by “real-time data warehousing?”‣ Zero latency, or just reduced latency?‣ Do I really mean “right-time data warehousing?”

• What can I use to reduce latency?‣ What does OBIEE offer?‣ What does the Oracle Database (and OWB)

offer?• Can I blend these different offerings?‣ Of course you can!‣ It’s the name of the presentation

What do I mean by “real-time”?

• Traditional data warehouses‣ Daily to weekly batch-load cycles‣ Key-metrics are always delayed‣ Information may not be “actionable”

• Real-time data warehouses‣ Up to the minute or hour reporting‣ Key-metrics are moving‣ Information is actionable

An interval will always exist between the occurrence of a measurable event and our ability to process that event as a “reportable fact”

!"##$%!&'"()!"#"$%&'(##%)*+)%'(,-*./")%0*1"%#(%2"%)3$"%(4%#0"%.""5%2"4($"%#0"+%/.)/)#%(.%$"*67#/,"%23)/."))/.#"66/8".'"9

!"#$%&#'()*#&)%+,-./)#$(/01-.23#4(&#%-#+)%*-#5#6(0)#-6)#)23.2)#/%2%3)/)2-7#-&%$-.(2#$(2-&(+#%2'#89:#*"*-)/*#%&)#&)%+#-./);<=

!"#/(>.+)#06(2)#.*#&)%+#-./)7#)?)2#*(/)#(@#/"#6(/)#)2-)&-%.2/)2-#3%'3)-*#%&)#&)%+#-./)#A#>1-#B6%-#%>(1-#>1*.2)**

.2-)++.3)2$)#*"*-)/*C

52#-6)#)%&+"#'%"*#(@#95#4(&#B%*#.-#$%++)'#')$.*.(2#*100(&-#>%$D#-6)2C<#&)0(&-.23#'%-%#@(&#-6)#$1&&)2-#/(2-6#B%*#%2#%$6.)?)/)2-=

8*#-)$62.E1)*#%2'#-)$62(+(3.)*#)?(+?)'7#$(/0%2.)*#/(?)'#-(#&)0(&-#(2#-6)#0&)?.(1*#'%"#>1-#B)&)#*-.++#&)+.%2-#(2#%2#(1-,(@,

6(1&*#>%-$6#0&($)**#-(#/(?)#'%-%#@&(/#-6)#*(1&$)#*"*-)/*#%2'#*-(&)#%2'#%33&)3%-)#.-#.2#-6).&#'%-%#B%&)6(1*)*#%2'#'%-%

/%&-*=

91-#B.-6#FG,6(1&#>1*.2)**#'%"*#%2'#-6)#2))'#-(#&)0(&-#%$&(**#/1+-.0+)#-./),H(2)*7#-6)#-&%'.-.(2%+#>%-$6#B.2'(B#.*#>).23

*E1))H)'#.2-(#2)%&#2(2,)I.*-)2$)=#J)B#'%-%#)I-&%$-.(2#%2'#+(%'#0%&%'.3/*#6%?)#%+*(#>))2#')?)+(0)'#-(#-&.$D+),@))'#&)0(&-.23

*"*-)/*=#91-#%&)#-6)*)#&)%++"#&)%+#-./)#A#%2'#'()*#.-#&)%++"#/%--)&C

52#&)%+.-"7#-6)&)#.*#%+B%"*#3(.23#-(#>)#%#')3&))#(@#+%-)2$"#>)-B))2#%2#)?)2-#6%00)2.23#%2'#.-#>).23#&)$(&')'#(2#%#-&%2*%$-.(2%+

*"*-)/7#0&(0%3%-)'#-(#-6)#&)0(&-.23#*"*-)/7#+(%')'7#%2'#@.2%++"#%33&)3%-)'#(2#%#'%-%#B%&)6(1*)#>)@(&)#>).23#&)%'"#-(#E1)&"=

8''#-(#-6%-#-6)#')+%"#>)-B))2#-6)#1*)&#.**1.23#%#E1)&"#-6%-#%$$)**)*#-6.*#2)B#2133)-#(@#.2@(&/%-.(2#%2'#-6)#&)*1+-#>).23

&)-1&2)'#-(#-6)#1*)&7#%2'#"(1#@.2'#%#'.*$)&2.>+)#+%3#>)-B))2#)?)2-#%2'#(>*)&?%-.(2=

K&1)7#-6)&)#%&)#-6.23*#"(1#$%2#'(#-(#/.-.3%-)#-6.*#+%3#>1-#-6)&)#B.++#%+B%"*#>)#%#$(10+)#(@#*-)0*#.2#-6)#0&($)**#B6)&)#&)'1$.23

(2)#-./)#0)&.('#.2$&)%*)*#%2(-6)&#A#@(&#)I%/0+)7#"(1#$%2#*6(&-)2#&)0(&-#)I)$1-.(2#-./)#>"#%33&)3%-.23#-6)#'%-%#-(#>)--)&

%2*B)&#-6)#E1)&"7#>1-#-6.*#*-)0#%''*#/(&)#-./)#-(#-6)#'%-%#+(%'#%2'#%33&)3%-)#06%*)=

L1&-6)&/(&)7#'.@@)&)2-#-"0)*#(@#.2$(/.23#'%-%#$%2#6%?)#'.@@)&.23#+%-)2$.)*=#L(&#)I%/0+)7#%#2)B#-&%2*%$-.(2#/%"#6%?)#%#*6(&-)&

+%-)2$"#-6%2#%#$6%23)#-(#%#&)3.(2%+#3&(10.23#(@#*-(&)*#-6%-#/%"#2))'#-(#>)#0&(0%3%-)'#-6&(136#/%2"#+%")&*#(@#6.*-(&.$%+

%33&)3%-.(2#-%>+)*#4*))#L.31&)#M<=

K6)#'.?.*.(2#>)-B))2#'%-%#%$E1.*.-.(2#%2'#'%-%#0&)*)2-%-.(2#0&)*)2-*#%#D)"#E1)*-.(2#-6%-#%2"#$(/0%2"#2))'*#-(#%2*B)&#B6)2

$(2-)/0+%-.23#/(?.23#-(#%#&)%+,-./)#95#*"*-)/N#%&)#"(1#O1*-#.2-)&)*-)'#.2#0&(?.'.237#%*#&%0.'+"#%*#0(**.>+)7#.2@(&/%-.(2#>%*)'

(2#%+&)%'"#%$E1.&)'#'%-%7#(&#'(#"(1#%+*(#2))'#-(#&)0(&-#(2#2)B+"#%'')'#@%$-1%+#.2@(&/%-.(2C

L(&#)I%/0+)7#%#B(&D)&#.2#%#/(>.+)#06(2)#$(/0%2"P*#$%++

$)2-&)#/%"#+.D)#-(#*))#.2@(&/%-.(2#%>(1-#Q6(B#3(('#%

$1*-(/)&#.*P#%2'#-6)#+.D)+.6(('#-6%-#*(/)(2)#B.-6.2#-6%-

$1*-(/)&P*#')/(3&%06.$#B(1+'#*B.-$6#-(#%2(-6)&#

*100+.)&=

K6.*#.2@(&/%-.(2#2))'*#-(#>)#0&)*)2-)'#E1.$D+"#4B6.+*-#-6)"

%&)#*0)%D.23#-(#-6)/<#%2'#0&)@)&%>+"#B.-6.2#-6)#$(2-)I-#(@#-6)

+.2),(@,>1*.2)**#%00+.$%-.(2#-6%-#/%2%3)*#-6)#$1*-(/)&

.2-)&%$-.(2=#R(B)?)&#.-#.*#12+.D)+"#-6%-#%2"#(@#-6)#/)-&.$*#1*)'

>"#-6)#$%++#$)2-&)#%3)2-#@(&#$1*-(/)&#0&(@.-%>.+.-"#%2'

0&)'.$-)'#$61&2#B(1+'#>)#.2@+1)2$)'#>"#)?)2-*#(@#-6)#+%*-#@)B

6(1&*=

S(2?)&*)+"7#%#*"*-)/#-6%-#.*#1*.23#)/>)'')'#95#@12$-.(2%+.-"

-(#%+)&-#-(#0(-)2-.%+#@&%1'#4*-($D#/%&D)-*#%2'#$&)'.-#$%&'

$(/0%2.)*#%&)#0(**.>+)#)I%/0+)#1*)&*<#/%"#B)++#2))'#-(

D2(B#%>(1-#10,-(,-6),/(/)2-#%$-.?.-"=

***+,-./0.12345,416,+537

FIGURE 1: BI latency

Recording Propagation Aggregation Query

Event Report

Real-time query latency

Business Case

• We want to be able to segment our reporting between “real-time” needs and static reports

‣ Most users will want real-time numbers delivered with dashboards

‣ Some users (financial-minded) don’t want numbers to change each time they run a report

• Use the standard Oracle BI Stack

‣ Oracle Database Enterprise Edition

‣ Oracle Business Intelligence Enterprise Edition

Latency versus Query Performance

OLTP Reporting

Federated OLTP/EDW Reporting

EDW with Realtime Component

Standard EDW

OLTP Reporting

Standard EDW

Latency

Query Performance

OLTP Reporting

Standard EDW

Latency

Query Performance

Oracle Next-Generation Reference Architecture

Requirement: Identifying and Capturing Change

Requirement: Change Data Capture

Identify Change

• Oracle CDC• OWB Trickle-feed

mappings• Trigger-based

capture• Recording

modification dates• MINUS Command

Identify Change

Propagate Change

mappings• Message Queues• Views over

database links• File movements

Identify Change

Propagate Change

Apply Change to Target

• OWB Mappings and Process Flows

• Other ETL Tools• Custom ETL code• Custom transaction

Identify Change

Propagate Change

Oracle Asynchronous Change Data Capture

Hot LogOnline redo logs mined locallyChange tables are updated and stored locally

Hot Log DistributedOnline redo logs are mined locallyChange tables are updated and stored remotely

Autolog OnlineOnline redo logs are mined remotelyChange tables are updated and stored remotely

Autolog ArchiveArchived redo logs are mined remotelyChange tables are updated and stored remotely

Hot LogOnline redo logs mined locallyChange tables are updated and stored locally

Latency

Hot Log DistributedOnline redo logs are mined locallyChange tables are updated and stored remotely

Autolog OnlineOnline redo logs are mined remotelyChange tables are updated and stored remotely

Autolog ArchiveArchived redo logs are mined remotelyChange tables are updated and stored remotely

Identify Change

Propagate Change

Identify Change

Propagate Change

Identify Change

Propagate Change

Identify Change

Propagate Change

•Trickle-feed mappings allow the development of Streams Capture and Apply processes as continuously running mappings.

• Layering in OLTP data to provide the real-time component

• General approach is to use a BI tool such as OBIEE

• No real database or modeling involvement

Classic Batch Load Star Schema: Process Flow

Classic Batch Load Star Schema: Fact Table Load

Federation with OBIEE

OBIEE Metadata and Data Layers

Relational, OLAP, File and Application Data Sources

• Physical Data Sources, suchas relational databases,OLAP Cubes and applicationsource adapters

• A Semantic Model made upof Metrics, dimensions, hierarchies and calculations, derived from...

Enterprise Semantic Model

• The Web Catalog containsdefinitions of reports, alertsdashboards, KPIs, based on...

• A Semantic Model made upof Metrics, dimensions, hierarchies and calculations, derived from...

Web Catalog

Enterprise Semantic Model

Flow of Data Through the Three-Layer Semantic Model

Simplification of the Data ModelIntegration of Disparate DataSources

Addition of Business Logic and Calculations

Logical Table Sources Map to Physical Models

Logical Table Sources Respect the Physical Model

WITH SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2, T43901.CUST_LAST_NAME as c3, T43971.SAL_AMT as c4from GCBC_CRM.CUSTOMERS T43901, GCBC_POS.POS_TRANS T43971, GCBC_POS.POS_TRANS_HEADER T43978where ( T43901.CUST_ID = T43978.CUST_ID and T43971.TRANS_ID = T43978.TRANS_ID and TO_DATE('2010-09-18 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T43978.TRANS_DATE ) union allselect T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T44105.AMOUNT as c4from GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY ) )),SAWITH1 AS (select sum(D3.c4) as c1, D3.c2 as c2, D3.c3 as c3from SAWITH0 D3group by D3.c2, D3.c3)select distinct 0 as c1, D2.c2 as c2, D2.c3 as c3, D2.c1 as c4from SAWITH1 D2

WITH SAWITH0 AS (select sum(T44105.AMOUNT) as c1, T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T45416.CALENDAR_YEAR as c4from GCBC_EDW.DATE_DIM T45416, GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T45416.DATE_KEY and T45416.SQL_DATE = TO_DATE('2003-01-09 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') ) group by T44042.CUSTOMER_FIRST_NAME, T44042.CUSTOMER_LAST_NAME, T45416.CALENDAR_YEAR)select distinct 0 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c1 as c5from SAWITH0 D1

• The BI Server understands from the Fragmentation content how to “optimize” the query

• Only one LTS is needed to provide the answer

Fragmentation with OLTP and EDW

Demonstration

• Strengths

‣ Low latency

‣ No complex modeling or ETL

‣ Incorporated into any standard dimensional EDW

• Weaknesses

‣ Performance

‣ Difficult to conform multiple systems

EDW with a Real-Time Component

• Modeling complimentary real-time structures in an EDW

• Use functionality of the Oracle Database and OWB

• Complex ETL and Database interaction

The Real-Time Partition

Table Partition

Fact Table

Table Partition

Fact Table

Physical Table

Fact Real-Time Partition

Table Partition

Fact Table

Physical Table

Fact Real-Time Partition

Physical Table

• All ETL processing of fact records are directed at the Real-Time Partition using Micro-Batch

• No indexes, small data set• Partition Exchange at the end of the day

Physical Table

Table Partition

Physical Table

Table Partition

Physical TableTable Partition

Early-Arriving Facts

• Grab the most recent dimension record when that entity already exists

• Use a default “Not Known Yet” record when they don’t

• Prior to the partition exchange, we run our late-arriving dimension loads

• We update the Real-Time Partition with the correct surrogate keys‣ OWB generates an efficient MERGE‣ Small data sets in the Real-Time Partition

What if Type 2 dimension changes haven’t arrived yet?

Real-Time Integration: Fact Table RT Load

Real-Time Integration: Correct Dimensions

Real-Time Integration: “Closing the Books” on the Day

Plugging it together with OBIEE (Non-Fragmentation)

• Using standard database functionality (UNION ALL)

• Could be done with a view• Using Surrogate Keys in the

RT Partition• OBIEE doesn’t have to

manage different LTS’s

EDW with a Real-Time Component

Demonstration

Summary

• Latency versus performance and scalability

• Latency and performance versus degree of difficulty

• Different variations on the two test cases

Stewart Bryson, Managing Director, Rittman Mead AmericaCollaborate, Orlando, April 2011

Real-Time DW with OBIEE and Oracle Database

real-time data warehousing using oracle database and obiee - collaborate'11

bi latency

realtime query latency

rittman mead bi forum

rittman mead blog

realtime numbers

realtime data warehousing

oracle databasestewart

oracle open world

Technology

obiee training | oracle obiee training | obie course |...

obiee publisher

obiee mate

obiee install

obiee & essbase · obiee charts with essbase data here is a...

sample obiee

data warehousing & business intelligence customer … ·...

oracle 11g and obiee data warehousing and analytics

obiee & essbase - mindstreamanalytics.com and essbase... ·...

obiee session

obiee performance

obiee introduction

obiee variables

working on projects remotely collaborate...

visualizing data using maps in obi 11g -...

obiee - how to access obiee

obiee hierarchies

obiee installation

oracle 11g data warehousing vincent chazhoor data warehouse...

easy connections...collibra – obiee integration v1.0...