obiee reporting against transactional schemas

T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com

Stewart Bryson, US Managing Director, Rittman MeadRittman Mead BI Forum, May 2012

Transactional Schemas with OBIEE 11g

Monday, 21 May 12

Who am I?

Monday, 21 May 12

Stewart Bryson, US Managing Director, Rittman Mead

Oracle ACEOracle BI/DW Architect and Delivery SpecialistCommunity Speaker and EnthusiastWriter for Rittman Mead Blog:

http://www.rittmanmead.com/blog

Developer of Transcend FrameworkEmail : [email protected] : @stewartbryson

Monday, 21 May 12

Who is Rittman Mead?

Monday, 21 May 12

Oracle BI&DW ExpertsConsulting, Training and SupportGlobal Company, Offices in the US, UK, India and

Australia

1 Oracle ACE Director, 2 Oracle ACEsLongest-running Oracle (BI) BlogVoted UKOUG BI Partner of the Year 2008, 2010, 2011Oracle Community Advocates

16 Sessions at Oracle Open World 2010 12 Sessions at Oracle Open World 2011

Twitter : @rittmanmead

Monday, 21 May 12

What Will We Talk About?

Monday, 21 May 12

Architectural decisionsDifferent types of transaction schemasApplication Express (Apex) Applications

Customer Tracking (CRM) Ask the Expert

See the RPD development processLearn some best practices along the way

Monday, 21 May 12

Why Do We Want to Do This?

Monday, 21 May 12

Data Warehouse is not an option Budgetary constraints Lack of conviction Failed BI projects in the past

Visionary Methodology Agile or iterative development Exadata

Paradigm Shifts Big Data Hadoop, MapReduce Endeca (?)

Because you can!

Monday, 21 May 12

Oracle Next-Generation DW Reference Architecture

Monday, 21 May 12

Staging Layer Change tables Reject tables External tables

Foundation Layer Typically third-normal form

(3NF) Process neutral Reporting-agnostic

Access and Performance Layer Dimensional model Star Schemas Process specific

Monday, 21 May 12

Recommend building a foundational layer instead of a replicated copy Insert-only copy of the

source No updates or deletes

Additional functionality Historical reporting Mimic slowly-changing

dimensionsFor simplicity sake, Im going

directly against the source for this presentation

Monday, 21 May 12

Oracle Business Intelligence Enterprise Edition (OBIEE) 11g

Monday, 21 May 12

Dashboards, Ad-hoc Reporting, Alerts, Microsoft Office Integration High quality graphical, role/user based views Multiple views of same data

Point and click ease of useCommon Enterprise Information Model

Unified semantic/logical view of data from multiple sources

Heterogeneous database access True enterprise deployment

Alerts, scheduling and distribution

Monday, 21 May 12

OBIEE Metadata and Data Layers

Monday, 21 May 12

Relational, OLAP, File and Application Data Sources

Physical Data Sources, suchas relational databases,OLAP Cubes and applicationsource adapters

Monday, 21 May 12

A Semantic Model made upof Metrics, dimensions, hierarchies and calculations, derived from...

Enterprise Semantic Model

Monday, 21 May 12

The Web Catalog containsdefinitions of reports, alertsdashboards, KPIs, based on...

A Semantic Model made upof Metrics, dimensions, hierarchies and calculations, derived from...

Web Catalog

Enterprise Semantic Model

Monday, 21 May 12

Flow of Data Through the Three-Layer Semantic Model

Monday, 21 May 12

Flow of Data Through the Three-Layer Semantic Model

Simplification of the Data ModelIntegration of Disparate Data Sources

Addition of Business Logic and CalculationsAddition of Aggregate Sources

Monday, 21 May 12

Two Types of Transactional Schemas

Monday, 21 May 12

Highly Normalized (3NF) Typical OLTP application Designed for high-scalability Hundreds or thousands of simultaneous

gets and puts Customer Tracking Application (CRM)

Monday, 21 May 12

Highly Normalized (3NF) Typical OLTP application Designed for high-scalability Hundreds or thousands of simultaneous

gets and puts Customer Tracking Application (CRM)

Highly Denormalized More denormalized than dimensional models Typically a single entity holding everything

about a transaction Weblogs, logging tables, etc. Our Ask the Expert Application

Monday, 21 May 12

Customer Tracking (CRM)

Monday, 21 May 12

Ask the Expert

Monday, 21 May 12

Ask the Expert

Monday, 21 May 12

Ask the Expert

Monday, 21 May 12

Ask the Expert

Monday, 21 May 12

Resolving Ambiguous (Multiple) Joins Between Tables

Monday, 21 May 12

A physical table may, by default, join to another physical table many times Multiple joins to a staff table for regional manager,

branch manager, etc. Multiple dates for order date, ship date, return date,

etc.

BI Server must have one, and only one, join route between each table

Therefore you must alias the joined table and join to each alias separately

Monday, 21 May 12

Extracting Simple Dimension Tables from 3NF

Monday, 21 May 12

Find an entity that we know we want to report on Customer makes a lot of sense for CRM Join our customer-specific entities in a Logical Table

Source (LTS)

Find a overall primary key for the logical dimension table Made up of multiple underlying tables Usually the key from the lowest level of granularity

Construct hierarchies that define the data Not much different than a dimensional model The logical dimension table now hides the

complexity

Monday, 21 May 12

Extracting Simple Fact Tables from 3NF

Monday, 21 May 12

Identify transaction tables as candidates Tables with important numeric values are easy Sometimes we have measure-less relationship

tables

Build the fact defining measures where needed Relationship tables might not have measures Kimball calls these factless fact tables Build a mock count measure

Transaction tables sometimes have textual attributes Build a logical dimension table for these Kimball calls these junk or mystery

dimensions

Monday, 21 May 12

WITH SAWITH0 AS (select sum(1) as c1, T367.ACTIVITY_TYPE as c2, T396.CATEGORY as c3, T476.ID as c4, T476.CUSTOMER_NAME as c5, T500.STATUS as c6from CUST_TRACK.EBA_CUST_CATEGORIES T396, CUST_TRACK.EBA_CUST_CUSTOMERS T476, CUST_TRACK.EBA_CUST_STATUS T500, CUST_TRACK.EBA_CUST_CUST_ACTIVITY T452, CUST_TRACK.EBA_CUST_ACTIVITY_TYPES T367where ( T367.ID = T452.ACTIVITY_TYPE_ID and T396.ID = T476.CATEGORY_ID and T452.CUSTOMER_ID = T476.ID and T476.STATUS_ID = T500.ID ) group by T367.ACTIVITY_TYPE, T396.CATEGORY, T476.CUSTOMER_NAME, T476.ID, T500.STATUS)select distinct 0 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c5 as c5, D1.c6 as c6, D1.c1 as c7, 0 as c8, 0 as c9from SAWITH0 D1order by c3, c4, c5, c6, c2

Monday, 21 May 12

WITH SAWITH0 AS (select sum(1) as c1, T367.ACTIVITY_TYPE as c2, T396.CATEGORY as c3, T476.ID as c4, T476.CUSTOMER_NAME as c5, T500.STATUS as c6from CUST_TRACK.EBA_CUST_CATEGORIES T396, CUST_TRACK.EBA_CUST_CUSTOMERS T476, CUST_TRACK.EBA_CUST_STATUS T500, CUST_TRACK.EBA_CUST_CUST_ACTIVITY T452, CUST_TRACK.EBA_CUST_ACTIVITY_TYPES T367where ( T367.ID = T452.ACTIVITY_TYPE_ID and T396.ID = T476.CATEGORY_ID and T452.CUSTOMER_ID = T476.ID and T476.STATUS_ID = T500.ID ) group by T367.ACTIVITY_TYPE, T396.CATEGORY, T476.CUSTOMER_NAME, T476.ID, T500.STATUS)select distinct 0 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c5 as c5, D1.c6 as c6, D1.c1 as c7, 0 as c8, 0 as c9from SAWITH0 D1order by c3, c4, c5, c6, c2

Monday, 21 May 12

Extracting Simple Facts and Dimensions

Does This Make Sense?

Monday, 21 May 12

Extracting Complex Dimension Tables from 3NF

Monday, 21 May 12

Dimensional models combine multiple entities in a single dimension Completely contained in a single hierarchy More than just removing snowflakes

Dimensional models use a Date DimensionMultiple hierarchies

Users want the data to rollup in different ways There is always more than one route from Grand

Total to Detail

Monday, 21 May 12

Transactional versus Dimensional

Monday, 21 May 12

Logical Fact

Logical Dimension

Monday, 21 May 12

Logical Fact

Logical Dimension

Monday, 21 May 12

Think more transactionally Is similarity to the OLTP system more

intuitive? Is it easier to model? Will we get better performance?

Monday, 21 May 12

Think more transactionally Is similarity to the OLTP system more

intuitive? Is it easier to model? Will we get better performance?

Think more dimensionally This is the way that business users think

about their data Modeling effort is about the same The BI Servers ability to prune unneeded

tables from an LTS makes performance negligible

Monday, 21 May 12

Combining Multiple Entities in a Single Dimension

Monday, 21 May 12

`

Monday, 21 May 12

`

Monday, 21 May 12

`

Monday, 21 May 12

`

Monday, 21 May 12

`

Monday, 21 May 12

WITH SAWITH0 AS (select sum(1) as c1, T476.CUSTOMER_NAME as c2from CUST_TRACK.EBA_CUST_ASSOCIATIONS T386, CUST_TRACK.EBA_CUST_CUSTOMERS T476, CUST_TRACK.EBA_CUST_CUST_ACTIVITY T452where ( T386.CUSTOMER_ID = T476.ID and T452.CUSTOMER_ID = T476.ID ) group by T476.CUSTOMER_NAME)select D1.c1 as c1, D1.c2 as c2, D1.c3 as c3 from ( select distinct 0 as c1, D1.c2 as c2, D1.c1 as c3from SAWITH0 D1order by c2 ) D1 where rownum

Building a Date Dimension

Monday, 21 May 12

Transactional systems do not have date dimensions Single record gets and puts Date functions are sufficient

Dimensional models use date dimensions We dont have one in our transactional schema We need one to build our date hierarchies OBIEE requires one for time-series functionality

Two options Build a date dimension in your reporting

environment joining dates to dates Construct a logical date dimension with what you

have

Monday, 21 May 12

Transactional systems do not have date dimensions Single record gets and puts Date functions are sufficient

Dimensional models use date dimensions We dont have one in our transactional schema We need one to build our date hierarchies OBIEE requires one for time-series functionality

Two options Build a date dimension in your reporting

environment joining dates to dates Construct a logical date dimension with what you

have

Monday, 21 May 12

WITH SAWITH0 AS (select sum(1) as c1, RTRIM(TO_CHAR(T452.ACTIVITY_DATE, 'MON')) as c2, TO_NUMBER(TO_CHAR(T452.ACTIVITY_DATE, 'yyyy'), '9999') as c3, T476.CUSTOMER_NAME as c4from CUST_TRACK.EBA_CUST_ASSOCIATIONS T386, CUST_TRACK.EBA_CUST_CUSTOMERS T476, CUST_TRACK.EBA_CUST_CUST_ACTIVITY T452where ( T386.CUSTOMER_ID = T476.ID and T452.CUSTOMER_ID = T476.ID ) group by T476.CUSTOMER_NAME, TO_NUMBER(TO_CHAR(T452.ACTIVITY_DATE, 'yyyy'), '9999'), RTRIM(TO_CHAR(T452.ACTIVITY_DATE, 'MON')))select distinct 0 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c1 as c5, 0 as c6from SAWITH0 D1order by c4, c3, c2

Monday, 21 May 12

Extracting Complex Fact Tables from 3NF

Monday, 21 May 12

Facts built from multiple sources are common in DW environments Sales table built from a CRM plus an e-commerce

application Activity fact tables that combine multiple types of

activity

This is fairly simple with ETL processing Multiple mappings loading the same fact table A single mapping doing complex union logic

Do we have the same capability available to us in the OBIEE Business Model?

Monday, 21 May 12

Complex Fact Tables: Multiple Activity Sources

Monday, 21 May 12

WITH SAWITH0 AS (select 1 as c1, T476.CUSTOMER_NAME as c2, T367.ACTIVITY_TYPE as c3, T452.ACTIVITY_DATE as c4from CUST_TRACK.EBA_CUST_ASSOCIATIONS T386, CUST_TRACK.EBA_CUST_CUSTOMERS T476, CUST_TRACK.EBA_CUST_CUST_ACTIVITY T452, CUST_TRACK.EBA_CUST_ACTIVITY_TYPES T367where ( T367.ID = T452.ACTIVITY_TYPE_ID and T386.CUSTOMER_ID = T476.ID and T452.CUSTOMER_ID = T476.ID ) ),SAWITH1 AS (select 1 as c1, T476.CUSTOMER_NAME as c2, T476.CREATED_ON as c3from CUST_TRACK.EBA_CUST_CUSTOMERS T476, CUST_TRACK.EBA_CUST_ASSOCIATIONS T386where ( T386.CUSTOMER_ID = T476.ID ) ),SAWITH2 AS ((select TO_NUMBER(TO_CHAR( TRUNC(D0.c4), 'yyyy'), '9999') as c2, D0.c2 as c3, 'Explicit' as c4, D0.c3 as c5, D0.c1 as c6from SAWITH0 D0union allselect TO_NUMBER(TO_CHAR( TRUNC(D0.c3), 'yyyy'), '9999') as c2, D0.c2 as c3, 'Implicit' as c4, 'Customer Account Creation' as c5, D0.c1 as c6from SAWITH1 D0)),SAWITH3 AS (select sum(D3.c6) as c1, D3.c2 as c2, D3.c3 as c3, D3.c4 as c4, D3.c5 as c5from

(continued...)

Monday, 21 May 12

WITH SAWITH0 AS (select 1 as c1, T476.CUSTOMER_NAME as c2, T367.ACTIVITY_TYPE as c3, T452.ACTIVITY_DATE as c4from CUST_TRACK.EBA_CUST_ASSOCIATIONS T386, CUST_TRACK.EBA_CUST_CUSTOMERS T476, CUST_TRACK.EBA_CUST_CUST_ACTIVITY T452, CUST_TRACK.EBA_CUST_ACTIVITY_TYPES T367where ( T367.ID = T452.ACTIVITY_TYPE_ID and T386.CUSTOMER_ID = T476.ID and T452.CUSTOMER_ID = T476.ID ) ),SAWITH1 AS (select 1 as c1, T476.CUSTOMER_NAME as c2, T476.CREATED_ON as c3from CUST_TRACK.EBA_CUST_CUSTOMERS T476, CUST_TRACK.EBA_CUST_ASSOCIATIONS T386where ( T386.CUSTOMER_ID = T476.ID ) ),SAWITH2 AS ((select TO_NUMBER(TO_CHAR( TRUNC(D0.c4), 'yyyy'), '9999') as c2, D0.c2 as c3, 'Explicit' as c4, D0.c3 as c5, D0.c1 as c6from SAWITH0 D0union allselect TO_NUMBER(TO_CHAR( TRUNC(D0.c3), 'yyyy'), '9999') as c2, D0.c2 as c3, 'Implicit' as c4, 'Customer Account Creation' as c5, D0.c1 as c6from SAWITH1 D0)),SAWITH3 AS (select sum(D3.c6) as c1, D3.c2 as c2, D3.c3 as c3, D3.c4 as c4, D3.c5 as c5from

(continued...)

Monday, 21 May 12

WITH SAWITH0 AS (select sum(1) as c1, TO_NUMBER(TO_CHAR( TRUNC(T476.CREATED_ON), 'yyyy'), '9999') as c2, T476.CUSTOMER_NAME as c3, T476.ID as c4, 'Implicit' as c5, 'Customer Account Creation' as c6from CUST_TRACK.EBA_CUST_CUSTOMERS T476, CUST_TRACK.EBA_CUST_ASSOCIATIONS T386where ( T386.CUSTOMER_ID = T476.ID and TO_NUMBER(TO_CHAR( TRUNC(T476.CREATED_ON), 'yyyy'), '9999') = 2012 ) group by T476.CUSTOMER_NAME, T476.ID, TO_NUMBER(TO_CHAR( TRUNC(T476.CREATED_ON), 'yyyy'), '9999'))select distinct 0 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c5 as c5, D1.c6 as c6, D1.c1 as c7, 0 as c8, 0 as c9from SAWITH0 D1order by c3, c4, c6, c2, c5

Monday, 21 May 12

WITH SAWITH0 AS (select sum(1) as c1, TO_NUMBER(TO_CHAR( TRUNC(T476.CREATED_ON), 'yyyy'), '9999') as c2, T476.CUSTOMER_NAME as c3, T476.ID as c4, 'Implicit' as c5, 'Customer Account Creation' as c6from CUST_TRACK.EBA_CUST_CUSTOMERS T476, CUST_TRACK.EBA_CUST_ASSOCIATIONS T386where ( T386.CUSTOMER_ID = T476.ID and TO_NUMBER(TO_CHAR( TRUNC(T476.CREATED_ON), 'yyyy'), '9999') = 2012 ) group by T476.CUSTOMER_NAME, T476.ID, TO_NUMBER(TO_CHAR( TRUNC(T476.CREATED_ON), 'yyyy'), '9999'))select distinct 0 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c5 as c5, D1.c6 as c6, D1.c1 as c7, 0 as c8, 0 as c9from SAWITH0 D1order by c3, c4, c6, c2, c5

Where did the union go?

Monday, 21 May 12

Extracting Complex Facts and Dimensions

Monday, 21 May 12

Modeling Highly-Denormalized Schemas

Monday, 21 May 12

Some applications are almost model-less Files Weblogs Thin applications

OBIEE doesnt accept a single logical table in the Business Model Need at least one logical fact table Need at least one logical dimension table Usually well have multiple logical dimensions

How would we do this in the OBIEE Business Model? Use multiple aliases to a single table for facts and

dimension LTSs? Use the same physical table directly for each LTS?

Monday, 21 May 12

Some applications are almost model-less Files Weblogs Thin applications

OBIEE doesnt accept a single logical table in the Business Model Need at least one logical fact table Need at least one logical dimension table Usually well have multiple logical dimensions

How would we do this in the OBIEE Business Model? Use multiple aliases to a single table for facts and

dimension LTSs? Use the same physical table directly for each LTS?

Monday, 21 May 12

WITH SAWITH0 AS (select sum(1) as c1, T140.CATEGORY as c2, T140.SUBJECT as c3, T140.ID as c4from ASKTOM.ATE_SUBMITTED_QUESTIONS T140group by T140.CATEGORY, T140.ID, T140.SUBJECT),SAWITH1 AS (select distinct 0 as c1, D1.c2 as c2, D1.c3 as c3, D1.c1 as c4, D1.c4 as c6from SAWITH0 D1),SAWITH2 AS (select D1.c1 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, sum(D1.c4) as c5from SAWITH1 D1group by D1.c1, D1.c2, D1.c3, D1.c4)select D1.c1 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c5 as c5 from ( select D1.c1 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, sum(D1.c5) over (partition by D1.c2) as c5from SAWITH2 D1order by c1, c2 ) D1 where rownum

Common to add another logical fact table with some of the same dimensional attributes

We have the Question information, but now we want to add the Review information

Build conformed logical dimensions out of disparate single attributes in two tables Submitter/Reviewer Submission Date/Review Date Submission/Review information

Monday, 21 May 12

Summary

Monday, 21 May 12

Myth: OBIEE is only for reporting against Star Schemas Siebel first acquired nQuire Server to do

effective reporting against Siebel CRM system.

The Key to effective OLTP reporting is creative use of Logical Table Sources (LTS)

OBIEE does table pruning only using what it needs to in the LTS

We dont have to use separate aliases when a table acts as both a fact and a dimension

Monday, 21 May 12

Stewart Bryson, US Managing Director, Rittman MeadRittman Mead BI Forum, May 2012

Transactional Schemas with OBIEE 11g

Monday, 21 May 12

obiee reporting against transactional schemas

Documents