oracle business intelligence 11g data-mart automation indicators... · “data-mart automation”...

Oracle Business Intelligence 11g

Data-Mart Automation

September 2013

© Peak Indicators Limited 2

AgendaData-Mart Automation

• Introduction

• Modelling Principles

• Scenario

• Option 1: Advanced Modelling

• Option 2: Aggregate Persistence

• Option 3: BI Server “Populate” Queries

• Peak ETA

• Summary



Introduction


Advanced Modelling with Multiple SourcesIntroduction

• Oracle BI is truly an “enterprise” Business Intelligence tool with great capabilities for handing:

• Multiple data sources

• Different types of data source

• Real-time vs Historical analysis

• Distribution via many channels

• Mobile / Desktop

• Complex Security

• But when you are asked to report from a blend of data sources, what is the right approach to take?


Advanced Modelling with Multiple SourcesAim of Presentation

• This presentation will cover the “Data-Mart Automation” capabilities of the Oracle BI Server

• We will look at the variety of ways in which Oracle BI can seamlessly combine multiple data sources together. For example:

• Federated Query

• Cross-Database Join

• Fragmentation

• Aggregate Persistence

• BI Server “populate” queries



Modelling Principles


Modelling PrinciplesThere are 3 Modelling Layers to Consider


Modelling PrinciplesPhysical Layer


Modelling PrinciplesBusiness Model Layer


Modelling PrinciplesPresentation Layer


Modelling PrinciplesSimple Subject Areas regardless of physical complexity!


Modelling PrinciplesReal-life Example

• Overcoming complexity in the RPD Business Model:


Modelling PrinciplesThe Business Model is always Star-Schema

• It doesn’t matter what type of data source you have…..

• It doesn’t matter how many data sources you have….

• The Business Model always looks like it is coming from a “star-schema” data-mart

• Facts only join to Dimensions, and Dimensions only join to Facts



Scenario


ScenarioCustomer with a variety of sources

• Let’s take this example situation where we have a customer with 3 data sources wanting to report in Oracle BI:

LegacySales

(pre-2000)

Sales Data Warehouse

(Year 2000+)

CRMService

(Real-time)

ORACLE BI


ScenarioHow can we achieve any of the following?

• Report on both Sales and Service in the same report?

• Report on both Legacy Sales and DW Sales at the same time?

• Report across 3 difference systems at the same time and with optimised performance?


ScenarioOut Starting Point – 3 distinct Subject Areas



Option 1: Advanced Modelling


Cross-Database JoinsOverview

• Cross-Database Joins are useful for joining a Fact table on one data source to a Dimension table on another

• Oracle BI does not extract “everything” from both sources!


Cross-Database JoinsRPD Modelling #1

• Even though the two tables are in different Physical Databases, you should create a Physical Join between them, just as if they were on the same database:



• Create a Logical Table for the “foreign” entity in the Business Model:



• Create a Logical Join between the two Logical Tables, setting the “Driving Table” to be the source that returns the fewest records:


Cross-Database JoinsTest

• The Session SQL will contain 2 physical queries

• The first against the “Driving” source

• The second to look up corresponding values in the secondary source


Cross-Database JoinsFurther Notes

• By default, Oracle BI limits usage so that a “drive join” can consume a maximum of:

• 100 parameters per query

• 200 queries in total

• A total of: 20,000 records

• These limits can be adjusted in the Database “Features” properties


Cross-Database JoinsPros and Cons

• Advantages:

• Relatively efficient method to join tables together in two different sources, without having to extract all the data from both

• Can perform well in the right situation

• Outer joins are supported

• Local/remote databases can be different types and structure

• Disadvantages:

• Lack of scalability – you cannot do this for millions of records

• BI Server has to do the join processing, not the database

• Relies on good data quality – the join has to work!

• Not suitable for complex models

• Choosing a “driving table” forces Oracle BI to optimise queries in the same way every time

• Open to abuse


Federated QueryOverview

• Federated Query enables you simultaneously query multiple data sources and view the data on the same report

• As long as all the facts share common dimensions

Facts 1 Facts 2Common

Dimensions


• The users “sees” one data source, but the data is coming from 2 different queries

• Oracle BI merges the summarised data together on the BI Server

Federated QueryOn a Report

SELECT “Year”, “Business Group”, “# Incidents”

FROM “CRM Service”

SELECT “Year”, “Business Group”, “# Orders”

FROM “Sales DW”


Federated QueryRPD Modelling #1

• Model your Logical Facts separately

• Each Logical Fact can come from a different data source

Sales DW

CRM Service



• Create logical joins between both Facts and the common Dimensions



• In the Dimension Logical tables, there are 2 LTS

• Each Logical Column has a mapping to both physical data sources

• The BI Server knows which LTS can be joined to which Fact LTS

Sales DW

CRM Service

Sales DW

CRM Service



• And that’s it!

• You subject area contains:

• A single set of Dimensions, each column can come from multiple sources

• 2 sets of Facts. Each set of Facts map to an individual source


Federated QueryPros and Cons

• Advantages:

• You can join multiple sources at any level of detail (no need to have a key at the most detailed level)

• Little impact on performance – you are just bringing together two summarised queries

• Quick to model and deliver value (compared to building a DW)

• Disadvantages:

• Relies on good data quality – the hierarchies have to be identical across all sources

• Tougher to implement when there are lots of transformation rules or data cleansing requirements

• Works well for summarised queries, but joining a low “detail” level will result in a performance hit


FragmentationOverview

• A “Fragmented” data source is one in which data is split up and stored in multiple places:

• The fragments don’t have to exist on the same data source

• The data sources could be of different types (e.g. Oracle, Excel, XML etc)

• The physical data structures do not have to be the same

• Examples:

• A customer has 3 different instances of Oracle EBS:

• EBS for Europe

• EBS for US

• EBS for Asia


FragmentationExample

• As with all types of data structure, the fact that the data is spread over multiple sources should be totally transparent to the End User

• Consider this example:

• Sales data is fragmented and can be obtained from two different sources:

• Data Warehouse for years 2001 onwards

• Excel spreadsheet for years prior to 2001


FragmentationPerformance

• With “Fragmentation”, Oracle BI will only query the necessary data sources

• For example, if the user drills down on the Year 1999 then Oracle BI will get the next set of results from the Excel Spreadsheet only

• OBIEE knows that the Data Warehouse only contains data from 2001 onwards, so does not initiate a query against it


FragmentationRPD Modelling #1

• To implement Fragmentation, you first need to build a Business Model where each Logical Column is mapped to all the sources

• You should end up with one Logical Table Source per fragment



• You then configure the “Fragmentation Content” for each Logical Table Source

• This informs OBIEE exactly which data is stored in each fragment



• The Fragmentation Content should be repeated across all your Logical Tables:


FragmentationFurther Notes

• The “Fragmentation Content” expressions can be reasonably complex:

• A combination of “AND” and “OR” operators

• Repository Variables can be used (but not Session Variables)

• Operators can be nested

• Expressions can involve calculations (e.g. VALUE + 1)

• Examples:• logicalColumn >= valueof(START_VALUE)+1 AND logicalColumn < valueof(MID_VALUE)

• EnterpriseModel.Period."Day" < VALUEOF("Snapshot Date") OR

EnterpriseModel.Period.MonthCode < VALUEOF("Snapshot Year Month") OR

EnterpriseModel.Period."Year" < VALUEOF("Snapshot Year") OR

EnterpriseModel.Period."Year" = VALUEOF("Snapshot Year") AND

EnterpriseModel.Period."Month in Year" < VALUEOF("Snapshot Month") OR

EnterpriseModel.Period."Year" = VALUEOF("Snapshot Year") AND

EnterpriseModel.Period."Monthname" IN ('Mar', 'Feb', 'Jan')


FragmentationPros and Cons

• Advantages

• Effective way to “union” many data sources together

• Handles different data sources with different structures

• BI Server only queries the data sources it needs to

• Disadvantages

• Best suited to smaller Business Models – does not handle large amounts of complexity (data cleansing etc)

• Once you set up Fragmentation, the whole Business Model ideally has to be fragmented – does not always cater for new requirements

• Not always a sustainable approach – in the long term the Business Model could become very complex and hard to maintain


Data-Mart AutomationOption 2: Aggregate Persistence


The Story So Far….Long Term Performance/Scalability

• All the techniques explored so far share a common set of disadvantages – mostly around complexity, scalability

• Furthermore, they rely on reporting direct against the source systems which may not have the infrastructure or data structures to support heavy duty reporting

• So what options are there for consolidating everything onto a single database?


Aggregate PersistenceOverview

• With minimal effort, OBI will automatically build and populate a series of aggregate tables on a chosen data source

• OBI will then automatically redirect any “summary” reports to use these aggregates

• New aggregates built without downtime

• OBI is 100% available whilst aggregates are being built

• Complex data-models transformed into optimised star-schemas


Aggregate PersistenceOverview

• In the case of Exalytics, the “aggregate” tables are placed in the TimesTen in-memory database or Oracle 12c In-Memory database located on the Exalytics server

• With data placed in-memory on the BI Server the result is complete elimination of disk and network bottlenecks

• Exalytics-only features include:

• TimesTen Columnar Compression

• Exalytics Summary Advisor


Aggregate PersistenceCustomer Example 1

• A Global provider of Mobile Satellite Communications:

• A 27x performance gain achieved using TimesTen in-memory aggregates


Aggregate PersistenceExample 2

• A UK Retailer:

• 88x performance gain across 9 dashboard achieved using in-memory aggregates


Aggregate PersistenceWithin the RPD

• Aggregate Persistence is a great utility for rapidly building a relational aggregation layer:

• it not only creates a set of physical aggregate tables, it populates them and then also models them into the RPD file

• The aggregate tables can exist on a separate database

• The process can take place whilst the RPD is on-line


• Before running the Aggregate Persistence scripts, run the “Model Checker” to identify any issues with your dimensional models e.g. non-unique keys

• File > Check Models > Complete

Aggregate PersistenceBI Admin Tool “Model Checker”


Aggregate PersistenceExample

• We are going to use Aggregate Persistence to consolidate Service and Sales data together in the same data-mart. Benefits include:

• More optimised data structures for CRM Service

• No more need to do cross-database joins


Aggregate PersistenceRPD Modelling #1

• Open up the "Sales DW" wized (Tools > Utilities) and choose the location of your script

Allows you to modify create scripts before

tables are created



• Choose your Fact columns and the levels at which you want to summarise data (e.g. by Month and Business Group)



• Do the same for any other aggregates you wish to create



• Your aggregate persistence script is now built:



• Run your sql script using NQCMD:• nqcmd -d [odbc to BI Server] -u weblogic -p [password] -s [script]



• Your new tables are now build, populated and modelled in the RPD



• The new aggregates are available immediately for use in the BI front-end:


Aggregate PersistencePros and Cons

• Advantages

• Extremely quick way to build an optimised data-mart, no design work required

• No ETL tool required

• Excellent re-use of code (using existing metadata in the RPD to build aggregates)

• Potential to deliver massive performance improvements e.g. 100x faster

• Disadvantages

• System generated table names / columns are ugly

• Full load only – no incremental mechanism provided out-of-the-box

• Quite hard to debug if it fails

• If it fails it requires manual work to clean up the RPD and database tables

• Lots of moving parts – tables and RPD metadata dropped and recreated each time


Data-Mart AutomationOption 3: BI Server “Populate” Queries


BI Server “Populate” QueriesOverview

• This final method of presenting data-marts to end users overcomes most of the issues outlined with previous methods

• It offers the following benefits:

• Incremental loading of aggregates via the BI Server

• Provides the ability to perform data cleansing

• Handles complex data models and logic

• Re-uses code generated by Aggregate Persistence (optional)

• All the performance gains of Aggregate Persistence

• More reliable – aggregates and metadata never get dropped

• No need to have “system generated” aggregate table names

• Supports both Oracle Database, TimesTen, SQL Server etc


BI Server “Populate” QueriesQuestion

• What is the difference between a Full Load and an Incremental Load?

1. A “Staging” table

2. A “WHERE” clause


BI Server “Populate” QueriesCustomer Example

• 6 different operational systems loaded into a central BI Data-Mart

• Extract/load driven by BI Server “populate” queries

Oracle

Business

Intelligence

“Global” BI Dashboards“In-Memory” optimisedBI Data-Mart

Sales UK Sales US

Finance UK Finance US

DesignsPayments


BI Server “Populate” QueriesExample RPD

A layer of “views” in the Physical Layer are used to extract data from source and populate a BI Data-Mart


BI Server “Populate” QueriesData Refresh with Aggregate Persistence

• The current method of reloading Exalytics using Aggregate Persistence:


BI Server “Populate” Queries#1 Did you know?

• Using the NQCMD command-line you can initiate various types of SQL against the BI Server:

• Logical Layer SQL

• Business Model Layer SQL

• Physical Layer SQL

• Database SQL (including PL/SQL blocks)

• Example Physical SQL to drop a table:

EXECUTE PHYSICAL CONNECTION POOL "BI Data-Mart"."BI Data-Mart Connection Pool"

BEGIN

EXECUTE IMMEDIATE 'DROP TABLE AGG_STG.AG_FACT_ORDERS_LN';

END;



• When you run a “Create Aggregates” command, behind the scenes the BI Server runs a series of “populate” commands to load each dimension/fact table

• The code can be obtained from the BI Server log file (nqquery.log)

populate "ag_Fact_Orders_Lin"

mode ( create table connection pool "Aggregates"."Aggregates Stage Connection Pool")

as select_business_model

"Sales DW"."Dim - Organization"."Organization Id" as "Organizati00001A59",

"Sales DW"."Dim - Time"."Month" as "Month00001C1C",

"Sales DW"."Fact - Orders Lines"."# Order Lines" as "Z_Order_Li00006355",

"Sales DW"."Fact - Orders Lines"."# Orders" as "Z_Orders00006356",

"Sales DW"."Fact - Orders Lines"."Quantity" as "Quantity00006357",

"Sales DW"."Fact - Orders Lines"."Order Amount" as "Order_Amou00006358",

"Sales DW"."Fact - Orders Lines"."Return Amount" as "Return_Amo00006359",

"Sales DW"."Fact - Orders Lines"."Cost of Sales" as "Cost_of_Sa0000635A",

"Sales DW"."Fact - Orders Lines"."Avg Unit Price" as "Avg_Unit_P0000635B",

"Sales DW"."Fact - Orders Lines"."Order Amount (Primary Markets)" as "Order_Amou0000635C",

"Sales DW"."Fact - Orders Lines"."Order Amount (Secondary Markets)" as "Order_Amou0000635D"

from "Sales DW"

Table Name

“Create” to create target table and then insert rows or

“Append” to just insert records (if table already exists)

The connection pool containing the target aggregate table



• The “populate” command initiates a “Business Model” SQL query

• It is possible to add a “WHERE” clause to filter the rows extracted from the source system

populate "ag_Fact_Orders_Lin"

mode ( create table connection pool "Aggregates"."Aggregates Stage Connection Pool")


"Sales DW"."Dim - Organization"."Organization Id" as "Organizati00001A59",

"Sales DW"."Dim - Time"."Month" as "Month00001C1C",






"Sales DW"."Fact - Orders Lines"."Cost of Sales" as "Cost_of_Sa0000635A",

"Sales DW"."Fact - Orders Lines"."Avg Unit Price" as "Avg_Unit_P0000635B",

"Sales DW"."Fact - Orders Lines"."Order Amount (Primary Markets)" as "Order_Amou0000635C",

"Sales DW"."Fact - Orders Lines"."Order Amount (Secondary Markets)" as "Order_Amou0000635D"

from "Sales DW"

where "Sales DW"."Dim - Time"."Month" = VALUEOF(CURR_MONTH)

1) select2) select_business_model3) select_physical

Example WHERE clause filter referencing a Repository Variable


BI Server “Populate” QueriesStep 1

• We need to create an empty “Staging” aggregate table

• It should be an identical copy of the target aggregate table

Oracle

Business Intelligence

11g

Target Aggregates

StagingAggregates

1


BI Server “Populate” QueriesStep 1 - Code


BEGIN

EXECUTE IMMEDIATE 'DROP TABLE ag_Fact_Orders_Lin_STG';

EXECUTE IMMEDIATE 'CREATE TABLE ag_Fact_Orders_Lin_STG

AS

SELECT * FROM ag_Fact_Orders_Lin

WHERE 1=2';

END;



• Obtain the relevant “populate” command from the BI Server query log file

• Modify it to “append” into the Staging aggregate table instead of the Target aggregate

• Add a WHERE clause filter to limit the rows extracted e.g. WHERE Quarter = CURR_QTR

Oracle


11g

Target Aggregates

StagingAggregates

2



SET VARIABLE DISABLE_CACHE_HIT=1, DISABLE_CACHE_SEED=1, DISABLE_SUMMARY_STATS_LOGGING=1,

INACTIVE_SCHEMAS='"BI Data-Mart".."BIDATAMART"':

populate "ag_Fact_Orders_Lin_STG"

mode (append table connection pool "BI Data-Mart"."BI Data-Mart Connection Pool")







"Sales DW"."Dim - Organization"."Business Group" as "Business_G0000746B",

"Sales DW"."Dim - Time"."Month" as "Month00007479"

from "Sales DW"

WHERE "Sales DW"."Dim - Time"."Month" = VALUEOF(CURR_MONTH);



• Incrementally update the Target Aggregate table from the Staging Aggregate. Performed in 2 stages:

1. Delete any records from the Target aggregate which need to be reloaded from the Staging aggregate (e.g. all records for the current quarter)

2. Insert all records from the Staging aggregate

• Both statements performed as part of a single database transaction

Oracle


11g

Target Aggregates

StagingAggregates

3




BEGIN

DELETE FROM ag_Fact_Orders_Lin

WHERE Month00007479 IN (SELECT Month00007479 FROM ag_Fact_Orders_Lin_STG);

INSERT INTO ag_Fact_Orders_Lin

SELECT *

FROM ag_Fact_Orders_Lin_STG;

COMMIT;

DBMS_STATS.GATHER_TABLE_STATS(ownname => 'BIDATAMART', tabname =>

'AG_FACT_ORDERS_LIN' , estimate_percent => 30 );

END;

Delete all records from the Target aggregate where the Month exists in

the Staging aggregate

Both commands done in a single database transaction

Once loaded, gather statistics


BI Server “Populate” QueriesThat’s it!

• Put all the commands into a single script (using Notepad) and then run it against the BI Server using nqcmd:

nqcmd –d AnalyticsWeb –u weblogic –p welcome1 –s inc_load.txt


• This script is for one table:

• Need to schedule & orchestrate:

• Data Warehouse Administration Console (DAC)

• Peak ETA

BI Server “Populate” QueriesRepeat for each table


http://www.peakindicators.com/index.php/peak-eta

Peak ETAIncrementally Load Aggregates in Parallel via the BI Server

Scheduling

Handling dependencies

Parallel running

Error handling

Surrogate keys

Alerting

Automatic code generation

http://www.peakindicators.com/index.php/peak-eta

Helping Your Business Intelligence Journey

oracle business intelligence 11g data-mart automation indicators... · “data-mart automation”...

Documents