No more unknown members!

Smart data load validation for Hyperion Planning using ODI

Ricardo Giampaoli

Rodrigo Radtke

About the Speakers

Giampaoli, Ricardo

● Oracle Ace Associate

● Master in Business Administration

and IT management

● Founder of TeraCorp Consulting

● EPM training instructor

● Essbase/Planning/OBIEE/ODI

Certified Specialist

● Blogger @ devepm.com

Radtke, Rodrigo

● Oracle Ace Associate

● Graduated in Computer

Engineering

● Software Developer Advisor at Dell

● ODI, Oracle and Java Certified

● Blogger @ devepm.com

TeraCorp is a company specialized in products and services

focused on EPM

TeraCorp mission is to create innovate solutions that helps

people, businesses and partners to exceed their goals reaching

their full potential.

Learn more @ www.teracorp.com.br/en

About TeraCorp

Planning uses RDBMS to store its metadata

information

● One main repository is used for Planning system

● One additional repository for each Planning App

Planning uses Essbase as Database

● The Planning App repositories has the same

Metadata information than Essbase Outline

The Metadata sync is done by “Refreshing” the

metadata from Planning to Essbase

Hyperion Planning Architecture

Store all Data used in Hyperion Planning

One Essbase App for each Planning App

One Essbase cube per each Planning Plan Type● Each cube has its own outline that

matches the Planning metadata

● Each cube has its own Data Files that stores the Planning data

Hyperion Essbase Architecture

In order to load data in Essbase, its mandatory to inform a

combination of members from all dimension and a data value

● Essbase reads data sources starting at the top and proceeding from left

to right

● Essbase gets the sparse member’s combination informed and

brings/creates the “block” in memory

● Essbase tries to load this member combination data into the block

● If all members from that combination are valid, the data is successfully

loaded

Essbase Data Load

Three ways to load data:● Using Essbase Load Data

● If a member is not valid, the process is aborted and no data is loaded

● Using Essbase Load Rules● If a member is not valid, the process continues (depending of

the configuration) and all data but the erroneous ones is loaded

● Using ODI● ODI is a powerful ETL tool used to integrate all different data

sources with Essbase using Java Essbase API to load data

Essbase Data Load

ODI sends chunks of data to Essbase based on a “Commit Interval”

defined in the Knowledge Module

If a member is not valid, Essbase cancel the entire Chunk of data and

ODI API switches to “Cell mode”, which sends one individual row to

Essbase per time, until the commit interval ends, making the process

extremely slow

Invalid member combination causes an error in the data load● “2015-03-30 18:00:52,234 DEBUG [DwgCmdExecutionThread]: Error occurred in sending

record chunk…Cannot end data load. Analytic Server Error(1003014): Unknown

Member [ACT001] in Data Load, [0] Records Completed 2015-03-30 18:00:52,234

DEBUG [DwgCmdExecutionThread]: Sending data record by record to Essbase”

Essbase Data Load using ODI

In order to prevent “Cell mode” to happen, we can:

● Check the metadata before the load process to guarantee that all

members are valid

We can do this dynamically and easily using ODI Constraints

(CKM) and Planning Metadata Repository

Essbase Data Load using ODI

ODI uses Knowledge Modules (KM) to interact with different

technologies

ODI Architecture

KM Type Description

LKM Loading Loads heterogeneous data to a staging area

IKM Integration Uses a given strategy to populate the target datastore from the Staging Area

CKM Check Checks data in a datastore for errors statically or during an integration process

RKM Reverse-

engineering

Retrieves the structure of a data model from a database. It is needed only for

customized reverse-engineering.

JKM Journalizing Creates the Change Data Capture framework objects in the source staging area

SKM Data Services Deploys data services that provide access to data in datastores

ODI Constraints checks automatically if the data inside a column

is valid according to a specific condition

Constraints are used by the CKMs to log all invalid data to a E$

table

Need to validate all metadata columns before load to Essbase to

guarantee that no invalid member will be sent (preventing “Cell

mode” to trigger)

ODI Constraints (CKM)

ODI interacts with Planning/Essbase using diverse KMs

● For Planning ODI has only a Load Metadata KM

● For Essbase ODI has a load and extract KMs for both Metadata and

Data

● ODI Knowledge Modules manipulates data to Planning/Essbase using

Java API

ODI does not have a CKM for Essbase

● Data validation needs to happen in the inbound tables, before the data is

sent to Essbase

ODI Constraints (CKM) and Essbase

CKM validation is tied to the DataStore (Inbound tables)

● For each inbound Table we need to create a set of constraints to

validate the data

● This makes difficult when we have more planning applications to load

data since we will have multiple inbounds jobs

If we have only one generic inbound table that contains one

column for each planning dimension (Distinct of all dimensions

from all Applications) we can have dynamic ODI Constraints to

validate any number of Hyperion Planning applications

ODI CKM and Inbound Tables

For each dimension column in this table, an

ODI constraint will be created to guarantee that

the data is valid for that dimension● Create the table respecting the Sparse dimensions first and

dense dimension later, this way we can load a entire block

each time

● The dense members are set before the data columns to

improve performance

● If the job loads an entire year the Period members could go

into columns (Loads an entire block per time)

● APP_NAME, CUBE and INTERFACE_NAME is used for

multiple application purposes

Inbound Generic table design

In order to validate against Essbase outline, we would need to

extract all outline members first

● One ODI interface per dimension

● Can be very slow in some large dimensions

Easier if we validate against Planning Repository (RDBMS)

● All metadata information from all dimensions is inside the Planning App

repository

● A lot faster to extract the metadata since it uses SQL

● Can be used even without the need of extracting the metadata

ODI Constraints (CKM) Estrategy

Option 1:

● Inbound table is in the same Database as Planning App Repository or

using DB Link

● Uses the Planning Repository SQL inside the ODI Constraints querying the

Planning App repository on the fly

Option 2:

● Inbound table in a different Database as Planning App Repository

● Needs to export all existing metadata from Planning App Repository to a temporary

table first and then create the ODI Constraints to query this temp table

Data Validation Architecture

All Planning App metadata is stored in only one table

Planning Repository (HSP_OBJECT)

Column Name Description

OBJECT_ID Object ID for all objects in planning.

OBJECT_NAME Stores all metadata description in Planning (e.g. Alias, Members)

OBJECT_TYPE Type of the Object (e.g. Entity, Account, Attribute…)

PARENT_ID Parent ID of the object. Used for build the parent/child relationship with OBJECT_ID

GENERATION Inform which generation that object belongs.

HAS_CHILDREN Inform if the object has or not a child

All existing metadata can be retrieved by a single query on

HSP_OBJECT

If we change the query a little we can bring all members at once

Repository Query

This query will return

all Accounts Hierarchy

This query will return

all Hierarchies

Not all members in HSP_OBJECT will be loaded to all Essbase

Cubes

● The member’s Plan Type defines in which Cube it will exist

Each Essbase Cube is represented by a Plan Type in Planning

Each Planning member may belong to multiple Plan Types

Data load occurs in one cube at a time

● That’s why we must find out which Plan Types a member

belongs in order to validate the data for the correct cube

Planning Repository

The properties of the members are stored in this table

Planning Repository (HSP_MEMBER)

Column Name Description

MEMBER_ID Same as Object ID.

USED_IN

Defines in what Plan Types (Cubes) that member will belong.

In case of a member exists in more than one Plan Type, Planning Sums the value

of all plan types its belongs

Plan Types: 1 2 3 4 5

Used In: 1 2 4 8 16

This table stores the Plan Types created in the Planning App

Planning Repository (HSP_PLAN_TYPE)

Column Name Description

PLAN_TYPE ID of the Plan Type

TYPE_NAME Name of the Plan Type

For each plan type that the member belongs

Planning SUMs the Used In

● If a Member exists in Plan Type 1 and 2 Used In 1 + 2 = 3

● All Plan Types = 1 + 2 + 4 + 8 + 16 = 31

To figure out if a member belongs to a plan type we

need to verify if its “Used In” is included in the SUM

● Plan Type 5 (Used in 16) exists in 31?

● Plan Type 1 (Used in 1) exists in 30?

● Which Plan Type exists in 20?

Planning Repository

Plan Type Used In

Plan Type 1 1

Plan Type 2 2

Plan Type 3 4

Plan Type 4 8

Plan Type 5 16

MOD returns the remainder

(modulus) of argument 1 divided

by argument 2

● If MOD of Used In per actual Plan

Type * 2 is >= actual Plan Type then

it exists in the SUM of Plan Types.

● Plan Type 5 exists in 31

● Plan Type 1 not exists in 30

● Plan Type 3 and 5 exists in 20

Planning Repository

All necessary validation data will come from these three tables

● A member that belongs to two Plan Types will appear two times

● MOD technique is used to discover which Plan Type that member

belongs to and “OR” is used to combine more than one Plan Type

Repository Query

This query multiplies all

members depending of

how many Plan type it

belongs

Another way to select a

Plan Type is using

BITAND

● BITAND computes an

AND operation of the bits

Planning Repository

Plan Type Used In Binary

Plan Type 1 1 00001

Plan Type 2 2 00010

Plan Type 3 4 00100

Plan Type 4 8 01000

Plan Type 5 16 10000

Three types of ODI constraints:

● Key: Checks if there are duplicated records or NULL

values in the data

● Reference: Checks the relationship (Foreign key) between two tables.

Both tables needs to be reverse-engineered inside ODI models

● Condition: Checks for any other specific rule using custom queries

Better to use Condition:

● No additional ODI models required (just the inbound table)

● More SQL flexibility

● Allow complex logic checks

ODI Constraints (CKM) Implementation

Validates directly against Hyperion Planning repository

● One constraint per dimension

● No need of any “extra” metadata extract

Inbound table and Planning Repository schemas needs to be in

the same database or using DB Link

● ODI needs to execute a SQL that access both tables (Inbound table and

Planning repository) in the same connection

Option 1: Constraints Against Planning App

2 changes in the

query for all

constraints

2 ODI variables

that dynamically

switches between

Planning

Applications and

Plan Types

Option 1: Constraints Against Planning App

Need to extract the metadata to a temporary table every time

One constraint per dimension

Inbound table and Planning Repository schemas may exist in

different Databases

Temp table will be created on the same Inbound table schema

Option 2: Constraints Against a Temporary Table

Option 2: Constraints against a temporary table

Gets all Dimensions at

same time

1 ODI variables that

dynamically switches

between Planning

Applications

Option 2: Constraints Creation

2 changes in the

query for all

constraints

2 ODI variables

that dynamically

switches between

Planning

Applications and

Plan Types

ODI Constraints Execution

When we open ODI

Operator, each constraint

validation will be shown

in the “Control” steps

Each step will show how

may invalid members

was removed from the

inbound data

Invalid data is sent to the

E$ error table

With only one inbound generic table, we will have only one

generic E$ table

● Stores all the POV and the data that fails the validation

● ODI_Cons_name, Interface_Name, App_Name, Cube and

ODI_Sess_NO identifies what was the error, from which package that

error came from and to which Planning app it should have loaded

Generic E$

Easy to add new Planning Applications

Generic Inbound tables makes easier to create Generic

components

● Send Email

● Text Logs (export from E$ table)

● Error Handling

● Fewer ODI components to Maintain

Generic Structure Benefits

Overall Architecture

Excel

Oracle

Sql

Server

XML

Teradata

Sources Stage Area

Inbound Table

Table 1

Table 2

Table 3

Table 4

Table N

Inbound

Generic

E$ Table

E$ Inbound

Generic

Generic Components

Error

Handling

Send

Email

Planning

Cube

1

Cube

2

Cube

3

Cube

N

QUESTIONS?

Ricardo Giampaoli – TeraCorpRodrigo Radtke de Souza - Dell

Thank you!