WHITE PAPER

How to Achieve Real-Time Analytics for E-Business Suite

Karthik MadaEmtec Inc.

Dean Alomerovic Humana Inc.

ABSTRACT

Oracle E-Business Suite mandatory reporting and business analytics increasingly require real-time transactional

data access in reporting and data warehouse environments. That presents a challenge in large multi-terabyte

systems. Oracle Active Data Guard, Golden Gate and Oracle Business Intelligence provide technology for

designing adequate solutions. Careful analysis is required to assess where each component performs to its full

strength and how they all fit together. Optimal solution is selected based on functionality, scalability, administration

and maintenance, and hardware resource requirements of the entire system.

CHALLENGES WITH E-BUSINESS SUITE REPORTING AND ANALYTICS

Oracle E-Business Suite plays a central role in many companies’ business landscape. Along with providing an

efficient OLTP environment, it is an indispensable resource for intelligent business decision making. To be efficient,

managerial and executive staff needs analysis and reports that convey the latest, real-time data. Even an hour of

delay may result in losing a client, revenue, or market expansion opportunity. Getting valuable information from

multi-terabyte E-Business Suite systems with minimum latency is a challenge.

As it is frequently common, there is no single solution that is optimal for all E-Business Suite implementations,

modules and dataset sizes. Therefore, we’ll explore multiple architectures and explain system constraints where

they fit well. In some cases, increasing capacity of the primary E-Business Suite environment may be all that is

needed. From return on investment (ROI) perspective, taking advantage of idle disaster recovery environments

through Oracle Active Data Guard is getting increasingly popular. In some other cases, nothing but reducing the

size of active datasets can help us achieve desired seconds-to-minutes reporting efficiency. Above all, Oracle

GoldenGate powerful real-time replication may be a key component there. It becomes obvious that overall

architecture efficiency is dependent upon full utilization and precise alignment of each component. Optimal solution

should be selected based on functionality, scalability, resource requirements and manageability of the entire

system.

©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com

OPERATIONAL VS ANALYTICS REPORTING

Oracle E-Business Suite with all of its 200 plus modules or applications is used for a huge variety of reporting

purposes. On high level, however, we can categorize these reports as either operational or analytics. Operational

reports are used for day-to-day business processes and decision making. They are typically based on an active

data set that contains the last quarter or two of transactions. The reports contain detailed information for the

active accounts and are used by managerial and business stuff to optimally steer daily operations. For operational

reporting, the data source can be either E-Business Suite directly or for larger implementations it may come from

an Operational Data Store.

On the other hand, analytics reports are based on larger data sets that usually include one to ten or more years of

transactional records. Executive staff uses them for strategic decision making and intelligence gathering. Data is

typically stored in a data warehouse and is aggregated in a way that enables efficient analysis. Functionally both

operational and analytics reports can coexist and be rendered by a single Oracle Business Intelligence Enterprise

Edition (OBIEE) instance, for example, and both typically offer drilldown capability to individual transactions.

WHERE IS TIME GONE AND WHY ARE REPORTS STALE?

When a report is executed against an Oracle database, it returns records consistent with the report start time.

Therefore, the report execution time is a measure of data staleness. In other words, if a report runs for two hours, it

will miss the last two hours of transactional data. For large volume data sets, more frequently than not transactional

records need to be transferred and transformed in some fashion to improve report runtime. Transformation time

adds to the total report lag time obviously. For example, if data is ETL-ed into an Operational Data Store (ODS) or

Data Warehouse (DW) environment and then a report is executed, the total lag time would be equal to the sum of

the ETL time and the report run time.

©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com

Figure 1. Components of report lag time

©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com

Transformation time and report run time are designated in the Figure 1 by tT and tR respectively. An ETL that runs

twice a day and has two-hour execution time would have tT of 14 hours. For Active Data Guard, tT would include

time to apply redo stream on the standby. For GoldenGate, tT comprises time to complete change data capture

(CDC) from the transaction logs, record CDCs in trail files, and deliver and apply them on the target. To reduce the

total report lag time, the goal is to minimize the sum of tT and tR.

SOLUTION 1: REINFORCE THE PRIMARY E-BUSINESS SUITE ENVIRONMENT

As a part of high availability architecture and performance requirements, it is typical for E-Business Suite

application tier to run on multiple nodes with web, forms, and concurrent processing distributed appropriately.

In a similar manner, in database tier we normally have a RAC where multiple DB instances may be individually

configured to handle specific OLTP or reporting workload. That is done by adjusting relevant DB parameters such

as parallelism and memory pools. In addition to E-Business Suite built-in reporting capability, companies typically

have Oracle Discoverer or similar tool that is multitoned for the same reasons. These powerful environments

are frequently capable of handling both transactional and reporting tasks. That is especially true for operational

reporting when data sets are not too large. If a need arises, the capacity can be increased by expanding nodes

pool in any tier. Advantage of this approach is the transformation time tT is equal to zero and there are no separate

environments to maintain. The solution is applicable when we have sufficient bandwidth in the database storage.

The solution is scalable and administration is not more difficult as a result.

SOLUTION 2: EMPLOY ACTIVE DATA GUARD STANDBY DATABASE FOR REPORTING

Disaster recovery requirements typically mandate companies to distribute E-Business Suite nodes across multiple

datacenters. In database tier, this is normally accomplished by having a dedicated standby database as shown in

the Figure 2. A standby is an exact replica of the primary database and remains idle until a disaster or a planned

switch-over occurs. The configuration is typically managed by Oracle Data Guard. For the last several years, there

has been increasing requirement to take advantage of this idle hardware from the return on investment perspective.

That is accomplished through Oracle Active Data Guard (ADG) that allows the standby to be open as a read-only

and thus be used for reporting needs while still continuously applying redo stream coming from the primary. ADG is

fully certified with E-Business Suite with caveat that the standby has to be a physical standby and not a logical one.

According to Doc ID 1070491.1, E-Business Suite is now certified to run many concurrent requests or reports

against a read-only standby. Certain requirements need to be met including that parallel concurrent processing

should be configured with some nodes and concurrent managers set up to handle ADG reports. It should be

noticed that these newly configured ADG concurrent managers still connect to the primary database; only the

reports run on the standby. Currently only Oracle Reports programs are supported to run on ADG. For reference,

full setup is described in detail in the document above. Additionally, the same Doc ID acknowledges that Oracle

Discoverer can officially be configured to take advantage of a read-only standby and provides the necessary steps.

©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com

Figure 2: Oracle Active Data Guard configuration

In addition to processing most of reporting tasks, ADG standby can substitute for the primary as the source of

numerous ETLs. Among them, standard ETLs distributed with both OBIEE 11g and 10g run against ADG with

no issues. In Oracle 12c, ADG has been enhanced to allow DML operations on global temporary tables and to

allow access to global or session sequences. These two enhancements have been crucial for enabling even more

reporting tools to run against the standby. On the top of that, ADG additionally can be used to run fast incremental

RMAN backups and helps prevent logical corruption on both primary and standby. Additional advantages of ADG

include a very low - a sub-second - latency in most environments with fast networks between primary and standby

thus resulting in transformation time tT being close to 0. The solution is applicable for operational reporting on

smaller and medium active datasets.

SOLUTION 3: USING GOLDENGATE AND ACTIVE DATA GUARD TOGETHER FOR REPORTING

For large E-Business Suite datasets, using technologies described so far, it may not be possible to achieve the

necessary response time for either operational or analytics reporting. These multi-terabyte datasets typically have

either transformation time or report time too high. In cases like this, Oracle GoldenGate can come to rescue as it

can help with both components, tT and tR. As Golden Gate has not been widely used with E-Business Suite, a

brief explanation of its functionality is in order.

GoldenGate is real-time, log based change data capture (CDC), transformation and replication software. It is

distinguished by very flexible, non-intrusive and low resource footprint architecture.

©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com

Three main groups of golden gate processes are extract, pump and replicat. Extracts are processes that capture

data changes from the source database transaction files. The Extract records these changes in a set of trail

files on the source database host. Pump processes transfer the trail files to the target environment from where

Replicat processes apply them to the targeted database. Manager process coordinates and administer the entire

environment.

So how do we take advantage of Golden Gate capabilities for operational and analytical reporting? For operational

reporting we typically create or instantiate an Operational Data Store (ODS) in a separate database. ODS includes

a subset of E-Business Suite tables that reporting is based on. To keep ODS lean, ODS tables are structured to

include only reporting columns form the corresponding source E-Business Suite tables. Furthermore, ODS does

not include all rows from the source tables but instead is populated only with the active data set that consists of

the last quarter or two of transactional data. A regular purging of aging records is scheduled on a weekly or monthly

basis.

Once ODS instantiation is complete, GoldenGate keeps replicating new changes from source to target tables

and thus keeps them in sync. The replication is very efficient with tT in seconds typically, but it may increase

significantly depending on the source transactions length. Operational reporting on the reduced dataset is quick

with tR ranging from a few seconds to a few minutes. Similarly, for analytical reporting GoldenGate can be

Figure 3. Oracle GoldenGate processes

©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com

configured to replicate changes from source to target in real-time manner. That way, large and long running ETLs

are completely avoided.

For GoldenGate replication, supplemental logging on the source database level is mandatory. It is employed for

tables without primary key or unique index. For those tables, supplemental logging causes additional entries to be

recorded in the redo stream so that the changed records can be identified on the target. Supplemental logging may

add up to 2% percent overhead on the database operation. One way to reduce that overhead is to make sure to

replicate from E-Business Suite only the schemas and tables of interest and nothing else. More frequently than not,

a set of lookup tables from applsys and apps schemas will need to be included in the replicated set as well.

It should be noticed that E-Business Suite as a whole still has some limitations regarding GoldenGate replication.

Reason? Datatypes ANYDATA and UROWID that E-Business Suite uses are not supported for direct GoldenGate

replication as additional logic is needed to get them fully functional in the target system. Furthermore, certain

E-Business Suite tables should be excluded from GoldenGate replication based on some twenty criteria identified

by E-Business Suite support team. A full list of exclusion criteria is given in Doc ID 1112325.1, but for illustration

purposes here are some: tables listed in DBA_QUEUE_TABLES, tables with row_size > 2MB, GL_INTERCO_%

tables, etc.

On the other hand, GoldenGate is fully functional with ADG and, as a result, it can be configured to replicate

E-Business Suite changes directly from the standby redo logs. Consequently, that helps eliminate any GoldenGate

impact on the primary environment.

ACTUAL IMPLEMENTATION OF GOLDENGATE AND ADG FOR REPORTING

Business requirement was to implement an efficient and scalable solution for near real-time E-Business Suite

financial reporting for operational and analytic purposes. The database was 6 TB in size and already had a disaster

recovery standby in place that was completely idle.

The first step of the solution was to configure the disaster recovery standby with ADG. Once that was completed,

the standby was used for all financial reports that execute fast enough to meet business requirements. Some

operational reports, however, absolutely needed a reduced dataset to meet a few-minute response time. As a

result, an ODS was established using GoldenGate that replicated directly from ADG. ODS was populated with

the last four month of GL transactions. GoldenGate also performed certain level of data transformation as a part

of replication to prepare data for faster reporting. For analytics reporting, an equivalent GoldenGate replication

implementation is under development.

On the application side, an existing OBIEE analytics instance was used to host the operational reporting

dashboards and reports as well. As a result of these changes, all major reporting was moved out of the primary

environment. Operational financial reporting from the ODS met the business requirements for real time data. The

reports included balance sheet, income statement and several others. This near real-time solution also helped us

retire an aging OBIEE 10g operational reporting that was based on standard ETLs executed four times a day. The

Figure 4 shows the complete solution architecture.

©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com

CHALLENGES AND LESSONS LEARNED FROM THE SOLUTION IMPLEMENTED

GoldenGate can be deployed to capture changes from ADG standby using either its online redo logs or archive

logs. To get replication underway as early as possible, the preference is to utilize online redo logs and GoldenGate

version 12.1.2.1.1 or higher should be used to accomplish that.

During its normal operation, GoldenGate needs access to archive logs for long running transaction. As a standard

practice, archive logs should stay in place for at least eight hours. However, if your E-Business Suite occasionally

has pretty long-running transactions, it is highly recommend considering adjusting that period accordingly. Also,

GoldenGate should be installed close the database server and should have local access to online redo and archive

logs. Otherwise, some replication architecture limitations may result.

A good GoldenGate health monitoring tools should be in place to help detect any problems as early as possible.

The tools should include status and statistics monitoring and alerting capability as a minimum. Additionally, a

versatile heartbeat mechanism should be in place designed to work with the most critical processes as well as

GoldenGate as a whole.

Remaining challenges are primarily related to long running E-Business Suite transactions that occasionally cause

extensive replication lags. Lag statistics shows us how well GoldenGate processes are keeping pace with the

Figure 4. Complete solution for E-Business Suite operational and analytics reporting

©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com

amount of data that is being replicated or transformed. Specifically, some general ledger journal entries posting

jobs submit a million of records in a single transaction. The jobs typically complete within an hour on E-Business

Suite side and generate a huge amount of redo entries. As a result, the extract chokes on such a large transaction

and falls hours behind. To alleviate the problem, Oracle Support has recommended a few GoldenGate configuration

changes such as turning off bound recovery and breaking large transactions into small chunks that commit sooner.

The recommendations have helped to a large extent but the core problem still persists and emerges during the

busiest times such as period closing.

CONCLUSION

E-Business Suite frequently plays a central role in managing the company’s business activities. In addition to being

an efficient OLTP system, it is an indispensable source for efficient decision making. Both operational and analytics

reporting increasingly require access to real-time E-Business Suite transactional data. Intelligent solutions are

necessary to meet such requirements. Solutions vary among diverse E-Business Suite implementations, dataset

sizes, and hardware capabilities. Oracle GoldenGate, ADG and OBIEE provide powerful platforms for building such

solutions. Careful analysis and the technical expertise are needed in order to arrive at optimal solution in each

individual case.

The actual implemented solution followed the same principals and led to establishment of an efficient real-time GL

reporting ODS with an existing OBIEE analytics instance. Tremendous performance improvements were achieved

as a result. For example, Balance Sheet and Income Statement reports for all companies used to take more than

30 minutes, and now are completing in less than five minutes. In more typical scenarios when reports are run at an

individual company level, both reports are rendered in a few seconds and in real-time.

ABOUT EMTEC

Emtec is the right size provider of technology-empowered business solutions for world-class organizations. Our

local offices, highly-skilled associates, and global delivery capabilities ensure the accessibility and scale to align your

technology solutions with your business needs. Our collective focus is to continue to build clients for life: long-term

enterprise relationships that deliver rapid, meaningful, and lasting business value.

CONTACT

For more information or questions on How to Achieve Real-Time Analytics for E-Business Suite, please contact us

at info@emtecinc.com