obiee performance

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

No Silver Bullets : OBIEE Performance in the Real World Robin Moffatt, Principal Consultant May 2014



The Requirement



The Problem

Lots of moving parts and complexity Overall solution crosses disciplines and job roles OBIEE developer Database developer DBA Server Admin Network SAN LDAP etc



The Additional Problem



The Only Answer

Evidence-based DESIGN and DIAGNOSTICS



Performance Beyond the Best Practices

Tear down the reliance on Best Practice, but with a viable, better, alternative instead.



Two-pronged approach

Do It Right, First Time What Makes A Performant OBIEE System? !

If Its Not Done Right, Know How To Figure Out What Is Broke Practical elements of diagnosing the causes of performance issues Methodical analysis - nose to tail



Resp

onse

time

Number of concurrent users

Performance: Response time is slow for one user Capacity: Response time degrades as user concurrency increases

Adding Capacity alone will not fix your performance problems Performance improvements can alleviate Capacity problems

Is your problem Performance or Capacity?



Nose to Tail Diagnostics

Request

Response

BI Server

BI Server

Presentation Services

Managed Server

Managed Server

JavaHost

Presentation Services

Web Browser

Web Browser

BI Plug-in

BI Plug-in

DWH

[ ... ]

[ ... ]

DWH

[ ... ]

[ ... ]

User WebLogic Server OBIEE system components Data Source(s)

Netw

ork

Netw

ork

Not all components listed, eg cluster controller



Time Profile

To understand why is it slow, we first must understand where is it slow !

Approach championed by Cary Millsap / Method-R Thinking Clearly About Performance (2010)

Performance improvement is proportional to how much a program uses the thing you improved.

Amdahls Law



Time Profile in Action

End-user reports a response time of 40 seconds. How can we make it faster?

Where did the time get spent?

Response

BI ServerManaged Server

Presentation ServicesWeb Browser BI Plug-in

User WebLogic Server OBIEE system components

DB Query 1 25 seconds

DB Query 2 5 seconds

Data Processing 10 secondsPage Generation

5 seconds

DWH

Data Source(s)DWH

Tim

e, s

econ

ds

Component



Time Profile in Action

Step Action Response Time (s) %

1 Physical SQL 1 execute on DB 25.00 62.5

2 Physical SQL 2 execute on DB [5.00] 3 BI Server combines dataset from DB results 10.00 25.0

4 Presentation Services generates page 5.00 12.5

Total 40.00 100.0

DB Query 1

DB Query 2

Data Processing

Page Generation

Time profile shows clearly : 1.Improve performance of Query 1 2.Push work into single query



Deep Dive into BI Server Time from nqquery.log

Elapsed

Response

Physical

Physical Query x

Physical Query 1 BI Server processing (federation, calculation, aggregation)

Send to client, wait for acknowledgement !Eg. Presentation Services generating pivot tables, table scrolling/paging, etc.

Com

pilat

ion

DB C

onne

ct

Logical Query Summary Stats: [...] Compilation time c (seconds)

Physical query response time p1 (seconds)

Physical query response time px (seconds)

Logical Query Summary Stats: Elapsed time e

Logical Query Summary Stats: [...] Response time r

Physical Query Summary Stats: [...] DB-connect time d (seconds) r - d - c - max(p) e - r



nqquery.log or Usage Tracking?

Step nqquery.log source Usage TrackingLogical Query Compilation c Logical Query Summary Stats: [...] Compilation time S_NQ_ACCT.COMPILE_TIME_SECDatabase connection d Physical Query Summary Stats: [...] DB-connect time n/aPhysical Query 1 response p Physical query response time S_NQ_DB_ACCT.TIME_SECPhysical Query 2 response p Physical query response time S_NQ_DB_ACCT.TIME_SECPhysical Query n response p Physical query response time S_NQ_DB_ACCT.TIME_SECLogical Query Response r Logical Query Summary Stats: [...] Response time S_NQ_ACCT.TOTAL_TIME_SECLogical Query Elapsed e Logical Query Summary Stats: Elapsed time n/aBI Server processing (federation, calculation, aggregation)

r-d-c-max(p) [derived] [derived]

Send to client, wait for acknowledgement (eg sawserver generating pivot tables etc)

e-r [derived] n/a



Usage Trackings blind spot

Usage Tracking is good, but it omits the Total Elapsed Time !!!!!!!

This can hide problems:

Elapsed

ResponseSend to client, wait for acknowledgement

nqquery.log: Logical Query Summary Stats: [...] Response time r Usage Tracking: S_NQ_ACCT.TOTAL_TIME_SEC

nqquery.log: Logical Query Summary Stats: Elapsed time e Usage Tracking: [Not Available]

uh oh!



Response

Iterative Approach

Be sure to account for all time, end-to-end (nose to tail) Expand time profile to focus on where the time is going in particular In reality, time profile based on BI Server alone will help in a lot of cases

Requestnqquery.logsawlog.log DB Explain plans, V$SQL, etcBrowser tools WLS logs

BI ServerManaged Server


User WebLogic Server OBIEE system components

DWH

Data Source(s)DWH

Netw

ork

Netw

ork



Iterative Approach

Most components have extended logging levels Presentation Services WebLogic Server Browser Tools



Digging Deeper with DMS Metrics

OBIEE supports Dynamic Monitoring System (DMS) !

Hundreds of low-level metrics !

Ranging from the obvious (BI Server Total Sessions) to the less obvious (Peak_DXEParameterizedImpl_Count) !

Once we have found WHERE the time has gone, DMS metrics are one way to help us find out WHY



Accessing DMS Metrics

DMS Spy opmn EM obi-metrics-agent + graphite + grafana

http://ritt.md/oma-intro



DMS Case Study

Query is running slowly, but its not clear why

CPU is only at 50% utilisation Only a handful of rows returned from the DB

What is the BI Server up to for this time?

Step nqquery.log source Time %Logical Query Elapsed e Logical Query Summary Stats: Elapsed time 14Logical Query Response r Logical Query Summary Stats: [...] Response time 13Logical Query Compilation c Logical Query Summary Stats: [...] Compilation time 0 0%Database connection d Physical Query Summary Stats: [...] DB-connect time 0 0%Physical Query 1 response p1 Physical query response time 4 29%BI Server processing r-d-c-max(p) [derived] 9 64%Send to client, wait for cursor close e-r [derived] 1 7%



DMS Provides the Proof

The Connection Pool has a fixed Capacity Connections will start to Queue once Capacity is reached

Physical connections limit is reached for connection pool "Star_orcl_Connection Pool 01". Start delaying the queries



Performance Diagnostics Approach

Capacity Examine capacity metrics (OS, DMS) over time Correlate with reported problems

Performance Build a time profile Dont know where to start? Try Usage Tracking or nqquery.log

-Long running queries -Logical queries returning lots of data to the client -Logical queries returning lots of data from the DB but small % to the client -Logical queries requiring lots of physical queries



So What Does Good Look Like?



This Is Not Best Practice

There is no one right way : It Depends c.f. ExtremeBI

The only best practice you should be using all the time is Use Your Brain. Steven Robbins / Tom Kyte



The Bucket List of Performance Gains

Dont Do It

Do It Only Once

Do It Less Often

Do It More Efficiently

(h/t Greg Rahn)



Database Pushdown (Do It More Efficiently)

Reduce the amount of work/data handled further up the stack Federation across sources can be a challenge c.f. BI Server caching

BI Server

Managed Server


DWH

[ ... ]

User WebLogic Server OBIEE system components Data Source(s)

Netw

ork

Netw

ork



Do You Need Database Pushdown?

BI Servers temp directory shows lots of work being done by BI Server Long time in BI Server processing phase Usage Tracking / Query logs : How much data the BI Server pulls back from the database as a percentage of rows returned to the user (low % is bad). How many database queries one report triggers - higher is generally less efficient

Step nqquery.log sourceLogical Query Elapsed e Logical Query Summary Stats: Elapsed timeLogical Query Response r Logical Query Summary Stats: [...] Response timeLogical Query Compilation c Logical Query Summary Stats: [...] Compilation timeDatabase connection d Physical Query Summary Stats: [...] DB-connect timePhysical Query 1 response p1 Physical query response timeBI Server processing r-d-c-max(p) [derived]Send to client, wait for cursor close e-r [derived]



Feeding the Excel Monster (Do It More Efficiently / Dont Do it)

Whats being done with the data once its in Excel? Could it be done in OBIEE instead? !

Alternatives to Export from Analysis/Dashboard: Dump direct from the database Use Logical SQL against BI Servers ODBC/JDBC interface directly Oracle have specific recommendations (DocID 1558070.1 p.13) -Favour CSV export over Excel -Favour BI Publisher export over OBIEE Analysis Export



Dont Do It / Do It Less Often

Filter Early Table Prompt != Filter

Report by Exception !

Make sure Dashboards have default/mandatory prompts(http://ritt.md/obi-prompts)

Yes, please do.

Be smart about Dashboard and Analysis design - dont cram everything into one page Less clutter : Better user experience & Better performance



Not forgetting

Do It More Efficiently Aggregation

Do It Less Often BI Server caching !

Database optimisation (partitioning, indexing, parallelism, statistics, etc)

Balanced Hardware configuration



OBIEE Performance in the Real World

Evidence-based DESIGN and DIAGNOSTICS !

Do It Right, First Time ! If Its Not Done Right, Know How To Figure Out What Is Broke Methodical analysis - nose to tail



#EOF email: [email protected] web: http://www.rittmanmead.com/author/robin-moffatt/ twitter: @rmoff



References & Further Reading

Cary Millsap - Thinking Clearly About Performance http://queue.acm.org/detail.cfm?id=1854041

Blog series: OBIEE and Performance http://ritt.md/obiee-performance

obi-metrics-agent http://ritt.md/oma-intro

Greg Rahn - The Core Performance Fundamentals Of Oracle Data Warehousing Balanced Hardware Configuration http://wp.me/p3cJT-by

Oracle documentation - Oracle Database 2 Day + Data Warehousing Guide - Balanced Hardware Configuration E25555-03

obiee performance

Documents