no silver bullets - obiee performance in the real world (rittman mead bi forum 2014)

33
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India) E : [email protected] W : www.rittmanmead.com No Silver Bullets : OBIEE Performance in the Real World Robin Moffatt, Principal Consultant May 2014

Upload: robin-moffatt

Post on 27-Jan-2015

110 views

Category:

Data & Analytics


3 download

DESCRIPTION

Are you involved in the design and development of OBIEE systems and want to know the best way to go about ensuring good performance? Maybe you've an existing OBIEE system with performance "challenges" that you need to diagnose? This presentation looks at the practical elements of diagnosing the causes of performance issues in OBIEE, and discusses good practices to observe when developing new systems.

TRANSCRIPT

Page 1: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

No Silver Bullets : OBIEE Performance in the Real World Robin Moffatt, Principal Consultant May 2014

Page 2: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

OBIEE Performance in the Real World

•Do It Right, First Time ‣What Makes A Performant OBIEE System? !

• If It’s Not Done Right, Know How To Figure Out What Is Broke ‣Practical elements of diagnosing the causes of performance issues ‣Methodical analysis - “nose to tail”

Page 3: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

The Requirement

Page 4: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

The Problem

•Lots of moving parts and complexity •Overall solution crosses disciplines and job roles ‣OBIEE developer ‣Database developer ‣DBA ‣Server Admin ‣Network ‣SAN ‣LDAP ‣etc

Page 5: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

The Additional Problem

Page 6: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Performance Beyond the “Best Practices”

•Tear down the reliance on “Best Practice”, but with a viable, better, alternative instead.

Page 7: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

The Only Answer

EVIDENCE-BASED DESIGN AND DIAGNOSTICS

Page 8: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Time Profile

•To understand why is it slow, we first must understand where is it slow !

•Approach championed by Cary Millsap / Method-R • “Thinking Clearly About Performance” (2010)

Performance improvement is proportional to how much a program uses the thing you improved.

— Amdahl’s Law

Page 9: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

“Nose to Tail” Diagnostics

Request

Response

BI Server

BI Server

Presentation Services

Managed Server

Managed Server

JavaHost

Presentation Services

Web Browser

Web Browser

BI Plug-in

BI Plug-in

DWH

[ ... ]

[ ... ]

DWH

[ ... ]

[ ... ]

User WebLogic Server OBIEE system components Data Source(s)

Net

wor

k

Net

wor

k

Not all components listed, eg cluster controller

Page 10: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Time Profile in Action

•End-user reports a response time of 40 seconds. •How can we make it faster?

‣Where did the time get spent?

Response

BI ServerManaged Server

Presentation ServicesWeb Browser BI Plug-in

User WebLogic Server OBIEE system components

DB Query 1 25 seconds

DB Query 2 5 seconds

Data Processing 10 secondsPage Generation

5 seconds

DWH

Data Source(s)

DWH

Tim

e, s

econ

ds

Component

Page 11: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Time Profile in Action

Step Action Response Time (s) %

1 Physical SQL 1 execute on DB 25.00 62.5

2 Physical SQL 2 execute on DB [5.00] —3 BI Server does work on DB results 10.00 25.0

4 Presentation Services generates page 5.00 12.5

Total 40.00 100.0

DB Query 1

DB Query 2

Data Processing

Page Generation

Time profile shows clearly : 1.Improve performance of Query 1 2.Push work into single query

Page 12: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Deep Dive into BI Server Time from nqquery.log

Elapsed

Response

Physical

Physical Query x

Physical Query 1 BI Server processing (federation, calculation, aggregation)

Send to client, wait for acknowledgement !Eg. Presentation Services generating pivot tables, table scrolling/paging, etc.

Com

pilat

ion

DB C

onne

ct

Logical Query Summary Stats: [...] Compilation time c (seconds)

Physical query response time p1 (seconds)

Physical query response time px (seconds)

Logical Query Summary Stats: Elapsed time e

Logical Query Summary Stats: [...] Response time r

Physical Query Summary Stats: [...] DB-connect time d (seconds) r - d - c - max(p) e - r

•http://ritt.md/nqquery

Page 13: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Usage Tracking’s blind spot

•Usage Tracking is good, but it omits the Total Elapsed Time !!!!!!!

•This can hide problems: !!

•Usage Tracking is great, but do be aware of this potential limitation to your visibility of where time has gone.

Elapsed

ResponseSend to client, wait for acknowledgement

nqquery.log: Logical Query Summary Stats: [...] Response time r Usage Tracking: S_NQ_ACCT.TOTAL_TIME_SEC

nqquery.log: Logical Query Summary Stats: Elapsed time e Usage Tracking: [Not Available]

uh oh!

Page 14: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Request

Response

Iterative Approach

•Be sure to account for all time, end-to-end (nose to tail) •Expand time profile to focus on where the time is going in particular • In reality, time profile based on BI Server alone will help in a lot of cases

BI ServerManaged Server

Presentation ServicesWeb Browser BI Plug-in

User WebLogic Server OBIEE system components

DWH

Data Source(s)

DWH

Net

wor

k

Net

wor

k

nqquery.logsawlog.log EM Active Reports, V$SQL, etcBrowser tools WLS logs

Page 15: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Digging Deeper with DMS Metrics

•OBIEE supports Fusion Middleware’s Dynamic Monitoring System (DMS) !

•Once we have found WHERE the time has gone, DMS metrics are one way to help us find out WHY !

•Hundreds of low-level metrics, ranging from the obvious (BI Server Total Sessions) to the less obvious (Peak_DXEParameterizedImpl_Count)

Page 16: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Accessing DMS Metrics

•DMS Spy •opmn •EM FMC •EM12c •obi-metrics-agent + graphite + grafana

http://ritt.md/oma-intro

Page 17: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Grafa-what?

OBIEE

DMS

OPMN

OS

obi-metrics-agent collectl

Graphite

carbon

whisper

graphite Grafana

•http://ritt.md/oma-intro •http://ritt.md/obiee-grafananqquery.log

extract.sh

Page 18: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

DMS Case Study

•Query is running slowly, but it’s not clear why

•CPU is only at 50% utilisation •Only a handful of rows returned from the DB

•What is the BI Server up to for this time?

Step nqquery.log source Time %Logical Query Elapsed e Logical Query Summary Stats: Elapsed time 16Logical Query Response r Logical Query Summary Stats: [...] Response time 15Logical Query Compilation c Logical Query Summary Stats: [...] Compilation time 0 0%Database connection d Physical Query Summary Stats: [...] DB-connect time 0 0%Physical Query 1 response p1 Physical query response time 4 25%BI Server processing r-d-c-max(p) [derived] 11 69%Send to client, wait for cursor close e-r [derived] 1 6%

[…][sessionid: d1d70000] [username: weblogic] -------------------- Rows 42, bytes 6914 retrieved from database query id: <<2779818>> [[[…]

Page 19: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Demo!

•The test driven by nqcmd - therefore just BI Server, no Presentation Services

export SA_NQCMD_ADVANCED=Yes nqcmd […] -q -T -o $(date +%Y-%m-%d-%H%M%S) -t 20 -ds 1 -td 30 -i 20 -w 5

Ramp up to 20 threads

Run at full-whack for 30 seconds

Launch a new thread every 20 seconds

Each thread runs a query every 5 seconds

“Open Sesame”

http://ritt.md/nqcmd-load-testing

Page 20: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Page 21: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

DMS Provides the Proof

•The Connection Pool has a fixed Capacity •Connections will start to Queue once Capacity is reached

Physical connections limit is reached for connection pool "Star_orcl_Connection Pool 01". Start delaying the queries

Page 22: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Resp

onse

tim

e

Number of concurrent users

•Performance: Response time is slow for one user •Capacity: Response time degrades as user concurrency increases

•Whilst Performance improvements can alleviate Capacity problems, adding Capacity alone will generally not fix your Performance problems

Is your problem Performance or Capacity?Ba

dG

ood

Page 23: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Performance Diagnostics Approach

•Performance ‣Build a time profile ‣Don’t know where to start? Try Usage Tracking or nqquery.log

-Long running queries -Logical queries returning lots of data to the client -Logical queries returning lots of data from the DB but small % to the client -Logical queries requiring lots of physical queries !

•Capacity ‣Examine capacity metrics (OS, DMS) over time ‣Correlate with reported problems

Queuing DMS metrics: Oracle BI DB Connection Pool/* -> Current Queued Requests Oracle BI PS Chart Engine -> Current Charts Queued Oracle BI PS Query Cache -> Current Queued Queries Oracle BI PS Thread Pools/* -> Current Jobs Queued Oracle BI Thread Pool/* -> Current Queued Requests

Page 24: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

So What Does “Good” Look Like?

Page 25: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

This Is Not “Best Practice”

•There is no one right way : It Depends •c.f.

•ExtremeBI •Exalytics

The only “best practice” you should be using all the time is “Use Your Brain”. — Steven Robbins / Tom Kyte

Page 26: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

The Bucket List of Performance Gains

★Don’t Do It

★Do It Only Once

★Do It Less Often

★Do It More Efficiently

(h/t Greg Rahn)

Page 27: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

‘Database Pushdown’ (Do It More Efficiently)

•Reduce the amount of work/data handled further up the stack •Federation across sources can be a challenge ‣c.f. BI Server caching

BI Server

Managed Server

Presentation ServicesWeb Browser BI Plug-in

DWH

[ ... ]

User WebLogic Server OBIEE system components Data Source(s)

Net

wor

k

Net

wor

k

Page 28: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Feeding the Excel Monster (Do It More Efficiently / Don’t Do it)

•What’s being done with the data once it’s in Excel? ‣Could it be done in OBIEE instead? !

•Alternatives to Export from Analysis/Dashboard: ‣Oracle have specific recommendations (DocID 1558070.1 p.13) -Favour CSV export over Excel -Favour BI Publisher export over OBIEE Analysis Export

‣Use Logical SQL against BI Server’s ODBC/JDBC interface directly ‣Dump direct from the database

Page 29: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Don’t Do It / Do It Less Often

• “Filter Early” •Table Prompt != Filter

•Report by Exception !

•Make sure Dashboards have default/mandatory prompts(http://ritt.md/obi-prompts)

Yes, please do….

•Be smart about Dashboard and Analysis design - don’t cram everything into one page •Less clutter : Better user experience & Better performance

Page 30: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Not forgetting …

•Do It More Efficiently ‣Aggregation

•Do It Less Often ‣BI Server caching !

•Database optimisation (partitioning, indexing, parallelism, statistics, etc)

•Balanced Hardware configuration

Page 31: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

OBIEE Performance in the Real World

•Do It Right, First Time !

•Methodical analysis - “nose to tail”

EVIDENCE-BASED DESIGN AND DIAGNOSTICS

Page 32: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

#EOF✴ email: [email protected] ✴ web: http://www.rittmanmead.com/author/robin-moffatt/ ✴ twitter: @rmoff

Page 33: No Silver Bullets - OBIEE Performance in the Real World (Rittman Mead BI Forum 2014)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

References & Further Reading

•Cary Millsap - “Thinking Clearly About Performance” ‣http://queue.acm.org/detail.cfm?id=1854041

•Blog series: OBIEE and Performance ‣http://ritt.md/obiee-performance

•obi-metrics-agent ‣http://ritt.md/oma-intro

•Greg Rahn - “The Core Performance Fundamentals Of Oracle Data Warehousing – Balanced Hardware Configuration” ‣http://wp.me/p3cJT-by

•Oracle documentation - “Oracle® Database 2 Day + Data Warehousing Guide - Balanced Hardware Configuration” ‣E25555-03