Performance Instrumentationfor PL/SQLWhen, Why, How

Karen Morton

Your speaker…

Karen MortonSr. Principal Database EngineerEducator, DBA, developer, consultant, researcher, author, speaker, …

Come see me…karenmorton.blogspot.comAn Oracle user group near you

Why guess?When you can know.

Method R.

“R” stands for response time.

Target theright task

11

Collect itsR

Details

22

Forecast, act33

Stop when cost of additional repairs exceeds

cost of the problem

Go to Step 144

The hardest part isgetting the “right”

information

Collecting properly scoped,Un-aggregated profile data

If you can’t measure it, you can’t manage it.

―David Garvin

There are two types of performance problems in this

world...

1Response time

problems.

2Inefficiencies that aren’t response time problems.

Yet.

Therefore...

You must be able to attack response time problems for

specific tasks that the business cares about.

Example...

Posting hurts.Fix it.

Why should you care?

You need to be able to attack inefficiencies that aren’t yet noticeable as user response

time problems.

Example...

Posting takes 3 hours.

It should take 2,but no user really

cares.

Why should *you* care?

Because

costs you money

waste

TCO $$

Waste

Labor $$

Hardware $$

Software $$

And waste...

...makesother work

go slower…

...even your fast stuff

recap...

You must be able to attack1. Response time problems2. Efficiency problems

Here’s whereinstrumentation

comes in

Why is this program slow?

SQL> exec p

PL/SQL procedure successfully completed.

SQL>

What is “slow”?

SQL> set timing onSQL> exec p

PL/SQL procedure successfully completed.

Elapsed 00:02:09:98SQL>

So we open up our code.

create or replace procedure p asbeginq;r;s;

end;/

Let’s find out.

create or replace procedure p ast0 number;t1 number;t2 number;t3 number;

begint0 := dbms_utility.get_time;q;t1 := dbms_utility.get_time ;dbms_output.put_line ('Procedure q: ' || to_char((t1 - t0)/100));r;t2 := dbms_utility.get_time ;dbms_output.put_line ('Procedure r: ' || to_char((t2 - t1)/100));s;t3 := dbms_utility.get_time ;dbms_output.put_line ('Procedure s: ' || to_char((t3 - t2)/100));dbms_output.put_line ('Total R : ' || to_char((t3 - t0)/100));

end;/

So now we know.

SQL> set timing onSQL> exec pProcedure q: 1Procedure r: 114Procedure s: 15Total R : 130

Elapsed 00:02:09:99SQL>

What was the co$t?

Response time increased by .01 seconds (From 129.98 to 129.99)

Code additions

4 variable declarations8 lines of code

These new lines of code are called

performance instrumentation.

With this data,we know where

to focus our attention.

Where would you startif you didn’t have this data?

What would happen ifyou started work on

procedure Q or S?

Procedure q: 1Procedure r: 114Procedure s: 15Total R : 130

Why guess?

When you can know.

Profiling is arequirement of good

development hygiene

It makes your appsfaster and cheaper

Without profiling…

Can’t tell where time goes “Tune” everything you

can Learn about everything

possible “Tuning” continues even

when improvement is impossible

With profiling…

Know where code spends time

Optimize only relevant code path

Learn first in high-impact areas

“Tuning” ceases when objective is achieved

Performanceinstrumentation

in real life

I just started a new joband need to drive 15 miles from

home to the office each day.

I decide to leave home at7:40am expecting to be

there by 8:00am.

Day 1

Depart7:40am

Arrive8:20am

Total Drive Time

40 minutes

Argh!It’s not good to belate for your first

day on a new job!

What am I going to do?

First, let me find out how

my drive time was spent.

Category Miles Elapsed Time (minutes)Neighborhood 1 3

Burke Centre Pkwy 2.5 8

Fairfax County Pkwy 6 12

I-66 2.5 8

Hwy 28 2 5

Westfields Blvd to Office 1.5 4

Total 15.5 40 minutes

Try another route?

Buy a faster car?

Hire private air transport?

Conclusion

Leave earlier to allowfor traffic fluctuations

Why guess?

When you can know.

What is instrumentationand why is it important?

Measures the attributes of a system

Creates monitoringmechanisms

Tracks and measuresperformance

What are the objectivesof proper instrumentation?

Easy toactivate

Capture timings ofbusiness-critical

tasks

Lightweight

Leveragesbuilt-in functionality

What are the benefitsof proper instrumentation?

Appropriatelyscoped traceinformation

More meaningfulinformation

This?

Or this?

This?

Or this?

Why did this take so long?What would happen if…?

Is this thing efficient?Am I done yet?

Easily answer questions like:

Provide final wordon disputes

DB

As

Dev

elop

ers

SAN

Adm

ins

Net

Adm

ins

OS

Adm

ins

Man

ager

s

Use

rs

How do Iinstrumentmy code?

Start withsupplied packages

DBMS_SESSION

set_identifier

identifies currentsession to maketracing easier

Lets instance know which end users

are on which connection

DBMS_SESSION.set_identifier(client_id =>'jqpublic~ora11r1')

LOGON Trigger

SYS_CONTEXT('USERENV', 'CLIENT_IDENTIFIER')SYS_CONTEXT('USERENV', 'SERVICE_NAME')SYS_CONTEXT('USERENV', 'IP_ADDRESS')SYS_CONTEXT('USERENV', 'TERMINAL')SYS_CONTEXT('USERENV', 'OS_USER')SYS_CONTEXT('USERENV', 'SESSIONID')

SELECT client_identifier,… FROM V$SESSION

…plus almost 40 other views

DBMS_APPLICATION_INFO

set_module

register business tasksand related logical

units of work

read_module

determine currentbusiness task and

logical unit of work

set_session_longops

monitor progressof long-running

operations

DBMS_APPLICATION_INFO.set_module(module_name=>'Order Entry',action_name=>'Get Order Items')

SELECT module, action, … FROM V$SESSION– or – FROM V$SESSION_LONGOPS

…plus almost 40 other views

DBMS_MONITOR

client_id_trace_enabledatabase_trace_enable

serv_mod_act_trace_enablesession_trace_enable

start extendedSQL tracing

DBMS_MONITOR.serv_mod_act_trace_enable(service_name=>'ora11r1',module_name=>'Order Entry',action_name=>'Get Order Items')

Trace file will contain

*** ACTION NAME:(Get Order Items) 2009-08-02 01:48:56.573

*** MODULE NAME:(Order Entry) 2009-08-02 01:48:56.573

SELECT * FROM DBA_ENABLED_TRACES

TRACE_TYPE SERVICE_MODULE_ACTIONPRIMARY_ID ora11r1QUALIFIER_ID1 Order EntryQUALIFIER_ID2 Get Order ItemsWAITS TRUEBINDS TRUEPLAN_STATS FIRST_EXECINSTANCE_NAME <null>

client_id_stat_enableserv_mod_act_stat_enable

gather performancemetrics

DBMS_MONITOR.client_id_stat_enable(client_id =>'jqpublic~ora11r1')

CLIENT_IDENTIFIER STAT_NAME TIME_SECS----------------- --------------------------- ----------jqpublic~ora11r1 DB CPU .009jqpublic~ora11r1 DB time .009jqpublic~ora11r1 sql execute elapsed time .007jqpublic~ora11r1 parse time elapsed .001. . .

SELECT * FROM V$CLIENT_STATS

SELECT * FROM V$SERV_MOD_ACT_STATS

when usingserv_mod_act_stat_enable

Where should Iplace instrumentation?

Set module & actionbelow the

BEGIN statement

Update module & actionfor any new

business tasks

Place above allEND and RETURN

statements(set to null)

Place inside allEXCEPTION

handlers(set to null)

Avoid placingcalls inside

LOOPs

Example...

PROCEDURE get_emp_simple_instr IS fnlist_stack fnlist_tab; lnlist_stack lnlist_tab; BEGIN DBMS_APPLICATION_INFO.set_module(module_name => 'Human

Resources’,action_name => 'Get Employees'); SELECT first_name, last_name BULK COLLECT INTO fnlist_stack, lnlist_stack FROM employees; DBMS_APPLICATION_INFO.set_module(NULL, NULL);

EXCEPTION WHEN OTHERS THEN DBMS_APPLICATION_INFO.set_module(NULL, NULL); DBMS_OUTPUT.PUT_LINE('get_emp_simple_instr => ERROR');

END get_emp_simple_instr;

08:00:00 08:05:00

08:00:00 08:02:04get_emp_simple_instr

08:00:00MODULE = ‘Human resources’ACTION= ‘get employees’

08:02:04MODULE = nullACTION = null

PROCEDURE get_emp_jobs_instr_flawed IS jtlist_stack jtlist_tab; lnlist_stack lnlist_tab;

BEGIN DBMS_APPLICATION_INFO.set_module(module_name => 'Human

Resources', action_name => 'Get Employees and Jobs'); get_emp_simple_instr; SELECT last_name, job_title BULK COLLECT INTO lnlist_stack, jtlist_stack FROM employees e, jobs j WHERE e.job_id = j.job_id; DBMS_APPLICATION_INFO.set_module(NULL, NULL);

EXCEPTIONWHEN OTHERS THEN DBMS_APPLICATION_INFO.set_module(NULL, NULL); DBMS_OUTPUT.PUT_LINE('get_emp_jobs_instr_flawed => ERROR');

END get_emp_jobs_instr_bad;

08:10:00MODULE = ‘Human resources’ACTION = ‘get employees and jobs’

08:12:04MODULE = nullACTION = null

08:10:00 08:15:00

get_emp_jobs_instr_flawed

08:10:00 08:12:0408:12:04 08:13:32get_emp_simple_instr

08:12:04Module/Action set to NULL

08:10:01MODULE = ‘Human resources’ACTION = ‘get employees’

08:13:32<end>

Problems can occurwhen one similarly

instrumented procedurecalls another

Remember to store module & action

in variablesand “reset” as needed

throughout code

PROCEDURE get_emp_instr_good IS fnlist_stack fnlist_tab; lnlist_stack lnlist_tab; preModuleName VARCHAR2(48) := NULL; preActionName VARCHAR2(32) := NULL;

BEGIN DBMS_APPLICATION_INFO.read_module(

module_name => preModuleName, action_name => preActionName); DBMS_APPLICATION_INFO.set_module(

module_name => 'Human Resources',action_name => 'Get Employees');

SELECT first_name, last_name BULK COLLECT INTO fnlist_stack, lnlist_stack FROM employees; DBMS_APPLICATION_INFO.set_module(

module_name => preModuleName, action_name => preActionName);EXCEPTION WHEN OTHERS THEN

DBMS_OUTPUT.PUT_LINE('HR_Package.get_emp_instr_good ERROR'); DBMS_APPLICATION_INFO.set_module(

module_name => preModuleName, action_name => preActionName);END get_emp_instr_good;

InstrumentationLibrary for Oracle

( ILO )

Used to simplify theprocess of instrumenting

your code

3 packagesILO_TASK

ILO_TIMERILO_SYSUTIL

RequiresOracle version

9.2.0.1or above

Makesinstrumenting

codeeasy

Encapsulates & simplifiesthe use of

dbms_application_infoand

dbms_session

Download from

SourceForge.net

Open Open Source!!

Can be modified tomeet your needs

Place 1st callbelow the

BEGIN statement

ILO_TASK.begin_task(module_name=>'Human Resources',action_name=>'Get Employees')

Change for any newbusiness tasks

ILO_TASK.begin_task(module_name=>'Human Resources',action_name=>'Get Jobs')

Place end callsabove all

END and RETURNstatements

and inside allEXCEPTION

handlers

ILO_TASK.end_task

That’sit!

PROCEDURE get_emp_instr_better ISfnlist_stack fnlist_tab;lnlist_stack lnlist_tab;

BEGINILO_TASK.BEGIN_TASK(module_name => 'Human Resources' ,action_name => 'Get Employees'); SELECT first_name, last_name BULK COLLECT INTO fnlist_stack, lnlist_stack FROM employees; ILO_TASK.END_TASK;

EXCEPTIONWHEN OTHERS THEN

DBMS_OUTPUT.PUT_LINE('get_emp_instr_better ERROR');ILO_TASK.END_TASK;

END get_emp_instr_better;

Turning on traceusing ILO

Activating trace is arun-time decision

The developer addsBEGIN_TASK & END_TASK

calls to the code

The DBA startstracing with

DBMS_MONITOR

The developer can starttracing with ilo_task

SET_MARK_ALL_TASKS_INTERESTING(TRUE, TRUE)

Casestudy

The Problem

Business critical tasktaking 40 seconds

MAX toleratedexecution time

15 seconds

Weeks spent searching for root cause and fix

Task was a singlePL/SQL packagewith 2,300 lines

of code

Code was executedover 100 times

per hour

Running onmulti-node RAC

Trace collection notvery helpful due toimproper scoping

Needed instrumentationto provide

proper scoping

Code contained36 business tasks

all marked within-line comments

In-line commentsused to guide

instrumentation with ILO

Spent40 minutes

to instrument36 tasks

Instrumentedcode placed

into production

Tracing activatedusing dbms_monitorfor the instrumented

module & actions

Problemcode

was executed

Tracingdeactivated

Trace fileretrieved

and profiled

Majority of time spent waiting forgc cr multi block request

Further drilldown showed one insert and one delete

to be largest “R” contributors

Reviewed the insert statementwhich was actually inserting

via a SELECT

1) Noted excessive LIO2) Plan showed use of full table scan

The Fix

Added index onts_row_seq

Problem code wasexecuted andtraced again

New response time10.118 seconds

Total time tofind and fix

the problem:

less than 1 hour

Why guess?

When you can know.

Summary

When should you consider performance

instrumentation?

Any time there is codewhose performance

you might evercare about.

Why shouldperformance

instrumentationmatter?

Because when it isn’t in place,

managing performanceis too complexand expensive.

How shouldperformance

instrumentationbe done?

By adding a fewextra lines of code

to registereach business task.

Performancedoesn’t need

to be hard

Let your softwarehelp you!

It’s a mindless taskyour computer is willing

to do for you…

…if you justask it to.

Why guess?

When you can know.

Thank you!

Questions & Answers