northeast ohio oracle users group - high performance plsql...•oracle asktom – –q&a site,...

Copyright©2015Oracleand/oritsaffiliates.Allrightsreserved.|

High Performance PL/SQL Bulk Processing, Function Result Cache and More

1

StevenFeuersteinOracleDeveloperAdvocateforPL/SQL

Oracle Corporation

Email:[email protected]:@sfonplsql

Blog:stevenfeuersteinonplsql.blogspot.comYouTube:PracticallyPerfectPL/SQL

Copyright©2015Oracleand/oritsaffiliates.Allrightsreserved.| Page2

DeepenYourPL/SQLandSQLExpertise• Takeadvantageofourcommunitywebsites.• OracleAskTOM– https://asktom.oracle.com

–Q&Asite,OfficeHourswithdatabaseexperts,andmuchmore• OracleDevGym– https://devgym.oracle.com

–Quizzes,workoutsandclassesforanactivelearningexperience• OracleLiveSQL– https://livesql.oracle.com

–24x7accesstothelatestreleaseofOracleDatabase,plusascriptlibraryandtutorials


KeyPerformanceFeatures• BulkprocessingwithFORALLandBULKCOLLECT• FunctionResultCache• ImprovedperformanceofPL/SQLfunctionsfromSQL• NOCOPY• Automaticoptimization


What’stheproblemwiththiscode?• Wehave,onaverage,10,000employeesperdepartment.

CREATE OR REPLACE PROCEDURE upd_for_dept (dept_in IN employees.department_id%TYPE,newsal_in IN employees.salary%TYPE)

ISCURSOR emp_cur IS

SELECT employee_id,salary,hire_dateFROM employees WHERE department_id = dept_in;

BEGINFOR rec IN emp_curLOOP

adjust_compensation (rec, newsal_in);

UPDATE employee SET salary = rec.salaryWHERE employee_id = rec.employee_id;

END LOOP;END upd_for_dept;


Row-by-row=Slow-by-slow?• ManyPL/SQLblocksexecutethesameSQLstatementrepeatedly,withdifferentbindvalues.–Retrievesdataonerowatatime.–PerformssameDMLoperationforeachrowretrieved.

• TheSQLenginedoesalottooptimizeperformance,butrow-by-rowprocessingisinherentlyslow.– But,but...aren'tSQLandPL/SQLsupposedtobeverytightlyintegrated?Let'stakealook"underthecovers.


Oracle server

PL/SQL Runtime Engine SQL Engine

PL/SQL blockProcedural statement executor SQL statement

executor

FOR rec IN emp_cur LOOPUPDATE employee

SET salary = ...WHERE employee_id =

rec.employee_id;END LOOP;

Performance penalty for many “context switches”

WhyRow-by-RowGoesSlow(Relatively)


BulkProcessinginPL/SQL• Thegoalisstraightforward:reducethenumberofcontextswitchesandyouimproveperformance.

• Todothis,Oracle"bundlesup"therequestsfordata(ortochangedata)andthenpassesthemwithasinglecontextswitch.

• FORALLspeedsupnon-queryDML.–Usewithinserts,updates,deletesandmerges.–Movedatafromcollectionstotables.

• BULKCOLLECTspeedsupqueries.–Canbeusedwithallkindsofqueries:implicit,explicit,staticanddynamic.–Movedatafromtablesintocollections.


BulkprocessingwithFORALLOracle server

PL/SQL Runtime Engine SQL Engine

PL/SQL blockProcedural statement executor SQL statement

executor

FORALL indx IN list_of_emps.FIRST ..list_of_emps.LAST

UPDATE employee SET salary = ...

WHERE employee_id = list_of_emps(indx);

Fewer context switches,same SQL behavior

Update...Update...

Update...Update...

Update...Update...

Update...Update...

Update...Update...

Update...Update...


ImpactofBulkProcessinginSQLlayer• ThebulkprocessingfeaturesofPL/SQLchangethewaythePL/SQLenginecommunicateswiththeSQLlayer.

• ForbothFORALLandBULKCOLLECT,theprocessingintheSQLengineisalmostcompletelyunchanged.– Sametransactionandrollbacksegmentmanagement– SamenumberofindividualSQLstatementswillbeexecuted.

• Onlyonedifference:BEFOREandAFTERstatement-leveltriggersonlyfireonce perFORALLINSERTstatements.–NotforeachINSERTstatementpassedtotheSQLenginefromtheFORALLstatement.

statement_trigger_and_forall.sql


BULKCOLLECTformulti-rowquerying

• Retrievemultiplerowsintoacollectionwithasinglefetch(contextswitchtotheSQLengine).– Depositthemultiplerowsofdataintooneormorecollections.

• NO_DATA_FOUNDisnot raisedwhennorowsarefetched;instead,thecollectionisempty.

• The"INTO"collectionsarefilledsequentiallyfromindexvalue1.– Thereareno"gaps"between1andtheindexvaluereturnedbytheCOUNTmethod.

• Onlyinteger-indexedcollectionsmaybeused.• Noneedtoinitializeorextendnestedtablesandvarrays.DoneautomaticallybyOracle.

SELECT * BULK COLLECT INTO collection(s) FROM table;

FETCH cur BULK COLLECT INTO collection(s) [ LIMIT number_of_rows ];

EXECUTE IMMEDIATE query BULK COLLECT INTO collection(s);


BULKCOLLECTwithImplicitCursor• An"unlimited"bulkcollectoperationthatpopulatesthecollectionwithallrowsfound.WhichcanleadtoPGAmemoryerrors!

DECLARETYPE employees_aat IS TABLE OF

employees%ROWTYPE;

l_employees employees_aat;BEGIN

SELECT *BULK COLLECT INTO l_employeesFROM employees;

FOR indx IN 1 .. l_employees.COUNTLOOP

process_employee (l_employees(indx));END LOOP;

END;

bulkcoll.sqlbulkcollect.tst

Declare a nested table of records to hold the queried data.

Fetch all rows into collection sequentially, starting with 1.

Iterate through the collection contents with a loop.


LimitrowsreturnedbyBULKCOLLECTCREATE OR REPLACE PROCEDURE bulk_with_limit

(deptno_in IN dept.deptno%TYPE)IS

CURSOR emps_in_dept_cur IS SELECT * FROM empWHERE deptno = deptno_in;

TYPE emp_tt IS TABLE OF emps_in_dept_cur%ROWTYPE;emps emp_tt;

BEGIN OPEN emps_in_dept_cur;LOOP

FETCH emps_in_dept_curBULK COLLECT INTO emps LIMIT 1000;

EXIT WHEN emps.COUNT = 0;

process_emps (emps);END LOOP;CLOSE emps_in_dept_cur;

END bulk_with_limit;

Use the LIMIT clause with the INTO to manage the amount of memory used with the BULK COLLECT operation.

Definitely the preferred approach in production applications with large or varying datasets.

bulklimit.sql


DetailsonthatLIMITclause• Thelimitvaluecanbealiteraloravariable.

– Isuggestpassingthelimitasaparameter togiveyoumaximumflexibility.

• Alimitof100seemslikeagooddefaultvalue.– Settingitto500or1000doesn'tseemtomakemuchdifferenceinperformance.

• Withverylargevolumesofdataandsmallnumbersofbatchprocesses,however,alargerLIMITcouldhelp.


WhentoconverttoBULKCOLLECT• PriortoOracle10g,youshouldconvertallmultiplerowfetchcodetoBULKCOLLECTs.

• On10.1andhigher,theoptimizerwillautomaticallyoptimizecursorFORloopstorunatperformancelevelssimilartoBULKCOLLECT.

• Soleaveyourcursorforloopsinplaceifthey...– containno DMLoperations.– seemtoberunningfastenough.

• ExplicitBULKCOLLECTsmayrunalittlefasterthancursorforloopsoptimizedtoreturn100rowswitheachfetch.

10g_optimize_cfl.sql


BULKCOLLECTConclusions• BULKCOLLECTimprovesperformanceofqueriesthatretrievemorethanonerow.

• UsetheLIMITclausetoavoidexcessivePGAmemoryconsumption.• Leaveittotheoptimizertospeedup"readonly"cursorFORloops.


FORALL• IntroductiontoFORALL• UsingtheSQL%BULK_ROWCOUNT• Referencingfieldsofcollectionsofrecords• UsingFORALLwithsparsely-filledcollections• HandlingerrorsraisedduringexecutionofFORALL


UseFORALLforrepeatedDMLoperations

• Convertloopsthatcontaininserts,updates,deletesormergestoFORALLstatements.

• Header(usually)looksidenticaltoanumericFORloop.– Implicitlydeclaredintegeriterator– Atleastonebindarraythatusestheiteratorasitsindexvalue.–UseINDICESOFandVALUESOFwhenbindarraysaresparse

PROCEDURE upd_for_dept (...) ISBEGIN

FORALL indx IN low_value .. high_valueUPDATE employee

SET salary = newsal_in

WHERE employee_id = list_of_emps (indx);END;

forall_timing.sqlforall_examples.sql


MoreonFORALL• UseanytypeofcollectionwithFORALL.• OnlyoneDMLstatementisallowedperFORALL.

–EachFORALLisitsown"extended"DMLstatement.• Thecollectionmustbeindexedbyinteger.• Thebindarraymustbesequentiallyfilled.

–UnlessyouusetheINDICESOForVALUESOFclause.• Indexescannotbeexpressions.

forall_restrictions.sql


Howmanyrowsweremodified?

• SQL%ROWCOUNTreturnstotalnumberofrowsmodifiedbyentireFORALL.–NottobereliedonwhenusedwithLOGERRORS.

• UsetheSQL%BULK_ROWCOUNTcursorattributetodeterminehowmanyrowsaremodifiedbyeachstatement.– A"pseudo-collection"ofintegers;nomethodsaredefinedforthiselement.

bulk_rowcount.sql


UsingFORALLwithSparseCollections• Priorto10.2,thebindingarraysinaFORALLstatementmust besequentiallyfilled.–UsingtheINlow..highsyntax.

• Now,however,youcanbindsparsecollectionsbyusingINDICESOFandVALUESOFintheFORALLheader.

10g_indices_of*.sql10g_values_of*.sql

PROCEDURE upd_for_dept (...) ISBEGIN

FORALL indx IN INDICES OF list_of_empsUPDATE employee

SET salary = newsal_inWHERE employee_id = list_of_emps (indx);


FORALLandDMLErrors• FORALLstypicallyexecutealargenumberofDMLstatements.• WhenanexceptionoccursinoneofthoseDMLstatement,thedefaultbehavioris:– That statementisrolledbackandtheFORALLstops.– All(previous)successfulstatementsarenot rolledback.

• WhatifyouwanttheFORALLprocessingtocontinue,evenifanerroroccursinoneofthestatements?

• JustaddtheSAVEEXCEPTIONSclause!


Example:FORALLwithSAVEEXCEPTIONS• Add SAVE EXCEPTIONS to enable FORALL to suppress errors at the statement level.CREATE OR REPLACE PROCEDURE load_books (books_in IN book_obj_list_t)ISbulk_errors EXCEPTION;PRAGMA EXCEPTION_INIT ( bulk_errors, -24381 );

BEGINFORALL indx IN books_in.FIRST .. books_in.LASTSAVE EXCEPTIONSINSERT INTO book values (books_in(indx));

EXCEPTION WHEN bulk_errors THEN

FOR indx in 1..SQL%BULK_EXCEPTIONS.COUNTLOOP

log_error (SQL%BULK_EXCEPTIONS(indx).ERROR_INDEX, SQL%BULK_EXCEPTIONS(indx).ERROR_CODE);

END LOOP;END;

Allows processing of all statements, even after an error

occurs.

Iterate through "pseudo-collection" of errors.

bulkexc.sql

If any exception is encountered, Oracle raises

-24381 when done.


SAVEEXCEPTIONSandFORALL

• TheSAVEEXCEPTIONSclausetellsOracletosave exceptioninformationandcontinueprocessingalloftheDMLstatements.

• WhentheFORALLstatementcompletes,ifatleastoneexceptionoccurred,OraclethenraisesORA-24381.

• YouthencheckthecontentsofSQL%BULK_EXCEPTIONS.

PROCEDURE upd_for_dept (newsal_in IN NUMBER,list_of_emps_in IN DBMS_SQL.NUMBER_TABLE) IS

BEGINFORALL indx IN list_of_emps_in.FIRST .. list_of_emps_in.LAST

SAVE EXCEPTIONSUPDATE employees

SET salary = newsal_inWHERE employee_id = list_of_emps_in (indx);

END;


SAVEEXCEPTIONSinDetail• Foreachexceptionraised,OraclepopulatestheSQL%BULK_EXCEPTIONSpseudo-collectionofrecords.– Therecordhastwofields:ERROR_INDEXandERROR_CODE.– ERROR_INDEX:theindexinthebindarrayforwhichtheerroroccurred.– ERROR_CODE:thenumber(positive)fortheerrorthatwasraised

• It'sapseudo-collection andonlysupportsasinglemethod:COUNT.• Soyouiteratefrom1toSQL%BULK_EXCEPTIONS.COUNTtogetinformationabouteacherror.

• Unfortunately,itdoesnotstoretheerrormessage.


ConvertingtoBulkProcessing• Let'stakealookattheprocessbywhichyougofrom"old-fashioned"codetoabulkprocessing-basedsolution.

• Fromintegratedrow-by-rowtophased processing• Challengesinclude:

–WithmultipleDMLstatementsinloop,howdoyou"communicate"fromonetotheother?

– AvoidexcessivePGAconsumption


The"OldFashioned"Approach• CursorFORloopwithtwoDMLstatements,trapexception,andkeepongoing.

CREATE OR REPLACE PROCEDURE upd_for_dept (dept_in IN employees.department_id%TYPE

, newsal_in IN employees.salary%TYPE)IS

CURSOR emp_cur ...;BEGIN

FOR rec IN emp_curLOOP

BEGININSERT INTO employee_history ...

adjust_compensation (rec.employee_id, rec.salary);

UPDATE employees SET salary = rec.salary ...EXCEPTION

WHEN OTHERS THEN log_error;END;

END LOOP;END upd_for_dept;

cfl_to_bulk_0.sql


Aphasedapproachwithbulkprocessing• Changefromintegrated,row-by-rowapproachtoaphasedapproach.

RelationalTable(s)

RelationalTable

Phase1:Bulkcollectfromtable(s)tocollection

Phase3:FORALLfromcollectiontotable

Phase2:Modifycontentsofcollectionaccordingtorequirements


Translatingphasesintocode• Thecfl_to_bulk_5.sqlfilecontainstheconvertedprogram,followingthephasedapproach.

cfl_to_bulk_0.sqlcfl_to_bulk_5_11g.sql

BEGINOPEN employees_cur;

LOOPfetch_next_set_of_rows (

bulk_limit_in, employee_ids, salaries, hire_dates);

EXIT WHEN employee_ids.COUNT = 0;

insert_history;

adj_comp_for_arrays (employee_ids, salaries);

update_employee;END LOOP;

END;

Phase1:GetData

Phase3:PushData

Phase2:MassageData

Phase3:PushData


Conclusions– BulkProcessing• FORALListhemostimportantperformancetuningfeatureinPL/SQL.

– AlmostalwaysthefastestwaytoexecuterepeatedSQLoperationsinPL/SQL.– Lookthatforprimeanti-pattern:Loopscontainingnon-queryDML.– Thenaskyourself:canIdoitentirelywithinSQL?Ifnot...bulkitup!

• Youtradeoffincreasedcomplexityofcodefordramaticallyfasterexecution.– ButrememberthatOraclewillautomaticallyoptimizecursorFORloopstoBULKCOLLECTefficiency.

–NoneedtoconvertunlesstheloopcontainsDMLoryouwanttomaximallyoptimizeyourcode.

• WatchoutfortheimpactonPGAmemory!


What'swrongwiththispicture? • .

SELECT * FROM materialized_viewWHERE pky = 12345;

What'sTaking SoLong?










Here'sthequestiontoaskoneself:

• Buttheyneedthedata!Anditneedstobetherightdata!What'sadevelopersupposedtodo?


TheWonderfulFunctionResultCache• TheFunctionResultCacheshouldbethecachingmechanismofchoiceforPL/SQLdevelopers.–Others:DETERMINISTIC,PGA(session-specific)caching

• Thiscacheis storedintheSGA;sharedacrosssessions;purgedofdirtydataautomatically

• Youcanandshoulduseittoretrievedatafromanytablethatisqueriedmorefrequentlythanupdated.– Staticdatasetslikematerializedviews(the"extremecase")– Samerowsfetchedmultipletimes(parametervaluesserveasuniqueindexintocache)

• EnterpriseEditiononly.


HowtheFunctionResultCacheWorks• AddtheRESULT_CACHEclausetoyourfunction'sheader.

– Yes,that'stheonlychangeyouhavetomaketoyourcode!

• Whenacallismadetofunction,OraclecomparesINargumentvaluestothecache.

• Ifnomatch,thefunctionisexecutedandtheinputsandreturndataarecached.

• Ifamatchisfound,thefunctionisnotexecuted;cacheddataisreturned.• Ifchangestoa"relieson"tablearecommitted,thecacheismarkedinvalidandwillbere-built.

11g_frc_demo.sql


PerformanceImpactofResultCache• TheresultcacheisstoredintheSGA.• SoweshouldexpectitbeslowerthanaPGA-basedcache.• ButaccessingresultcachedatadoesnotrequiregoingthroughtheSQLengine.

• Soitshouldbemuchfasterthanexecutingaquery.– EvenifthestatementisparsedandthedatablocksarealreadyintheSGA.

• Let'sfindout!

11g_emplu*.*


ResultCache– ThingstoKeepinMind- 1• Ifyouhaveuncommittedchangesinyoursession,dependentcachesareignored.– Thecachewillnot overrideyourownchangeddata.

• Cachingisnotperformedforcomplextypes:recordswithCLOBs,collections,etc.– ButOracleisoptimistic!

• Thecacheisnot relatedtoSQLstatementsinyourfunction.– ItonlykeepstrackoftheinputvaluesandtheRETURNclausedata.

11g_frc_demo.sql


ResultCache– ThingstoKeepinMind- 2• Youcannotusetheresultcachewithinvokerrightsprogramunitsuntil12.1.

– Bypassexecutionoffunctionbody,Oraclecannotresolvereferencestoobjects- thewholepoint ofIR.

• Functionswithsession-specificdependenciesmustbe"result-cached"withgreatcare.– Virtualprivatedatabaseconfigurations– ReferencestoSYSDATE,relianceonNLS_DATE_FORMAT,timezonechanges– Applicationcontexts(callstoSYS_CONTEXT)

• Solution:movealldependenciesintoparameterlist.11g_frc_vpd.sql

11g_frc_vpd2.sql


ManagingtheResultCache• Oracleoffersanumberofwaystomanagetheresultcacheandtuneittoyourspecificapplicationneeds:

• RESULT_CACHE_MAX_SIZEinitializationparameter– Ifthecacheistoosmall,thentheLRUalgorithmnegatesthepointofthecache.

• DBMS_RESULT_CACHEmanagementpackage• v$RESULT_CACHE_*performanceviews• Warning:useofthefunctionresultcachecanresultinlatchcontentionthatcancausedatabaseperformancedegradation.

show_frc_dependencies.sp


FineGrainedDependenciesin11.2• Oraclekeepstrackoftabledependenciesonaper-resultlevel.

– Eachresultcachedcouldhaveadifferentsetofdependencies.

• Achangetoatablecouldinvalidatejustasubsetoftheresultsinthecache.– It'snot allornothing- whenyourfunction'sdifferentlogicpathscould"hit"differenttables.

11g_frc_dependencies.sql11g_frc_dependencies2.sql


MakeItEasyonYourself• Ihopeyouwillagreethattheresultcacheisagreatfeature.• Buthoweasywillitbeforyoutoapplyit?• Ifyouwrite/duplicatequeriesthroughoutyourcode,upgradingwillbeexpensiveandslow.

• Ifyouhideyourqueriesbehindfunctions,youhaveasinglepointofdefinition,andyoucanupgrade"instantly."

11g_frc_encapsulation.sql


TheNOCOPYParameterHint• Bydefault,OraclepassesallOUTandINOUTargumentsbyvalue,notreference.– ThismeansthatOUTandINOUTargumentsalwaysinvolvesomecopying ofdata.– AllINargumentsarealwayspassedbyreference(nocopying).

• WithNOCOPY,youturnoffthecopyprocess.– For"large"formalparameters,suchascollections,youmayseeaperformancebenefit.

• Compile-timewarningswillnotifyyouofopportunitiestousethishint.

nocopy*.*


OptimizingFunctionExecutioninSQL• Thatseemslikeanawfullygoodidea!• Twomethods:

–WITHclausethatdefines afunction–UDFpragma

• WITHFUNCTION:defineafunctiondirectlywithinyourSQLstatement.–MosthelpfulforavoidingredundancyinlongSQLstatements

• UDFpragmadeclarativelytellsthecompilerto"prepare"thefunctionforexecutionfromSQL.– Reducesthecostofthecontextswitch

12c_with_function*.sql12c_udf*.sql


AutomaticOptimization– SettoLevel3• Thedefaultoptimizationlevelis2.

– It'sbeen2since10.1,andit'safineoptimizationlevel.

• But11.1introducedautomaticsubprograminlining– level3–Workswithnestedsubprogramsonly!

• RecommendationfromPL/SQLdev team:alwaysmoveoptimizationlevelupto3.–Onlydownsideyouarelikelytoseeisincreasedcompiledcodesize.Thatshouldnotbeanissue.

– YoucanalsouseINLINEpragmaforselectiveinlining.

11g_inline*.sql


PL/SQLOptimization• Makesureyouintegratethe"bigticket"performancefeaturesasyoudevelop.–NativeSQL,bulkprocessing,FORALL,NOCOPY,UDF,optlevel3...

• Focusprimarilyoncorrectnessandmaintainability.• Thenidentifybottlenecks.

– DBMS_HPROF,DBMS_PROFILER• Thenperformmoregranularoptimizationwithinidentifiedsubprograms.

northeast ohio oracle users group - high performance plsql...•oracle asktom – –q&a site,...

Documents