Performance-based Budgeting

CLEAR Training Materials

Manual

1

PerformanceBudgeting

MarcRobinson

2

ContentsTableofAcronyms........................................................................................................................................5Introduction..................................................................................................................................................71.BudgetingandtheBudgetProcess...........................................................................................................82.OverviewofPerformanceBudgeting......................................................................................................12ExpenditurePrioritization.......................................................................................................................12ManagingforResults..............................................................................................................................13ModelsofPerformanceBudgeting.........................................................................................................14

3.PerformanceInformationFundamentals...............................................................................................20PerformanceConcepts:theResultsChain..............................................................................................20Outcomesvs.ExternalFactors................................................................................................................23Outputsvs.OutcomesandActivities......................................................................................................24PerformanceIndicators..........................................................................................................................24

4.PerformanceMeasuresandtheBudget.................................................................................................28EnsuringthatIndicatorsareUseful........................................................................................................30LinkingFundingtoOutcomes.................................................................................................................31LinkingFundingtoOutputs.....................................................................................................................32

5.PerformanceMeasurementSystems......................................................................................................35EnsuringthatPerformanceInformationisCostEffective......................................................................39

6.EvaluationandPerformanceBudgeting.................................................................................................42EvaluatingProgramEffectiveness...........................................................................................................43EfficiencyReview....................................................................................................................................45MakingEvaluationRelevanttotheBudget............................................................................................45EvaluationandtheBudget:TheChileanExample..................................................................................46InternalversusExternalEvaluations.......................................................................................................48

7.PerformanceAuditing.............................................................................................................................508.ProgramBudgetingFundamentals.........................................................................................................52DefiningPrograms...................................................................................................................................53PerformancePressureandProgramBudgeting.....................................................................................54ProgramPerformanceInformation........................................................................................................55

3

ProgramTitlesandObjectives................................................................................................................56ProgramStatements...............................................................................................................................58TheProgramHierarchy...........................................................................................................................58Numberandsizeofprograms.................................................................................................................59DevelopingPrograms..............................................................................................................................60IntegratingStrategicPlanswiththeProgramBudget............................................................................61

9.ProgramAppropriationandExpenditureControl..................................................................................6410.ProgramAccountingandCosting..........................................................................................................68BudgetClassificationandChartofAccounts..........................................................................................68FinancialManagementInformationSystems.........................................................................................69ProgramCostAllocation.........................................................................................................................72AdministrationPrograms........................................................................................................................75SalaryCostsandPrograms......................................................................................................................78CostAllocationandtheProgramHierarchy...........................................................................................78ProgramCostEstimationinBudgetPreparation....................................................................................79

11.SpecialTopicsinProgramBudgeting....................................................................................................82ProgramsandOrganizationalStructure.................................................................................................82MinistryBoundariesandInternalSupportServices...............................................................................85BroaderConflictsbetweenProgramandOrganizationalStructure.......................................................87ProgramStructuresinDevelopingCountries.........................................................................................88FunctionsandPrograms.........................................................................................................................89ProgramsandFunctions:aCaseStudy...................................................................................................92ProgramsandLevelsofGovernment......................................................................................................93

12.FormulaFundingandPurchaserProviderSystems..............................................................................95LimitstoFormulaFundingandPurchaserProvider................................................................................96

13.TargetsandPerformanceBudgeting....................................................................................................98GamingandPerverseEffects................................................................................................................100

14.PublicFinancialManagementReformFoundations...........................................................................10215.ExpenditurePrioritizationandPerformanceBudgeting.....................................................................105SpendingReview...................................................................................................................................105ExpenditurePrioritizationProcessesMoreGenerally..........................................................................109PlanningandPrioritization....................................................................................................................111

4

ExpenditurePrioritizationattheSectorandMinistryLevels...............................................................11116.AccrualAccountingandPerformanceBudgeting...............................................................................114WhatareAccrualAccountingandAccrualBudgeting?........................................................................114TheComplexityofAccrualAccountingandBudgeting.........................................................................119AccrualAccountingandProgramBudgeting........................................................................................120AccrualAccounting,PurchaserProviderandFormulaFundingSystems.............................................121

17.ParliamentandPerformanceBudgeting.............................................................................................124ParliamentandPerformanceAccountability........................................................................................124ParliamentandExpenditurePriorities..................................................................................................126

18.MediumTermBudgetingandPerformanceBudgeting......................................................................127NatureandBenefitsofMTBudgeting..................................................................................................127ForwardEstimates................................................................................................................................128MTCeilings:Fixedvs.Indicative?..........................................................................................................129MTBudgetingandPerformanceBudgeting..........................................................................................131TheMediumTermExpenditureFramework(MTEF)............................................................................132DevelopingCountriesandtheMTEF....................................................................................................134

19.ImplementingGovernmentWidePerformanceBudgeting...............................................................13720.TheManagingforResultsContext......................................................................................................14121.ConcludingComments........................................................................................................................143Glossary.....................................................................................................................................................144References................................................................................................................................................150

5

TableofAcronymsAA AccrualaccountingABC ActivitybasedcostingAOB AccrualoutputbudgetingBC BudgetclassificationCOA ChartofaccountsCOFOG ClassificationoftheFunctionsofGovernmentDRG DiagnosticrelatedgroupFF FormulafundingFMIS FinancialmanagementinformationsystemGBE GovernmentbusinessenterpriseIFMIS IntegratedfinancialmanagementinformationsystemIT InformationtechnologyLOLF LoiorganiquerelativeauxloisdefinancesMFR ManagingforresultsMoF MinistryoffinanceMT MediumtermMTEF MediumtermexpenditureframeworkMTFF MediumtermfiscalframeworkOECD OrganizationforEconomicCooperationandDevelopmentPB PerformancebudgetingPFM PublicfinancialmanagementPM PerformancemeasurementPSA PublicServiceAgreementSAI Supremeauditinstitution

6

ZBB Zerobasebudgeting

7

IntroductionThismanualisdesignedtoaccompanythecoursesonperformancebudgetingdeliveredattheregionalCentersforLearningonEvaluationandResults(CLEAR).Itpresentssummariesoftheessentialcontentforeachofthecoretopicscoveredinthosecourses.Thismanualcanthereforebeusedonastandalonebasis.However,itmaybeusedtobestadvantageinconjunctionwiththesupportofthekeyreadingslistedattheendofeachsection.

8

1.BudgetingandtheBudgetProcessGovernmentbudgetingistheallocationanduseofresources,andassociateddecisionsabouthowtheresourcesusedwillbeacquired,bythatpartofthepublicsectorwhichisfinancedprimarilybycompulsorychargessuchastaxes1.Equivalently,governmentbudgetingbudgetingforshortinwhatfollowscanbesaidtobeabouttheallocationofresourcesfortheprovisionofservicesandtransfersonanonmarketbasis,wherenonmarketmeanseitherfreeorgreatlybelowcost.Thegovernmentbudgetisthereforeafinancialplanwhichcoversthosepublicsectoragenciessuchasministriesandthesocialsecuritysystemwhichareprimarilytaxfinanced.Itdoesnottypicallycoverpubliccorporations,whichnormallymanagetheirfinancesonamoreautonomousbasis.2Budgetingisaprocessinvolvingfourstages:Aggregatefiscalpolicyformulation:thisreferstothedeterminationofthegovernmentsoverarchingobjectivesforthebudgetdeficit,debtandotherrelevantfiscalaggregates3,whichshouldthenbetranslatedintodecisionsaboutthedesiredlevelsofaggregaterevenueandexpenditure.Budgetpreparationandenactment:inthisstageofthebudgetprocessthegovernmentdecideshowmuchfundingitwillprovidetowhichagenciesandforwhichpurposes.Thisisgivenformalexpressioninthebudgetlawandbudgetregulations,whichareenactedbythelegislatureandthehighestexecutiveorgansofgovernment(i.e.thepresidentorcabinetofministers).Budgetexecution:thisreferstothecarryingoutoftheexpenditureplandevelopedinthebudgetincludingtheenteringofcontractsandexpenditureoffunds.

1Weusethetermtaxasshorthandtoincludeothercompulsoryrevenuesourcessuchasmandatoryleviesorfees,andfines.Budgetfinancingviaborrowingmaybeconsideredtobefinancingbydeferredtaxes.2Publiccorporationsarefinancedprimarilybythepriceswhichtheychargetheconsumersoftheirservicespriceswhicharevoluntaryinthatconsumersarenotlegallyobligedtopurchasetheservicesconcerned.Thebudgetnormallyonlyrelatestopubliccorporationsinsofarastheyeitherreceivetaxpayersubsidiesortheirprofitsareasupplementarysourceofbudgetfunding.3Dependingonthepolicyframeworkofthecountryconcerned,theotherfiscalaggregatesintermsofwhichfiscalpolicyisframedmightinclude,e.g.,thelevelofnetworth(governmentassetsminusliabilities)orthepercentageofrevenue/GDP.

9

Accounting,AuditingandReporting:thiscoversthepreparationofaccountingrecordsofgovernmentspendingandrevenue,theirauditingbybothinternalandexternalauditors,andtheprovisionofreportsonbudgetexecutiontogovernmentagencies,ministers,parliamentandthepublic.Budgetinghasthreekeyobjectives Soundmacrofiscaloutcomes:thisisprimarilytheproductofgoodaggregatefiscalpolicy.

Aggregatefiscalpolicyneedstoensurefiscalsustainabilitywhichmeansessentiallythatdeficitsanddebtdonotgetoutofcontrolbecausethisisimportantforeconomicconfidenceandstability.Ifbudgetingdoesnotkeepdebttosustainablelimits,financialmarketscanpotentiallyloseconfidenceinthesecurityoflendingtogovernment.Theresultmaybeasuddenincreaseintheinterestrateswhichgovernmentisforcedtopayand,intheextreme,anunwillingnesstolendtogovernment.Inadditiontofiscalsustainability,aggregatefiscalpolicycanplayanimportantstabilizationpolicyroleinparticular,throughprovidingsupporttotheeconomythroughadditionalspendingorlowertaxesduringarecession.

Appropriateprioritizationofexpenditure:thisreferstotheallocationoffundstothesectorsandprogramswhicharemosteffectiveinmeetingsocialneeds.Itmeans,forexample,thatifacountryisfacingamajornewhealthchallenge,additionalfundswillbeallocatedinthebudgetforprogramsdesignedtotacklethatchallenge.Thecorollaryofthisisthatfundsaremovedawayfromsectorsandprogramswherespendingislowpriority.

Serviceeffectivenessandefficiency:thismeans,firstly,thatgovernmentservicesare

deliveredefficientlyand,secondly,thattheyaredesignedandmanagedsoastomaximizetheireffectiveness.

Performancebudgeting,aswewillsee,focusesdirectlyonimprovingtheabilityofbudgetingintodeliveronthesecondandthirdoftheseobjectives.4Effectivenessandefficiencyarethereforekeywordshere.Effectivenessreferstotheextenttowhichaservicedeliversthebenefitswhichitissupposedtodelivertosociety.Forexample,anHIV/AIDSpreventioncampaigniseffectiveifitsucceedsinreducingtherateofnewHIV/AIDSinfections.Aswillbeexplainedlater,effectivenessisabouttheextenttowhichprogramsachievetheirintendedoutcomes.Efficiency,ontheotherhand,referstodeliveryofservicesatthelowestpossible4However,indoingso,itcanalsoindirectlyimprovemacrofiscaloutcomes.

10

cost,withoutsacrificingquality.Expresseddifferently,efficiencyisaboutminimizingwasteintheproductionofgovernmentservices.Budgetpreparationshouldstartwith,andbecarriedoutwithinthecontextof,goodaggregatefiscalpolicy.Inotherwords,decisionsabouthowmuchfundingtoprovideagenciesshouldbeconstrainedbyclearpolicyaboutthelevelofaggregateexpenditurewhichis,givengovernmentrevenues,consistentwithgovernmentsdeficitanddebtobjectives.Keyaspectsofbudgetpreparationandenactmentinclude: Thedeterminationofexpenditurepriorities, Estimatesofspendingrequirementsparticularlyassociatedwithongoingcommitments

suchasthegovernmentwagebill,capitalprojectsunderway,andexpenditureobligationsonmandatoryitemssuchaspensionsandothersocialsecuritypayments,

Theformulationofcurrentandcapitalbudgets,whichrequiredifferentbudgetpreparationmethodsbutwhichshouldatthesametimebelinked.5

Theformulationofbudgetfundingdecisionsintolegalauthorizations(controltotals)whichcanbeappropriatedbytheparliamentorpromulgatedbytheexecutivegovernment.

Thewayinwhichbudgetpreparationisorganizeddifferconsiderablybetweendifferentcountries.Somekeyaspectsofbudgetpreparationwhichhaveaparticularlyclosebearingonperformancebudgetingsuchasexpenditureprioritizationandmediumtermbudgetingmethodsarediscussedinthismanual.Budgetexecution,ontheotherhand,canbeseenasaclearlydefinedsequenceofstages,asfollows: Authorizationstage:basedontheparliamentarybudgetappropriations,thefinance

ministrycommunicatestoministriestheirexpenditureentitlements.Thenatureoftheseauthorizationsdiffersinternationally.Insomecountries,authorizationsaremadeonamonthbymonthbasisthatis,eachmonthministriesaretoldhowmuchmoneytheywillhavetospendinthecomingmonth.

5Buildingmajornewcapitalassetssuchasschools,hospitalsandroadshasimportantimplicationsforfuturecurrentexpenditure,whetheronstaffandsuppliestorunthefacilitiesconcerned,oronthemaintenanceofthoseassets.

11

Commitmentstage:acommitmentisacontractorsomeotherobligationtomakeafuturepayment.

Verificationstage:ensuringthattheserviceorgoodspreviouslycontractedforhavebeendelivered,andmeetthespecificationsofthecontract.

Paymentauthorizationstage:inthelightofverification,approvalforthepaymenttobemadeisgivenbyanofficialindependentofthepersonswhoinitiatedthecommitment.

Paymentstage:thepaymentisactuallymade. Accountingstage:eachpaymentisenteredintotheaccountingsystemanditsrelevant

characteristicsrecorded.Inawelldevelopedsystem,theaccountingsystemwillalsorecordthecommitmentsstage.

Auditingstage:theaccountsareauditedfirstlybyinternalauditors(withinthespendingministryconcerned)andsubsequentlybyexternalauditors(theauditorgeneralorauditcourt).

Performancebudgetingaimstochangebudgetpreparationfundamentally,linkingitmuchmoresystematicallytotheeffectivenessandefficiencyofexpenditure.Indoingso,however,italsobringsaboutequivalentlymajorchangesinbudgetexecution.KeyReadingsPotter,B.andJ.Diamond(1999),GuidelinesforPublicExpenditureManagement(Washington:IMF),availableathttp://www.imf.org/external/pubs/ft/expend/index.htm.SchiavoCampo,S.&D.Tommasi(1999),Managinggovernmentexpenditure(Manila:AsianDevelopmentBank),obtainableathttp://www.adb.org/documents/manuals/govt_expenditure/.

12

2.OverviewofPerformanceBudgetingPerformancebudgetingaimstoimprovetheeffectivenessandefficiencyofpublicexpenditurebylinkingthefundingofpublicsectororganizationstotheresultstheydeliver.Itusessystematicperformanceinformation(indicators,evaluations,programcostingsetc)tomakethislink.Theimpactofperformancebudgetingmaybefeltinimprovedprioritizationofexpenditure,andinimprovedserviceeffectivenessand/orefficiency.PBusuallyalsoemphasizesgivinggovernmentagenciesandtheirmanagersgreaterflexibilityintheuseofresourcesthantheywouldtypicallyhaveundertraditionaltightlycontrolledpublicmanagementsystems.Akeyelementofthisisgreaterflexibilityinthechoiceofthemixofinputswhicharetobeusedtodeliverservices(e.g.howmuchlaborinputvs.externallysourcedinputs,themixoftypesofexternallysourcesuppliesandservicesused).Animportantimplicationofthisistheneedformoreflexibilityhumanresourcesmanagement,atopicdiscussedinalatersection.TheincreasedinternationalinterestinperformancebudgetinghasbeenpromptedinpartbyarecognitionthatitisalltooeasyinGovernmenttolosesightofthefundamentalobjectiveofdeliveringpositiveoutcomestothecommunity.Publicsectororganizationswhicharefinancedbytaxesandothercompulsorychargeslackthemarketdisciplineswhichcompelcommercialenterprises,particularlythoseoperatinginhighlycompetitivemarkets,tobecustomeroriented.Politicalaccountabilitythroughtheelectoralprocessis,ofcourse,extremelyimportant,butisnotnecessarilysufficienttoensurethatpublicsectororganizationsarehighlyfocusedupontheresultstheydeliver.ExpenditurePrioritizationExpenditureprioritizationisoftenquiteweakingovernment.Aparticularconcernisthetendencyformuchongoingprogramexpendituretoescapeseriousscrutiny,andforbudgetarydecisionstobemainlyfocusedonwhatnewspendinginitiativestoadopt.Thetermincrementalismiswidelyusedtodescribethistendency.6Ifprioritizationprocessesarenotwelldeveloped,governmentstypicallyrely,ifandwhentheyneedtomakespendingcuts,onacrosstheboardcutsthatis,cutsofthesamepercentage6Formoreonincrementalism,seeRobinsonandBrumby(2005).

13

toallministries.Acrosstheboardcutsarebydefinitionindiscriminateandinferiortoaselectiveapproachwhichwouldtargetcutsatlowerpriorityorlesseffectiveprograms.Improvedexpenditureprioritizationcanhelptoachievemoresustainablepublicfinances.If,forexample,spendingcutsareneededtoachievefiscalconsolidation(i.e.togetdeficitsanddebtundercontrol),thecapacitytotargetcutsselectivelyatthelowestpriorityprogramswillhelptomakethosecutsmoresustainable(i.e.thereducethepressurestoreversethem),thusincreasingtheprobabilitythattheimprovementtotheoverallstateofpublicfinanceswillbeenduring.ManagingforResultsPerformancebudgetingshouldbeviewedinthebroadercontextofasetofrelatedmanagingforresults(MFR)reforms.MFRcanbedefinedastheuseofformalperformanceinformationtoimprovepublicsectorefficiencyandeffectiveness.Itsfundamentalstartingpointismaximumclarityabouttheoutcomeswhichgovernmentisattemptingtoachieve,andabouttherelationshipofoutputs,activitiesandresourcesusedtothosedesiredoutcomes.GoodstrategicplanningandbusinessplanningareanessentialelementofMFR.MFRalsotendstoemphasizetheexantestipulationofperformanceexpectationsforagencies,workunitsandindividualsthroughtheuseofperformancetargetsandstandards.AstandardelementofthestrategichumanresourcesmanagementcomponentofMFRistheintroductionofstrongerperformancebasedextrinsicincentives(rewardsandsanctions)forpublicofficials.Typically,thisisaccompaniedbygreaterflexibilityofemployment,includinggreatercapacitytosanctionordismisspoorperformers,andgreatereaseintransferringorterminateemployeesinprogramswhichthegovernmentiseliminatingorcuttingback.ExamplesofotherelementsofMFRreformswhichhavebeenintroducedinmanycountriesinclude: Customerorientationmeasures:anexampleofsuchameasureisclientrightscharters,

whichdefineclientservicerights(e.g.timelinessofservice,whattypeofserviceetc). Markettypereforms:thisreferstoarangeofmeasureswherebypublicserviceproviders

aremadetooperateinamorebusinesslikemanner,underthepressureofcertaintypesofmarketforcesormechanismswhichsimulatemarkets.Competitivetenderingwherea

14

serviceprovidedbyagovernmentagencyisthrownopentoprivatecompetitionviatenderisoneexample.Purchaserprovidersystemsareaperformancebudgetingmodelwhichfallsintothecategoryofmarkettypereform.

ModelsofPerformanceBudgetingThereisnotjustonemodelofperformancebudgeting,butarangeofdifferentmodels.Alllinkfundingandresults,butindifferentways.Someperformancebudgetingsystemsprogrambudgetingandzerobasebudgeting,forexamplearegovernmentwidesystems.Ontheotherhand,someareonlyintendedtoapplytoparticularsectorsofexpenditureorcategoriesoforganization(e.g.aformulafundingsystemapplyingtoschools).Performancebudgetingsystemsdiffertosomeextentintheirobjectives.Someplaceaimprincipallytoimproveexpenditureprioritization,whileothersaremainlyfocusedoninimprovingserviceeffectivenessand/orefficiency.Reflectingthedifferentformsoflinkagebetweenresultsandfundingwhichtheyseektobuild,performancebudgetingsystemsalsodifferinrespecttothetypeofperformanceinformationuponwhichtheyprimarilyrely.Thisappliesbothtothetypeofperformancemeasureswhichtheyuse,andalsotowhethertheymakeuseofevaluationaswellasindicators.Asdiscussedearlier,allformsofperformancebudgetinghaveincommontheideaofgreatermanagerialfreedom,particularlyinrespecttothechoiceofinputmix.Themostbasicformofperformancebudgetingisthatwhichusesperformanceinformationsystematicallyinthepreparationofthegovernmentwidebudget.Acommontoolusedforthispurposeisprogrambudgeting,inwhichexpenditureisclassifiedinthebudgetbyobjectives(outcomesandoutputs),ratherthansolelybyeconomiccategories(suchassalaries,suppliesandcommunicationcosts)andorganizationalcategories(e.g.ministryanddepartmentwiththeministry).Theprimaryobjectiveofprogrambudgetingisimprovedexpenditureprioritization.However,byincreasingtheattentionpaidduringbudgetpreparationtospendingministry

15

performance,thistypeofperformancebudgetingalsoaimstoincreasethepressureonministriestoimproveefficiencyandeffectiveness.7Itshouldbeemphasizedthat,intheterminologyadoptedhere,performancebudgetingisagenericconceptofwhichprogrambudgetingisoneformofperformancebudgeting.Thisneedstobeemphasizedbecausetherearesomewhoattempttodistinguishprogrambudgetingfromperformancebudgeting.Zerobasebudgeting(ZBB)isessentiallyavariationofprogrambudgetingwhich,initsoriginalform,calledforthecomprehensivereviewandprioritizationofallexpenditureonacontinuingbasis.Toachievethis,allprogramsweretobedecomposedforthepurposesofbudgetpreparationintodecisionpackages(alsoknownasserviceincrements)whichwouldprovidechoiceabouttheextenttowhicheachprogrammightbecutbackor,alternatively,givenincreasedfunding.Ideally,thesedecisionpackageswouldcoverallmajoroptions,eventoa100percentcutintheprogramconcerned(hencethezerobase).Priorityrankingswouldbeattachedtothesedecisionpackagesagain,onthebasisofperformanceinformationandtheserankingswouldthenbeusedtoensurethattheavailablelevelofrevenuefundedthosedecisionpackageswhichwereofhighestpriority.Inthisway,ZBBaimedtogoevenfurtherthanprogrambudgetinginimprovingexpenditureprioritization.ThefundamentalproblemwithZBBwasthepracticalimpossibilityofcomprehensivelyreviewingallexpenditureeachyear.Inotherwords,thezerobaseexaminationofeachprogramwasnotreallypracticable.Selectiveexpenditureappraisalisallthat,inpractice,canbeachieved.Inrecognitionofthis,ZBBevolvedintowhatissometimescallalternativebudgeting.Ratherthanseekingtoexaminealloptionsforeachprogram,extendingeventoaonehundredpercentcut,alternativebudgetingtriedtomakethetaskmanageablebyconfiningtheprocesstotheexaminationeachyearofamuchnarrowerrangeofoptionsforcutsorincreasestoeachprogramforexample,theoptionsof15,10and5percentcuts,andperhapsalsoof5or10percentincreases.

7Programbudgetinghasthepotentialtocontributesignificantlytoaggregateexpenditurediscipline,andtherebytofiscaldiscipline.Bygivinggovernmentanimprovedcapacitytoidentifylowpriorityorineffectiveprogramswhichcanbecut,itmakesiteasiertocreatethefiscalspacenecessarytorespondtonewpolicyprioritieswithoutunduegrowthinaggregatespending.Moreover,ifandwhenfiscalconsolidationisnecessary,betterprioritizationmakesitpossibletotargetspendingcutssoastominimizetheirsocialcostandincreasetheirsustainability.

16

Underbothprogrambudgetingandzerobasebudgeting,thelinkbetweenresultsandfundingisaratherloose,ratherthanmechanical,one.Theresultsachievedbyprograms(ordecisionpackageswithprograms)isconsideredcloselyinthebudgetpreparationprocess,butthereitisnotthecasethatpoorresultsautomaticallymeanscutsor,conversely,thatgoodresultsautomaticallymeansincreasingfunding.Sincethe1980s,arangeofnewerformsofperformancebudgetinghavebeendevelopedwhichaimtobuildatighterlinkagebetweenfundingandresults.Thesenewerformsofperformancebudgetingallaimtoincreasethepressureonpublicagenciestoimprovetheeffectivenessand/orefficiencyoftheserviceswhichtheydelivery.Toachievethese,theyrelyonthreemechanisms,usedeitherinisolationorincombination.Thefirstofthesemechanismsislinkingbudgetstoperformancetargets.Settingperformancetargetswhetherforministries,workunitsorindividualsis,asnotedabove,ageneralMFRtheme.Itiswhenbudgetfundingislinkedinsomemannertoperformancetargetsthatwearetalkingaboutaformofperformancebudgeting.ThebestexampleofatargetbasedformofperformancebudgetingistheUKPublicServiceAgreement(PSA)systemasitoperatedbetween1998and20048.Underthissystem,targetsweresetonatriennialbasisaspartofamultiyearbudgetingframework.Inotherwords,highleveltargetsforoutcomesandoutputs(seethenextsectionforexplanationsofthesekeyconcepts)weresetasanintegralpartoftheprocesswhich,everythreeyears,determinedthecorefundinglevelswhicheachministrywouldreceiveforthecomingthreeyearperiod.Targetsweresetinthelightoffunding,andfundinginthelightofperformanceagainsttargets.Themajorquestionthatthisraisestobediscussedlaterinsection13istheconcretenatureofthislinkbetweenfundingandtargets.Targetsettingraisesarangeofissues,whichwillalsobediscussedindetaillatersection13.Oneoftheseistheproblemofselectingtherightnumericalvalueforthetarget.Inprinciple,ofcourse,thetargetshouldbeneithertootoughnortooeasy.However,itisoftennoteasyfortheministryoffinance(MoF)orothercentraldecisionsmakers,withoutadetailedknowledgeoftheareaofservicedeliveryconcerned,tosetthetargetatanappropriatelevel.Anotherissuesraisesbytargetsettingistheissueofbehavioraldistortions,anexampleofwhichisthedangerthat,instrivingtofulfillatargetsetintermsofonedimensionofperformance,theagencymay8SeeP.Smith,PerformanceBudgetinginEnglandinRobinson,PerformanceBudgeting.

17

putlesseffortintoanotherdimensionofperformancewhichisnotfactoredintotheperformancetarget.9Thesecondmechanismusedtocreatingatighterlinkbetweenfundingandresultsisformulafunding,whichisthesubjectofsection12.Formulafundingmeansthattheleveloffundingtoagovernmentagencyisdetermined(inpartorwhole)asamathematicalfunctionofsomeexplicitvariables.Asimpleexampleisaformulatoestimateschoolfundingrequirementsoverthemediumtermasafunctionofthenumberofstudentsandthecostperstudent(basedondemographicprojections).Thisillustratestheideaofacostbasedformula,thesimplestversionofwhichbasesfundingonthequantityofoutputtimestheunit(average)costperoutputas,forexample,inaschoolfundingsystemwherefundingforeachschoolisdeterminedprimarilybyitsnumberofstudentsmultipliedbyanamountintendedtocoverthecostofoneyearseducationofonestudentattherelevantstageofschooling(e.g.$4000peryearninestudentperyear)..Inthiscase,theformulaisbeingusedonlyasabudgetestimationtoolthatis,toestimatehowmuchfundingshouldbeprovidedtoaparticularserviceareaortospecificserviceproviderunits(e.g.specificschools).Notallformulafundingarrangementscanbesaidtobeformsofperformancebudgeting.Itisonlywhentheformulalinksfundingtoresults(outcomesand/oroutputs),anddoessowiththeintentionofboostingefficiencyand/oreffectiveness,thatformulafundingbecomesatoolforperformancebudgeting.Thiswouldbethecase,forexample,iftheformulawasbeingusedineffecttosetaperformancetargetfortheagency.Asaconcreteexampleofthis,supposeaministrywasfundedonaformulabasisto,say,vaccinateonemillionpersonsat$20pervaccinationinthecomingyear,andwasthenheldtoaccountforifitspendallthemoneybutonlyvaccinated600,000people.Thethirdmechanismusedbythenewerperformancebudgetingmodelsisperformancefundingincentivestoagencies.Anexampleofthisisperformancebonusfunding,whichoperatesbygivingagenciesadditionalpaymentsformeasuredgoodperformance(orperhapsalsofundingreductionsasafunctionofmeasuredpoorperformance).Anexampleisuniversityfundingsystemswhichoperateinmanycountriesinwhichpublicuniversitiesaregivenbonusfunding,overandabovetheircorefunding,basedonperformanceagainstmeasuressuchasthepercentageofgraduateswhoareinprofessionalemploymentsixmonthsaftergraduation.As9Forexample,ifthetargetissetintermoftheunitcostofservicedelivery,theagencymightsacrificequalityinordertocutscosts.

18

thisexampleshows,mostperformancefundingincentivesarealsoformsofformulafunding,becausethefundingincentiveprovidedisdeterminedbyamathematicalformulalinkedtoaperformanceindicator.10Performancefundingincentiveslikeformulafundingarealmostinvariablyusedasinstrumentforsectoralperformancebudgetingsystems(i.e.coveringsomesectorofgovernment,suchasschools,orhospitals.Theaboveoutlinesthreekeymechanismsusedbynewerperformancebudgetingsystemstocreatetighterlinksbetweenfundingandresults.Asindicatedabove,thesemechanismsmaybeusedinisolationorincombination.Animportantexampleofthecombinedapplicationofwhoeofthesemechanismsisthepurchaserprovider(PP)model.PPisacombinationoftheprinciplesofincentivepaymentsandformulafunding(thesecondandthirdofthemechanismsidentified).Inapurchaserprovidersystem,apublicsectoragencyisfundedasifitwereanarmslengthproviderofgoodsorservices.Governmentfundstheagencybypayingapriceperunitfortheagencysoutputs(andsometimes,butmorerarely,outcomes).Iftheagencyscostofproductionexceedsthepriceitispaid,theagencymakesaloss.Conversely,itretainsanyprofitmadebyproducingatacostbelowtheprice.InaPPsystem,fundingisonlyprovidedforresultsdelivered.Finally,thepricepaidisideallybasedon,partlyorinwhole,onsomemeasureofwhatitshouldcostanefficientproducertodelivertheoutputconcerned.Theaimofthissystemistomotivatestrongperformanceviastrongfinancialperformanceincentives.Themostsuccessfulexampleofapurchaserprovidersystemisthediagnosticrelatedgroupsystemofhospitalfunding.Broadlyspeaking,thisisasysteminwhichhospitalarefundedprimarilyforthetreatmentstheydelivertopatients.Servicesarecategorizedintoanumberofdifferentoutputtypes(DRGs),andaspecificpriceisattachedtoeachDRGcategory.Suchasystemprovidesapowerfulincentiveforefficiency.Atthesametime,itraisesconcernsaboutthedangerofcostcuttingwhichunderminesthequalityoftreatmentorhasotherperversebehavioraleffects.TheexperienceoftheDRGsysteminthisrespectis,however,encouraging.Accrualoutputbudgeting(AOB)representedan(unsuccessful)attempt,inNewZealandandAustralia,toapplythepurchaserproviderprincipletothegovernmentaswhole,asopposedtoaparticularsectorsuchashospitals.TheideaofAOBwasthatministrieswouldbefundednot10Notethatinthiscase,theformulaisnotlinkedtothecostsoftheperformancedelivered(forexample,noonewouldpretendtoknowtheadditionalcostinvolvedinboostingthegraduateemploymentrateby,say,5percent).

19

bybeinggivenbudgets,butwouldratherbepaidfortheoutputstheydeliver,asiftheywereexternalcontractorstogovernment.Itwasintendedthatgovernmentwouldbeatoughpricesetter,seekingtocreatethetypeofpricingpressurewhichoccursinacompetitivemarket.Becauseinthismodelministriesweresupposedtoactlikebusinesses,itwasnecessarythattheyusebusiness(i.e.accrual)accountinghencetheaccrualinaccrualoutputbudgeting(thelinkwithaccrualaccountingisdiscussedinasection16).Italsobecamenecessarytodistinguishgovernmentspurchaserrolevisvisitsagenciesfromitsownershiprole.TheAOBexperienceisdiscussedindetailinthesection12.KeyReadingsRobinson,M(2007),PerformanceBudgetingModelsandMechanismsinM.Robinson(ed.)PerformanceBudgeting:LinkingFundingandResults.

20

3.PerformanceInformationFundamentalsOutcomesandoutputsplayacentralroleinallmodelsofperformancebudgeting,anditisessentialforanydiscussionofperformancebudgetingthattheseandrelatedconceptsareclearlyunderstood.PerformanceConcepts:theResultsChainIntheresultschainframework,outputsareproducedusinginputs(resources)viaactivitiesandprocesses,andoutputsgenerateoutcomesforthecommunity.

TheResultsChain

Inputs(Resources)

Activities(Processes)

Outputs

IntermediateOutcomes

HighlevelOutcomes

Outputsaregoodsorservicestheproductswhichaministryorothergovernmentorganizationdeliverstoexternalparties.Thisusuallymeansservicesdeliveredtoorforthedirectbenefitofthecommunity,althoughsomeoutputsareserviceswhichministriesdelivertootherministries.Examplesofoutputsinclude:medicaltreatments;advicereceivedbyfarmersfromagriculturalextensionofficers;studentstaught;andpolicecriminalinvestigations.Most

21

governmentoutputsareservices,andforthisreasonthetermsoutputandservicewillbeusedinterchangeablyinthismanual.Outcomesaretheintendedimpactsofoutputsmoreprecisely,thechangesbroughtaboutbypublicprogramsuponindividuals,socialstructures,orthephysicalenvironment.Healthinspectionsofrestaurantsareanoutput,theintendedoutcomeofwhichisthatfewerdinersfallsick.Criminalinvestigationsareapoliceoutput,andreducedcrimetheoutcome.Manygovernmentservicesaimtoachievemorethanoneoutcome.Forexample,schooleducationaimstoincreasethelevelofeducationofthepopulation.Butitalsoaims,amongstotherthings,toimproveeconomicperformance.Bothahigherlevelofeducationandastrongereconomyareoutcomes.Becauseitisbymeansofthefirstofthesethatthesecondisachieved,amoreeducatedpopulationissaidtobeanintermediateoutcome,andastrongereconomyahigherleveloutcome.Therelationbetweenproximateandhighleveloutcomesisoneoflogicalcausality(i.e.theproximateoutcomesinducethehighleveloutcomes).Theoutcomeselementisthedistinctivelypublicsectorcomponentoftheresultschain.Bycontrast,thewayinwhichoutputsareproducedisconceptualizedintheresultschaininexactlythesamewayasintheprivatesector:outputsareproducedbytheuseofinputsinproductionactivitiesandprocesses.Forexample,thetreatmentwhichseriouslyinjuredpersonreceivesinhospitalinvolvestheuseofasetofinputs(skilledstaff,operatingequipmentandfacilities,medicalsupplies,electricityetc)andasetofactivitiesincludinganesthesia,surgeryandnursing,aswellassupportingactivitiessuchassuppliesandfacilitymanagement.Inputs,asthisexampleindicates,refertoallinputs,assetsandcapabilitieswhichareormaybedrawnonintheproductionprocesstodelivertheoutputsandoutcomesdesired.Althoughinputsistheconventionalresultschainterm,andthereforewillbeusedhere,thetermresourcesactuallycapturesbetterthescopeofwhatisreferredto.Thusinputswhichcontributetothecapabilitytodeliverresultsincludenotonlyequipmentandbuildingsby,forexample,organizationalcultureandstaffmorale.11

11Weare,inotherwords,talkingaboutmuchmorethanthefactorsofproductionofeconomics101(labor,landandcapital),andalsotodrawattentionthattheinputsanorganizationcandrawnuponarenotonlypurchasedfromoutside,butcreatedwithintheorganization.

22

Physicalassetsareinputswhicharesometimesmistakenlythoughtofasoutputsasin,forexample,thenumberofroadsorbridgesprovidedbygovernmenttothecommunity.Butitisnottheroadsandbridgeswhichareoutputs,butrathertheservicewhichcitizensgetfromtheseroadsandbridges.Thismeans,forexample,thatpassengermilestraveledisanoutputmeasurewhilekilometersofroadisaninputmeasure.12Thetermactivitiesmayseemselfexplanatory,butconfusionbetweenactivitiesandoutputsisverycommon.Someexamplescanhelpavoidthisconfusion: Inahospital,anesthesiaandcleaningareactivitiesratherthanoutputsbecausetheyare

componentsoftheoverallserviceprovidedtothepatient,ratherthanthecompleteservice.Thepatientcantrecoverthroughanesthesiaorcleaninginisolation,anditisonlyviathecombinationofallthenecessaryactivitiesthatthecompleteservice(theoutput)isdelivered.Moregenerally,anoutputmustbecapableofdeliveringtheintendedoutcome.

Busdrivingisanactivity,whereaspassengertripsaretheoutputs.Similarly,teachingisan

activityratherthananoutput,andtheoutputisstudentstaught.Inthesecases,anactivityproducesmultipleoutputs.13

Asmentionedabove,outputsaredefinedasservicesprovidedbyministriesorothergovernmentagenciestoexternalparties.Soifoneministryprovidesservicestoanotherministryforexample,training,orofficeaccommodationtheseareministryoutputsbecausetheotherministryisaclientexternaltotheministryprovidingtheservice.However,aservicewhichonepartofaministryprovidestoanotherpartofthesameministryforexample,whentheeducationministryshumanresourcesgrouprecruitsteachersfortheprimaryschoolgroupisnotanoutputbutasupportservice.1412Thisshouldbequalifiedbynotingthat,ifaseparatepublicworksministryisresponsibleforconstructingtheroadsandbridgesandthendeliveringthemtothetransportministrywhichmanagesthem,thentheroadsandbridgeswouldindeedconstituteoutputsofthepublicworksministrybecauseitisdeliveringthemtoanexternalclient.Butifthetransportministrymanagestheconstructionitself,theroadsandbridgesarenotoutputs.Ineithercase,theoutputprovidedbythetransportministryistheuseoftheroadbytravelers.13Or,potentially,asillustratedbythecaseofbusdriving,nooutputsatall.Thiswouldbethecaseifthebusranbutnopassengersgotonboard.14Somewouldarguethatwhatconstitutesanoutput,asopposedtoasupportservice,dependsonyourorganizationalperspective,andthatfromthepointofviewofthehumanresourcesgroup,recruitmentisanoutputwhichtheydeliver.Thisisavalidobservation.However,intheperformancebudgetingcontext,theperspectivenormallyadoptedisthatoftheministryorotherdistinctgovernmentorganization.Thepointistofocusentireministriesontheservicestheydelivertoexternalclients,ratherthanoninternalactivities.

23

Theresultschainisaverywellestablishedconceptualframeworkforthepublicsectorwhichgoesunderarangeofnames,includingtheprogramlogicmodelandlogicalmodel.Althoughitsconceptsarestandardized,andhaveclearmeaningsattachedtothem,thereisonesignificantdivergenceofusageworthmentioning:thepracticeadoptedbysomeofdividingoutcomes,asdefinedabove,intooutcomesandimpacts.Thedistinctioniseitheroneoftimelines(impactsbeingmorelongterm),oristhedistinctionbetweenintermediateandhighleveloutcomes.Thisterminologyisnotusedhere,becauseinpracticethereisoftennocleardistinctionatthemarginsbetweenoutcomesandimpacts.Oneotherminorpointofterminology:intermediateandhighleveloutcomesgobyawiderangeofothernames(immediateoutcomes,proximateoutcomes,endoutcomes,ultimateoutcomes).Theideais,however,thesame.Outcomesvs.ExternalFactorsOutcomesarethechangesbroughtaboutbygovernmentintervention.Ifthelevelofmalariafallsforreasonswhichhavenothingtodowithgovernmentactionsbecause,forexample,thereisadroughtwhichreducesmosquitonumberssubstantiallythisfallisnotagovernmentoutcome.Neitherisitanoutcomeiftherateofeconomicgrowthincreasessubstantiallybecause,andonlybecause,theworldeconomyisverybuoyant.Thefallinmalaria,ortheboostinthegrowthrate,areoutcomesonlytotheextentthattheyaretheresultofgovernmentactions.Outcomesthereforeneedtobedistinguishedfromtheconsequencesofexternalfactors.Externalfactorsarefactorsbeyondthecontrolofgovernmentwhichinfluencethecharacteristicsofindividuals,socialstructuresorthephysicalenvironmentwhichthegovernmentistryingtochange.Thelevelofrainisthereforeanexternalfactorimpactimpactingonthemalariarate.Thestateoftheworldeconomyisanexternalfactorimpactingontherateofdomesticeconomicgrowth.Inassessingoutcomes,thechallengeistodistinguishtheimpactofexternalfactorsfromthatofthegovernmentintervention.Externalfactorsarealsosometimesknownascontextualfactorsorconfoundingfactors.Externalfactormaybepartoftheexternalenvironmentinwhichtheprogramoperates,ortheymaybecharacteristicsoftheclient.Anexampleoftheformeristheimpactofthestateoftheeconomyonthesuccessrateofaprogramfortherehabilitationofinjuredworkersingettingits

24

clientsbackintotheworkforce.Anexampleoftheclientcharacteristictypeofexternalfactorscanbeseenineducation,wheretheeducationallevelsattainedbychildrenaredeterminednotonlybythequalityoftheeducationtheyreceiveatschool,butbyexternalfactorssuchastheirintelligence,nutritionandlevelofparentalsupport.Expresseddifferently,thesetypesofexternalfactorsimpactuponoutcomevariablessuchastheliteracyandnumeracyrates(seethenextsectionfortheconceptofanoutcomevariable).Thismakesitdifficulttocomparetheperformanceoftwodifferentschoolsbysimplylookingatlevelsofstudenteducationalattainment,becausedifferencesinthesocioeconomicmixofstudentpopulationscanmakesuchcomparisonscompletelymisleading.Outputsvs.OutcomesandActivitiesItisusefultoclarifyfurthertherelationshipbetweenoutputsandoutcomes.Tobeconsideredanoutput,anoutputmustbecapableofachievingitsintendedoutcome,butthisdoesnotnecessarilymeanthateachunitofoutputwillbesuccessfulinactuallyachievingtheoutcome.Considertheexampleofamedicaltreatmentofanaccidentvictimwhodiesnotwithstandingreceivingexactlythebesttreatmentinatimelymanner.Itwouldnotbereasonabletosayunderthesecircumstancesthatthepatientdidnotreceiveaservice(output).Thesamepointapplies,say,toeducation,wheredespitereceivingexcellentteaching,certainstudentsmayfailtolearnmuch,whileothersinthecasewilldoverywell.Anassociatedpointisthat,tobeanoutput,aservicemustbecomplete.Considertheexampleofacar,whichisonlyaproduct(i.e.anoutput)onlywhenalltheparts(wheels,engineetc)areattachedanditcanthereforefunctionasintended.Anothergoodexampleistetanusinjections,whereitisnecessarytohaveafullcourse(ofthreeinjections)inordertoacquireimmunity.Thetetanustreatmentoutputsshouldthereforebemeasuredintermsofnumbersofpersonswhoreceivethecompletecourse.Otherwise,onewillendupcountingactivities(e.g.incompletecourseoftetanustreatments)whichareincapableofyieldinganybenefittotheclient/community.PerformanceIndicatorsPerformanceinformationreferstoinformationonresultsachieved,and/orcostsofachievingthoseresults.Therearetwobasictypesofperformanceinformation:performancemeasuresandevaluation.Inthisandthenextcoupleofsections,wefocusonmeasures.Afterthat,

25

evaluationwillbediscussed.Itshould,however,beemphasizedattheveryoutsetthatitisamistaketothinkthatperformancebudgetingisonlyabouttheuseofperformancemeasuresinthebudgetprocess.Performanceindicatorsarequantitativemeasureswhichprovideinformationontheeffectivenessandefficiencyofprogramsandorganizations.Thereisnodifferencebetweenaperformanceindicatorandaperformancemeasurebothtermsareusedinterchangeablyinthismanual.Wealsoneedtobecarefulnottoconfuseobjectives,indicatorsandtargets.AnobjectiveisastatementofwhatoneistryingtoachieveforexamplereducingdeathfromHIV/AIDS.Bycontrast,aperformanceindicatorisquantified(e.g.thepercentageofthepopulationwhichisHIV/AIDSpositive,orthenumberofpersonsdyingannuallyfromHIV/AIDS).Atargetgoesonestepfurtherandsetsapreciseaimtobeachievedbyaspecificdate(e.g.reducingthepercentageofHIV/AIDSpositivepersonsinthepopulationbyatleastonethirdby2020).Performanceindicatorsshouldbeselectedaccordingtotheextenttowhichtheyare: Relevant Representative Costeffective Comparable MinimizeperverseeffectsAnindicatorisrelevantwhentheaspectofperformanceitseekstomeasureisimportanttotheobjectivesofitsusers.Relevancedependsonwhotheuseris.Inselectingindicatorstopresenttoparliamentandthepublic,forexample,indicatorsofinternalprocesses(suchastimetakentofilljobvacancies)wouldnotberelevant.Anindicatorisrepresentativetothedegreetowhichitsucceedsinmeasuringthedimensionofperformancewhichitseekstomeasure.Representativeindicatorsaregoodproxiesforwhattheyaretryingtomeasure.Ameasureofaveragetimetakentoanswerclientphonecallsis,forexample,apoorproxyforthequalityofclientserviceifwhathappensisthatcallsareansweredquicklybuttheclientisleftonholdforanextendedperiod.Anindicatorismorerepresentativethelessitvariesforreasonswhichhavenothingtodowithperformance,suchastheimpactofuncontrollableexternalfactorsandstatisticaluncertainty.Toberepresentative,anindicator

26

shouldalsobeunambiguous.Thatis,itshouldbeclearwhetherachangeintheindicatormeansthatperformancehasimprovedordeteriorated.Anindicatoriscosteffectiveifthebenefitsofusingaspecificindicatorexceeditscostofcollection,processingandverification.Performancemeasurementsystemsarenotcostfree.Likeeverythingelse,theyshouldthereforepassabenefit/costtest.Acomparableindicatorisonewhichisusedbysimilarserviceproviders,orbythesameorganizationinthepast.If,forexample,thereisaninternationalstandardmeasureforaserviceforexample,aswiththemeasuresprovidedinEuropebythecommonsetofsocialpolicyindicatorsthenusingthatmeasuremakesitpossibletomakeperformancecomparisonswithothercountries.Similarly,performancecomparisonsbecomemucheasierifallrelevantentitiese.g.alllocalgovernments,orallhospitalsausethesameindicators.Comparabilityovertimeisusefulbecausecontinuingtousethesameindicatorsthathavebeenusedinthepastmakesitpossibletoidentifytrendsinbeforewhichmightbeimpossibletodetectiftheperformancemeasureshavechanged.Perverseeffects,whicharediscussedfurtherbelow,areadverseconsequencesofperformancemeasures.Examplesare: Inseekingtoimprovemeasuredliteracyandnumeracy,schoolsmightreducetheirfocuson

otherimportantdimensionsofeducation,suchastheteachingofforeignlanguagesorsocialvalues,

Supposethetimelinessofhospitalaccidentandemergencywardtreatmentismeasuredbythepercentageofpatientswhoaretreatedwithin4hours.Thiscanpotentiallyproducetheperverseeffectthatanypatientwhocannotbetreatedwith4hoursmaybekeptwaitingconsiderablylonger,withprioritybeinggiventopatientswhohavearrivedlaterbutwhocanstillbetreatedwithin4hours.

Perverseeffectsareinparttheconsequenceofindicatordesign:forexample,theomissionofsomekeyaspectsofperformancewhich,withimproveddesign,couldhavebeencaptured.Inthecaseofhospitalwaitingtimes,theperverseeffectreferredtowouldbeavoidedbyusingameasureofaveragewaitingtimeinsteadofthemeasureofthosetreatedwithinaparticular(arbitrary)timeperiod.Thesefivecriteriaforgoodperformanceindicatorsareideals,andfewgoodindicatorswillscorehighlyonallofthem.Sometimes,ahighratingononecriterionwillcomeatthepriceofa

27

lowerratingonanother.Outcomemeasuressuchasthecrimerate,forexample,scoreveryhighontherelevancecriteriabutlesswellonrepresentativenessbecausetheyaregreatlyinfluencedbyexternalfactorsbeyondthecontrolofgovernment.Inthissection,wehavefocusedonperformanceconceptsandperformanceinformationingeneral.Inthespecificcontextofperformancebudgeting,however,whatwearereallyinterestedinistheperformanceinformationevaluationsaswellasindicatorswhichisusefulfor,andusableby,budgetdecisionmakers.Animportantpointtobearinmindhereisthatdecisionmakerscanonlyabsorbandusealimitedandcarefullyselectedamountofinformation.Itisthereforeessentialnotonlytobehighlyselectiveinchoosingahandfulofkeyperformanceindicatorsrelevanttobudgetdecisionmakers,butalsotoputtheseindicatorstogetherwithanalysiswhichinterpretsandexplainsthem,drawingoverallconclusionsaboutperformance.Thisanalysisshoulddrawcloselyontheconclusionsofformalevaluationsandotherreviewwork.KeyReadingsRobinson,M(2007),InformingPerformanceBudgetingandResultsInformation,inM.Robinson(ed.)PerformanceBudgeting:LinkingFundingandResults.HMTreasuryetal(2001),ChoosingtheRightFabricaFrameworkforPerformanceInformation,obtainableathttp://archive.treasury.gov.uk/performance_info/fabric.html.RoyalStatisticalSociety(2003),PerformanceIndicators:Good,BadandUgly,obtainableathttp://www.rss.org.uk/pdf/PerformanceMonitoringReport.pdf.

28

4.PerformanceMeasuresandtheBudgetTheindicatorsofmostvalueforperformancebudgetingwilltendtodiffersignificantlyfromtheindicatorsusedforotherpurposes.Whatwillgenerallybeofgreatestvaluetobudgetdecisionmakersindeterminingappropriateprogramfundinglevelswillbeindicatorsoftheresultsachievedbyprogramstheoutcomesthattheyachieveandtheoutputswhichtheydelivertoachievetheseoutcomes.Theywillrepresentonlyasubsetoftheindicatorsusedinternallyformanagerialpurposesbyministries,whichwillincludenotonlyresultsorientedindicators,butalsoactivityandinputindicatorsfocusedontheinternalprocesses,capacitiesandresourcesoftheministry.Therearethreemaintypesofoutputindicators:indicatorsofoutputquantity,indicatorsofoutputquality,andindicatorsofefficiency.Outputquantityindicatorsmeasurethevolumeofserviceprovided.Examplesofoutputquantityindicatorsare: Numberofvaccinationscarriedout, Numberofmalariapronedistrictssprayed, Numberstudentstaughtatseventhgrade, Numberofplanningapplicationsdetermined.Outputqualityindicatorsprovideinformationontheextenttowhichtheserviceisofatypewhichislikelytoachieveitsintendedoutcome.Twoexamplesofmorereadilymeasurablequalityindicatorsaretimelinessindicatorsandclientsatisfactionindicators.Examplesoftimelinessindicatorsare: Averagewaitingtimeofahospitalpatientbetweenarrivalandtreatment, Averagetimeforaplanningapplicationtobedetermined, Averageresponsetimeofthefirebrigadetoafire.Clientsatisfactionmeasurescanbeofvarioustypes,fromsimplemeasuresofthelevelofsatisfactionfeltbytheclient,tomoretargetedmeasuressuchas:

29

Clientratingsofthecourtesyoftheserviceprovider, Thepercentageofclientswhoconsiderthattheyobtainedtheservicewhichtheywereseeking.Efficiencyindicatorsmeasure,ideally,theextenttowhichtheserviceisdeliveredatlowcostwithoutsacrificingquality.Unitoutputcostoneimportanttypeofefficiencyindicator,althoughthisisnotaperfectmeasureasitdoesnotnecessarilyholdqualityconstant.Theunitcostissimplythetotalcostofdeliveringtheoutputdividedbyoutputquantity.Examplesofunitcostmeasuresare: Costpervaccination(includingdelivery), Costperplanningapplicationdetermined, Costpervisaapplicationprocessed.

Laborproductivityindicatorsareanotherimportanttypeofefficiencyindicator.Examplesoflaborproductivityindicatorsinclude: Pensionbenefitapplicationsprocessedperstaffmember, Averagestafftimetakentoadministerapracticaldrivinglicensetest.Inrespecttooutcomemeasurement,thebiggestchallengearisesfromthedistinction,discussedinthelastsection,betweenoutcomesandexternalfactors.Asnoted,outcomesarethechangeincharacteristicofindividuals,socialstructuresorthephysicalenvironmentbroughtaboutbygovernmentintervention,andthechallengeistoseparateouttheimpactofgovernmentinterventionfromthatofotherfactorsbeyondgovernmentcontrol.Inpractice,manyoutcomeindicatorsareinfactmeasuresofwhatcanbecalledtheoutcomevariablethatis,thecharacteristicsofindividuals,societyorthephysicalenvironmentwhichpublicprogramsseektochange.Measuresoftheoutcomevariabledonotdistinguishtheimpactofthegovernmentsinterventionfromthatofexternalfactors.Agoodexampleofthistypeofoutcomeindicatorisacrimeratestatistic(e.g.thenumberofrobberiesperthousandofpopulation).Weknowthatthecrimerateisnotonlyinfluencedbygovernmentinterventionssuchaspolicing,butisalsogreatlyaffectedbysocialandeconomicdevelopmentsbeyondgovernmentcontrol(e.g.thelevelofpoverty).Crimeratestatisticsmakenoattempttoeliminatetheimpactofthelattertypeofexternalfactor.

30

Theperfectoutcomemeasurewouldbeonewhichexcludestheimpactofallexternalfactorsandwhichmeasuresonlythechangeintheoutcomevariablebroughtaboutbygovernmentintervention.Developingsuchmeasuresis,however,exceedinglydifficult.Inafewareas,modernperformancemeasurementtechniqueshavegonequitealongwayindevelopingoutcomemeasureswhichremove,toatleastsomeextent,theeffectsofexternalfactors.Agoodexampleofthisisthesocalledvalueaddededucationperformancemeasures,whichattempttoadjustforstudentcharacteristics.Forexample,valueaddedschoolleaguetablespresentmeasuresofthecomparativeperformanceofschoolswhichadjustforthesedifferencesinstudentpopulation.However,althoughthisandothertechniqueshavebeendevelopedtoobtainbetteroutcomeindicatorswhicharelesseffectedbyexternalfactors,theoverallscopeforeliminatingexternalfactorsfromoutcomemeasuresisquitelimited.Asaresult,thegreatmajorityofoutcomemeasuresareaffected,tovaryingdegrees,byexternalfactors.EnsuringthatIndicatorsareUsefulTobeusefultocentraldecisionmakers,whoinvariablyhavegreatdemandsontheirlimitedtime,programperformanceinformationneedstobereadilydigestible.Acoupleofkeyprogramindicatorsmay,forexample,bemoreusefultotopdecisionmakersthanacomprehensivecompendiumofdozensofindicators.Thedetailisofmoreinteresttoprogrammanagersthantothecenter.Anotableinnovationinthiscontexthasbeenthedevelopmentofsummarymeasureswhichincorporateawiderangeofperformanceinformationintooneormoreoverallperformanceratingsfortheprogram.AgoodexampleofthisisthemeasuresofprogramperformancewhichwasdevelopedundertheUSProgramAssessmentRatingTool(PART)whichoperatedundertheBushpresidency.UnderthePARTsystem,theUSfinanceministry(theOfficeofManagementandBudget)setaboutratingallfederalgovernmentprogramsoveraperiodof5years.Eachprogramwasratedonascaleof4ratings,rangingfromeffectivetonoteffective(therewasalsoaresultsnotdemonstratedrating,usedwherethereisinsufficientinformationtoformajudgment).Thesesummaryratingswereintendedtobemuchmoreinformativeandreadilyunderstoodthanthelargebodyofmoredetailedmeasuresandevaluationswhichunderpinnedthem.Theprogramratings,andthereasoningbehindthem,wereallmadepublic(onthewebsiteExpectMore.gov).Moreover,thePARTsystemwasdesignedfromtheoutsetasatoolforperformancebudgeting,becausetheprogramratingswereusedtoinformthepreparation

31

ofthepresidentsbudgetproposaltoCongress(andwerealsointendedtoinfluenceCongressbudgetdecisions).SummaryperformancemeasuresofwhichPARTisbynomeanstheonlyrecentinternationalexamplehavetheenormousadvantageofreadycomprehensibility.Theycanalsobefarmoreeffectivethanmoredetailedindicatorsinputtingrealperformancepressureongovernmentagencies.Thisshouldnot,however,blindustotheveryconsiderabletechnicalchallengesinvolvedinappropriatelydefiningsummarymeasures.Inappropriateselectionofconstituentvariables,orpoortechnicaldesigninrespecttotheaggregationofthesevariables,canproducesummarymeasureswhicharequitemisleading.LinkingFundingtoOutcomesInconsideringtheroleofperformancemeasuresinbudgetingandfundingdecisions,itisimportanttoexplicitlyconsidercertainkeychallengeswhichariseinlinkingoutcomeand/oroutputstofunding.Thesehaveanimportantbearingonthechoiceofmodelofperformancebudgeting.Westartbyconsideringtheoutcomes/fundinglink,andafterthatlookattheoutput/fundinglink.Inrespecttooutcomes,theimpactofexternalfactorsonmeasuredoutcomevariableshasmajorimplicationsforthetypeoflinkagebetweenoutcomesandfundingwhichitmaybepossibletocreateinaperformancebudgetingsystem.Asnotedabove,mostoutcomeindicatorsdonotseparateorseparateonlytoalimiteddegreetheoutcomesachievedbygovernmentinterventionfromtheimpactofuncontrollableexternalfactors.Whentheinfluenceofexternalfactorsisextensive,itmaybedifficultorevenimpossibletopredicttheoutcomewhichanyparticularlyleveloffundingofapublicservicewillproduce.Thisisamajorobstacletoattempttocreateatightlinkbetweenfundingandoutcomesforexample,bybasingapurchaserprovidersystemonoutcomes.Forthisreason,performancebonusfundingbasedonoutcomestendstobeusedonlytoprovidesmalladditionalfundingrewards(e.g.5percentadditionalfunding),becauseiftherewastoostrongalinkbetweenfundingandunpredictableoutcomes,thefinancialstabilityofthegovernmentagencyconcernedwouldbeadverselyaffected.Atthesametime,outcomemeasuresarecrucialtoprogrambudgeting,andmustconstituteakeyelementinexpenditureprioritizationdecisions.Thisraisesthequestionofoutcomestargetsandfunding.TheUKPSAsystemreliedheavilyonoutcomeindicators,overmanyofwhichthegovernmenthadlimitedcontrol.How,andtowhat

32

extent,canonelinkfundingtotargetswhichareonlypartiallycontrollable?Clearlytherehastobeflexibilityinthefunding/targetrelationshipundersuchcircumstances.Thereare,however,manywhotaketheviewthatflexibilityisnotenoughandwhodisputethewisdomofseekingtolinkbudgetdecisionstooutcometargets.Intheirview,targetsettingingovernmentshouldbeconfinedmainlytooutputsorothervariablesoverwhichgovernmenthasahighdegreeofcontrol(e.g.activities).LinkingFundingtoOutputsThemainfocusinPBsystemsuchasformulafundingandpurchaserprovideristhecreationoflinksbetweenthequantityofoutput(i.e.volumeofservicesprovided)andtheleveloffunding.Formanyoutputsproducedbygovernment,thereisamuchstrongerlinkbetweenfundingprovidedandoutputsdelivered(ordeliverable)thanisthecaseforoutcomes.Thisisparticularlytrueforstandardizedoutputs,whichareoutputswhereeveryclientreceivesprettymuchthesamelevelofservice,sothatunitcostshouldthesame.15However,quiteafewgovernmentservicesarenotstandardized.Theyare,rather,heterogeneousoutputs.Thismeansthatthelevelofserviceprovidedtodifferentclients,orindifferentcases,isdeliberatelyvariedsoastoaddressdifferencesinclientconditionsorcircumstances.Policecriminalinvestigationsareaclassicexampletheamountofeffortputinpercase,evenforthesametypesofcase(e.g.murderinvestigations)variesenormouslydependingonthecircumstancesofthecase.Eveninschooleducation,whichisquitestandardizedforthegreatmajorityofstudents,heterogeneityispresentwhenadditionalteachingandcareactivityisdevotedtochildrensufferinganintellectualorphysicaldisability.Substantialheterogeneityunderminethepredictabilityoftherelationshipbetweenthefundinglevelandtheoutputswhichtheagencycanbeexpectedtoproduce,thusalsoinfluencinghowtightalinkitispossibletocreatebetweenfundingandoutputs.Thisparticularlyaffectsthescopeforapplyingformulafundingandpurchaserprovidersystems,asdiscussedinsection12.Thismeansthattheseformsofperformancebudgetingcanonlybeappliesselectivelytotherighttypesofservices.

15Atleastifweleaveasidecostdifferencearisingfromfactorslikegeography.

33

Thereisoneothertypeofserviceforwhichtightlinksbetweenoutputsandfundingareproblematic.Thisiscontingentcapacityoutputs,ofwhichafiredepartmentisagoodexample.Thefiredepartmentmaintainscapacitytoprovideatveryshortnoticeanoutput(firefighting)forwhichthedemandishighlyunpredictable.Itwouldbeunrealistictoseektobuildaverycloselinkbetweenthenumberoffiresattendedbythefireserviceandtheleveloffunding.Fireservicescannotthereforebefundedonaperoutputbasis,butmustinsteadbefundedinsuchawayastodeliveracertainlevelofcapacitytofightfires.Thediscussiontothispointhasfocusedonoutputquantity.Thereisalsothequestionofpotentiallylinkingfundingtooutputquality.Itwouldbehighlydesirabletobeabletoincludetheoutputqualityaswellasquantitydimensionin,forexample,apurchaserproviderfundingarrangement.Oneoftheconcernsaboutperverseeffectsraisedbypurchaserprovidersystemsisthat,infundingonlyforoutputquantity,onecreatesincentivesforagenciestocutcostsbyreducingquality.Includingaqualitycomponentinfundingcould,inprinciple,resolvethisproblem.Inpractice,however,thisisnoteasy,giventhelimitstoourcapacitytomeasurequalityandtheconsequentlyhighlyimperfectnatureofmostqualitymeasures.Ingeneral,thebesthopeforlinkingfundingtooutputqualityisthroughsomeelementofperformancebonusfundingbasedonqualitymeasures(similartooutcomebonuses)inotherwords,byaddingontoasysteminwhichthemainfundingisbasedonoutputquantityasmalladditionalelementofqualitybasedfunding.Finally,thereisthequestionoflinkingfundingtoactivitiesassuggestedthosewhoadvocateactivitybasedbudgeting.Thiswillbediscussedfurtherinthesessiononformulafunding.Akeyquestionwhicharisesinbasingfundingonactivities,however,isthatofhowfartheactivitieswhicharebeingfundingnecessarilyleadtoresults(outputsandoutcomes)whichthepubliccaresabout.Rememberthatwhileoutputsareservices,activitiesaremerelyworkprocesses(seesection1).Anorganizationwhichisinternallyfocusedratherthanexternallyfocusedonresultsmayputexcessiveemphasisonactivitiesfortheirownsake.Tofundsuchanorganizationpartlyonthebasisofinternalsupportactivitiessuchasmeetingsheldorpolicydocumentsdevelopedcouldworsenthisproblem.Inconclusion,thetightnessofthelinkwhichcanbecreatedbetweenresultsandfundingdependsontheextenttowhichtheunderlyingconnectionbetweenfundingprovidedandtheresultsonecanexpectisclearcut.Iftherelationshipisanuncertainonedueto,forexample,externalfactorsorheterogeneitythenatightlinkmaybeimpossible.Thisiswhy,forexample,

34

itishardtousethepurchaserprovidermechanismtofundorganizationsforoutcomes.Conversely,programbudgetingsystemwillworkevenwhenthereisconsiderableuncertaintyintherelationbetweenresultsandfunding,preciselybecauseprogrambudgetinglinksfundingtoresultsonlyfairlyloosely.Thisiswhyprogrambudgetingiswidelyusedasthebasisforwholeofgovernmentperformancebudgetingsystem,whereasmechanismssuchasformulafundingpurchaserprovidercanonlyworkwhenappliedtocertainsectorsortypesofservices(e.g.hospitalsorschools).Theissuesdiscussedinthissectionalsohaveanimportantbearingontheroleofindicatorsversusevaluation:namely,thattheimperfectionsofperformancemeasuresmeanthatonecannotrelyuponmeasuresalonefortheperformanceinformationtobeusedforexpenditureprioritization.KeyReadingsRobinson,M(2007),ResultsInformationinM.Robinson(ed.)PerformanceBudgeting:LinkingFundingandResults.

35

5.PerformanceMeasurementSystemsTheperformancemeasurement(PM)systemreferstothemechanismsfordatacollection,theprocessingofdataintoindicators,thevalidationofthoseindicators(i.e.ensuringthatthenumbersarereliableandarenoteithermanipulatedorstatisticallyquestionable),andtheirpresentationtousers.APMsystemisasubsetofthebroadermonitoringandevaluationsystem.Indiscussingtheperformanceinformationrequirementsofperformancebudgeting,itneedstobeborneinmindthatperformancebudgetingisnotorshouldnotbeanisolatedreform.Asemphasizedearlier,itispartofasetofbroaderreforms,oftenreferredtoasmanagingforresults,whicharedesignedtofocuspublicmanagementmoreonresultsdelivered,andlessoninternalprocesses.Thesebroaderreformsincludecivilservicereformsdesignedtoincreasethemotivationandincentivesofpublicemployees;organizationalrestructuringtoincreasethefocusonservicedeliveryandimprovecoordination(e.g.creationofagenciesandreductionofnumbersofministries);andinstitutionalchangetostrengthenpublicaccountabilityforperformance.Actionontheseandarangeofrelatedfrontsisnecessaryiftheefficiencyandeffectivenessofpublicexpenditureistobetangiblyimproved.ImprovedperformanceinformationisfundamentaltoeachoftheseelementsofMFRreformbecauseallneedtobeunderpinnedbybetterperformanceinformation.Forexample,acrucialingredientinmakingthecivilservicemoreperformanceorientedistheimprovementofinformationontheactivitiesandoutputsofindividuals,workgroupsandagencies.Thisimpliesthatgovernmentwideperformanceinformationstrategyshouldbedesignedtomeetnotonlytheneedsofperformancebudgeting,butofMFRprocessesmoregenerally.Itisnotappropriate,forexample,toattempttodevelopasystemofperformancemeasurementaimedexclusivelyatbudgetingapplications,andentirelyseparatesetsofmeasuresusedforcivilservicemanagement,accountabilityorotherpurposes.Thegovernmentwideperformanceinformationsystemshouldbedevelopedasanintegratedwhole.Forexample,ifdatacollectionreliesonsurveys,itwillingeneralbemuchbettertouseonesurveytocollectarangeofdifferentinformationfromaparticulartargetgroupthantoconductanumberofseparateuncoordinatedsurveys.

36

Efficiencyintheoperationoftheperformancemeasurementsystemisparticularlyimportantbecausetherearesignificantfinancialandhumanresourcecostsintheproductionofpertinentandreliableperformanceindicators.Thefirststageintheproductionofperformanceindicatorsisdatacollection.Themostreadilyavailablesourceofrawdataforindicatorsisusuallyclientservicerecordssuchashospitaltreatmentrecords,schoolenrolmentandattendancedata,anddataonregulatoryapprovalsgranted.AgoodPMsystemwillneedtherefore,firstly,toimprovethereliabilityofclientservicerecordsand,secondly,toexpandthetypeofdatacollectedinclientsservicerecordssothatitprovidesthebasisforabroadersetofindicators.Clientrecordsprovidesomeinformationaboutoutcomes.Forexample,inschools,examinationandgraduationsarerecorded,andinhospitalsdeathsarerecorded.However,ingeneral,clientservicerecordsprovideinformationmainlyaboutthevolumeofservices(quantityofoutputs)deliveredtoclientsandtheactivitieswhichtheseoutputscomprise.AgoodPMsystemthereforeneedstogobeyondtheconstraintsofclientservicerecordsinordertoconstructmoreandbetteroutcomeandoutputqualitymeasures.Inthisrespect,theuseofsurveys(including,butnotconfinedto,clientsatisfactionsurveys)isthekeyelement.Surveydatacannotonlyprovidegoodmeasuresofoutcomevariables.Itcanalsoprovidemoretimelymeasures,bycapturingoutcomevariableswhichwillonlymuchlatermanifestthemselvesindemandforgovernmentservices.Forexample,surveybaseddataonalcoholandtobaccoconsumptionratesprovidesamuchearlierwarningoftheextentoftheproblemwhichpreventativehealthservicesneedtoaddressthandolevelsofalcoholandtobaccorelateddisease,whicharisesonlyafteryearsofabuse.Surveyscanpickupmeasurevariablesrelevanttooutcomeswhichtaketimetomanifestthemselves.Thesecondstageintheproductionofindicatorsisdataprocessingi.e.thetransformationofrawdataintoperformanceindicators.Akeyquestionhereisthechoiceofprocessingtechnology.Thereisafullrangeofoptionshere,rangingfromthemosthightechthroughtotherelativelysimple.Atoneendofthespectrum,performancemeasurementmaybebuiltintoalargeIntegratedFinancialManagementInformationSystem(seethesectiononprogramcostingandaccountingformoreonIFMISs).Theadvantageofthisifthesystemworksproperlyisthattheperformanceindicatorsaredirectlylinkedtotherelevantfinancialdata(e.g.programperformanceindicatorsareautomaticallylinkedtoprogramexpendituredata;outputquantitymeasurestooutputcostsetc).

37

Attheotherendofthespectrum,itisperfectlypossibletomanageaperformancemeasurementsystembasedonExcelspreadsheetsandanAccessdatabase.Thisisasimplerandlowercostoption.SomeITexpertswillsuggestthatincorporatingperformancemeasuresintheIFMISshouldbethepreferredapproach.However,themorecomplexandmultifacetedIFMISsare,themorechancetheyhaveofneverworking,ornotworkingwell.Thisisparticularlytrueindevelopingcountries.Inmanycountries,thesimpleroptionwillworkbetter.Thethirdstageintheproductionofindicatorsisindicatorvalidation,whichreferstotheassuranceofthereliabilityoftheindicators.Thisisimportanttosafeguardagainsterrorsandmanipulation,aswellasagainstmethodologicaldeficiencieswhichmaketheindicatorsunreliable.Therearetwoaspectsofindicatorvalidation.Thefirstconcernsthevalidationofthecapacityofagencysystemstoensureindicatorqualityforexample,theadequacyoftraininganddataentrychecks.

PerformanceMeasurementProtocolsTheUKRoyalStatisticalSocietyproposedtheideaofaPerformanceMeasurementProtocolinitsexcellent2003reportPerformanceIndicators:TheGood,theBadandtheUgly,suggestingthat:Beforeintroducingperformancemonitoringinanypublicservice,aPMprotocolshouldbewritten.Thisisanorderlyrecordnotonlyofdecisionsmadebutalsoofthereasoningorcalculationsthatledtothosedecisions.APMprotocolshouldcoverobjectives,designconsiderationsandthedefinitionofPIs,samplingversuscompleteenumeration,theinformationtobecollectedaboutcontext,thelikelyperversebehaviorsorsideeffectsthatmightbeinducedasareactiontothemonitoringprocess,andalsothepracticalitiesofimplementation.Proceduresfordatacollection,analysis,presentationofuncertaintyandadjustmentforcontext,togetherwithdisseminationrules,shouldbeexplicitlydefinedandreflectgoodstatisticalpractice.Theotheraspectofindicatorvalidationistheauditofindicatorsthatis,theassessmentofthereliabilityofindicatorsbypersonsseparatefrom,andindependentof,theunitsresponsiblefortheirproduction.Auditmaybeinternal(i.e.carriedoutbyanauditunitwithintherelevantministry),orexternal.Externalmaymeanthattheauditiscarriedoutbyagovernmentwidebodyindependentoftheministrywhichproducestheindicators,oritmaymeangoingonestepfurtherauditbyabodyindependentofexecutivegovernment(thatis,asupremeauditinstitution).

38

Therearetwobroadapproachestotheauditingofperformanceindicators.Thefirstistheauditingofspecificindicators.Theotherapproachisasystemsverification,whichfocusesonauditreviewoftheperformancemeasurementsystems(includingqualityassuranceprocesses)withineachagencywiththeobjectiveofforminganopinionastowhetherthesesystemsaresuchastobelikelytoproducereliableperformanceindicators.Aquiteseparateissuenottobeconfusedwiththeauditofperformancemeasures(whichfocusesonassessingtheirreliability)isthatofexternalreviewoftheappropriatenessoftheindicatorschosenbyspendingministries.Thechoiceofmeasuresforperformancebudgetingpurposescannotbelefttospendingministriesalone,becausetheperformanceinformationhastoservetheneedsofcentralbudgetdecisionmakers.Forexample,ifwhentheenvironmentministryisaskedtocomeupwithkeyperformanceindicatorsforitsconservationprogram,itsuggeststheuseofameasureofthenumberofpolicieswhichitdevelops,itshouldbefirmlytoldthatanactivityindicatorsuchasthisisnotsatisfactory,andthatitneedstodevelopoutcomemeasures(suchasindependentestimatesoftheremainingpopulationsofendangeredspecies).TheMoFand/orothercentralagenciesmustthereforeprovideguidancetospendingministriesaboutthetypesofindicatorstheyneedtodevelop,andmustsubsequentlyreviewingtheappropriatenessofindicatorssuggestedbyministries.Thequestionofwhetherthesupremeauditinstitutionshouldbecomeinvolvedinquestioningthechoiceofindicatorismorecontroversial.Insomecountriesthishappens(e.g.theUS,wheretheGovernmentAccountabilityOfficefrequentlycritiquesindicators).Bycontrast,inBritishlikeparliamentarysystems,itisfrequentlyassertedthattocritiquethechoiceofindicatorswouldbetogotooclosetothecritiqueofgovernmentpolicy,whichissomethingwhichsupremeauditinstitutionsaretraditionallybarredfromdoinginthosesystems.ThefinalstageofthePMsystemisthepresentationoftheindicators.Thisneedstobedoneinsuchamannerastomaketheindicatorsasreadilyunderstandableandusableaspossible.Usabilityfirstandforemostdependsontheselectiveofarelativelysmallnumberofindicatorswhichareofgreatestrelevancetotheuserconcerned.Thereisnothingwhichreducesthevalueofindicatorsmorethanpresentingauserwithhundredsofindicatorsmostofwhichareoflittleinteresttotheuser,butthroughwhichheorsheisobligedtoworkinordertofindwhatisofinterest.

39

Itiscrucialinreportingperformanceindicatorsthattheybeplacedintocontext,ratherthanpresentedsimplyasnumbersinisolation.Themeans,inparticular,theinclusioninaperformancereportnotonlyoftheindicatorsthemselves,butofanarrativediscussionofthetrendswhichtheindicatorsshowand,inassociationwiththat,informationonandanassessmentofextentoftheimpactofrelevantexternalfactors.Abestpracticeinexternalperformancereportingwhichnotallcountrieswillnecessarilybeabletoimplement,givenfinancialconsiderationsistocreateamultitieredsysteminwhichuserscanstartatthebroadestlevelofhigherleveloutcomeindicatorsandthendrilldowntotwoormorelowerlevelsofmoredetailedindicators,attheministryandprogramlevel(manyofwhichwillbemorefocusedonoutputs,activitiesandinput/resources).

NeedforMoreRobustPMSystemsinAfricaAprobleminAfricancountries,andperhapsinsomeotherRegions,isthatalthoughsectorministriescollectarangeofperformanceinformation,thequalityofdataisoftenpoor.Thisispartlybecausetheburdenofdatacollectionfallsonoverworkedofficialsatthefacilitylevel,whomustprovidethedataforotherofficialsindistrictofficesandthecapitalbutwhorarelyreceiveanyfeedbackonhowthedataareactuallybeingused,ifatall.Thisleadstoanotherchickenandeggproblem:Dataarepoorpartlybecausetheyarentbeingused;andtheyrenotusedpartlybecausetheirqualityispoor.Insuchcountriesthereistoomuchdata,notenoughinformation.Thus,anotherlessonfortheinstitutionalizationofagovernmentM&EsystemistheneedtobuildreliableministrydatasystemstohelpprovidetherawdataonwhichM&Esystemsdepend.Anauditofdatasystemsandadiagnosisofdatacapacitiescanbehelpfulinthissituation.Itwouldprovidethestartingpointforanynecessaryrationalizationofdatacollectionsorimprovementsintheirquality.Itwouldgivethedatathecredibilitynecessarytobeused.KeithMcKay(2007),HowtoBuildM&ESystemstoSupportBetterGovernment,WorldBankIndependentEvaluationGroup.

EnsuringthatPerformanceInformationisCostEffectiveThedevelopmentofperformanceinformationsystemsisnotsimplyamatterofdevelopingthebestandmostcomprehensiveresultsandcostinformationpossible.Rather,itisaboutabenefit/costjudgment.Performanceinformationdoesnotcomefree.Itiscostlyinbothfinancialandhumancapacitytermstodesign,buildandthenoperateonacontinuingbasisthesystemsconcerned.Carefuljudgmentsthereforeneedtobemadeabouthowfartogoinrespecttochoicessuchasthenumberofperformancemeasurestobedeveloped,aswellas

40

relatedquestionssuchasthesophisticationofprogramevaluationmethodologyandcostingmethodologies.Thesechoicesfaceeventhewealthiestcountries.Buttheyareparticularlypressingforcountrieswithmorelimitedfinancialandskilledhumanresources.Suchcountriesshouldbeparticularlyselectiveandstrategicinthedevelopmentofperformancemeasures.Asdiscussedlater,theyshouldalso,inmanycases,makeuseofquitesimpleprogramevaluationmethodologies.And,asmentionedabove,theyshouldnotseektodevelopefficiencymeasureswhichrequirerelativelycomplexmanagerialaccounting(e.g.allocationofindirectcoststooutputs).Thetemptationofadoptingwhatappearatthetimetobecuttingedgepracticeswhetheritbeaccrualaccountingandbudgetingatpresent,orpurchaserprovidermodelstenyearsagoshouldalsobestudiouslyavoided.Itisalsoaconsiderablechallengetobuildthecapacitynecessarytooperateagoodperformanceinformationsystem.Forperformancebudgetingpurposesthisdemands,inparticular,greatchangeintheskillsetandcompetencesoftheMoF.Ratherthanbeingandexclusivelyeconomic/accountingbody,theMoFmustdevelopcompetenceinpolicyanalysisandinthedevelopmentofperformanceinformationtosupportthatpolicyanalysis.Onlyinthiswaycanitdevelopthecapacitytoadviseexecutivegovernmentwellaboutexpenditureprioritychoices,inordertomakeeffectiveperformancebudgetingpossible.ImplementationstrategyincludinginrelationtoPMsystemsisdiscussedinsection19.However,itisworthnotingherethatmanydevelopingcountriesareunrealisticintheirexpectationsastohowlongitshouldtaketodevelopacomprehensiveperformancemeasurementsystem.CountrieswhichareparticularlyadvancedinthisareasuchastheUKandUSAtookdecadestogetwheretheyaretoday.Itisaseriousmistakefordevelopingcountrieswhicharerelativelynewtoperformancemeasurementtosetouttodevelopthousandsofprogramindicatorsinveryshorttimeperiods(e.g.23years).Itisfarbettertostartoffmodestly,focusingonarelativelysmallnumberofthemostrelevantindicators.Crucially,muchattentionshouldbepaidtoensuringthatcollection,processingandvalidationsystemsareadequateinrespecttothatselectivegroupofindicators.Thisismuchmorevaluablethangeneratingamuchlargergroupofindicatorswhichcannotbereliedupon.KeyReadings

41

UKNationalAuditOffice(2000),GoodPracticeinPerformanceReportinginExecutiveAgenciesandNonDepartmentalPublicBodies,obtainableathttp://www.nao.org.uk/publications/9900/good_practice_in_performance.aspx.USGovernmentAccountingStandardsBoards(2005)GovernmentServiceEffortsandAccomplishmentsPerformanceReports:aGuidetoUnderstanding,obtainableathttp://www.seagov.org/sea_gasb_project/sea_guide.pdf.

42

6.EvaluationandPerformanceBudgetingPerformancebudgetingisoftenrepresentedasbeingonlyabouttheuseofperformanceindicatorsinthebudget.Thisiswrong,becauseitoverlooksthecruciallyimportantroleofevaluation.Itisforthisreasonthatthroughoutthismanualreferenceismadetotheperformanceinformationbaseofperformancebudgeting,ratherthantoperformanceindicatorsalone.EvaluationisthesubjectofanotherCLEARcourse,andthegeneralnatureandmethodologiesofevaluationarethereforediscussedhereonlyverybriefly.Furtherinformationcanalsobefoundinthereadingslistedattheendofthissection.Ingeneralterms,however,evaluationistheformalassessmentofprograms,projects,organizations,orpoliciesusingsystematicmethodologies,withtheintentionofformingasobjectiveanassessmentaspossibleoftheirefficiency,effectiveness,designormanagement.Onecandistinguishbetweenretrospective(expost)evaluations,whichareevaluationsofprogramsetcwhicharealreadyoperating,andprospective(exante)evaluations,whichareappraisalsofpossiblenewprogramsetcbeforetheyareimplemented.

ADefinitionofEvaluationThesystematicandobjectiveassessmentofanongoingorcompletedproject,programorpolicy,itsdesign,implementationandresults.Theaimistodeterminetherelevanceandfulfillmentofobjectives,developmentefficiency,effectiveness,impactandsustainability.Anevaluationshouldprovideinformationthatiscredibleanduseful,enablingtheincorporationoflessonslearnedintothedecisionmakingprocessofbothrecipientsanddonors.Evaluationalsoreferstotheprocessofdeterminingtheworthorsignificanceofanactivity,policyorprogram.Anassessment,assystematicandobjectiveaspossible,ofaplanned,ongoing,orcompleteddevelopmentintervention.KeithMcKay(2007),HowtoBuildM&ESystemstoSupportBetterGovernment(WorldBankIndependentEvaluationGroup).

Evaluationcansupportthebudgetprocessbyhelpingeitherto: Identifyprogramsorcomponentsofprogramswhichcanpotentiallybecut:thismeans

programswhicharenotcosteffectiveandwhichcannotreadilybemadecosteffectivethroughpolicydesignormanagementchanges,

Identifysavingswhichcanbemadebyimprovingtheefficiencyofservicedelivery.

43

EvaluatingProgramEffectivenessTheevaluationofprogrameffectivenesshasaparticularlyimportantroletoplayinthoseformsofperformancebudgetingwhichfocusontheallocationofresourcesinthegovernmentwidebudget,ofwhichprogrambudgetingisthemostimportantform.Insuchsystems,theprimaryfocusisuponmakingbudgetingasperformanceinformedaspossible.Aswehaveseen,thismeansinparticularthat: Decisionsaboutexpenditureprioritizationwheretoallocatelimitedtaxresourcesare

informedbygoodinformationonprogrameffectiveness,

Decisionsaboutfundingforspecificministriesandagenciesandinparticulardecisionsontheirrequestsforadditionalresourcesareinformedbyreliableinformationonhoweffectivelytheministryoragencyhasusedfundingithasreceivedinpastbudget.

Evaluationiscrucialinthiscontextbecauseperformanceindicatorsarefrequentlyinsufficientinisolationtopermitjudgmentsonprogramoragencyeffectiveness.Aswehaveseen,someprogramoutcomescannotbemeasured,orcanbemeasuredonlyveryimperfectly,andmanyoutcomeindicatorsareheavilycontaminatedbyexternalfactors.Evaluationisveryimportantasameansofmakingjudgmentsaboutthelikelyimpactofexternalfactorsonoutcomevariable.Itcanenableustomakesomeusefuljudgmentaboutprobableeffectivenessofprogramsevenwhenwelackanyoutcomemeasures(e.g.viaprogramlogicevaluationseebelow).Moregenerally,whetherforbudgetaryoranyotherpurpose,performanceindicatorsalonerarelysuffice(seebox).

RoleofEvaluationinaBalancedM&ESystem

ThemostprevalenttypeofperformanceassessmentpracticeisindicatorbasedmonitoringPerformance[indicator]monitoring,inandofitself,representsarelativelycrudewaytoinformdecisionmaking.Inmanycases,thereisaneedforamuchmorenuanced,indepthunderstandingoftheprocessesinvolvedinparticularprogramsorpolicies,whichevaluationsaremuchbetterequippedtoprovide.Giventhespecificstrengthsandweaknessesofmonitoringandevaluation,whicharepotentiallycomplementarytoeachother,theidealapproachisonethatreliesonanappropriatebalancebetweenthetwotypesofactivity.ArielZaltsman(2006),ExperiencewithInstitutionalizingMonitoringandEvaluationSystemsInFiveLatinAmericanCountries,WorldBankIndependentEvaluationGroup.

44

EvaluationofeffectivenessalsoanimportantroletoplayinsystemsofbudgetlinkedperformancetargetslikedtheBritishPublicServiceAgreementssystem.Asnotedearlier,insuchsystemstargetsaresetasanintegralpartofthebudgetprocess,withtheaimofensuringthatthestringencyofthetargetsisrelatedtotheextentoffundingprovided.Aftertheevent,scrutinyofactualperformanceagainsttargetisveryimportantinthistypeofsystem,anditisherethatevaluationcanmakeanimportantcontribution.Forexample,whenthetargetsareoutcometargets,acrucialpartoftheassessmentofperformanceagainsttargetisconsiderationofwhether,say,afailuretomeetatargetisthefaultoftheagencyconcerned,orwhetheritisinsteadduetotheimpactofunanticipatedexternalfactors.Evaluationcanbeveryusefulforthispurpose.Moregenerally,evaluationhasasignificantroletoplayinanyperformancemanagementsystemwhichplacesconsiderableemphasisonthesettingofperformancetargets,irrespectiveofhowcloselytargetsettingislinkedtothebudget.Therearetwotypesofevaluationwhicharerelevanttotheassessmentofprogrameffectiveness: Outcomeevaluations(alsooftenreferredtobyevaluationspecialistsasimpact

evaluations):Outcomeevaluationsaimtodirectlymeasuretheeffectivenessofprogramsbytakingoutcomeindicatorsandusingsophisticatedmethodstoassessthelinkbetweentheprograminterventionandthemeasureschangesinoutcomeindicators,intheprocessexplicitlydealingwithproblemssuchasexternalfactors.Themethodsemployedincludesocalledexperimentaltechniques(whichentailthecomparisonofprogrambeneficiarieswithcontrolorcomparisongroupsattwoormorepointsintime),andtheuseofregressionanalysistoseparatetheeffectofexternalfactorsfromothercausalfactorswhichimpactonoutcomeindicators.

Evaluationsofprogramlogic:theseassesswhethertheprogramisdesignedinsuchawayastomakeitlikelythatitwillachieveitsintendedoutcome.Toevaluateprogramlogic,thefirststepistoclarifyexactlyhowtheprogramissupposedtoachieveitsoutcomes.Expressedintermoftheresultschain,thekeyquestionsare:Whatintermediateoutcomesistheprogramexpectedtodeliver?Howisitthatthoseintermediateoutcomesareexpectedtogenerate,orcontributeto,theprogramsintendedhigherleveloutcomes?Oncetheprogramlogicisclarified,thenextstepistoaskwhetheritisreasonabletoassumethattheprogramwillachieveitsintendedoutcomes.Forexample,givenwhatweknowaboutrelevanteconomictheory,isitreasonabletoassumethataspecificindustrypolicywilldeliveritsintendedoutcomes?

45

EfficiencyReviewTheuseofevaluationtoidentifysavingsfromimprovedefficiencyofservicedeliveryinrelevanttoanyformofbudgeting.Inthiscontext,theformofevaluationwearetalkingaboutisefficiencyreview.Totheextentthatefficiencyreviewsgobeyondrecommendingstepstoimproveefficiencytoprovidingquantifiedestimatesofpotentialsavings,theyareofpotentiallygreatvaluetoministriesoffinance.Efficiencyreviewhasaparticularrelevancetoformsofperformancebudgetingsuchasformulafundingandpurchaserprovidersystems,inwhichfundingisbasedontheunitcostsatwhichthefundingauthoritybelievetheservicedeliveryagencyshouldbeabletoproduceoutputs.Specifically,thefindingsofefficiencyreviewcaninformthatfundingauthorityaboutwhatunitcosttouseforfundingpurposes.Forexample,supposethatthegovernmenthasbeenfundingthehealthministryforvaccinationsattheministryspresentunitcost$20perhead,butthatanefficiencyreviewindicatesthatthiscurrentunitcostisgreaterthanitshouldbeandcouldbereducedby10percentovertwoyears.Underthesecircumstances,itwouldbereasonableforthefundingauthoritytoplantograduallyreducetheamountitpaysto$18.Althoughefficiencyreviewsareusefulforthistypeofperformancebudgeting,evaluationsofeffectivenessclearlyarenot.Thisisbecausethesetypesofperformancebudgetingfocusaredirectedtoimprovingefficiencyratherthanimprovingtheallocationofresources.MakingEvaluationRelevanttotheBudgetInformingthebudgetprocessisnotbyanymeanstheonlyroleofevaluation.Broadlyspending,onecandistinguishvaluationforbudgetarypurposesfromevaluationforpolicy/managementimprovementpurposes.Thelattertypeofevaluationaimstohelpinstitutionsimprovepolicydesignthatis,helpinstitutionstochangethenatureoftheservicestheydelivertothecommunitysoastomakethemmoreeffectiveinachievingtheirintendedoutcomeswithoutfocusingonidentifyingprogramswhichcanbecut.Oritcanaimtohelpinstitutionsimproveprocessesandmanagementsoastomakethedeliveryofservicesmoreefficient,butwithoutafocusonquantifyingbudgetarysavingswhichcanbemadeasaresultofsuchefficiencyimprovements.Agoodperformancebudgetingsystemthereforerequirestheconductofselectedevaluationsspecificallyintendedtoinformthebudgetprocessthatis,designedtogivebudgetdecisionmakersbetterinformationuponwhichtobasebudgetdecisions.Expresseddifferently,itcannotbeassumedthat,simplybecauseagovernmentconductssubstantialevaluationwork,thatthisevaluationworkwillnecessarymeettheneedsofperformancebudgeting.

46

Thishasimplications,firstly,fortheselectionoftopicstobeevaluation.Ifevaluationisgoingtomakeitsmaximumpotentialcontributiontoresourceallocation,theprograms(orelementsofprograms)andtopicschosenforevaluationshouldbethosewhichappearprimafacielikelytoyieldbudgetarysavings.Secondly,budgetlinkedevaluationneedstodeliveritsfindingsquicklyandattherighttimetobetakenintoaccountinbudgetarydecisions.Itisimportantheretobearinmindthatevaluationscanbeconductedingreatdepth,ifdesired,makingextensiveuseofsurveys,interviewandotherdatagatheringtechniques,incombinationwithquitesophisticatedanalytictechniques.Suchindepthevaluationsmaygeneratemorereliableresults,butittendstotakeconsiderabletime,whichmaymeanthatitisnotwellgearedtoservingtheneedsofbudgetdecisionmakers.Becauseoftheimportantoftimelyinformation,considerableemphasisisbeingplacedthesedaysonrapidevaluationsforbudgetarypurposes.Rapidevaluationsareevaluationscarriedoutinquiteshorttimeframes(e.g.36months).Rapidevaluationstendtofocusontheevaluationofprogramlogicratherthanoutcomeevaluation.Asindicatedabove,evaluationsofprogramlogicaimtoassesshowplausibleitisthattheprogramwouldgenerateitsintendedoutcomes.Theyareinprincipleinferiortooutcomeevaluations.However,outcomeevaluationstendtotakeconsiderablemoretimetoproduceandtocostconsiderablymore.Thespeedandlowcostcharacterofrapidevaluationmakesitveryattractiveinpracticaltimes,particularlybutbynomeansexclusivelyinlowincomecountries.Evaluationcanmakeitsgreatestcontributionwhenlinkedtosystematicspendingreview(SR),whichisdiscussedinSection15.SRreferstothesystematicscrutinyofongoingprogramexpendituretoidentifyoptionsforcuts.TheMoFwillingeneralmanageandconductspendingreview,providingadviceonpotentialcutstothepoliticalleadership.Intheprocess,theMoFcanmakebestuseofevaluationbycommissioningrapidevaluationsfocusedonprogramswhichitorthepoliticalleadershipsfeelsmaybeofdoubtfuleffectiveness.EvaluationandtheBudget:TheChileanExampleThebestdevelopedexampleofbudgetlinkedevaluationlinkedtospendingreviewisthatwhichhasbeenprogressivelydevelopedinChilefromthelate1990s.InChile,performancebudgetingispartofabroaderperformancemanagementsystem,knownasthe"systemofevaluationandmanagementcontrol".Thissystemaimstoimprovetheeffectivenessofpolicymakingandmanagementthroughoutcentralgovernment,tocreateperformanceincentivesforcivilservants,andtomakethebudgetresultsoriented.ThedegreeofusewhichChilem