service delivery indicators for strengthening local monitoring of … · 2015. 7. 24. · service...
TRANSCRIPT
Service Delivery Indicators for Strengthening Local Monitoring of Rural Water Service Delivery in Uganda
PoIntS foR actIon
• MinistryofWaterandEnvironment(MWE)toadoptalimitednumberoftheproposedServiceDeliveryIndicators(SDIs)thatfocusontheactuallevelofservicedeliveredtosupplementtheGoldenIndicators.
• MWEshouldtrackadherenceofwaterservicestonationalnormsandstandards
• MWEandrelevantpartnerscarryoutanational-levelsurveyandusethereportsasinputfortheJointTechnicalReviewprocesstoprovideanopportunityforallsectorstakeholderstoassesstheusefulnessoftheSDIs.
• Organiseawarenessraisingandtrainingactivitiesforsectorprofessionalsrepresentingthewholerangeofpotentialusers:MWE,TSUs,DWOs,DonorsandNGOs
This paper presents a framework of service delivery indicators (SDIs) for monitoring rural water services in Uganda. This has been developed for use by the sector to broaden the scope of the existing national monitoring system, the golden indicators, beyond tracking performance of systems to actual services delivered. The SDIs were developed based on sector norms, standards and guidelines set by the Ministry of Water and Environment. This paper describes the process of developing the indicators, and shows how they complement the golden indicators, tracking critical service parameters that were not captured so far, such as reliability, users’ satisfaction or performance of districts. The additional information from SDIs for instance highlights that a district may have a functionality rate of 82% and yet users are only assured of getting water at any time from 70% of the systems.
BRIEfInG PaPER November 2014
Background and rationale for SDIsTheperformanceofUganda’sWater,SanitationandHygiene(WASH)sectorismonitoredthroughtrackingof11“golden”indicators.DataontheseindicatorsiscollectedbyDistrictLocalGovernments(DLGs)staff,undertheguidanceoftheMinistryofWaterandEnvironment(MWE).TheMWEcompilesandanalysesthisinformationintheannualWaterandEnvironmentSectorPerformanceReportwhichispresentedanddiscussedamongsectorstakeholdersduringyearlyJointSectorReviews(SsoziandKerstinDanert,2012).Althoughveryhelpfulfornationalstakeholderstotrackgeneralprogressinthesector,thegoldenindicatorshaveanumberoflimitations:
lBydefinition,theyonlycaptureasmallpartofallissuesrelatedtoruralwater.Outofthe11goldenindicators,7arerelatedtoruralwater,focusingonaccesstowater,percapitainvestments,waterquality,andfunctioningoftheserviceprovideri.Theydon’tincludeinformationonaspectsthattheliteratureonruralwatersupplyidentifiesasbeingimportant,suchasthelikelihoodofsustainabilityofwaterfacilities,servicelevels,users’satisfaction,ortechnicalbackstoppingprovidedtowaterserviceproviders.
lBecauseofthelimitednumberofparameters,theydonotallowmakingcorrelationsthatcouldinformthereasonsbehindtheprogressandtrendsthatthegoldenindicatorstrack.
lTheyhavebeendesignedforusebynationalpolicy-makersanddecision-takers,anddonotprovidesufficientinformationtoallowlocalactorssuchasstaffsofDistrictWaterOffices(DWOs)ofTechnicalSupportUnits(TSUs)iitotakeinformeddecisionsanddirectremedialactionsatlocallevel.
InviewoftheshortcomingsoftheGoldenIndicators,IRC/Triple-SinUgandatogetherwiththeMWEdevelopedServiceDeliveryIndicators(SDIs),withtheaimofsupplementingthealreadywell-establishednationalmonitoringframework.TheSDIsshouldprovideadeeperunderstandingoftheresultsonthegoldenindicators,andthereforeallowenhancedmonitoringofruralwaterservicesdelivery,sothatactioncanbetakenatlocalandnationallevel.
Forthepurposeofthedevelopmentofthisframework,aSteeringCommitteewasestablishedwithmembershipfromgovernmentalandnon-governmentalbodies.ThetechnicaltoolsandplatformaredesignedwithsupportfromMakerereUniversityinKampala.Thispaperpresentsthe
proposedframeworkformonitoringusingtheSDIs,whichencompassesmorethanthemeredefinitionofmonitoringindicators.Itentailsalsothedefinitionoftheprinciplesforthemonitoring,thestructureandcontentoftheSDIsthemselvesandadetailedmethodologyfordatacollection,guidelinesandformatsforanalysisandpresentationofresultsfromthecollecteddata.
Key principles and approach for the development of SDIs in UgandaProcess,scopeandkeyconsiderationsforthedevelopmentofSDIsTheSDIsweredevelopedbycomparingtheGoldenIndicatorstothevariousruralwatersubsectorpoliciesandguidelinesincludingservicedeliverynorms,standardsandprescribedpractices,androlesandresponsibilitiesofstakeholders.Inthisway,weidentifiedgapsnotcoveredbytheGoldenIndicatorsthattheSDIscouldfill.Forinstance,withrespecttoservicelevels,thegoldenindicatorscaptureinformationonwaterquality,accesstoanimprovedfacilityanditsfunctionality.SDIscouldsupplementthat,withdataondistanceofthefacility,quantityofwateraccessed,reliabilityofthefacilityandqualityofthewatersource.
Unit of analysis and scope of applicationThepurposeoftheSDIsistoprovideaquickunderstandingofthesituationoftheruralwatersupplysector,whichaffects–positivelyornegatively–thedeliveryofsustainablewaterservices.Forthis,SDIscanprovidethestatusofservicesatacertainmomentintimeand,whendatacollectionisrepeated,measurechangesovertime.However,theSDIsare
Example of complementarity of SDIs and golden indicators on service delivered
TheMWEsectorperformancereport2013showsthatfunctionalityforruralsystemsinKabaroleandLiradistrictsreach82%and74%respectively.However,preliminaryanalysisoftheSDIsshowsthat,ifwetakeintoaccountreli-abilityofthefacilities,only70%ofruralwatersystemsinKabaroleandLiraarefunctioningandreliableiii.Sincefunctionalityisderivedfromcalculatingthenumberofwaterpointsdeliver-ingwateratthetimeofspotcheck,anindica-toronreliabilityhelpsaddingatimedimen-sion,showingthatactuallyfortwothirdofthefacilitiesusersareassuredtofindwaterattheirwaterpoint.
notmeanttobeusedfortrackingtheperformanceofeverysinglewaterfacility,servicemanagerorserviceauthorityiv.Theyratherprovideinformationofthemaintrendsinabroadergeographicalarea,likeadistrictorsub-county.Theunitofanalysisisthereforethesub-countyordistrict.ThisalsoimpliesthatdataontheSDIsiscollectedonasampleofwatersystems.TheSDIsarealsodesignedinsuchawaythattheyareapplicabletothetwomainruralservicedeliverymodelsfoundinUganda,soastofacilitateaggregation.
GiventhatthemainpersonswhowouldusetheseSDIarestaffofwaterserviceauthorities,datacollectionandanalysistoolshavebeenkeptsimpleandquicktoapply.Forinstance,thedatacollectionprotocolisdesignedinsuchawaythatenumeratorsdonothavetogotohouseholdstoconductinterviews,butratheraskquestionstouserswhocomeandcollectwater.Datacollectionquestionsarekeptassimpleaspossible,oftensimplyrequiringayes/noansweroraresponsethatcanbeselectedfromalistofpre-setoptions.
Table 1. Parameters monitored by Golden Indicators and SDIs for rural water services in UgandaLevel Golden Indicators (GIs) Service Delivery Indicators (incl. sub-indicators feeding into these indicators; (sub)
indicators specific to WSSBs)
Service deliv-ered
Water quality (E. Coli) Water quality (E. Coli; TDS; Turbidity)
Not included in GIs Water quantity (Water quantity delivered; Water quantity accessed)
Access (distance to improved water facility)
Accessibility (Number of users; Distance to water facility; Walking time)
Functionality water facility Reliability (Uptime water facility)
Not included in GIs Quality of water source (Non exposure to pollution; No seasonal variations water qual-ity; No seasonal variations of yield)
Users’ level Not included in GIs Users’ satisfaction with the service delivered (Users’ satisfaction with water quality; Users’ satisfaction with water quantity; Users’ satisfaction with accessibility; Users’ sat-isfaction with reliability)
Not included in GIs Users’ sense of ownership of water facility (Users’ financial contribution to O&M; Cleanliness water facility surroundings)
Service manage-ment level
Management (actively functioning WSC/WSSB)Gender (women in key positions in the WSC/WSSB)
Service manager and operator composition and activeness (Activeness service man-ager; Gender; Viability of scheme operator)
Not included in GIs(although some aspects may be cap-tured under the WSC/WSSB being active)
Service manager’s and operator’s performance of tasks (Financial records; Collec-tion user fees; Cost recovery; Meetings with users; Preventive maintenance; Scheme operator’s reports; Meetings with tap committees; Relationships with service authority and support)
Not included in GIs Service manager and operator internal governance (Records on decision points; Transparency on O&M fund; Transfer water fees to WSSB)
Service author-ity and support mechanisms
Not included in GIs District Water Office (DWO) staffing (No staff DWO)
Equity (Deviation from the District average no. of persons per water point)
District planning (Planned investment based on equity)
Per Capita Investment Cost
Not included in GIs Community mobilisation pre-construction (Signed Memorandum of Understanding; Community capital cash contribution; Land agreement)
Not included in GIs Support and supervision to service managers by service authority and Handpump Mechanics / HPMs (Reactivation of service managers; Responsiveness DWO to major breakdowns; Responsiveness HPMs to breakdowns)
Not included in GIs Support and supervision to service managers by Umbrella Organisation (Support visits; Responsiveness to major breakdowns)
Not included in GIs Construction supervision (Quality construction supervision)
Not included in GIs Monitoring (Use of water facilities’ functionality data; Monitoring of service managers’ activities)
Not included in GIs District coordination (Functionality of District Water and Sanitation Coordinating Com-mittee)
Not included in GIs Responsiveness of TSU (Responsiveness of TSU to request for support from DWO)
Structure and content of the Service Delivery Indicators in UgandaSDIsdescribethewayruralwatersupplyservicesaredeliveredandsupportedacrossthefollowinglevels:
1.Userslevel:Thisreferstotheirsatisfactionwiththeserviceandtheirsenseofownershipofthefacility,measuredthroughproxyindicatorssuchasfinancialcontributiontoO&Mandcleanfacilitysurroundings;
2.Servicedelivered:Coveringwidelyrecognisedserviceslevelparameters–waterquality,quantity,distanceandreliabilityofthefacility(KayseretAl.,2013)–aswellasqualityofthewatersource(i.e.nonexposuretopollutionandseasonalvariations);
3.Servicemanagementlevel:Referringtothecompositionandactivenessofservicemanagers,theirperformanceintheirtasksandresponsibilities,aswellastheirinternalgovernance;
4.Serviceauthorityandsupportmechanismslevel:Focusingonperformanceofserviceauthorityfunctionsbeforeconstruction(planningphase,communitymobilisation),duringconstruction(supervision)andafterconstruction(supportandsupervisiontoservicemanagers,monitoring),aswellasgeneralconditionsfortheperformanceofthesefunctions(DWOstaffing,districtcoordination).
ThedetailedcontentoftheSDIsispresentedinTable1indicatingalsohowtheyexpandontheGoldenIndicators.
Table1highlightshowSDIscanprovideadditionalinformationtothegoldenindicators,differingmainlyinthefollowing:lTheSDIscontainthecompletesetofgenerallyacceptedservicelevelparameters,ascomparedtothethreecoveredbytheGIs
lTheSDIscontaininformationaboutusers’satisfaction,anareacompletelynotcoveredbyGIs
lArguablythebiggestdifferenceliesintheservicemanagementandsupportlevel,wheretheSDIscontainmoredetailedinformation,whereastheseareweaklycoveredbytheGIs.
framework for SDIs data collection and analysis
Sampling StrategyKeepingthecostsoftheapplicationoftheSDIslow
isakeyconsiderationwhendesigningthemonitoringframework.Thisimportantaspectwascentralinthedesignofthesamplingstrategythatworksasfollows:
lServiceauthoritylevel:allserviceauthoritiesaresampled
lServicemanagementandservicelevel:thesearecollectedforwaterpointsbutnotforallinageographicarea.Thedesignedsamplingstrategyallowsasmallsamplesizewhilestillensuringgoodaccuracyoftheresults.Basedonanumberofassumptions,i.e.thatthesub-countywouldbethelowestgeographicalunitforanalysisoftheresults,andthatalowerprecisionisacceptableaslongasaccuracyvishigh,aminimumsamplesizeissetat11waterpoints.Thismeansthatforeachdistrict,datahastobecollectedfromallsub-counties,with11pointsourcesand(whenavailable)at11tapsvisitedineachsub-county.
lUsers.Foreachwaterpoint,10userswhocometofetchwaterareinterviewed;inordertoallowrepresentativenessoftheusers,interviewsareconductedatdifferentmomentsoftheday,andevery5thvisitorofthefacilityisinterviewed.
Thisapproachhastheadvantagesofbeingastandardstatisticallybasedsampling,thatreducesthecostsoffullsurveys,withveryreliableestimatesatthedistrictlevel(within10%pointsconfidenceinterval)andwithin5%pointsconfidenceintervalatthenationallevel.Themaindisadvantagesofsuchasmallsamplesizeisthatthesamplingstrategyneedstobestrictlyadheredto,toensureaccuracyoftheresults,andthatserviceauthoritiescannottakeactionsonspecificwaterpointsthatfalloutsidethesample.
Data collection methodology and toolsTheinformationrequiredfortrackingthesub-indicatorsandindicatorscomesfromvarioussources,sothatthebestinformedindividual/groupisinterviewedforeachspecificissue:
lHouseholds(representedbythepersoncollectingwateratthefacility)forsomeoftheindicatorsrelatedtotheservicedelivered,users’satisfaction,andperformanceoftheservicemanager
lWSC/WSSB(representedbyatleasttwomembersofthecommittee/board)fortechnicalinformationonthewaterpoint,someoftheindicatorsrelatedtotheservicedelivered,users’paymentfortheservice,performanceofsomeservicemanager’stasks,supportprovidedbythe
serviceauthority
lStaffsofsub-countyandDWOforperformanceofsomeoftheserviceauthoritytasks
lUmbrellaOrganisationStaffformattersrelatedtopost-constructionsupporttoWSSBs
Standardmobilephones(non-smartphones)areusedtorecordthedatacollected.Apartforthequestionsdirectedtotheserviceauthorityandsupportinstitutions(DWO,sub-countyandumbrellaorganisation),theentrypointfordatacollectionisthewaterpoint.Mostofthedata,particularlyatthewaterfacilitiesaswellasinterviewswithWSCs/WSSBsandwaterusersiscollectedbylocalHandpumpmechanics(HPMs)whoarefamiliarwiththeareasandlocationofwaterpoints.
Inaddition,mostofthemhavealreadybeeninvolvedincollectingdatafortheDWOtofeedintothegoldenindicatorsandalreadyareconversantwithdatacollectionusingmobilephones.TheHPMsparticipatedinatwodaytrainingonthedatacollectionprotocolandonhowtousemobilephonestocollectdata.Forthefirstroundofdatacollection,whichtookplaceinDecember2013,theinformationfromstaffsoftheDWOs,sub-countiesandumbrellaorganisationswasgatheredbyTriple-Sstaff.
Thedatacollectedisthendirectlystoredonanonlineplatformthatisaccessibletofieldsupervisorstomonitorinrealtimeincomingdata.Theplatformisnowaccessibletoallrelevantstakeholders.Dataprocessing:calculationandpossibleaggregationoftheSDIsvalues
Eachsub-indicatoristrackedthroughoneorseveralmeasurableparameters(alsoseeAdanketAl.,2013).Foreachwaterpoint,avalueisallocatedtoeachsub-indicator:l“0”whentheresponseisnegative,orunderaset
benchmark,or
l“1”whentheresponseispositive,orreachingthebenchmark.
Thetablebelowshowsexamplesoflinksbetweensub-indicatorsandparametersandhowscoringisdone.
Valuesforeachsub-indicatorarecalculatedatthelowestgeographicalunitforanalysis,i.e.thesub-countyfordataonusers,servicedeliveredandservicemanagementandthedistrictfortheonesonsupportauthority.Itisobtainedbycalculatingtheaverageofthescoresforthe11waterpoints(pointsourcesortaps),convertedintoapercentage.Forinstance,ifforthesub-indicator“Walkingtime”,7waterpointsscoreda“1”and4a“0”,thevalueofthesub-indicatoris64%.Eachsub-countyhenceobtainsavalueforeachsub-indicatorandcanbecomparedtoothersub-counties.
Thevalueofagivenindicatorissimplyobtainedbycalculatingtheaverageofthevaluesofthecorrespondingsubindicators.Furtheraggregationcanbedonetoobtainavalueforeachindividuallevelofservicedelivery(servicedelivered,users’satisfactionandsenseofownership,servicemanagers,andserviceauthorityandsupportmechanisms).Forinstance,thevalueforservicedeliveredistheaverageofthevaluesforthe5individualindicators(Waterquality,Waterquantity,Accessibility,ReliabilityandQualityofwatersource).Again,thevalueobtainedinasub-countyoradistrictcanbecomparedwiththeonesobtainedforotherareas.Resultscanbepresentedforpointsources,forpipedschemes,orforallsourcestogether.Aggregationscanalsobedoneforvariousgeographicallevels:sub-county,district,ornational.
Data analysisAnalysisofdatacollectedinDecember2013shallbefinalisedinApril2014.Resultsforeachsub-indicator,eachindicatorandeachlevelofservicedeliverywillbeobtainedthroughdownloadingofshort(1-2pages)automaticallygeneratedPDFreportsthatpresentkey
Table 2. Sample of the water Service Delivered Indicators & scoring system at facility level
Indicator Sub Indicator Measurable parameter
Score
Accessibility No of users of water facility
No of regular users registered by the committee
No. users ≤ norm = 1No. users > norm = 0(norm is 300 users for a borehole, 200 for a shallow well or a protected spring, and 150 for a tap)
Distance to water facility
Distance between household and water facility for 10 users
At least 80% of users are within 1 km = 1Less than 80% of users are within 1 km = 0
Walking time Time spent (minutes) by 10 users for a round trip to the facility
At least 80% of users do a round trip within 30 min = 1Less than 80% of users do a round trip within 30 min = 0
graphsandtablesandareaccessibleonline.Oncethesereportsareaccessible,staffsofDWOsandsub-countieswillinterpretthemwithsupportfromTriple-SandtheMWE.ThejointinterpretationofwhattheSDIsmeanshallleadserviceauthoritiestothedesignofremedialactionsbasedontheidentificationofgapsandissuesinservicedelivery.
Way forward and conclusionThispaperhaspresentedanadditionalframeworkformonitoringruralwatersuppliesthatcomplementsthewell-knownGoldenIndicators.ThetestingoftheSDIshasshownthattheyhavepotentialtogenerateinformationonactualservicelevels,reliabilityoftheservice,usersatisfactionandperformanceofserviceauthoritiesthatisnotcapturedbythegoldenindicators.AsaresulttheSDIsprovideopportunityfor
identifyingissuesandgapsacrossthe‘entireservicedeliverychain’thathavebeenoverlooked.
Thepapershowsevidenceonhowtrackingreliabilityofruralwatersystemstellsthewaterusers’storyhiddenbehindfunctionalityfigures.Despitetheadvancementintheprocessofdevelopingtheindicators,thereisstillcontentionamongsectoractorsthatparameterstrackedbySDIsareverymanyandthatthesystemmaynotbeeasytoreplicatebygovernmentgivenitsavailableresources.Theongoingprocessofanalysisofdatafromthefirstroundofdatacollectionwillbeusedtofurtherprioritisetheindicatorstoselectthosethatproducethemostrelevantandactionabledata.
References
1. Adank,M.,Smits,S.,Bey,V.,Verhoeven,J.andPezon,C.(2013)Developmentanduseofservicedeliveryindicatorsformonitoringruralwaterservices.IRCSymposium2013:MonitoringSustainableWASHServiceDelivery(http://www.irc.nl/page/79345).IRC:TheHague.
2. Kayser,G.L.,Moriarty,P.,Fonseca,C.andBartram,J(2013),DomesticWaterServiceDeliveryIndicatorsandFrameworksforMonitor-ing,Evaluation,PolicyandPlanning:AReview.InternationalJournalofEnvironmentalResearchandPublicHealth,2013,Vol.10,pp.4812-4835.
3. Ssozi,DisanandKerstinDanert(2012),NationalMonitoringofRuralWaterSupplies.HowtheGovernmentofUgandadiditandlessonsforothercountries.FieldNoteNo2012-4,RuralWaterSupplyNetwork.
About IRC
IRC is an international think-and-do tank that works with govern-ments, NGOs, entrepreneurs and people around the world to find long-term solutions to the global crisis in water, sanitation and hygiene services. At the heart of its mission is the aim to move from short-term interventions to sustainable water, sanitation and hygiene services.
With over 45 years of experience, IRC runs projects in more than 25 countries and large-scale programmes in seven focus coun-tries in Africa, Asia and Latin America. It is supported by a team of over 80 staff across the world.
For more information please visit www.ircwash.org
Visiting addressAqua Complex, 1st FloorPlot 2220 Port Bell Road, Kitintale
Postal addressP.O Box 40398Kampala, Uganda
tEL: +256 758 [email protected]/uganda
End Notes
i. There are two main service delivery models for rural water in Uganda: point sources (boreholes, shallow wells and protected springs) and piped schemes. The service manager (also called service provider) is the body that manages and delivers the water service to a defined population in a defined service area, taking care of operation, maintenance and administration (book keeping, tariff collection, etc.) of the system. For rural water services in Uganda, the service manag-ers usually are community-based Water Source Committees (WSCs) for point sources and Water Supply and Sanitation Boards (WSSB) for piped schemes.
ii. TSUs are regional units established by the MWE to build capacity and offer back-up support to district local governments in their fulfil-ment of roles and responsibilities as regards to WASH services.
iii. A water point is considered as reliable when it provides water 95%
of the time or only breaks down for a maximum period of 2 weeks in the whole year.
iv. The service authority is the body with legal responsibility for guaranteeing a water service in a defined area, fulfilling functions such as planning, coordination, oversight of services. A service author-ity usually is responsible for technical assistance to service managers although this can be contracted out. In Uganda, service authority functions are split between two administrative levels: districts and sub-counties.
v. Accuracy is the trueness of the result, the proximity of measure-ment results to the true value; this should not be confused with preci-sion, which consist of the repeatability or reproducibility of the meas-urement. Accuracy is determined by the sampling strategy which sets how representative the sample is, while precision is largely determined by the sample size and to some degree to the sampling strategy.