summative phase report -...

i SUMMATIVEPHASEREPORT- THE DISASTERS & EMERGENCIES PREPAREDNESS PROGRAMME

EVALUATION

3 MAY 2018

SUMMATIVE PHASE REPORT

ii EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT

Phuong Pham, PhD, MPH*Vandana Sharma, MD, MPH*Rebecca Hémono, MPHJessica Jean-Francois, MUP, M.S. EdJennifer Scott, MD, MPH

With contributions from:Root Change, Valsa Shah, MS, and Piotr Bialowolski, PhD

*Co-leads for the Report

Suggested Citation:

Pham P, Sharma V, Hémono R, Jean-Francois J, Scott J. (2018) The Disasters and Emergencies Preparedness Programme Evaluation: Summative Phase Report, May 3 2018. Harvard Humanitarian Initiative.

© Harvard Humanitarian Initiative, 2018The views presented in this paper are those of the authors and do not necessarily represent the views of the DEPP programme, its partners or donor.

All rights, ownership and copyright of data and materials are retained by the Harvard Humanitarian Initiative.

Report design © Alice Hale / Action Against Hunger

1 SUMMATIVEPHASEREPORT-TAblEOfcOnTEnTS

ACRONYMS 2

ACKNOWLEDGEMENTS 4

EXECUTIVE SUMMARY 5 InTRODUcTIOn 6 MAiN FiNDiNGS 12 cOnclUSIOnSAnDREcOMMEnDATIOnS 18

cHAPTER1INTRODUCTION AND CONTEXT 32

cHAPTER2 EVALUATION METHODOLOGY 49

cHAPTER3RELEVANCE AND VALIDITY OF DESIGN 79

cHAPTER4RELEVANCE AND EFFECTIVENESS OF INTERVENTIONS 93

cHAPTER5EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS 128

cHAPTER6EFFICIENCY AND VALUE FOR MONEY 147

cHAPTER7SUSTAINABILITY AND LIKELIHOOD OF IMPACT 164

cHAPTER8HUMANITARIAN PREPAREDNESS AND RESPONSE CASE STUDIES 183

cHAPTER9CONCLUSIONS AND RECOMMENDATIONS 208

TABLE OF CONTENTS

2 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT

ACRONYMS

ADCAP AgeandDisabilitycapacityProgramme

CDAC-N communicatingwithDisasterAffectedcommunitiesnetwork

CLA countrylearningAdvisor

CSO civilSocietyOrganisation

CWC communicatingwithcommunities

DAC DevelopmentAssistancecommittee

DEPP DisastersandEmergenciesPreparednessProgramme

DFID DepartmentforInternationalDevelopment

DRC DemocraticRepublicofthecongo

DRR DisasterRiskReduction

EWEA ImprovedEarlyWarning-EarlyActions toStrengthenDisasterPreparednessEthiopia

FE financialEnablersProject

GBV Gender-basedViolence

HERR HumanitarianEmergencyResponseReview

HHI HarvardHumanitarianInitiative

IDI In-depthInterview

IDP InternallyDisplacedPerson

INGO Internationalnon-governmentalorganisation

IRB InstitutionalReviewboard

KAP Knowledge,AttitudesandPractices

L/NNGO local/nationalnon-governmentalorganisation

LPRR linkingPreparednessResilienceandResponse

M&E MonitoringandEvaluation

MEL Monitoring,Evaluationandlearning

NAHAB nationalAllianceofHumanitarianActorsbangladesh

NGO non-governmentalorganisation

NHN nationalHumanitariannetwork

NPAC nonProjectAttributablecosts

OCHA UnitednationsOfficeforthecoordinationofHumanitarianAffairs

PHEP Gambella PublicHealthEmergencyPreparednessinGambella

3 SUMMATIVEPHASEREPORT-AcROnyMS

PIP ProtectioninPractice

ROI ReturnonInvestment

SEPS StrengtheningEmergencyPreparednessSystemsinMyanmar

START STARTnetwork

STP ShiftingthePower

T1 TimePoint1(formativephase)

T2 Timepoint2(interimphase)

ToR TermsofReference

TD TalentDevelopment

TSC TransformingSurgecapacity

UEWEA UrbanEarlyWarningEarlyAction

UK UnitedKingdom

UN Unitednations

VAWG ViolenceAgainstWomenandGirls

VFM ValueforMoney

WASH Water,SanitationandHygiene


ACKNOWLEDGEMENTS TheHarvardHumanitarianInitiative(HHI)evaluationteamextendsthankstoActionAgainstHunger, who commissioned this report, and Dr. Robina Shaheen and Hannah Wichterichwhomanaged theexternalevaluation.Wealso sendourdeepgratitude toniamhGibbons(managementsupport),Rootchange(networkAnalysis),ValsaShah(ValueforMoney),andPiotrbialowolski,Ph.D.(Statisticalanalyses)fortheircontributions.Wethankthemanyparticipantsinthisevaluation,includingcountry-basedDisastersandEmergenciesPreparednessProgramme(DEPP)projectteams,headquarters-basedDEPPstakeholdersandthose involved inDEPP’sleadership and governance.We are also grateful to communitymembers inMyanmar andEthiopiawhotookthetimetorespondtohouseholdandcommunitysurveys.

Particular thanks are due to our research partners who implemented data collection andcontributedtheirexpertiseandknowledgeinspecificcountriesandelementsoftheevaluation:thebusaracenterforbehavioralEconomics(Kenya),TheEthiopianPublicHealthAssociation(Ethiopia),InnovationsforPovertyAction(Myanmar,thePhilippines).WethankT-consinSouthSudanfortheirwillingnesstopartnerwithus.

WeareappreciativeofthetimeandeffortputinbytheleadershipandresearchadministrationteamsattheHarvardHumanitarianInitiative,bethIsraelDeaconessMedicalcenterandtheHarvardT.H.chanSchoolofPublicHealth,whoseon-goingsupporthasbeencrucialtothisevaluation.lastly,wewouldliketoacknowledgeourinterns,SarahThang,Julianneylan,AminaGoheer,noorZanial,MichaelDeStAubinandGinacianconefortheirvaluablesupportonthisreport.

5 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy

EXECUTIVESUMMARY


INTRODUCTION Thisreportprovidesthesummativeresultsfromthethree-yearexternalimpactevaluationoftheDisastersandEmergenciesPreparednessProgramme(DEPP)conductedbyateamattheHarvardHumanitarianInitiative(HHI).TheDEPPwasa£40millionprogrammefundedbytheDepartmentforInternationalDevelopment(DfID)thataimedtostrengthenskillsandcapacityand improvethequalityandspeedofhumanitarianresponse incountriesthatareatriskofnaturaldisastersoremergencies.

The DEPP was delivered by two non-governmental Organisation (nGO) consortia, theSTARTnetwork,whichreceived£27million,andthecommunicatingwithDisasterAffectedcommunities (cDAc-n) network from 2015-2018, which received £3 million. In addition,£10millionwasreservedforan innovationwindowwhichwasseparately implementedandevaluated. TheDEPP comprised14 individual projects implemented in one ormore of 101 priority countries: South Sudan, Kenya, Ethiopia,Mozambique,Democratic Republic of thecongo(DRc),Jordan,Pakistan,bangladesh,MyanmarandthePhilippines,witheachprojectoperating in somebutnotallof thecountries.Onlyoneproject (thelearningProject)wasimplementedinall10countrieswhilefiveprojects(PublicHealthEmergenciesPreparedness(PHEP)inGambella,ImprovedEarlyWarningEarlyActions(EWEA)inEthiopia,StrengtheningEmergencyPreparednessSystemsinMyanmar,UrbanEarlyWarningEarlyAction(UEWEAinKenya),financialEnablersinthePhilippines)wereimplementedinonlyonecountryeach.The14DEPPprojects,theirconsortiamembers,locationsandbudgetsarepresentedinTable1.

The Disasters and Emergencies Preparedness Programme (DEPP)

• Investment:£40million

• Location: 10countries

• Duration: 3years

• Number of preparedness projects:14humanitariancapacitybuildingprojects

• Overall objective:Toimprovequalityandspeedofhumanitarianresponseincountries thatareatriskofnaturaldisastersorhumanitarianemergencies

Key approaches: capacitybuildingoflocalandnationalhumanitarianstaffandcommunities,earlywarning systemdevelopment, supportingcollaborativeactionand strengtheningnet-works

DEPP PROJECTPROJECT

CONSORTIA MEMBERS

COUNTRIES** DESCRIPTION BUDGET (£)

TALE

NT

DEV

ELO

PMEN

T (T

D)

SavethechildrenUK(lead),OxfamGb,ReliefInternational,cHSAlliance

bangladesh,DRc,Ethiopia,Jordanand

Kenya,lebanon

Toproducehigh-qualityprofessionalsatalllevelswhoarebetterequippedtotackletheissuessurroundingcomplexemergencies,helpingtoensurethattherightpeopleareintherightplacedoingtherightthingstoassistdisaster-affected

communities.

5,985,087

1 Indonesiawasinitiallyselectedasan11thDEPPfocalcountry,butnoselectedprojectsplannedtoimple-mentactivitiesthere.


SHIF

TIN

G TH

E PO

WER

(STP

) ActionAidInternational(lead),cAfOD(lead),christianAid,Tearfund,concernWorldwide,

Oxfam

bangladesh,Pakistan,

Ethiopia,KenyaandDemocraticRepublicofcongo(DRc)

Tosupportlocalactorstotaketheirplacealongsideinternationalactorsinordertocreateabalancedhumanitariansystemthatismoreresponsiveandaccountabletodisaster-affected

communities.

4,876,637

THE

LEAR

NIN

G

PRO

JECT

*

ActionAgainstHungerAll DEPP countries

Toevidencetheextenttowhichpreparingpeopleisaneffectiveandefficientapproachtodisastermanagementandbrokerinternalandexternalrelationshipsforlearningaboutwhatisandisnot

workingincapacityexchange.

3,343,375

BETT

ER D

IALO

GUE,

BE

TTER

IN

FORM

ATIO

N,

BETT

ER A

CTIO

N

(CD

AC-N

PRO

JECT

)

WorldVision(lead),cDAcnetwork,ThomsonReuters

foundation,bbcMediaAction,Internews

bangladesh,SouthSudan,ThePhilippines

Toensurethattwo-waycommunicationisapredicable,coordinatedandresourcedcomponentofhumanitarianresponseinordertocontributetoimprovementineffectivedeliveryofassistanceto

disasteraffectedcommunities.

3,000,000

TRAN

SFO

RMIN

G

SURG

E CA

PACI

TY

(TSC

)

ActionAidInternational(lead),Acf,christianAid,cAfOD,cARE,IMc,IslamicRelief,MuslimAid,Plan,Savethechildren,Tearfund

Pakistan,thePhilippines,RegionalHubinbangkok,Thailand

Tostrengthencivilsocietysurgecapacityatinternational,regionalandlocallevels,contributingtoadiverseanddecentralisedthirdsectorpillarbetterabletocomplementexistingUnited

nations,Redcrossandgovernmentstructuresinordertohelpcommunitiesincreaseresilience,

reduceriskandimprovecrisisresponse.

2,482,824

ALER

T

HelpAgeInternational(lead),care

international,HandicapInternational,IslamicReliefWorldwide,concernWorldwide,Oxfam,coventryUniversity

ThePhilippines,Pakistan,bangladesh,Kenya,

Mozambique,Somalia,Haiti

Todevelopasystemofapproachesandtoolsfordisasterpreparednessthatincreasestheabilityoforganisationstorespondimmediately,effectively

andappropriatelywhenadisasterstrikes.

1,987,000

FIN

ANCI

AL

ENAB

LERS

(F

E) OxfamGb(lead),Tearfund,christianAid

ThePhilippines

Totransferhumanitariancapacity,autonomyanddecision-makingtoorganisationsclosertopeopleaffectedbycrisis,asawayoffacilitatingmore

effectiveandappropriateaid.

1,637,745

AGE

AND

D

ISAB

ILIT

Y CA

PACI

TY

BUIL

DIN

G

(AD

CAP)

HelpAgeInternational(lead),cbM,

DisasterReady.org,HandicapInternational,IfRc,OxfordbrookesUniversity,RedRUK

Kenya,Pakistan

Toensureolderpeopleandpersonswithdisabilitybenefitfromimprovedaccesstoservices,asaresultofrecognitionbyhumanitarianactorsoftheirspecificneedsandincreasedcapacityamongsthumanitarianactorstodeliverinclusive,

accessibleandappropriateresponse.

1,045,159

LIN

KIN

G

PREP

ARED

NES

S RE

SPO

NSE

AN

D

RESI

LIEN

CE (L

PRR)

christianAid(lead),ActionAid,concern,HelpAgeInternational,King’scollegelondon,MuslimAid,Oxfam,Saferworld,World

Vision

bangladesh,Kenya,DRc,Pakistan,thePhilippines,colombia,Indonesia***,Myanmar

Todesignandrolloutprogrammingapproacheswhichstrengthentheresilienceofpeoplelivinginfragilestatesandbeneficiariesofhumanitarian

assistance.

1,002,964

STRE

NG

THEN

ING

EM

ERG

ENCY

PR

EPAR

EDN

ESS

SYST

EMS

IN

MYA

NM

AR (S

EPS) christianAid(lead),

DcA,RegionalIntegratedMulti-HazardEarlyWarningSystemforAfricaandAsia

(RIMES)

Myanmar

Tostrengthenthelinksamongnational,subnationalandlocal-levelpreparednessin

Myanmar,withaprimaryfocusoncapacitatinglocalcommunitiesandstructurestoaccessinformationandlinkwiththeon-going

establishmentofpreparednessandearlywarningsystemsinthecountry.

925,698


PRO

TECT

ION

IN

PRAC

TICE Oxfam(lead),

IRc,WorldVisionInternational

DRc,lebanon,Turkey,

SouthSudan,Myanmar,PakistanandthePhilippines

ToenablenationalnGOstoimplementprotectionactionsindisasterandconflictresponses,developnewtypesofpartnershipsandcollaborationsbetweenprotectionactorsandinfluencethe

internationalprotectionarchitecturesoitismoreinclusiveofnationalnGOs.

870,137

IMPR

OVE

D EA

RLY

WAR

NIN

G – E

ARLY

AC

TIO

NS T

O ST

REN

GTH

EN

DIS

ASTE

R PR

EPAR

EDN

ESS

IN ET

HIO

PIA

(EW

EA)

OxfamGbonbehalfofAfricacllimatechangeResilienceAlliance(lead),christianAid,nationalDisasterRiskManagement

commission,nationalMeteorologicalAgency

Ethiopia

Tocontributetoimprovedemergencypreparedness,timelinessandqualityofrisk

informationandresponseactionsbycommunity,government(localtofederal)andnGOsina

coordinatedmanner

784,255

URB

AN E

ARLY

W

ARN

ING

, EAR

LY

ACTI

ON

(UEW

EA)

concernWorldwide(lead),Oxfam,KenyaRedcrossSociety

Kenya

Toimproveurbanearlyactionbyimprovingthealignmentoflocalandmunicipalgovernanceinstitutionsandresponseagenciestowardsidentifyingrelevanttriggersandimplementingearlyresponsebasedonthese,supportedbyanincreaseintheallocationoffinancialsupportfromkeydonorstowardsurbanEarlyWarningEarly

Action.

675,000

PUBL

IC H

EALT

H

EMER

GEN

CIES

PR

EPA

RED

NES

S IN

GA

MBE

LLA

(P

HEP

)

christianAid(lead),AmrefHealthAfrica,EthiopiaMinistryofHealth,nationalMeteorologyAgency

EthiopiaTostrengthenearlywarning,preparednessandpromptresponseofthehealthsectortopublic

healthemergencies.548,694

Table1:ProjectsintheDisastersandEmergenciesPreparednessProgramme2

*Formerly known as the Monitoring, Evaluation and Learning Project (MEL).

**Includes non-DEPP focal countries where projects were implemented (i.e., Lebanon, Somalia, Haiti, Colombia, Turkey).

***Indonesia was initially selected as an 11th DEPP focal country but none of the selected projects planned implementation in Indonesia. However, over the course of the programme, LPRR ultimately did implement some activities in this country.

TheDEPP’slogicalframeworkoutlinedthefollowingoutputs,outcomesandimpactindicatorsforthethree-yearprogrammepresentedinTable2below.

STATEMENT

OU

TPU

TS

1.Improvedknowledgeandunderstandingofindividualsbysharingbestpracticeofhumanitarianpreparednessandresponse

2.Improvedpreparednesssystemsforearlyactionwithcommunitiesatriskofdisasters

3.Increasednumberofcoalitions,partnershipsandnetworkswhichworkingtogether,areabletoaddresshumanitarianneedsinawiderangeofemergencysituations

4.Improvedinstitutionalarrangementsandpolicyenvironmentssothatnationalsystemsforhumanitarianresponseandpreparednessarebettersupportedandmoresuitable

2 TheSTARTnetwork.DisasterandEmergenciesPreparednessProgrammeInfographic.August2015.Accessedon18february2016athttp://www.start-network.org/wp-content/uploads/2015/08/DEPP-In-fographic.pdf.


OU

TPU

TS 5.Strengthenedevidencebaseforwhatworkstohelpbuildhumanitariancapacityatscale

6.Human-centereddesignapproachestestedandimplementedacrossfourinnovationslabs,leveragingnon-traditionallocalactorstodeliverpreparednessapproacheswhicharecommunity-appropriate,leadingtomorepreparedandengagedcommunities

OU

TCO

MES

1.IncreasedandstrengthenedemergencypreparednesscapacityinDEPPcountries,focusingonstrengtheninglocalhumanitariancapacityandchampioninglocalisationinamannerconsistentwiththeGrandbargainandtheWorldHumanitarianSummit

IMPA

CT

1.DEPPreducessufferinganddeliversbetterhumanitarianservices,throughimprovedpreparedness,forpeoplearoundtheglobeaffectedbydisastersandemergencies

Table2:SummaryofDEPPprogramme-levellogicalframework

Thefinalversionoftheprogramme-level logical frameworkwasapprovedbyDfID in2017and fully implementedbyAugust2017, severalyearsafter thestartof theprogrammeandthecommencementoftheexternalevaluation.Whiletherewereseveraldraftversionsoftheprogramme-levellogicalframeworkinplacebeforethen,theyrequiredsignificantstrengthening3.ThereforewiththeagreementfromDfID,theevaluationwasnotdesignedinconjunctionwiththelogicalframeworkanddoesnotreportagainstit(Seebelowforfurtherdetails).Inaddition,outputarea6intheprogrammelogicalframeworkrelatestotheinnovationwindow,referredtoastheDEPPInnovationlabs.Thoughoriginallyconceivedtobeimplementedsimultaneouslywiththe14DEPPprojects,duetodelays,itsimplementationbeganrecentlyandwillcontinuefortwoyears,endingin2019.Asitwasnotimplementedduringthetimeframeoftheexternalevaluation,DEPPInnovationlabswasnotpartofthisexternalevaluation.Aseparateevaluationhasbeencommissioned.p

DEPP THEORY OF CHANGE AND SIMPLIFIED CAUSAL CHAINDuring theevaluation inceptionphase, theevaluation teamassessed theDEPPprogrammetheoryofchange(Annex3)presentedinthebusinesscase(Annex1).Theprogrammetheoryofchangewasfoundtobecomplexandnolongerrepresentedwellthesuiteofprojectsastheyhadevolvedovertime.AprocesswasundertakentocreateamoresimplifiedbutconceptuallyappropriaterepresentationoftheDEPPthatcouldserveasthebackboneoftheevaluationapproachand to reportagainstduringeachevaluationphase.figure1 illustrates theDEPPprogramme causal chain thatwas developed and thatwas used to develop a programme-level logical frameworkandsetof indicators for theevaluation4.Thecausalchain illustrates

3 forexample,DfID’sAnnualReviewoftheDEPP,March2018states:“Thereisstillmoreworktodoonthelogframe–includingfinalisingtheoutcomeindicatorsandincludinganoutputwhichtracksvalueformoney.”

4 Page17oftheDEPPbusinesscasestated,“Throughacommissionedevaluation,wewilldeveloptheTheoryofchangeinmoredetailaspartoftheevaluationinceptionstage,andwewilltestourcoreassumptionsthroughtheDEPPevaluation.”TheevaluationteamrevisedtheTheoryofchangetoenableafeasibleevaluationdesignasmandatedintheinceptionphaseandtheTermsofReference.TherevisedcausalchainwasapprovedbytheEvaluationSteeringcommitteeandthusbecamethebasisfortheeval-uationdesign,includingtheevaluationframeworkandindicators.notethatwhiletherewereseveraldraftprogrammelogicalframeworksinplacein2016,thefinalversionwasnotapproveduntil2017.


thecausallinksbetweenthe14DEPPprojectsandtheiractivities,fourexpectedevaluationoutputareas(1.capacitybuilding,2.collaboration,3.learningand4.EarlyWarningSystemDevelopment),expectedoutcomes inboth theshort term (improvedhumanitariancapacity,improvedpreparedness)andlongterm(increasedeffectivedeliveryofhumanitarianresponse).Hypothesiseddownstream impacts (mortality,morbidity,economic impactandrecovery)arealsodepictedbelow,butarenotexpectedtobeachievedwithintheprogramme’sthree-yearcycleandthuswereconsideredoutsideofthescopeoftheoverallevaluation.Thecausalchainalsospecifiesthattheactivities,outputsandoutcomesareexpectedtooccurattheindividual,organisational, community and government levels. This also captures the variety of DEPPbeneficiariesthatweretargetedateachoftheselevels(includingindividualhumanitarianstaff,humanitarian organisations, communties, governments etc). This impact evaluation focusedonlyonassessingoutputs,aswellasshort to long-termoutcomesat thesedifferent levels. notethattheoutput,outcomeandimpactareasintheprogrammelogicalframework(Table1)arequitesimilartotheevaluationteam’soutputandoutcomeareasasdepictedinfigure1.5 However,theindicatorsareunique(Seechapter2forfurtherdetail).

figure1:SimplifiedDEPPprogrammecausalchain

5 Programmeoutput1correspondstoevaluationoutput1,programmeoutput2toevaluationoutput4,programmeoutput3toevaluationoutput2,programmeoutput4doesnotdirectlycorrespondwithanyevaluationoutputarea,programmeoutput5correspondstoevaluationoutput3,programmeoutput6doesnotcorrespondtoanyevaluationoutputarea,programmeoutcome1correspondswithevaluationoutcome1,andprogrammeimpact1correspondswithevaluationimpact1.Programmeoutputarea6correspondstotheinnovationwindowwhichisnotwithinthescopeofthisevaluation.impact1.

DEPPProjects

HumanitarianCapacity

Preparedness

EffectiveDelivery ofHumanitarianResponse

MortalityMorbidityEconomic ImpactRecovery

LEVELS OF ACTION

IndividualsCommunitiesOrganisationsGovernment

INPUTS &ACTIVITIES

OUTPUTS SHORT TERMOUTCOMES

LONG TERMOUTCOMES

IMPACT

Collaboration

CapacityBuilding

Early WarningSystem

Development

Learning


PURPOSE Thethree-yearexternalimpactevaluationoftheDEPPaimedto:

1. improveprogrammeeffectivenessandenhancelearning;and

2. assesstheextenttowhichtheDEPPoverallhasprovidedanefficientandeffectiveapproachtostrengtheningresponsecapacity.

The evaluation also served to provide accountability and learning for programme, projectand external stakeholders. These stakeholders include DEPP project consortia members,DEPPbeneficiaries,programmeandproject-levelstaff,aswellasexternalstakeholders fromhumanitariannGOs,internationalorganisations,theUn,andgovernments.

Theevaluationwasimplementedinfourphases:aninceptionphaseduringwhichtheevaluationframeworkandmethodologyweredesigned;aformativephasetoevaluatetheimplementationof theDEPPby assessing the relevance of outputs and the efficiency and effectiveness ofprogrammedelivery;aninterimphasetoassessshort-termoutcomes;andasummativephasetoassessintermediateoutcomesandpreliminaryindicatorsoflikelihoodofimpact.

The research design for the external evaluation was guided by a set of questions andmethodologicalapproachoutlinedintheTermsofReference(ToR)(Annex4)andrefinedintheinceptionreport.Itusedamixedmethodsquasi-experimentaldesigntoassessimpactattheprogrammelevel.Itwasnotdesignedtoassessindividualprojectimpacts(whichwereassessedby each project’s own independent evaluation), or the £10million innovationwindow.Theexternalevaluationmethodology includedaminimumsetofevaluationactivitiescomprisinga document review and remote data collection across all countries, aswell as an intensivesetofevaluationactivities in fourcountries (Myanmar, thePhilippines,KenyaandEthiopia).Theintensivesetofevaluationactivitiesincludedquantitativeandqualitativedatacollectioncomprisingorganisational-levelassessmentsandcommunity-levelassessments;aneconomicValue for Money (VfM) assessment; analysis of humanitarian preparedness and responsenetworks;humanitarianresponseassessment/casestudies;observationandsitevisits. Inthefourintensivesetcountries,theevaluationteampartneredwithlocalresearchorganisationswhoimplementedthedatacollection.TheseincludedthebusaracenterforbehavioralEconomics(Kenya),TheEthiopianPublicHealthAssociation(Ethiopia),andInnovationsforPovertyAction(Myanmar,ThePhilippines).Datawerecollectedattwotimepointsthatwereroughly12monthsapart–timepoint1(T1)duringtheformativephaseoftheevaluation,andtimepoint2(T2)intheinterimphase,inordertoassesschangesovertime.Intotal,634documentswerereviewed,and2542quantiativesurveys,and133qualitativeinterviewswereconductedatT1and3291quantitativesurveysand149qualitativeinterviewswereconductedatT2.EvaluationfindingsrelatedtoT1werepresentedintheformativephasereportandthissummativereportpresentsT2findingsaddressingthefiveevaluationquestionsandsub-questions.Thesummativereportincludesfindingsfromallphasesoftheexternalevaluation(november2015–May2018).

Thesummativefindingscorrespondtothekeyevaluationquestionsinthefollowingfiveareas:

1) RelevanceandValidityofDesign;2) RelevanceandEffectivenessoftheInterventions;3) EffectivenessofManagementArrangements4) EfficiencyandValueforMoney(VfM);5) SustainabilityoftheInterventionandlikelihoodofImpactoftheProgramme.

Thekeyfindingsarepresentedbelow.


MAIN FINDINGS 1. RELEVANCE AND VALIDITY OF DESIGN6

The established five results areas (objectives) of theDEPPwere appropriate, and the rightpeople in the rightplaceswere targeted.The focusonbuildingnationalcapacity, improvingpreparedness and targetingvulnerable groups to ensure their inclusionduring humanitarianresponseactivitiesisconsistentwithneedsidentifiedduringliteratureandevidencereviews,aswellaswithnewerglobalcommitmentsrelatedtohumanitariansystemreformandlocalisation7 ofaid.At ahigh level, theprogramme’sobjectiveswereclear, relevant, alignedwithDfID’shumanitarian priorities and aim to fill a clear gap in humanitarian capacity. However, theprogramme’sthree-yeartimeframewasunrealistictomeettheseobjectives.

Thecountryselectionprocesswasiterative,basedonappropriatecriteriabutwasnotoptimalduetolackofstrategicdirectionandobjectivesattheportfoliolevelandultimatelysomekeycountriesatriskofhumanitariancrisesmaynothavebeenincluded.Theportfolioofinterventionsineachcountryvariedwhichcouldhaveledtolostopportunitiestomaximiseimpactandefficiencieswithineachsetting.

Thedesignprocess,atleastintheinitialphase,wasnotlogicalandcoherent.Theseissuesledtothedesignandselectionofmanyprojectsthatwereretrofittedtothebusinesscaseorselectioncriteria.localinvolvementisacriticalstepwithinthedesignprocessthatwasnotadequatelyconsidered,leadingtochallengeswithrespecttolocalownershipandstakeholderbuy-inandpotentiallyreducingwiderimpactoftheDEPP.Thedesignprocesswasparticipatorybutlocalbeneficiariesandstakeholderswerenotadequatelyinvolved.

Insufficient time and resources were allocated during the design phase to ensure projectconsortiahadthenecessarytimeandspacetogrow,andthatkeystakeholders,especiallyatthelocallevel,couldbeinvolvedinaparticipatoryway.Individualsandagenciesdidnotalwayshave the capacity to collaborate and facilitation and capacity building around collaborationcould have contributed to healthiermore effective collaborations.The design phase duringthesecondroundofSTARTnetworkprojectsaddressedmanyoftheseweaknessesandwasledby in-country teamsensuring localbuy-in, andeventually smootherandtimelierprojectimplementation.

ResourcingfortheDEPPattheportfoliolevelwasconsideredsufficient,butprojectbudgetsweretight,with insufficientnon-project attributable costs (nPAcs), inadequateallocationoffundsforcollaborativeactivitiesandvariableallocationformonitoringandevaluation(M&E).Resourceswere also needed during the design phase to ensure that smaller agencies andin-countrystaffcouldparticipateinameaningfulway.

Designattheprogrammelevelwasnotadequatelyconsideredandprojectsweredesignedinisolationwith limited localconsultationandbuy-in.Despite theflaws in thedesignprocess,

6 fullevaluationfindingsarepresentedinchapter3.7 Thereisnogloballyaccepteddefinitionoflocalisation.Onesuggesteddefinitionis:“Aidlocalisationisa

collectiveprocessinvolvingdifferentstakeholdersthataimstoreturnlocalactors,whethercivilsocietyorganisationsorlocalpublicinstitutions,tothecentreofthehumanitariansystemwithagreaterroleinhumanitarianresponse.Itcantakeanumberofforms:moreequitablepartnershipsbetweeninternationalandlocalactors,increasedand“asdirectaspossible”fundingforlocalorganisations,andamorecentralroleinaidcoordination.Underpinningthisisthequestionofpower.localisationrequiresashiftinpowerrelationsbetweenactors,bothintermsofstrategicdecision-makingandcontrolofresources.”Morethanthemoney-localisationinPractice,GroupeURD,Trocaire,2017.


theselectedprojectsdidaddresscountryneedsonce theywerecontextualisedduring theirimplementation periods. The objectives, systems and processes, the programme theory ofchange,definitionofkeyterms,andexpectationsabouthowprojectsareintendedtointerlinkandinteractwerenotdevelopedandarticulatedattheoutset.Genderinclusionwasrequiredbutspecificguidelineswerenotclearlyarticulatedduringtheprojectdesignstage.

Responsetoemergencieswasnotadequatelyconsideredduringthedesignphase–theprogrammewasdesignedtoimproveemergencypreparednessbutmechanismstoenableorfacilitatecontributiontohumanitarianresponseeffortswerenotexplicitlyintegratedintotheprogramme(leadingtoreducedpotentialimpact).Despitethis,DEPPcontributedtoatleast42humanitarianresponsesin11countries.

2. RELEVANCE AND EFFECTIVENESS OF THE INTERVENTIONS8

Implementationdelayswereuniversalacross theDEPPprojects.Delaysweredueto limitedcontextualisationandinvolvementofin-countryteamsinthedesignphase,underestimationoftimerequiredforcollaboration,andadministrativeandcontractualbottlenecks.Thesedelaysledtoshorterperiodsforprojectactivitiesandreducedpotentialprogrammeimpact.

ThenumberofindividualsexposedtoDEPPcapacitybuildingactivities(n=33,388)exceededtheultimatetargetoftheprogrammeoftraining4,200individualsby795%.Thegendertargetwasmissedbyabout5%(i.e.,45%ofthesetraineeswerewomen).Examplesoftheseactivitiesincludes in-person trainings, developmentof trainingpackages andguidelines, developmentof earlywarning systems, collaboration activities such as learning events and conferences,developmentofcasestudiesandevidencesharinganddissemination.

DEPPcontributed to increasedcapacitybuildingefforts, collaboration,earlywarningsystemdevelopment,andlearninginbeneficiaryorganisations.However,outcome-levelchangeswithrespecttoknowledgechange,andinpreparednesslevelswerevariable.Overall,therewerenosignificantimprovementsinknowledgeoncorehumanitariancompetenciesofhumanitarianstaff,onself-reportedknowledgeondisasterpreparedness,responsetodisastersandemergenciesorageordisability-relatedissuesindisastersdespitetheincreasedexposuretocapacitybuilding.

Themosteffectiveindividual-levelcapacitybuildingapproachreportedbybeneficiariesinvolvedin-persontrainingcombinedwithastrategytoreinforceknowledge(simulations,mentoring,coaching).flexiblefundingapproaches,thoughinitiallyconsideredhighrisk,werefoundtohavehighpotentialofeffectiveness.Distanceorremotelearningwasnotreportedtobeaseffectiveasin-personapproaches.

Therehavebeensomesignificant improvements in individual,organisationalandcommunitypreparedness among DEPP beneficiaries, especially among local organisations. However,bothDEPP and non-DEPP participantswere exposed to capacity building activities relatedtoemergencypreparednessandresponse,andinsomecasessimilarimprovementshavealsooccurred in thecomparisongroups. Thechanges in thecomparisongroupmayormaynothavebeenrelatedtoexposuretoothercapacitybuildingactivities.Overall,formostindicators,the Difference-in-Difference analysis showed no overall net effect of the DEPP. However,qualitativedatasuggestssomeimprovementsinemergencypreparednessthatmaybedifficulttodetectwithquantitativeindicators.

8 fullevaluationfindingsarepresentedinchapter4.


Documentreviewandin-depthinterviewsdemonstratecontributionoftheDEPPtoat least42differentemergencyresponsesin11countries.TherewerenodocumentedcontributionstoresponsesinMozambique,oneofthecountrieswiththefewestDEPPprojects(n=1).Thelargestnumberofresponsecontributions (n=16)occurred inthePhilippineswhichalsohadthehighestnumberofDEPPprojectsimplemented(n=6).notethatKenyaandPakistanalsohadsixDEPPprojectsbutinthesespecificcontexts,DEPPcontributedtosubstantiallyfewerresponsesthaninthePhilippines.ThehighperformanceinthePhilippinesintermsofabsolutenumberofhumanitarianresponseeffortswaslargelydrivenbythefinancialEnablersprojectwhich cumulatively involved contribution towards 14 different responses (out of 16DEPP-supported responses in thePhilippines).Qualitativedata collected in thePhilippines,KenyaandEthiopiasuggestmoretimely,locallydrivenresponsesinthesecontexts;qualitativedatawerenotavailabletosystematicallyassessresponsesinmanyothercountries.However,itwasevidentthat inmanycasesDEPPbeneficiarieswereunabletoresponddueto lackoffundsallocatedforemergencyresponseactivities.Incontrasttothequalitativeindicators,quantitativedataonperceptionsofimprovedspeedandeffectivenessofresponseactivitiesdidnotshowasignificantmeasurableeffectoftheDEPP.

3. EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS9

collaborative approaches were consistently reported through in-depth interviews andquantitative data to be the most appropriate, preferred approach to deliver humanitariancapacitybuilding, but respondents identified challenges in engagement, communication andcoordination.Qualitativedata triangulatedwithdataextracted fromproject andprogrammedocuments demonstrate that as a delivery mechanism, collaborative and consortia-basedapproacheswereparticularlyeffectivewhenthefollowingconditionsweremet:existenceofjointobjectives,valuesandcommonwaysofworking;collaborationswereorganicnotforced;sufficienttime,space,resources,capacityandwilltofosterhealthycollaborativerelationships;and streamlined decision-making and contractual processes, and fewer consortia partners.Quantitativenetworkdatademonstratedthatthetopthreeareaswhereorganisationswithinthe humanitarian networks in the four intensive set countries (Ethiopia, Kenya, Myanmarand the Philippines) collaborated were advocacy, community capacity building and projectimplementation.

Thenetworkstructuresineachofthefourintensivesetcountrieswerefoundtobesignificantlydifferentfromtheoutset.Therewassomeevidenceofstrengthenedhumanitarianresponsenetworksamongthethreeofthefourintensivesetdatacollectioncountries(thePhilippines,Kenya and Ethiopia, while the network in Myanmar remained unchanged). The Philippinesnetworkwasfoundtobehighlylocallydrivenwith70%oflinkswithlocal/nationalnGOs(l/nnGOs); network size increased betweenT1 andT2, but the network became less densewith a smaller number of connections. There was evidence of network growth in both inEthiopiaandKenya.Thenetwork inEthiopiawas internationallydrivenwith little inputfromlocalorganisations,whileinKenya,thenetworkwasbalancedbetweenlocalandinternationalactors.ThenetworkinMyanmarwasverysmall,isolatedanddominatedbyinternationalactors.

findingsoncollaborationweremixed.collaborationfrequency(aproxyindicatoroftrust)andquality of relationships increased in Kenya and the Philippines, but decreased inMyanmarandEthiopiaovertime.collaborationwithintheDEPPcohortincreasedinMyanmarandthePhilippines,butbothdidnotincreaseengagementwithnon-DEPPactors.InEthiopiaandKenyathenumberofrelationshipsbetweenDEPPandnon-DEPPorganisationsincreasedindicatingthatDEPPinstitutionsarereachingbeyondtheirDEPPpartnershipsintothebroadersystem.



basedonthenetworkanalysis, therewasnoquantitativeevidenceof increased localisationover the courseof theprogrammewithin the four intensive set countries. In thisparticularanalysis, localisationwas assessed bymeasuring the proportion of relationshipswith local/nationalorganisationswithin thehumanitariannetworks.While therewasnochange in thismetricinEthiopia,thePhilippinesorKenyaovertime,inMyanmar,thehumanitariannetworkactuallybecamemoredominatedbyinternationalnGOs(InGOs)overtime.However,qualitativedata(seechapter7)anddatafromdocumentreviewsuggestthatchangesinattitudesaroundlocalisationhaveoccurred.Takentogetherthesedatademonstratethatattitudinalchangeshaveoccurredbuthavenotyettranslatedintoquantitativebehaviouralchangerelatedtolocalisation.

Inaddition, insufficienttimepassedbetweenthedatacollectionpointstosufficientlyassessandtestthehypothesisthatstrengthenednetworksandgreatercollaborationleadtobetteremergencypreparedness.DataatT1pickedupnetworkeffectscausedbyever-shiftingstrategiescommontofirst-yearimplementation.T2likelycapturedsomedistinctiveprogrammeimpactsofDEPP,butsufficienttimetofollownetworkchangerequiresatleastanadditional12months.

According toqualitativedata, inter-project collaborationcontributed to improved sharingoflearningandevidencebutthisdidnotnecessarilytranslatetobehaviouralchange.

4. EFFICIENCY AND VALUE FOR MONEY10

focusingoneconomyandefficiency,Value forMoney (VfM) showsgoodpotential for theprogrammewithareasforimprovement.Goodindicatorsofcosteconomywereevident,thoughsomebudgetswereleanwithinsufficientresourcesforportfoliomanagementandcollaborationactivities.

Thecollaborativemodelnecessarily lends itself toadegreeof inefficiency,hightransactionscostsandslowinformationflows.Ontheflipside,thismodelhadsubstantialbenefitsandforsomeprojectsthebenefitshavelikelyoutweighedthecosts.forotherprojectsthesamebenefitsofthemodelcouldhavebeenachievedatlowercostandgreaterefficiencyifthecontracting,M&Esystemandcostingofactivitieshadbeenmoreconsciouslyconsideredfrominception.Thecollaborativemodeltiedtohierarchicallongdeliverychainshaslikelycompromisedeconomyandefficiencytoadegree.

Therewereshortfallsintermsofsystemsandresourcessetupforefficiencyingovernanceandstrategy,portfoliomanagement,decision-makingandconsortiumarrangements.Thespendingon management functions at the programme level were too economical and would havebenefittedfromsomefundsandtimebeingredeployedfromthelearningProjecttotheDEPPmanagementteamintheformofM&Eexpertise,bothattheprogrammeandproject levels.If,however,independencewasdesired,thenmoredirectlinkageandcommunicationbetweenthemanagementteamandthelearningProjectwherebydatafromM&Etransmittedregularlyanddirectlytothemanagementteamthroughbothautomaticreportingordirectdataaccessthroughadashboardfollowedbyregularmeetingscouldhavebeenconsidered.

PositivefindingsintermsofVfMreporting,adaptivemanagementandcollaborativewaysofworkingwere identified.Aftertheformativephaserecommendationto increaseuseofVfMindicators, the DEPPmanagement teammade a strong effort to increase the project staffcapacity aroundVfM concepts and reporting through the implementation of two half-daytraining events. Subsequently, some projects carried out ad hoc reports onVfM andmostprojectsincorporatedVfMasacomponentoftheirfinalevaluations.

There are no strong quantitativeVfM findings to suggest that preparedness improved the



efficiency of humanitarian response. Analysis of the empirical data showed no significantdifferencebetweenDEPPandcomparisonorganisationswithregardstoperceivedimpactofDEPPon institutionalspeedandcostofresponse,andtheextenttowhich institutionalandpolicyenvironmentaffectedthespeedandcostofresponse.

costper result indicator analyseswereundertaken toestimate theexpenditure incurred toachieveoutcomesrelatedtocapacityandpreparedness.Whilereturnoninvestmentanalysesassessespotentialorprojectedreturnoninvestment,thecostperindicatoranalysescomparesactual expenditure incurred per result achieved.This analysis demonstrated goodVfM perresultactuallyachieved.ThehighestVfMwasforMyanmar,whichdemonstratedacostperpercentage point increase in perceived organisational preparedness of only £ 1,138. It notadvisable tomakedirect comparisonsbetweencountriesbecauseof thedifferences incostbaseswithincountries,andthemanyotherfactorsthatcomeintoplay.notably,Kenyaalsohadaverylowunitcostperresultarea.fortheindividualresponseperformanceindicatorinthePhilippines,theannualunitcostfigureforeachtrainedindividualisroughly£30.Itisdifficulttomake avalue judgment on this figure because there is limited basis for comparison, butgiventhefactthatasignificantimprovementinthesatisfactionratingwasobserved,andfromexperiencewithotherprojectsthat£30isnotoutsidethereasonablerange,thisappearstobeagoodVfMfinding.

5. SUSTAINABILITY OF THE INTERVENTION AND LIKELIHOODOF IMPACT OF THE PROGRAMME11 DEPP has contributed to strengthening national emergency preparedness systems in someoftheprogrammecountries,butnotall.forexample, theUrbanEarlyWarningEarlyAction(UEWEA)projecttodetecturbanemergenciesandenablerapidresponsehasbeenadoptedbythenairobicountygovernmentinKenya,whohavealsocommittedtoaddressingurbanfoodsecurityissues.ThePublicHealthPreparednessprojectinGambella,Ethiopia,workedwiththegovernment to improve disease and outbreak surveillance, leading to improved surveillanceandreportingatvariouslevelsofthehealthsysteminseveraldistrictsandinthestrengthenedcapacitytotestforpathogensattheregionallaboratory.InPakistan,managementofthesurgeplatformdevelopedunder theTransforming Surgecapacity project has been takenover bythenationalHumanitariannetwork(nHn),withamemorandumofunderstanding(MOU)inplacewith the localgovernmentbodytoensure implementationafter theprojectends.TheMarsabit county Government in Kenya adopted the linking Preparedness Resilience andResponse(lPRR’s)project’sconflict-sensitiveapproachduringhumanitarianresponse(includingduringthe2016-2017droughts) thusprovidingadditionalevidenceofDEPP’scontributionstonationalpreparednesssystems.Inothersettings,however,thedegreetowhichtheDEPPhas influencednational preparedness systems andgovernmentdisasterplanshasbeen lessclear.Thepositiveinfluencesthathavebeenidentifiedareimportantandrepresentsignificantinvestmentandefforts.However,itshouldbenotedthatgenerallytheseeffectshavebeenonasmallergeographicscale–oftenatthedistrictorcountylevelandinmostcases,eachpositiveexampleofchangewasdirectlyattributabletoanindividualproject.

While the DEPP business case underscored the disproportionate impact of disasters andhumanitarianemergenciesonwomeninparticularandemphasisedthattheprogrammewouldstrategicallyaddress inclusion, at least in regard togenderandviolenceagainstwomenandgirls (VAWG), sucha focuswas largelyabsent fromtheprogramme.Across theprogramme,two projects specifically focused on inclusion, Protection in Practice (PIP) which focusedon protection, and Age and Disability capacity Programme (ADcAP) which focused on



strengtheninginclusionoftheelderlyandpeoplewithdisabilitiesinemergencyresponse.bothof theseprojectsmadestrides tobuildprotectionand inclusioncapacityand insomecasesinfluencedtheworkofotherDEPPprojects.ADcAPinparticularstronglychampionedinclusionefforts includingatprogramme-leveleventsandhelpedshapeprogramme-leveldialogueonthis issue.However, the extent towhich priority groups (women, children, the elderly, andpeoplewithdisabilitiesandothervulnerableormarginalisedgroups)wereholisticallyincludedacross theprogrammeand inclusionefforts adequatelymonitoredwasminimal.The lackofmainstreamingofinclusionofgenderandprioritisedgroupsacrosstheprogrammeresultedinafragmentedprogramme-wideapproachandunevenresults.WhiletherewasnoevidenceofindividualknowledgechangeoninclusionofvulnerablegroupsoronVAWG,therewassomeevidenceoforganisationalchangewithrespecttoinclusion,inparticularinEthiopiaandamonglocal organisations in intensive set countries. InEthiopia, therewas a statistically significantincreaseininclusionoftheelderlyandpeoplewithdisabilitiesindesignandimplementationofpreparednessprogramminginDEPPorganisations,aswellasastatisticallysignificantincreasein organisationswith policies on inclusion.Therewas also a statistically significant increasein theproportionof localorganisationsacrossall intensivesetcountries thathave inclusionpoliciesinplace.

Previous external evaluation reports from the interim and summative phase documented aconsistentlackofdisaggregateddataandminimallydocumentedgenderconsiderationsacrossprojects.Recommendationstostrengthengenderreportingwithinprojectdataweretakenupaftertheformativephaseevaluationreport.Therewasanincreaseindatadisaggregationbygender over the course of the programme.However, reporting on other prioritised groupsremained low,making itdifficult toassessthetrueextenttowhichthesegroupshavebeenreachedbycapacitybuildingorotheractivities.

TherehavebeenseveralconcreteexampleswhereinDEPPhasimpactedgovernmentpoliciesor systemsor increasedpolitical commitment.forexample,UEWEAproject’s advocacyandsensitisationefforts led to anamendmentof theDisaster andEmergencyManagementAct2015(DEMAct)toincludefoodsecurityandanMOUwiththenairobicitycountyGovernmentto ensure the government’s role in addressing urban food security issues12. The linkingPreparednessResilienceandResponseprojectdemonstratedchangesattheinstitutionallevel.for example, christianAid adopted the updated resilience framework13, and aViolence toPeacestrategy14.Despitetheseexamples,therewasnostatisticallysignificantchangeacrosstheDEPPinpolicy-relatedquantitativeindicatorsinthefourintensivesetcountries.changesmaytakesignificanttimetooccurandquantitativedatamaynotfullycaptureothertypesofpolicychangeorshiftsinthelandscapethataremoredifficulttomeasurequantitatively.

changesinquantitativeindicatorsrelatedtolocalisationhavenotyetoccurredwithinthreeoutoffour intensivecountries. InEthiopia,however,thepercentageofDEPPorganisationswithpolicies inclusivetol/nnGOsincreasedby16%betweenT1andT2.Inaddition,attitudinalchangestowardslocalisationhavealsooccurred,includinginhowInGOsconsider,address,andinvolvelocalactorsandcommunities.ThisadoptionofthelocalisationapproachisconsideredtobeDEPP’smostsignificantchangebykeyinformants.Examplesofthistypeofchangeweredescribed in bangladeshwhere local actors became involved in decision-making processeswithinalargenationalpreparednessplatform,andinthePhilippineswherelocalorganisationswerethefirsttomobiliseandrespondtothecrisisinMarawi.

There is some evidence of benefits being embeddedwithin organisations and systems and

12 UrbanEarlyWarningEarlyActionfinalEvaluationReport.13 https://www.christianaid.org.uk/sites/default/files/2017-05/resilience-framework.pdf.14 https://www.christianaid.org.uk/resources/about-us/tackling-violence-building-peace-global-strate-

gy-2016.


potential for longer-term effects but this is dependent on the extent to which programcomponentsareabletocontinuebeyondtheendofDEPP.Thelikelihoodofsustainabilitywasenhancedforprojectsorprojectcomponentsthatdemonstratedthefollowingcharacteristics: builtonexistingworkandexistingpartnershipsand/orconsortia;partneredwiththenationalandlocalgovernment;hadand/ordevelopedapolicyoradvocacyelement;workedtowardssystems-levelchange;paidgreaterattentiontoexitplansearlierintheprojectcycle;developedtools, guidelines, or systems that fill an important gap; involved beneficiaries and/or localstakeholders, and exhibited good VfM; built linkages with other entities and other DEPPprojects;andimplementedincontextsthatweremorefertileforchange.

CONCLUSIONS AND RECOMMENDATIONS TheDEPPprogrammewasdesignedin2014,andwasaheadofitstimeinmanyways,especiallywithitsstrongfocusonlocalisationandmulti-stakeholderconsortia.Itincludedamixofbothflagshipprojectsusingmoretraditionalapproaches(i.e.,in-personindividualcapacitybuilding)withsomemoreinnovativeelements(i.e.,a£10millionInnovationWindow;flexiblefundingmechanismstofinancebothlocalcapacitybuildingbutalsoemergencyresponse;pooledsurgeplatforms;urbanearlywarningsystemsandinclusionoffoodsecurityasanemergency).

TheDEPPsufferedfromanumberofdesignchallengesincludingathree-yeartimeframethatwasunrealistictomeettheobjectivesanduniversal implementationdelays.ThedecisionbyDfID tono longer fund theprogramme limits thepotential long-term impactof theDEPP.Despitethisdecision,theDEPP’sfocusonemergencypreparednessandlocalisationisstillarelevantone.Italignswellwiththegrowingdialogueamongtheglobalhumanitariancommunityandwithmore recent policy commitments such as theGrand bargain and thecharter forchangewhichadvocateforincreasedvoiceforlocalactors,andforrebalancingofpowerandrelationshipswithintheexistinghumanitarianarchitecture.

Despite its initialdesignflaws,positivefindingsrelatedtothecollectiveactionoftheDEPPand to individual DEPP projects have been documented. These include the following keyachievements:

• 33,388 individuals were exposed at varying degrees to capacity building based onprojectreports

• Stakeholders across the programme noted that themost significant change due totheDEPPwasachangeinattitudestowardandincreasedsupportoflocalisation(i.e.,contributingtotheGrandbargain)

• changesincapacityoflocalorganisations(intermsofsystemsandprocesses,fundraising,conductingneedsassessments)andtheiremergencypreparedness levels,that ledtooutcomessuchasbecomingeligibleforUnpooledfunding,successinsecuringexternalemergencyresponsefunds,andbeinggivenleadershiproles

• Significant changes in organisational policy with respect to inclusion of vulnerablegroups(Ethiopia,localorganisationsacrossallintensivesetevaluationcountries),andlocalisation(Ethiopia)

• Someevidenceofstrengthenednetworksandincreasedcollaboration(Ethiopia,Kenya,thePhilippines)


• contributiontoatleast42humanitarianresponsesin11countriesandsomequalitativeevidenceinseveralsettings(ThePhilippines,Kenya)ofimprovedspeed,efficiencyandinclusivenessofemergencyresponse

• cross-country learning in termsof theamountof learningdocumentsproducedandsharingofideasacrosscountriesandprojects

• Strongcross-projectcollaborationinthePhilippinesdemonstratingbenefitsofcollectiveaction

nevertheless,agreater impact,asoriginallyenvisioned,could likelyhavebeenachievedhadtherebeenamorecohesivedesign,alongertimeframe,amorerobustprogrammeM&Esystemandamorestrategicapproachwithregardstotypeofcapacitybuildingimplementedateachlevelofaction(individual,organisational,community,systems)andthebalancebetweentheselevels.Overall,theexternalevaluationfoundthefollowingfindingsthatneedtobeexaminedfurtherinfutureevaluationsandprogrammingrelatedtoemergencypreparedness:

• noquantitativeevidencetodatethatthethree-yearprogrammehasledtomeasurableimpactwithregardstomoreefficient,andtimelyhumanitarianresponse

• noquantitativeevidencetosupport thehypothesis thatstrengthenednetworkscanlead to improved emergency preparedness and response (mainly due to the shortobservationperiodavailabletoassessthis)

• noquantitativeevidencethatDEPPimprovedindividualknowledgeoncorehumanitariancompetencies,preparedness,inclusionofvulnerablegroups,orprotectionissuesinthefourintensivesetcountries

• fewchangesinquantitativepolicy-relatedindicators,thoughthereweresomeexamplesofimprovedpoliciesdocumentedbyprojects

• Toowideofavarietyofindividualcapacitybuildinginitiativesandlackofstrategyontypeofcapacitybuildingapproachesateach level (individual,organisational, system,community)andeffectdesiredateachlevel

The three years allotted for an emergency preparedness programme to achieve fivemacroresults in10countrieswasanambitioustimelineandagenda.The levelofexposure to theDEPP programme activitieswas unevenly distributed across the 10 DEPP focus countries;eachcountry receivedonlya subsetof the14projects.forexample, threeDEPPcountriesbenefittedfromsixprojectseach,whiletwocountrieshadonlyoneprojecteach(notincludingthelearningProject;seeTable4.7forthefulllistofprojectnumberspercountry).ThisunevenlevelofexposuretoDEPPcompoundedwithvaryingcountry-specificcontextualfactorshasledtovaryingresultsinspecificcountries.HadtheDEPPbeenabletocontinueforanotherthreeyearsormoreandbeenabletointegrateitslearningsintoarevisedprogrammestrategy(i.e.,permittingtimetorestructureitsmanagementstructure,scaleupprojectsthatshowpromiseandreviseprojectsthatarelesspromising),therewouldbeamorefeasibletimeperiodtoobtainmoreconclusivefindingsaboutDEPP’sachievementstowardsitsfiveresultareas.


RECOMMENDED ACTIONS FOR PROGRAMMECOMPONENTS

Given the key evaluation findings, there is scope to replicate and further test, or scale upsomecomponentsoftheprogramme(Table3).ThiswillensurethatthepreviousinvestmentinDEPPissustained,andthatanyadditionalresourceswillbeusedeffectively,efficientlyandstrategically. based on evaluation findings that have been triangulated using multiple datasources,noneoftheDEPPprogrammecomponentshavebeenrecommendedtobeceasedandnotbepursuedinthefuture.ThisinitselfisastrongindicationthattheDEPPcoretheoryof change and initial assumptions areworthy of being re-examined for future investment.Specially,ninecomponentswererecommendedtobemodifiedandre-tested,whileeightwererecommendedtobereplicatedindifferentcontextsandpotentiallyonalargerscale,enablingmorerigorousevaluations(Table3)notethatinaprogrammeeachofthesecomponentswouldnotbeexpectedtofunctioninisolation,andtheinteractionbetweencomponentsshouldalsobeconsidered.

DEPP PROGRAMME COMPONENT

RECOMMENDED ACTION KEY FINDING / LESSONS LEARNED

Individual Capacity Building Modifyandre-test

• Prioritisemulti-prongedapproaches(in-person+coaching,mentoring,simulations)

• Qualityoftrainingsshouldbeprioritisedoverquantityofindividualstrained(webinarscanreachalotofpeoplebutmaybelesseffective)

• considerstructuralbarrierswithinorganisationsthatmayimpedeapplicationoflearning

Organisational Capacity Building Modifyandre-test

• Prioritiseapproachesthatpermitorganisationalself-assessmentsandcapacitybuildingapproachestailoredtoorganisations’needs

• focusonstrengtheningofadministrativepolicies,proceduresandsystems(i.e.,humanresources,procurement,financesystems)hasbeeneffectiveforl/nnGOs

• Approachesshouldbebetteradaptedtodifferentcontexts

• Accesstofundingiskeyfororganisationstoputlearningintopractice

• conductcomparativeassessmentsoftheeffectivenessofdifferentapproaches

Community Capacity Building Modifyandre-test

• fewprojectsaddressedthis,butthosethatdidindicatedsomepotential;furtherevidenceisneededonhowbesttobuildcommunitycapacity

Systems Capacity Building Modifyandre-test

• fewprojectsdirectlyaddressedthis,buttherehasbeensomeevidenceofqualitativeattitudinalchangethatcouldbeaprecursortoeventualsystems-levelchange;moretimeandfurtherevidenceareneededonhowbesttobuildcapacityatthesystemslevel

Early Warning Systems

Replicateandtestonlargerscaleindiffering

contexts

• UEWEA’surbansurveillancesystemshouldbetestedinotherurbansettings

• Otherearlywarningsystemprojectscouldbeadaptedforandimplementedinothercontextswithsufficientprojecttimelines


Pooled Surge Platforms


contexts

• Pooledsurgeplatforms,especiallynational-levelplatformshaveshownsomeevidenceofeffectiveness.Pooledsurgeplatformsshouldensurealocallydrivendesignprocessandimplementationtoensurecontextualisationandlocalownership

• Replicateandtesttheseindifferentcontextstogeneratemorerigorousevidenceontheireffectiveness

Flexible Funding Mechanisms


contexts

• flexiblefundingmechanismswerefoundtobeapromisingapproachtosupportl/nnGOsself-directedcapacitybuildingefforts.Inaddition,flexiblefundswereessentialtotranslatingemergencypreparednessintotimelyandefficientresponseactivities,especiallyforl/nnGOs

• flexiblefundingmechanisms(bothforcapacitybuildingandresponseefforts)shouldbereplicatedinothersettings,andmorerigorouslyevaluated

Multi-stakeholder Platforms Modifyandre-test

• Theseplatformsweremoreeffectiveincontextswhereexistinghumanitarianstructuresweremorematureandwithsupportinggovernments.

• furthermodificationandtestingoftheapproachisneededtounderstandhowbesttoutilisethisstrategyandinwhichspecificsettings

Consortia Modifyandre-test• consortiathatbuildonexistingrelationships,commonvaluesandwaysofworking,andthataresmallerwithinclusionofl/nnGOsshouldbefavoured

Strengthening Networks

Modifyapproachandre-test

• Useknowledgeanddataoncountrynetworkstocreatemoretargetedapproachestostrengthennetworks;allowenoughtimetotestwhetherstrongernetworkscanleadtoincreasedemergencypreparedness

Advocacy and Policy

Mainstreamacrossprogramme

• Someadvocacyeffortswereimplementedbuttherewasnosystematicadvocacycomponentattheprogrammelevel.Itwasrecognizedthatadvocacyandpolicychangeareneededtoovercomebarrierstochangeandtoincreaseprogrammesustainability

• Ensureacohesiveprogramme-widestrategyrelatedtoadvocacyandpolicy

Learning Project Modifyandre-test

• Theapproachofhavinganindividualprojectdedicatedtolearningandgeneratingevidenceisuniquebutfindingsweremixedduetothelackofprogrammemonitoringandthelackofadirectlinktothemanagementteam.Alternativemodelsforanindependentlearningprojectcouldbeenvisionedandpotentiallyassessedinafutureprogramme

Evidence Generation Modifyapproach

• Prioritisequalityoverquantity

• Includeempiricalevidencegeneration

• Adaptevidenceuseandknowledgetranslationstrategies

Protection Mainstreamacrossprogramme

• Ensureacohesiveprogramme-widestrategyrelatedtoprotection

Inclusion of Prioritised Groups


• Ensureacohesiveprogramme-widestrategyoninclusionofprioritisedgroups(suchaswomen,theelderlyandpeoplewithdisabilities)

Localisation

contextualiseandreplicateandtestonlargerscaleindiffering

contexts

• Definelocalisation(whatitis,whichentitiesareconsideredlocal);andtestapproachestoenableInGOstotakeamoresupportiveroletowardsl/nnGOs,andbetteroperationaliselocalisationindifferentcontexts


DO NOT PURSUE FURTHER

MODIfyAnDRE-TESTcOnTExTUAlISEAnDREPlIcATEInAnOTHERSETTInG

ORMAInSTREAMAcROSSPROGRAMME

Table3:RecommendedactionsforeachDEPPprogrammecomponent

Inaddition,anumberofbestpracticesrelatedtotheDEPPprogrammecomponentshavebeencompiledtoguidefutureprogrammedesignandimplementation.ThesebestpracticescanbefoundinAnnex11.

Specificconclusionsandrecommendationsrelatedtoeachevaluationquestioncanbefoundbelow.

1. RELEVANCE AND VALIDITY OF DESIGN CONCLUSION 1: WhiletheobjectivesandtargetsoftheDEPPwereappropriate,relevantandalignedwithDfIDpriorities,theDEPPdesignprocesssufferedfromnumerousweaknessesthathindered itspotentialfor impact.AnyfutureprogrammewouldrequireasignificantredesignbasedonthelessonslearnedandtheevidencegeneratedfromtheDEPP.forfuturedesign,programmeresultsthatarerealisticwithinathree-yeartimeframemustbeset,ortheprojecttimeframeshouldbeextendedtoallowformoretimetomeetlonger-termoutcomes.Amorelogical,coherentdesignprocess,which is locally led,drawsonneedsassessments,andusesparticipatoryapproachesshouldbeprioritisedtoensurethatprojectsarecontextualisedandbasedonexistingneedsatthecountrylevel.Thedesignprocessshouldadequatelyconsiderprojectalignmentandcomplementarityaswellasrisksattheprogrammelevel.Itshouldalsoensurethedevelopmentandimplementationofprogramme-levelsystemsandprocesses,suchasarobustM&Esystem,toensurecoursecorrectionandthatprojectsfunctionasaportfoliorather than as standalone projects. This includes elaborating a programme-level theory ofchange,definitionofkeyterms,cross-projectlinkages,andstreamlinedinclusionofgenderandprioritised groups alongwith appropriate considerationof cross-cutting themes. Emergencypreparednessandresponseprojectsshouldalsoconsiderurbancontextsand issuessuchaspovertyandfoodinsecuritywhichlieatthehumanitarian-developmentnexus.budgetsshouldincludehigherallocation fornPAcs,programmemanagement functionsandM&Eacrossallprojects.

RECOMMENDATION 1.1: [DEPP PROGRAMME BOARD AND DFID] The design process should be re-envisioned to consider alternative programme models and governance structures. To increase the local relevance and effectiveness of theprogramme, the design process should be locally driven from the outset (i.e., led bylocalstakeholders),andincludesufficienttimeandfundsforprojectdesign,localisation,consortiadevelopmentandcollaboration.

Potential Model 1: The process could involve several stages: After the overallprogrammatic business case and theory of change are developed, global goalscould be established and local andnational ownership prioritised.A programme-level focal point could be appointed in each country to conduct contextualisedneedsassessmentswhichwould feed into thedevelopmentof theportfolio-levelstrategy.Theproject designprocesswouldoccur at the country level, under theoversightofthefocalpointsandtheProgrammeboard.Thisapproachwoulddrivethe localisation agenda and allow for country-based organisations to determinethemostneededinitiativeswithinthegoalsoftheoveralltheoryofchangeaswellasriskassessment.Itwouldhelpalignprojectstowardcommonprogrammegoals,


identifycomplementaryelementsofdifferentprojects,mainactorsandthosethatneedtobebolsteredaswellasmoreriskyprojectsworthyofsupport.Itwouldalsoensuresufficient investment ineachcountry inordertoaccomplishthetheoryofchange,andwouldenablemorecomprehensiveassessmentsofrisks.Also,areturnoninvestmentassessmentcouldbeconductedtoinformthefinalinvestment.focalpointscouldworkwitharegionalorcountryM&EandlearningAdvisortodevelopandimplementprogramme-andproject-levelM&Esystems.Asaprogramme-levelactor, the focalpointwouldhelpbuildcohesivenessandenhance thevisibilityofDEPPasaunifiedprogrammeineachcountry.

Potential Model 2: Alternatively, consider a multi-phase programme in which aseriesofpilotinterventionsaredevelopedandimplementedonasmallscalewithstrong but targeted M&E systems integrated to evidence changes and provideaccountabilitymechanisms.Successfulpilotinterventionscouldbescaledupduringthesubsequentphase,buildingonthelessonscapturedduringthepilot.Astrongprogramme- and project-levelM&E system during the scale up phasewould beintegraltomeasuringchangeandensureregulartrackingofoutputstomodifytheprogrammewhenneeded. Inaddition, this typeofmulti-phaseprogrammecouldincludeapre-pilotphasewhereseedfundingisprovidedtodeveloppromisingideas,toconceptualiseprojectsandtobuildconsortiaaroundtheprojects.

RECOMMENDATION 1.2: [DEPP PROGRAMME BOARD AND DFID] The programme portfolio should be balanced in terms of risk, types of activities, size of projects, geographic scope and cost. To maximise impact, the link between emergency preparedness and response should be better articulated; preparedness programmes should strongly consider embedding flexible funding mechanisms in order to facilitate contribution to humanitarian response by local actors. Theportfolioshouldincludeamixofmoreestablished ideasandprojects, those thatareconsidered riskierandwitha higher likelihood of failure, aswell as innovative elements. Prioritising humanitarianneed,examiningcountrydisasterandemergencyprofiles,aswellasaligningwithoverallprogramme strategy during country selection process would ensure a more optimal,balancedsetoffocuscountries.forafundofthissize,concentratingonasmallergroupofprojects(atleastinanyscaleupphaseofamulti-phaseprogramme),andpotentiallyfeweroverallcountriesmayhelptoensurethatinvestmentsarenotspreadtoothinandthatthereisbothsufficientinvestmentandalargeenoughnumberofprojects ineachcountry to reach a criticalmass and achieve systems-level changes.A complementaryandstrategicmixofactivitiesattheprogrammelevelandespeciallyatthecountryleveliswarranted tomaximise impacts;but this tooneeds tobe focused.forexample, theDEPP’sopenapproachledtotoomanydifferenttypesofcapacitybuilding,toogreatanemphasisonindividualcapacitybuilding,andlackofconsensusondesiredeffectsateachlevel.Instead,amoredeliberateandevidence-basedglobalcapacitybuildingstrategyisneeded,withmorebalancebetweenlevelsofaction(individual,organisational,systems),aswellasconsiderationofstructuralbarrierstochangethatmighthinderbehaviourchangeor operationalising organisation-level changes. Inmany cases advocacy, policy changeor other system-based approachesmay be critical to easing someof these obstacles;integratingaprogramme-wideadvocacyorpolicycomponentshouldbefurtherexplored.Emergencypreparednessmustnotbethoughtofinisolationandmechanismstoembedlinkagestohumanitarianresponse(includingresponsefunding)areneeded.


RECOMMENDATION 1.3: [PROGRAMME BOARD AND DFID] Portfolio-level harmonisation should be prioritised, including mainstreaming of gender and inclusion of prioritised groups. Toensurealignmentoftheprojectsandacohesiveprogramme-wideapproach,portfolioobjectives,aswellaskeyterms(e.g.,collaboration,preparednessandlocalisation),andapproaches(e.g.,typesofcapacitybuildingefforts)shouldbefullydefined,and thedefinitionsstandardisedacross theprogramme.cross-cutting themesthatareprioritisedbythedonorand/orprogrammeandprojectstakeholders, suchasgenderandaddressingVAWG,mustbegivenadequateattentionandembeddedacrossthe programme. Inclusion of prioritised groups and gender considerations are equallyimportantandshouldbemainstreamed.

RECOMMENDATION 1.4: [PROGRAMME BOARD AND DFID] Ensure objectives are realistic and feasible within the programme’s time frame. Develop programmeswithrealistic objectives that are feasiblewithin a three-year time frame.Thesemost likelywould focus on outputs and shorter-term outcomes. Alternatively, consider creatinglonger (i.e., five-yearor longer)programmetimelines if longer-termoutcomesmustbeincluded.WhilelongerprogrammetimelinesmaynotcorrespondtocurrentDfIDfundingcycles,amulti-phaseprogrammecouldbeaneffectivesolutionthatiscompatiblewithfundingtermsaswellasactivitiesaimedatlonger-termimpact.

RECOMMENDATION 1.5: [PROGRAMME BOARD AND DFID] Ensure M&E processes are integrated within the programme design from the outset and that there is an appropriate balance between M&E and learning activities. Itisimperativethataprogramme-levellogicalframeworkbeinplacepriortoimplementationofanyprogrammeor project activities.Development of this framework should occur in parallelwith theprogrammeandwithinputfromlocalstakeholders.Ideally,indicatorsshouldbestreamlinedacross theprogramme,andshouldcontainacoresetofprogramme indicatorsagainstwhichallprojects report,withpotentiallyaseriesofadditionalsite-specificorproject-specificindicatorswhererelevant.IncorporatequantitativeVfMindicatorswithinthecoresetofindicatorstoensureVfMandefficiencycanbetrackedacrosstheprogramme.TheM&Esystemshouldenablethecollectionandreportingofdatadisaggregatedbysexandotherprioritisedgroups. Inat leastoneprogrammefocuscountry,considerprioritisinga randomised controlled trial (RcT) or other rigorous evaluation design to evaluateprogrammeeffectiveness.Thiswouldrequireinvolvingtheevaluationteamintheinitialstagesoftheprogrammedesignprocesstoensurealignmentoftheevaluationquestions,randomisationoftheinterventionsites,andadequatebaselinedatacollection.Thiswouldallowfortestingoftheeffectivenessoftargetedcomponentsoftheprogrammeaswellastheprogrammeoverall,isolatingtheroleofdifferentcontextualfactors.Technologicalsolutions suchasdashboardswith real-timedatavisualisationsof key indicators couldhelpenhanceroutinemonitoring,andallowformorerapidcoursecorrection.SufficienttrainingforprogrammeandprojectfocalpointswithrespecttotheM&Esystemmustbeprovidedandappropriatemechanismstoensurehigherqualitydatamustbeinplace.learningis importantbutrequiresawell-designedM&Esysteminordertobeofmostvalue.

RECOMMENDATION 1.6: [DFID] Re-examine the development of the business case, as well as its timing and its content, and broaden definitions of disasters, emergency preparedness and response.foranyfutureprogrammeatthescaleoftheDEPP,tofacilitateprojectdesign,thebusinesscaseshouldprovidetimelyandsufficientdetailssuchasfocuscountries, the criteria for country selection, a set of definitions of key concepts (suchas collaboration, institutional arrangements, emergency preparedness and localisation),andaconcisesummaryof theDfIDpolicyon inclusionofprioritised targetgroups inprogrammeandprojectdesign.furtherconsiderationsforhowtothinkaboutemergencypreparednessinurbancontextsandconflict-affectedsettingsareneeded.considerations


forissuessuchaspovertyandfoodinsecuritywhichlieattheintersectionofdevelopmentandhumanitariansectorsshouldbeconsideredwhendesigningemergencypreparednessandresponseprojectsdependingonthecontext.

2. RELEVANCE AND EFFECTIVENESS OF THE INTERVENTIONS

CONCLUSION 2: DEPPinterventionswereoverwhelminglyfoundtoberelevantandappropriateandweredemonstrated tofill importantgaps thathavenotbeenpreviouslywelladdressed(surveillance of emergencies in urban settings, conflict-sensitive emergency preparedness,inclusionofaginganddisability,etc).However,start-updelayslimitedprojectimplementationperiodsandreducedpotentialimpact.Optimisingthedesignprocessasdescribedaboveandstreamliningadministrativeandcontractualprocesswouldensuresmoothandtimelyprojectstart-upandminimiseimplementationdelays.Attheprogrammelevel,therewasnoevidenceof improved knowledge on core humanitarian competencies, inclusion, or emergencypreparedness due to the DEPP but significant changes in organisational capacity andpreparedness among DEPP beneficiaries occurred, in particular among local organisations.furthermore,DEPPorganisationscontributedtoimprovedresponseinseveralcontexts(ThePhilippines,KenyaandEthiopia).Agreaterimpactcouldlikelyhavebeenachievedwithalongerimplementationperiod,andwitheasieraccesstoemergencyresponsefunding.Thereisscopeforcertainprogrammecomponentstobereplicatedorscaledupormodifiedandfurthertested.

Conclusion 2a: Collaboration and Consortia

functioning of consortiawas suboptimal and contributed to implementationdelays and slow decision-making. A smaller, more strategic set of consortiamembers, with more consistency between UK and in-country members foreach projectwould streamline project delivery and should be considered infutureprogrammes. Inclusionofmorel/nnGOswithin consortiamighthelpshiftpowerbalancesandcontributetolocalisation.collaborationswithexternalstakeholdersincludinggovernmentandUn,thoughessential,weredifficultandlessonslearnedshouldinformfutureprogramming.IntwosequentialdiarrhoealoutbreaksinKenya,theconsortiamodelandcollaborationwiththegovernmentenabled coordinated effective response activities that improved with eachsubsequentemergency.

Conclusion 2b: Capacity Building

Overall individual training targets at the programme level were exceededsuggesting successful implementation of DEPP capacity building activities.However,thesedataincludedindividualswhoparticipatedincapacitybuildingeffortswithminimal exposure such aswebinars and forums.Accessibility totrainingremaineduneven.Thetargetofincluding50%womentraineesacrosstheprogrammewasnotmet,andsometrainingswereonlyavailableinEnglish.Structuralbarriers,suchasorganisationalandmanagementarrangements,wereakeybarriertoapplyingandintegratingknowledgegainedfromDEPPcapacitybuildingeffortsat theorganisational levelandshouldbeaddressed in futureprogrammes. Multi-pronged capacity building efforts that include practicalapproachestosustainandapplyknowledgewereperceivedtobemosteffective,butquantitativedatashownoevidenceofknowledgechange.Organisational-levelcapacitybuildingeffortswhereseveralstrategieswerecombinedandtailoredtothespecificgapsofeachorganisationwereperceivedtobeeffective,andevidenceofincreasedorganisationalcapacityoflocalorganisationswasdemonstratedinanumberofsettings.


Conclusion 2c: Evidence Generation and Learning

Generationoflearningandevidencewashighintermsofquantity,butevidencewasmainlyanecdotalandlackinginscientificrigor.Arguablytoomuchemphasiswasplacedongenerationoflearningproductscomparedtoroutinemonitoring.Differentialreportingbyprojectsagainsttheprogramme-levellogicalframework(once itwas finalised and fully implemented inAugust 2017) occurred, andtherewasevidenceofweakqualityassurance.Inaddition,therewasalackofdisaggregateddatacollectedwithrespect to importantsubgroups.Sharingoflearningdidoccurbut there is noevidence that this translated tobehaviourchange.learningeventswereausefulapproachtosharelearning,howeveranincreasedfocusonthedisseminationofevidenceattheseeventswouldhavebeenvaluable.ThelearningPlatformwasanimportantresource(thatnowservesaspartof theDEPP’s institutionalmemory)but itwasnotusedconsistentlyacrossprojectsandfeaturedvaryingamountsofevidenceandlearningsharedbyeachproject.Suchplatformsshouldbemoreheavilypromotedtoincreaseuse by in-country partners, represent resources from all projects and ensureavailabilityofresourcesinrelevantlanguages.TheDEPPlearningProjecthadbeenoriginallyconceptualised to takeon theM&Efunctionsof theDEPP inadditiontolearning.Whileauniqueidea,thismodeldidnotworkwell–M&Efunctionsandresponsibilitywouldhavebeenbetterplacedattheprogrammemanagementlevel.Placingthesefunctionsasoneprojectalongsidetheotherswithlimitedauthorityledtodiffusionofaccountabilityandresponsibility.Asastandalonelearningproject,ithashadmixedresults,butcouldbere-envisionedforfutureprogrammes.

Conclusion 2d: Early Warning Systems

Early warning systems are important as they provide (in theory) accurate,predictive and timely data to support emergency preparedness. However,findingswithrespecttoimplementationofearlywarningsystemsaremixed.Sixdiverseearlywarningsystemsweredevelopedandfivewerefullyoperationalatthetimeofthisreport.However,becauseofimplementationdelays,thelengthoftimethesesystemswerefunctionalwasverylimited.Severalofthesystemshavebeenwellintegratedintoappropriatestructurescontributingtoincreasedlikelihood of sustainability and impact. Only two systems (UEWEA, PHEPGambella)havebeenusedinanemergencyresponse.UEWEAinparticularwasshowntobeaneffectivesystemthatwasabletodetectseveralemergenciesthatwouldhaveotherwisegoneunnoticedininformalsettlementsaroundnairobiandthiscontributedtoimprovedtimelyandeffectiveresponse.Thissystemhadbeenoperationalthelongest,andalsoestablishedaconcreteexitstrategy,withthelocalcountygovernmenttakingovermanagementofthesystem.Thisprojectbenefittedfromalonger“effective”timelinecomparedtotheotherprojectsasittookforwardworkfromapreviousproject15thatconceptualisedthesystem,developedandtestedtheindicatorsandestablishedgovernmentcollaborations.Withoutthatexistingworktobuildon,UEWEAwouldlikelyhavefacedsimilarchallengesastherestofthecohortintermsofestablishingafunctionalsystemwiththeprogrammeperiod.

15 IndicatorDevelopmentforSurveillanceofUrbanEmergencies(IDSUE).UrbanEarlyWarningEarlyActionfinalEvaluationReport,p1.


Conclusion 2e: Emergency Preparedness and Response

There are some emerging examples in several countries of perceivedimprovementsinemergencypreparednessattheorganisationalandcommunitylevel.DocumentreviewhasdemonstratedthattheDEPPhascontributedtoatleast42responsesin11countriesandthereareseveralself-reportedexamplesofstrengthened,morelocalisedresponse.

RECOMMENDATION 2.1 [DEPP PROGRAMME BOARD] The Learning Project and its placement should be re-examined, with its role, responsibilities, objectives and accountability mechanisms clearly defined at the outset of any future programme. Programme-level M&E should sit within programme-level management, but the casecould bemade to test a differently structured standalone learning project in a futureprogramme.furtherconsiderationstoensureabalancebetweenquantityandqualityoflearningandevidencegenerated,includingtheburdenandtimerequirementsplacedonprojects,andthetimingofM&Eactivitiestobeginbeforeprojectsstartimplementingareneeded.

RECOMMENDATION 2.2 [DEPP PROGRAMME BOARD] Project-level M&E systems should directly feed into programme-level systems (via a common, core set of indicators), and should be in place prior to implementation with timely reporting to permit agile course correction. StrongerM&Esystems includingcollectionofempiricaldatawouldsupport projects in accessing additional funding andyield better quality projectswitha greater likelihood of impact. Accountability mechanisms, including accountability tobeneficiaries,shouldbestrengthened.

RECOMMENDATION 2.3 [DEPP PROGRAMME BOARD] To increase the effectiveness of capacity building interventions, emphasis should be placed on quality not on quantity. Attheindividuallevel,approachesthatbuildskillsandreinforcelearning,andcombineseveralstrategies,shouldbeprioritised.Attheorganisationallevel,approachesthataretailoredtoidentifiedgapsandaimtostrengthensystemsandprocessesshouldbe emphasised. flexible funding mechanisms providing small or in some cases micrograntsforcapacitybuildingwereonesuccessfulapproachtoempowerlocalorganisationsto build their capacity and should be considered in future programmes. Systems-levelcapacitystrengtheningshouldalsobefurtherpursued.furtherattentiontothebalanceofactivitiesacrossthesedifferentlevelsandthedesiredeffectateachlevelwillbeneededto maximise impacts. Structural barriers need to be addressed, and capacity buildinginterventionsshouldbebettertailoredtospecificcontexts.

RECOMMENDATION 2.4 [DEPP PROGRAMME BOARD] Development of early warning systems should be taken forward within future programmes but only when sufficient implementation periods are available. Athree-yeartimeframewasinsufficienttodesignandoperationaliseanearlywarningsystem;morerealistictimelinesareneededtoensurecompletionofthesystemandallowingtimetotestthesystemwithinemergencyresponses (or simulations). community members and government officials should beactivelyinvolvedindesigningthesurveillancesystems,beneficiarycriteria,andresponsepackagedetails.consensusonindicatorsandearlywarningsystemsshouldbereachedin collaboration with community and government stakeholders. Early and sustainedengagementshouldbeemphasisedtofostertrustandstrengthenpartnerships.

RECOMMENDATION 2.5 [DEPP PROGRAMME BOARD] Future programmes must consider emergency preparedness and response on a continuum with more attention paid to the response component. The programme should strategise on how best toenableimprovedresponsethroughemergencypreparednessactivitiesandensuresuch


mechanisms are embedded within the programme, rather than expecting responsecontributiontooccurorganicallyasanaturalby-productofpreparednessactivities.Thiscouldtakevariousforms.flexibleresponsefundscouldbebuiltintotheprogramme(andpotentially the individual projects) to provide a rapidmechanism for local programmebeneficiariestoimplementresponseactivities.Asthisapproachattheprojectlevelhasworkedwell inseveralcontexts itshouldbeviewedasapromisingstrategyworthyofreplicationinmoresettingsandtestedasaprogramme-levelcomponent.Alternatively,orinaddition,moreformallinkswithexistingemergencyresponsefunds(suchastheSTARTfund)couldbepursued.Inaddition,furtherthinkingaroundemergencypreparednessinconflict-affectedsettingsandinurbancontextsisneeded.

RECOMMENDATION 2.6 [DEPP PROGRAMME BOARD] To leverage the global momentum around localisation of humanitarian response and build on the contribution of DEPP in advancing this agenda, more nuanced considerations on localisation are needed. This includes articulating a clear definition ofwhat localisationmeans,whichentitiesareconsidered local andwhat the implicationsof thesedefinitionsare fromapractical standpoint. Strategically, laying out and potentially testing approacheswhichcouldenableInGOstotakeamoresupportiverole,andbetteroperationaliselocalisation(movefromtheoreticaltoactual).Organisationalpoliciesaroundlocalisationcouldbeonetarget,butunderstandinghowtooperationalisethepoliciesintoactualprocesseswouldbeanimportantelement.Additionally,thehumanitarianlandscapeindifferentsettingsanditsrelationshiptolocalisationneedtobeconsideredtoensurethatappropriate,contextuallyrelevantstrategiestosupportlocalisationareimplemented.finally,localisationfromthedonorperspectiveshouldbefurtherexploredtounderstandstrategieswhichmightenabledonorstomoreeasilysupportlocalcounterpartswithinthecontextofriskaversionandduediligenceprocesses.

3. EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS CONCLUSION 3: collaborationishighlyvaluedandDEPP’sconsortiaapproachwasuniversallydeemedtobethemostappropriate,preferreddeliverymechanismforhumanitariancapacitybuildingandemergencypreparednessandresponseactivities.Therewassomeevidencethatthe consortia model enabled coordinated, timelier response in some settings. Globally,effectivenessofconsortiacouldbe increased infutureprogrammesthroughasmaller,morestrategic set of consortia members, improved communication, opportunities for bettercoordination of activities, and provision of sufficient resources for strategic and organisedcollaborations. Inclusion of more l/nnGOs within the consortia could contribute to thelocalisationagenda.networkanalysesdemonstratedevidenceofstrengthenedhumanitarianresponse networks in the Philippines, Kenya and Ethiopia, while the network in Myanmarremained unchanged.A range of different types of emergency preparedness and responsenetworksexistinthesecountries,fromhighlyisolated(Myanmar)tohighlydistributed,locallyled(ThePhilippines).Inanyfutureprogrammeaimingtostrengthennetworks,thesedifferencesinthesize,scopeandprioritiesofnetworksacrosscountriesshouldbeappropriatelyconsideredduringthebothdesignand implementation.Humanitarian landscapesandexistingnetworksacrossfocalcountriesshouldbeconsidered,particularlyinthedesignoftheprojectsandattheprogrammeleveltoleverageexistingcollaborationsandlocalpartnerships.Targetingofnetworkmembers for capacity building and evidence sharing should be more strategic and includerelevantkeyinfluencers.

RECOMMENDATION 3.1 [DEPP PROGRAMME BOARD] In future programmes, refine and optimise the consortium model building on lessons from the DEPP, ensuring context-specific considerations. Workingeffectivelywithcollaborativestructuresinthe


futurewillrequirebetterunderstandingofwhattypesofconsortiaareappropriateandeffectivewithin different contexts. Overarching factorswhich should be incorporatedinprogrammesof thisnature include supportingcollaborative structures thatbuildonexistingsuccessfulrelationshipsandthatworkwithasmallernumberofmembers,andthatworktowardsstrategiccollaborations.Toincreasethespeedofdecision-makingandeffectiveness of consortia, streamlined contracting and reporting processes should beestablished.Ensuringl/nnGOsareincludedinconsortia,andassessmentofstrategiestoaddressriskaversionamongdonorstosupportingsuchconsortiaareneeded.

RECOMMENDATION 3.2 [DEPP PROGRAMME BOARD] Network assessments could better inform targeted strategies to enhance collaboration and to effectively strengthen emergency preparedness and response networks. country-specificnetworkmappingandassessmentsshouldbeundertaken,whenappropriate,tofacilitatetheuseofmoretargetedapproachestobuildingnetworksandpartnerships,suchasthroughtheidentificationandtargetingofkeyinfluencers.Deeperexplorationonthelevelofinfluenceandroleofdifferentorganisations(i.e.,asresourcehubs,knowledgebrokersetc.)withinthenetworkinfocalcountrieswouldbeextremelyvaluabletoinformstrategiesaroundwhichorganisationstotargetandhow.Theseassessmentscouldalsoenableoptimisationofcollaborationwithlocalandnationalactors.forexample,localisationeffortscouldberefinedinthePhilippineswherenationalnGOsarealreadyverystronglyinvolvedwithinemergencypreparednessandresponsenetworks.

4. EFFICIENCY AND VALUE FOR MONEY (VFM) CONCLUSION 4:Goodindicatorsofcosteconomywereevident,butinsomecasesprojectbudgetsweretooleanwithinsufficientresourcesforconsortiamanagementandcollaborativeactivities.Attheportfoliolevel,resourcesdedicatedtoportfoliomanagementandcollaboration,M&EandtonPAcswereinsufficientwithrespecttodesiredprogrammeobjectives.Infutureprogrammes,morestrategicallocationoffundsshouldbeundertaken.Inaddition,systemsandresources for governance, strategy, portfoliomanagement, decision-making and consortiumarrangementsshouldberestructuredtoimproveefficiency.Adaptivemanagementprocessespermittedcoursecorrectionand revisionsofprojectplans to reflectchangingcontexts, andVfMreportingimprovedoverthecourseoftheprogramme.futureprogrammesshouldbetteradoptandstandardiseVfM indicators inorder tocollectdata for internalprojectpurposes,cross-projectcomparisonsandgenerallearning.TherewerenostrongVfMfindingstosuggestthat emergency preparedness improved the efficiency of humanitarian responsewithin theprogrammeduration.

RECOMMENDATION 4.1 [DEPP PROGRAMME BOARD] Streamline contractual processes, management decisions and flow of funds. Whilerecognisingtheimportanceofsubcontractingprocessesandorganisations’internalprocedures,futureprogrammingshould takesteps tominimiseandstreamlinecontractualprocesses.forexample,pre-agreementlettersofcommitmentcouldhelpclarifysomecontractualissuespriortothestart of projects.Theprogrammemanagement could alsoprovide a template for sub-contracts and consider setting a standardtimeline for contract turnaround. If projectsare structuredwith a global grant holder and in-country host or partner, streamliningmanagementdecisionsandflowoffundssothatthesameorganisationserves inbothroleswheneverpossiblecouldbeconsidered.


RECOMMENDATION 4.2 [DEPP PROGRAMME BOARD AND PROJECTS] Adopt and standardise VFM indicators at programme and project levels and implement routine reporting. DfIDguidanceonVfMinhumanitarianprogrammesorotherrelevantVfMguidanceshouldbeusedtoenhanceVfMmonitoringandreportinginfutureprogrammes.AdoptingVfMindicatorsattheprogrammelevel,aswellastheprojectlevelwillensurethatVfMcanbecloselytrackedandanalysedatbothlevels.

RECOMMENDATION 4.3 [DEPP PROGRAMME BOARD] In a future programme, strategically allocate funds to achieve programme objectives, with greater funds designated for programme management costs, M&E and NPACs. foralargeportfolio,programmemanagementcostsshouldbeintherangeofaround10%,toenablesufficientresourcestoundertakeitskeyfunctionsofmanagement,M&Eandstrategictasks.nPAcscanbeupto20%andashighas25%,butshouldbeconsideredrealisticallytoensurethatin-countryandbackofficecostsarecovered.WithrespecttoM&E,ensuresufficientresources are available at the both the programme and project levels to efficientlyimplementbothroutinemonitoringandmorerigorousevaluations.

5. SUSTAINABILITY OF THE INTERVENTION AND LIKELIHOODOF IMPACT OF THE PROGRAMME CONCLUSION 5: Insomesettings,DEPPhascontributedtostrengtheningnationalemergencypreparedness systems but typically on a small geographic scale, and by individual projectconsortia.TherehavebeenseveralconcreteexampleswhereDEPPhasimpactedgovernmentpoliciesorsystemsor increasedpoliticalcommitment.Stakeholdersnotedashift inthewayorganisationsconsider,addressandinvolvelocalactorsandcommunities,andattributeditasoneofDEPP’smostsignificantcontributions.However,therewasalackofadetailedstrategyfortheinclusionofgenderandprioritisedtargetgroupsattheprogrammelevelresultinginafragmentedprogrammewideapproachandunevenresults. Inaddition,cross-cuttingthemessuchasaddressingVAWGwerenotadequately integrated intotheprogramme.Ratherthanonlyincludingoneortwoprojectswithaninclusionandprotectionfocuswithinaportfolio,amoreintegratedapproachinvolvingmainstreamingacrosstheprogrammeshouldbeprioritised.Thisshouldentaildisseminationofdetailedexpectationsandguidelinesoninclusionofgenderandother prioritised groups and reporting requirements during thedesignphase to ensurethatprojectsaredevelopedaccordingly.Aswell,consistencyinimplementationofgenderandinclusionconsiderationswithregularmonitoringshouldbeestablished.Inadditiontoinclusion,sustainability planning was weak and was not clearly planned or documented at both theprogrammeandprojectlevels.ThisfailureseemstohavebeenlinkedtoexpectationsaroundthepossibilityofreceivingadditionalfundsfromDfIDforasecondphaseoftheDEPP,asarticulatedinthebusinesscase,combinedwithlackofguidancefromtheprogrammeonsustainability.Inanyfutureprogramme,moredeliberate,strategicsustainabilityplanningisneededatalllevels,beginning during the design phase, andwith transparency around the possibility of furtherfunding.Attheprogrammelevel,thereissomeevidencethatbenefitshavebecomeembedded–withexamplesofsystemandpolicychangeaswellasstrengthenednationalpreparednesssystems, albeit on a small geographic scale. However, the potential for longer-term effectsisdependentontheextenttowhichdifferentcomponentsareabletocontinuebeyondthecloseoftheprogramme.futureprogrammesshouldprioritiselongerimplementationperiodstoincreaselikelihoodofimpact,andincorporategovernmentcollaboration,systems-levelchange,policyoradvocacycomponents,involvementofbeneficiariesandgoodVfMinordertopromotesustainability.


RECOMMENDATION 5.1 [DEPP PROGRAMME BOARD AND PROJECTS] Develop a strategy for inclusion of gender and prioritised target groups at programme level, ensure projects adopt an approach which aligns with this strategy and monitor implementation at all levels. Inthegovernancecriteria,considerbroadeningthegenderstatementtoastatementoninclusionofprioritisedtargetgroups.Developguidelinesforprojectsonrequirementsforthisstatement.Thiscould include linkstokeydocumentsonbestpracticesandthelawandpoliciesgoverninginclusionofprioritisedtargetgroupsin development programming and project design. Ensure gender considerations areconsistentlyimplementedandreported.

RECOMMENDATION 5.2: [DEPP PROGRAMME AND PROJECTS] Within any future programme, integrate deliberate, strategic sustainability planning that takes a more holistic view on sustainability going beyond simply securing funding streams. Programme and project sustainability plans should be developed as early as possible,preferablyinthedesignphaseandupdatedregularly.However,forriskierprojectswithahighpossibilityoffailure,sustainabilityplanningdoesnotmakesenseandisnotagooduseofresourcesuntilresultscanbedemonstrated.futureprogrammesshouldincreasethe likelihoodofsustainabilityand longer-term impactsbybuildingonpreviouseffortsand existing partnerships/consortia, working with governments (when appropriate),incorporating a policy or advocacy element, developing sustainable outputs such astools, guidelines or systems, increasing cross-project linkages, fostering sustainablerelationships that can continue beyond theDEPP, improving beneficiary engagement,andstrengtheningfeedbackmechanismsforthecommunity.GoodVfMshouldalsobeprioritisedtomaximisesustainability.Programmevisibilityatthecountrylevelshouldbemaximised through increasedadvocacy initiativesand thedevelopmentof streamlinedcommunicationwithexternalstakeholders.


INTRODUCTION& CONTEXT

cHAPTER1

33 SUMMATIVEPHASEREPORT-InTRODUcTIOnAnDcOnTExT

Eachyeartheseverityandcomplexityofnaturalandman-madedisasterscontinuestoincrease,making recovery and reconstruction efforts more challenging and costly. foreseeing thistrend, theHumanitarianEmergencyResponseReview(HERR)16urgedtheUKgovernment’sDepartment for International Development (DIfD) to invest in innovative approaches todisasterpreparednessinordertobuildlocalresilience,stating“Ifwearetomeetthechallengesahead,wehavetobe‘aheadofthecurve’ratherthanalwaysbehind;preparingfordisasters,aswellasreactingtothem.”Atthattime,lessthan5%ofallhumanitarianfundingwasinvestedindisasterpreparedness.This constituted less than1%ofOfficialDevelopmentAssistance(ODA),despitethefactthat“earlyresponseisfarmorecosteffectivethanlatehumanitarianresponse” and better preparedness is critical to more timely response17. The HERR alsohighlighted that “the levelofprofessionalism in thehumanitariansectorneeds tobe raisedthrough better investement in skills and training.” In response to theHERR,DfID createdthebusinesscase18fortheDisastersandEmergenciesPreparednessProgramme(DEPP)(SeeAnnex1)andallocated£40millionoffundingtostrengthenskillsandcapacitytoimprovethequalityandspeedofhumanitarianresponse incountriesthatareatriskofnaturaldisastersoremergencies.TheDEPPaimedtoincreasepreparednessatthelocal,regionalandnationallevelstoenablecountriestobebetterequippedtorespondtodisaster.19Itfocusedonbuildingcapacityof“nationalactorswhoareusuallythefirstonthesceneofadisaster”,anapproachnowreferredtoaslocalisation(seetheboxesbelow).

TheDEPPwasdeliveredbytwonon-governmentalOrganisation(nGO)consortia,theSTARTnetwork,(receiving£27million),andthecommunicatingwithDisasterAffectedcommunities(cDAc-n)network, (receiving £3million), to implement 14 capacity building projects. Theremaining£10millionwasreservedforaninnovationwindowwhichwasdeveloped,implemented,andmanagedseparatelywithalatertimeline.The14DEPPprojectswereimplementedinoneormoreof1020prioritycountries:SouthSudan,Kenya,Ethiopia,Mozambique,DemocraticRepublicofthecongo(DRc),Jordan,Pakistan,bangladesh,MyanmarandthePhilippines,witheachprojectoperatinginsomebutnotallcountries.Onlyoneproject(thelearningProject)wasimplementedinall10countrieswhilefiveprojects(PublicHealthEmergenciesPreparedness(PHEP)inGambella,ImprovedEarlyWarningEarlyActions(EWEA)inEthiopia,StrengtheningEmergencyPreparednessSystemsinMyanmar,UrbanEarlyWarningEarlyAction(UEWEAinKenya),financialEnablersinthePhilippines)wereimplementedinonlyonecountryeach.The14DEPPprojects,theirconsortiamembers,locationsandbudgetsarepresentedinTable1.1.

THE DISASTERS AND EMERGENCIES PREPAREDNESS PROGRAMME (DEPP)• Investment: £40 million

• Location: 10 countries

• Duration: 3 years

• Number of preparedness projects: 14 humanitarian capacity building projects

• Overall objective: To improve quality and speed of humanitarian response in countries that are at risk of natural disasters or humanitarian emergencies

Key approaches: Capacity building of local and national humanitarian staff and communities, early warning system development, supporting collaborative action and strengthening networks

16 HumanitarianEmergencyResponseReview(HERR),https://www.gov.uk/government/publications/hu-manitarian-emergency-response-review.

17 businesscaseInterventionSummary:DisastersandEmergenciesPreparednessProgramme,DfID.18 Ibid.19 ibid.20 Indonesiawasinitiallyselectedasan11thDEPPfocalcountry,butnoselectedprojectsplannedtoimple-

mentactivitiesthere.


BET

TER

D

IALO

GU

E,

BET

TER

IN

FOR

MA

TIO

N,

BET

TER

A

CTI

ON

THE

LEA

RN

ING

PR

OJE

CT

FIN

AN

CIA

LEN

AB

LER

S

SHIF

TIN

G

THE

POW

ER

TALE

NT

DEV

ELO

PMEN

T

TRA

NSF

OR

MIN

G

SUR

GE

CA

PAC

ITY

PRO

TEC

TIO

N

IN P

RAC

TIC

E

ALE

RT

AG

E &

DIS

AB

ILIT

Y

CA

PAC

ITY

PR

OG

RAM

ME

(AD

CA

P)LI

NK

ING

PR

EPA

RED

NES

S,

RES

PON

SE A

ND

R

ESIL

IEN

CE

PUB

LIC

HEA

LTH

EM

ERG

ENC

IES

PREP

AR

EDN

ESS

IN G

AM

BEL

LA

STR

ENG

THEN

ING

EM

ERG

ENCY

PR

EPA

RED

NES

S SY

STEM

S IN

M

YAN

MA

R

IMPR

OV

ED E

AR

LY

WA

RN

ING

- EA

RLY

A

CTI

ON

ETH

IOPI

A

UR

BA

N E

AR

LY

WA

RN

ING

, EA

RLY

AC

TIO

N

3-YEARPROGRAMME

£40MILLIONFUND 14

PROJECTS10COUNTRIES

45ORGANISATIONS

KENYA

PAKISTAN

BANGLADESH

PHILIPPINES

DRC

MOZAMBIQUE

ETHIOPIA

MYANMAR

SOUTH SUDAN

JORDAN

The Disasters and Emergencies Preparedness Programme uses collective networks to improve the quality and speed of humanitarian response in countries at risk of natural disaster or con� ict related humanitarian emergencies.

£500

,000

£784

,256

£954

,698

£675

,000 £3

,343

,375

£1,6

55,9

20

£3,0

00,0

00

£4,8

76,6

36

£5,9

85,0

87

£2,4

82,8

24

£804

,634

£1,9

87,0

00

£1,0

43,6

73

£978

,987


EMERGENCY PREPAREDNESS

Emergency preparedness refers to “the abil-ity of governments, professional response organisations, communities and individuals to anticipate and respond effectively to the impact of likely, imminent or current hazards, events or conditions. It means put-ting in place mechanisms which will allow national authorities and relief organisations to be aware of risks and deploy staff and resources quickly once a crisis strikes.”21

Emergency preparedness focuses narrow-ly on improving response to disasters, but it belongs to a wider set of activities to reduce the likelihood and impact of disas-ters on people’s lives called ‘Disaster Risk Reduction’ (DRR). These include activities focused on prevention, mitigation and response to humanitarian emergencies.22

The IASC Common Framework for Pre-paredness23 describes the following com-ponents of emergency preparedness:

• Hazard and risk analysis and early warning systems

• Institutional and legislative frame-works

• Resource allocation and funding• Coordination• Information management and com-

munication• Preparedness and contingency / re-

sponse planning• Training and exercises• Emergency services, standby arrange-

ments and prepositioning

The DEPP aimed to focus on those areas related to ‘people and systems’, in partic-ular hazard, risk and early warning; infor-mation management and communication; contingency/preparedness and response planning; and training and exercises.24

LOCALISATIONThere is no globally accepted definition of ‘localisation’ of humanitarian aid. The defini-tion suggested by Trocaire and Group URD is holistic and appears to align very closely to the DEPP approach to localisation: “Aid localisation is a collective process involving different stakeholders that aims to return local actors, whether civilsociety organisa-tions or local public institutions, to the centre of the humanitarian system with a greater role in humanitarian response. It can takea number of forms: more equitable partner-ships between international and local actors, increasedand “as direct as possible” funding for local organisations, and a more central role in aid coordination. Underpinning this is the question of power. Localisation requires a shift in power relations between actors, both in terms of strategic decision- making and control of resources.” 25

The OECD states that supporting local humanitarian responders contributes to the following:26

• Early response and access (including to small scale crises below the threshold for international aid)

• Improved acceptance of humanitarian aid (such as in conflict-affected areas)

• Cost effectiveness• Links with development• Increasing accountability

The growing calls to increase support for local humanitarian responders is also re-flected in a number of policy commitments including:

• Good Humanitarian Donorship, Princi-ple 827

• The Grand Bargain, Workstream 228

• The Sendai Framework for Disaster Risk Reduction29

• The 2030 Agenda for Sustainable Development30

21 https://www.humanitarianresponse.info/en/coordination/preparedness/what-preparedness.22 ibid.23 Inter-AgencyStandingcommittee(IASc)commonframeworkforPreparedness,2013.24 businesscaseInterventionSummary:DisastersandEmergenciesPreparednessProgramme,DfID.25 Morethanthemoney-localisationinPractice,GroupeURD,Trocaire,2017.26 OEcD,localisingtheResponse,2017.27 GoodHumanitarianDonorshipPrinciples,https://www.ghdinitiative.org/ghd/gns/principles-good-practice-

of-ghd/principles-good-practice-ghd.html.28 TheGrandbargain,https://interagencystandingcommittee.org/grand-bargain-hosted-iasc.29 SendaiframeworkforDisasterRiskReduction,https://www.unisdr.org/we/coordinate/sendai-framework.30 The2030AgendaforSustainableDevelopment,https://sustainabledevelopment.un.org/post2015/transformingourworld.


DEPP PROJECTPROJECT

CONSORTIA MEMBERS

COUNTRIES** DESCRIPTIONBUDGET

(£)TA

LEN

T D

EVEL

OPM

ENT

(TD

)SavethechildrenUK(lead),OxfamGb,Re-liefInternational,cHS

Alliance

bangladesh,DRc,Ethiopia,Jordanand

Kenya,lebanon

Toproducehigh-qualityprofessionalsatalllevelswhoarebetterequippedtotackletheissuessur-roundingcomplexemergencies,helpingtoensurethattherightpeopleareintherightplacedoingtherightthingstoassistdisaster-affectedcommu-

nities.

5,985,087

SHIF

TIN

G TH

E PO

WER

(STP

) ActionAidInternational(lead),cAfOD(lead),christianAid,Tearfund,concernWorldwide,

Oxfam

bangladesh,Pakistan,Ethio-pia,KenyaandDemocraticRepublicofcongo(DRc)

Tosupportlocalactorstotaketheirplacealong-sideinternationalactorsinordertocreateabalancedhumanitariansystemthatismorere-sponsiveandaccountabletodisaster-affected

communities.

4,876,637

THE

LEAR

NIN

G

PRO

JECT

*

ActionAgainstHungerAllDEPPcoun-

tries

Toevidencetheextenttowhichpreparingpeopleisaneffectiveandefficientapproachtodisastermanagementandbrokerinternalandexternalrelationshipsforlearningaboutwhatisandisnot

workingincapacityexchange.

3,343,375

BETT

ER D

IALO

GUE,

BE

TTER

IN

FORM

ATIO

N,

BETT

ER A

CTIO

N

(CD

AC-N

PRO

JECT

)

WorldVision(lead),cDAcnetwork,Thom-sonReutersfounda-tion,bbcMediaAction,

Internews

bangladesh,SouthSudan,ThePhilippines

Toensurethattwo-waycommunicationisapredi-cable,coordinatedandresourcedcomponentofhumanitarianresponseinordertocontributetoimprovementineffectivedeliveryofassistanceto

disasteraffectedcommunities.

3,000,000

TRAN

SFO

RMIN

G

SURG

E CA

PACI

TY

(TSC

)

ActionAidInternational(lead),Acf,christianAid,cAfOD,cARE,IMc,IslamicRelief,MuslimAid,Plan,Savethechildren,Tearfund

Pakistan,thePhilippines,RegionalHubinbangkok,Thailand

Tostrengthencivilsocietysurgecapacityatinter-national,regionalandlocallevels,contributingtoadiverseanddecentralisedthirdsectorpillarbetterabletocomplementexistingUnitednations,Redcrossandgovernmentstructuresinordertohelpcommunitiesincreaseresilience,reduceriskand

improvecrisisresponse.

2,482,824

ALER

T

HelpAgeInternational(lead),careinterna-tional,HandicapInter-national,IslamicReliefWorldwide,concernWorldwide,Oxfam,coventryUniversity

ThePhilippines,Pakistan,ban-gladesh,Kenya,Mozambique,Somalia,Haiti

Todevelopasystemofapproachesandtoolsfordisasterpreparednessthatincreasestheabilityoforganisationstorespondimmediately,effectively

andappropriatelywhenadisasterstrikes.

1,987,000

FIN

ANCI

AL

ENAB

LERS

(F

E) OxfamGb(lead),Tear-fund,christianAid

ThePhilippines

Totransferhumanitariancapacity,autonomyanddecision-makingtoorganisationsclosertopeopleaffectedbycrisis,asawayoffacilitatingmore

effectiveandappropriateaid.

1,637,745

AGE

AND

D

ISAB

ILIT

Y CA

PACI

TY

BUIL

DIN

G

(AD

CAP)

HelpAgeInternational(lead),cbM,Disaster-Ready.org,HandicapInternational,IfRc,

OxfordbrookesUniver-sity,RedRUK

Kenya,Pakistan

Toensureolderpeopleandpersonswithdisabilitybenefitfromimprovedaccesstoservices,asare-sultofrecognitionbyhumanitarianactorsoftheirspecificneedsandincreasedcapacityamongsthumanitarianactorstodeliverinclusive,accessible

andappropriateresponse.

1,045,159

LIN

KIN

G

PREP

ARED

NES

S RE

SPO

NSE

AN

D

RESI

LIEN

CE (L

PRR) christianAid(lead),

ActionAid,concern,HelpAgeInternational,King’scollegelondon,MuslimAid,Oxfam,Saf-erworld,WorldVision

bangladesh,Kenya,DRc,Pakistan,thePhilippines,colombia,Indonesia***,Myanmar

Todesignandrolloutprogrammingapproacheswhichstrengthentheresilienceofpeoplelivinginfragilestatesandbeneficiariesofhumanitarian

assistance.

1,002,964


STRE

NG

THEN

ING

EM

ERG

ENCY

PR

EPAR

EDN

ESS

SYST

EMS

IN

MYA

NM

AR (S

EPS)

christianAid(lead),DcA,RegionalIntegrat-edMulti-HazardEarlyWarningSystemforAfricaandAsia(RIMES)

Myanmar

Tostrengthenthelinksamongnational,subnation-alandlocal-levelpreparednessinMyanmar,withaprimaryfocusoncapacitatinglocalcommunitiesandstructurestoaccessinformationandlinkwiththeon-goingestablishmentofpreparednessand

earlywarningsystemsinthecountry.

925,698

PRO

TECT

ION

IN

PRAC

TICE Oxfam(lead),IRc,

WorldVisionInterna-tional

DRc,lebanon,Turkey,SouthSudan,Myan-mar,PakistanandthePhilip-

pines

ToenablenationalnGOstoimplementprotectionactionsindisasterandconflictresponses,devel-opnewtypesofpartnershipsandcollaborationsbetweenprotectionactorsandinfluencethe

internationalprotectionarchitecturesoitismoreinclusiveofnationalnGOs.

870,137

IMPR

OVE

D EA

RLY

WAR

NIN

G – E

ARLY

AC

TIO

NS T

O ST

REN

GTH

EN

DIS

ASTE

R PR

EPAR

EDN

ESS

IN ET

HIO

PIA

(EW

EA)

OxfamGbonbehalfofAfricacllimatechangeResilienceAlliance(lead),christianAid,nationalDisasterRiskManagementcommis-sion,nationalMeteoro-

logicalAgency

Ethiopia

Tocontributetoimprovedemergencyprepared-ness,timelinessandqualityofriskinformationandresponseactionsbycommunity,government(localtofederal)andnGOsinacoordinatedmanner

784,255

URB

AN E

ARLY

W

ARN

ING

, EAR

LY

ACTI

ON

(UEW

EA)

concernWorldwide(lead),Oxfam,KenyaRedcrossSociety

Kenya

Toimproveurbanearlyactionbyimprovingthealignmentoflocalandmunicipalgovernanceinsti-tutionsandresponseagenciestowardsidentifyingrelevanttriggersandimplementingearlyresponsebasedonthese,supportedbyanincreaseintheallocationoffinancialsupportfromkeydonorstowardsurbanEarlyWarningEarlyAction.

675,000

PUBL

IC H

EALT

H

EMER

GEN

CIES

PR

EPA

RED

NES

S IN

GA

MBE

LLA

(P

HEP

)

christianAid(lead),AmrefHealthAfrica,EthiopiaMinistryofHealth,nationalMete-orologyAgency

EthiopiaTostrengthenearlywarning,preparednessandpromptresponseofthehealthsectortopublic

healthemergencies.548,694

Table1.1:ProjectsintheDisastersandEmergenciesPreparednessProgramme31

*Formerly known as the Monitoring, Evaluation and Learning Project (MEL).

**Includes non-DEPP focal countries where projects were implemented (i.e., Lebanon, Somalia, Haiti, Colombia, Turkey).

***Indonesia was initially selected as an 11th DEPP focal country but none of the selected projects planned implementation in Indonesia. However, over the course of the programme, LPRR ultimately did implement some activities in this country.

With14individualprojects,theDEPPwasbroadinscopeandcomplexindesign.Eachprojectimplementedadifferentsetofactivities,andworkedwithindifferentstructures.Theprojectsalso targeted various levels of action (or beneficiaries) including individuals, communities,organisations and governments. The programme utilised a number of strategies includingcollaboration, innovation, scale, decentralisation, complementarity, learning and addressingcross-cuttingissues.ThefiveoverallobjectivesortargetedresultareasoftheDEPPasoutlinedinthebusinesscasearepresentedintheboxbelow32.Whileeachprojectworkedtowardsoneormoreoftheseresultsareas,asawholethecollectiveactionoftheprogrammeaimedtoaddressallfive.

31 TheSTARTnetwork.DisasterandEmergenciesPreparednessProgrammeInfographic.August2015.Accessedon18february2016athttp://www.start-network.org/wp-content/uploads/2015/08/DEPP-In-fographic.pdf.

32 InallcountriesexceptJordan,thecombinedprojectscoveredallfourDEPPobjectives.TalentDevelop-mentwastheonlyprojectimplementedinJordananddidnotcoverthe3rdand4thobjectives.


DEPP RESULT AREAS1. Improved knowledge and understanding of individuals by sharing best practice for

humanitarian preparedness and response

2. Improved preparedness systems for early action with communities at risk of disasters

3. Increased number of coalitions, partnerships and networks which, working together, are able to address humanitarian needs in a wide variety of emergency situations

4. Improved institutional arrangements and policy environments so that national systems for humanitarian response and preparedness are better supported and more sustain-able

5. Strengthened evidence base for what works to help build humanitarian capacity at scale

A3-year external impact evaluationof theDEPPprogrammewas commissionedbyActioncontrelafaim(Acf)(seeToRinAnnex4),andconductedbyanevaluationteam(Annex2)attheHarvardHumanitarianInitiative(HHI).Thisreportpresentsthesummativeresultsoftheimpactevaluationincudingfindingsfromallphasesoftheevaluation.

DEPP THEORY OF CHANGE AND SIMPLIFIED CAUSAL CHAIN During theevaluation inceptionphase, theevaluation teamassessed theDEPPprogrammetheoryofchange(Annex3)presentedinthebusinesscase(Annex1).Theprogrammetheoryofchangewasfoundtobecomplexandnolongerrepresentedwellthesuiteofprojectsastheyhadevolvedovertime.AprocesswasundertakentocreateamoresimplifiedbutconceptuallyappropriaterepresentationoftheDEPPthatcouldserveasthebackboneoftheevaluationapproachandtoreportagainstduringeachevaluationphase.figure1.1illustratestheDEPPprogrammecausalchainthatwasdevelopedandthatwasusedtocreateaprogramme-levellogicalframeworkandsetofindicatorsfortheevaluation33.Thecausalchainillustratesthecausallinksbetweenthe14DEPPprojectsandtheiractivities,fourexpectedoutputareas(1.capacitybuilding,2.collaboration,3.learningand4.EarlyWarningSystemDevelopment),expectedoutcomesinboththeshortterm(improvedhumanitariancapacity,improvedpreparedness)andlongterm(increasedeffectivedeliveryofhumanitarianresponse).Hypothesiseddownstreamimpacts (mortality, morbidity, economic impact and recovery) are also depicted within thecausalchain,andbasedonstakeholderinterviewswerenotexpectedtobeachievedwithintheprogramme’s3-yearcycleandthuswereconsideredoutsideofthescopeoftheevaluation.Thecausalchainalsospecifiesthattheactivities,outputsandoutcomesareexpectedtooccurat the individual, organisational, community and government levels. This also captures thevarietyofDEPPbeneficiariesthatweretargetedateachoftheselevels(includingindividualhumanitarian staff, humanitarian organisations, communties, governments etc). This impactevaluationfocusedonlyonassessingoutputs,aswellasshorttolong-termoutcomesatthesedifferentlevels.

33 Page17oftheDEPPbusinesscasestated,“Throughacommissionedevaluation,wewilldeveloptheTheoryofchangeinmoredetailaspartoftheevaluationinceptionstage,andwewilltestourcoreassumptionsthroughtheDEPPevaluation.”TheevaluationteamrevisedtheTheoryofchangetoenableafeasibleevaluationdesignasmandatedintheinceptionphaseandthetermsofreference.TherevisedcausalchainwasapprovedbytheEvaluationSteeringcommitteeandthusbecamethebasisfortheevalu-ationdesign,includingtheevaluationframeworkandindicators.notethatwhiletherewereseveraldraftprogrammelogicalframeworksinplacein2016,thefinalversionwasnotapproveduntil2017.


figure1.1:SimplifiedDEPPprogrammecausalchain

DEPP PROGRAMME-LEVEL LOGICAL FRAMEWORK AfinalversionoftheDEPPprogramme-levellogicalframeworkwasapprovedbyDfIDin2017andfullyimplementedbyAugust2017,severalyearsafterthestartoftheprogramme.ThefinalversionofthelogicalframeworkispresentedinTable1.2below.Whiletherewereseveraldraftversionsoftheprogramme-levellogicalframeworkinplacebeforethen,theyrequiredsignificantstrengthening34.Therefore,withagreementfromDfID,becausetheexternalevaluationhadalreadybegunbefore thefinalisationof the logical framework, itdoesnot reportagainst it.However, it is important to note that the output, outcome and impact areas in this logicalframeworkarequitesimilartotheevaluationteam’soutputandoutcomeareasasdepictedinfigure1.1.35Theindicators,however,areunique(SeeTable2.1intheMethodologysectionforfurtherdetail).

34 forexample,DfID’sAnnualReviewoftheDEPP,March2018states:“Thereisstillmoreworktodoonthelogframe–includingfinalisingtheoutcomeindicatorsandincludinganoutputwhichtracksvalueformoney.”

35 Programmeoutput1correspondstoevaluationoutput1,programmeoutput2toevaluationoutput4,programmeoutput3toevaluationoutput2,programmeoutput4doesnotdirectlycorrespondwithanyevaluationoutputarea,programmeoutput5correspondstoevaluationoutput3,programmeoutput6doesnotcorrespondtoanyevaluationoutputarea,programmeoutcome1correspondswithevaluationoutcome1,andprogrammeimpact1correspondswithevaluationimpact1.Programmeoutputarea6correspondstotheinnovationwindowwhichisnotwithinthescopeofthisevaluation.

DEPPProjects

HumanitarianCapacity

Preparedness

EffectiveDelivery ofHumanitarianResponse

MortalityMorbidityEconomic ImpactRecovery

LEVELS OF ACTION

IndividualsCommunitiesOrganisationsGovernment

INPUTS &ACTIVITIES

OUTPUTS SHORT TERMOUTCOMES

LONG TERMOUTCOMES

IMPACT

Collaboration

CapacityBuilding

Early WarningSystem

Development

Learning


STATEMENT INDICATOR TARGET (MARCH 2018) ASSUMPTIONSO

UTP

UTS

1.Improvedknowledgeandunderstandingofindividualsbysharingbestpracticeofhumani-tarianpreparednessandresponse

1.1#ofDEPPhuman-itariancapacitybuildingprogrammes,countriesandstrategiesthatutilisearangeofimplementa-tionmethodstoimprovethecapacityofindivid-ualsattheinternational,nationalandsub-nationallevels

12humanitariancapacitybuildingprogrammesarerevised,scaledupordevelopedin12coun-tries,usingamixof6strategiesincluding:shortcourses,longcourses,trainingoftrainers,men-toring/coaching,e-learn-ingandwebinars

capacitybuildingac-tivitieswillincludeadiversityandinclusionapproach. Individualhumanitarianworkerscantransmitlearningintoactionwith-intheirorganisationsandwideroperatingenviron-ments. chosenapproaches,methodologies,andin-novationsarethecorrectonestobuildcapacityinanincreasinglycomplexhumanitarianecosystem. Shortcourse=lessthanonemonth longcourse=morethanonemonth

1.2#ofstaffacrossmul-tiplehumanitariandisci-plineswhoaccessqualityDEPPhumanitarianca-pacitybuildingactivitiesofvaryingdurationandqualificationonarangeofsubjects

Atleast4200individuals(50%male,50%female)haveaccessedqualityDEPPhumanitarianca-pacitybuildingactivitiesacross10subjectstovariouslevelsofaccred-itation(includingshortcourses,long,courses,e-learningandToT)

2.Improvedprepared-nesssystemsforearlyactionwithcommunitiesatriskofdisasters

2.1#ofcountrieswithpreparednesssystemsforcommunities,agen-ciesandgovernmentinstitutionsdevelopedorstrengthened,includingearlywarningsystems,preparednessplans,col-laborativesurgerosters

Preparednesssystemsinuseinatleast7countriesandatleastthreegov-ernmentinstitutions,andcapacitybuiltinrespon-sibleinstitutionsatcom-munityandnationallevelandtestedviasimulationinatleastonecountryorwherearesponseisrequired

Asystemsapproachtopreparednessissuitedtoidentifiedcountriesandpopulations,aswellmul-tipleformsofhazards. Onceequippedwithacoordinatedsystemstakeholdersareabletotakeearlyaction. Resultsexpectedfromthedeploymentofthesesystemswillhaveapos-itiveimpactonincentiv-isingpreparednessandearlyaction. note:Thelanguageusedforthisindicatorhasbeenchosenonpur-posetoclarifythatthepreparednesssystemsdevelopedbytheDEPParenotnationalpre-parednesssystems.Theorganisationsthatwillusethesepreparednesssystemswillincludelocal,regionalandnationalgovernmentbodies,In-GOs,nationalnGOs,andotherinternationalactors

2.2.Improvedprepared-nesssystemsforearlyactionwithcommunitiesatriskofdisasters

6systematiccommu-nicationapproaches(includingearlywarningbulletinsandsurveillancereports)operationalinatleast3countriesatmultiplelevelsandtestedviasimulationinatleastonecountryorwherearesponseisrequired


OU

TPU

TS

3.Increasednumberofcoalitions,partnershipsandnetworkswhichworkingtogether,areabletoaddresshuman-itarianneedsinawiderangeofemergencysit-uations

3.1formalpartnershipsareestablishedandmain-tainedwithawiderangeofnationalandinterna-tionalactors(bothwithinandoutsidetheDEPPconsortium)thatincreaseactionandcommitmenttoaddresshumanitarianneedsinemergencies

Partnershipsbetweenatleast200nationalandinternationalhumani-tarianactors(includingnationalnGOs,interna-tionalnGOs,academicinstitutions,governmentagencies,privatesectorcompanies,andRedcross/crescentSocieties)demonstrateincreasedactionandcommitmenttoaddresshumanitarianneedsinemergenciescollaboratively Atleast70%oftheseac-torscommittomaintain-ingatleastoneoftheirpartnershipsbeyondtheendoftheprogramme,inordertocontinueaddressinghumanitarianneedscollaboratively.

Modelsofcollaborationbeingtestedcontributetoimprovingeffective-nessinpreparednesswork. collaborativemecha-nismscreateddonotreinforcesystematicdisempowermentandexclusionofcrisisaf-fectedpopulations,firstresponders,localandnationalhumanitarianleadership. Platforms,forumsandcommunitiesofprac-ticeactivelyreflectonstrengthsandweakness-esofdifferentapproach-essothattheyremainrelevantmechanismsforthehumanitariansys-tem.Aspectsofexistingandnewmechanismsthatdon’tworkcanbeaddressed,andthereiswillingnesstochange.

3.2collaborativemech-anismscontributetoimprovedlearning,effi-ciencyandcoordinationofDEPPstakeholdersforemergencyprepared-ness,programmingandresponse

Analysisof140collabo-rationstoriesandprojectfeedbackindicatethatcollaborationhasledtoimprovedlearning,effi-ciencyandcoordination

4.Improvedinstitutionalarrangementsandpolicyenvironmentssothatnationalsystemsforhu-manitarianresponseandpreparednessarebettersupportedandmoresuitable

4.1numberofstrategies(includingactionplans,guidelinesandtools)coveringvariousthe-maticareasadoptedbynationalandinternationalactorsinarangeofcoun-triestoachievemoreappropriatehumanitarianresponseforcrisisaffect-edpopulations

13project-levelstrate-gies(includingorganisa-tionalactionplans,guide-lines/standardsandtools)around7thematicareasareimplementedacross11countries

Strategiesforadvocacyactivitiesarethecorrectonessuitedtothewiderhumanitariansector. Vulnerablegroupsandcrisisaffectedpopula-tionsvoicesareconsis-tentlyintegratedintoinfluencinghumanitarianpreparedness. Initiativesandinnovationareabletoinfluencepolicyandpracticeofor-ganisationsorthewiderhumanitariansector.

4.2DEPPadvocacyandinstitutionalchangeactivitiestargetcorehumanitarianthemessuchasageanddisability,protectionandcommuni-catingwithcommunities

8corehumanitarianthemestargetedthroughadvocacyandinstitution-alchangesactivities


OU

TPU

TS

5.Strengthenedevi-dencebaseforwhatworkstohelpbuildhu-manitariancapacityatscale

5.1EvidenceacrosstheDEPPisgeneratedandcapturedsystematicallyandcontributestofutureuptakeoflessonslearnedtohelpbuildhumanitari-ancapacityatscale

Atleast200learningresourcesthatpromotetheuptakeofbestprac-ticeandlessonslearnedfromtheDEPPareproducedandshared,coveringabroadrangeofpreparednesssubjectareasincludinghuman-itariancapacitybuild-ing,resiliencebuilding,preparednesssystems,humanitariansurge,local-isation,collaboration,ageanddisabilitycapacitybuildingandprotectionmainstreaming EveryDEPPprojectdemonstratesatleastoneexampleofsignif-icantadaptiveprojectmanagementbasedonrecommendationsfromDEPPlearningresourcesonhowtoimprovehu-manitarianprogramming.

currentproblemsandgapsintheevidenceonhumanitariancapacitybuilding/enhancementforpreparednesscanbeidentifiedandsolutionscanbefacilitatedwithintheprogramme. Thereiswillingnessinidentifiedcountriesandthewiderhumanitariansectortoacceptevidencegatheredtofeedbackintoarangeofcapaci-ty-building/enhancementeffortsandactivities. Itwillbepossibletoreferencelearningandevaluationstomeasurethedegreeofuptakebyprojectsandthewiderhumanitariansector. newandexistingDfIDpreparednesscapacitybuildingprogrammesdesignand/orimplemen-tationapproachesareinformedbytheevidencefromDEPP.

5.2DEPPMElprojectpromotesgeneration,sharinganduseofevi-denceandlearning

DEPPMElprojectpro-videsprojectsupporttoallother13DEPPproj-ects,co-creates13re-searchpieceswithDEPPpartners,managestheDEPPexternalevaluationinitssummativephaseanddelivers20learningeventsacross9countries

6.Humancenteredde-signapproachestestedandimplementedacrossfourinnovationslabs,leveragingnon-tradition-allocalactorstodeliverpreparednessapproach-eswhicharecommuni-ty-appropriate,leadingtomorepreparedandengagedcommunities

6.1communitiesareactivelyidentified,en-gagedandinvolvedinallstagesoftheinnovationsprocessandoutputsareassessedusingcommu-nity-basedevaluationmethods.

Relevant/usercommu-nitieshaveevaluatedlabactivitiesforrele-vanceandusefulness.Independentcommunityperceptionsurveydatagatheredacrossfourlabstocomplementcontex-tualisedparticipatorymethodsforcoherentcomparisonsandreport-ing.

communityengagementinthedesignprocessleadstogenerationofnewideasanddevelop-mentofmoreappropri-atepreparednessmech-anisms. communitiesaremoti-vatedtoremainengagedwiththeinnovationpro-cessbecauseofthevaluetheyattributetobeinginvolvedindevelopingmoreusefultools.


OU

TPU

TS

6.cont.

6.2DEPPlabsinthefourcountriestest,pilotandsupportinnovationsthroughaninnovationprocess,learningfromwhichiscuratedandsharedwiththebroadernetwork.

Atleast48projectssupportedandtestedacrossthefourlabs.Scaledinnovationsevaluatedandsharedacrossfourcountries.lessonslearnedaboutnon-traditionalsectorengagementandcon-textualisedapproachestocommunitycentereddesigninemergencypre-parednesssharedacrossthebroadernetwork.finalevaluationincludesexternalassessmentofsustainability,qualityandeffectivenessofcommu-nityledinnovationpro-cessesbyrelevantlocalinnovationsactors.

Someinnovationshavealreadybeenpioneered,whichcanbeidentifiedandscaledupmorequicklywithlabsupport. labsupportmechanismsincludingtraining,fund-ingandmulti-sectoralpartnershipsareade-quatetoprogressinnova-tionstoapointofgreaterviability. Sharinglessonslearnedwithnetworkmemberscontributestobehaviourchangewithinthesector.

OU

TCO

MES

1.IncreasedandstrengthenedemergencypreparednesscapacityinDEPPfocuscountries,focusingonstrengthen-inglocalhumanitariancapacityandchampi-oninglocalisationinamannerconsistentwiththeGrandbargainandtheWorldHumanitarianSummit

1.Proportionofre-cipientsofqualityhumanitariancapacitybuildingactivitiesthatdemonstrateimprovedknowledge,attitudesorbehavioursincorehu-manitarianareas

Outofsampletakenfrom13humanitari-ancapacitybuildingprogrammes,70%willdemonstrateincreasedknowledge,attitudesandbehaviours

Oneassumptionisthattrainingstaffwillresultinincreasedorganisationalpreparedness-wehavetriedtoaddressthisbyincludingoneindicatoraroundstaffknowl-edge(Indicator1)and1aroundnGOreportsofcapacitystrengthen(Indi-cator4) coordinationmecha-nismsandpreparednesssystemsarewellintegrat-edintoorganisationsandstaffreceiveadequatetraining. Strategiestoaccomplishinstitutionalchangearebothdiverse,flexibleandeasytointegrateintoexistingprogramming,withappropriatesupportandtrainingprovided.


OU

TCO

MES

1.cont.

2.DEPPcoordinationmechanismsandpre-parednesssystemspro-moteefficientresponsetoemergencies,withclearpoliciesandproce-duresandinstitutionalsupport

feedbackon100%ofpilotmechanismsandsystemsindicatesthat70%ofasampleofstakeholdersreportin-creasedpreparedness,andthatthesurroundingpoliciesandproceduresareadequate. AnalysisfromaDEPPreturnoninvestmentstudydemonstratesthatDEPPsystemspromoteefficientresponsetoemergencies.

3.DEPPagencies,withaparticularemphasisonlocalandnationalnGOs,implementstrategieswhichresultinimprovedprogrammingforhumani-tarianpreparedness

AllDEPPagenciesdemonstrateimprovedhumanitarianprogram-ming(includingpolicies,plansandlearning) l/nnGOssupportedbyDEPPreportmorein-creasedcapacitytopre-pareforandrespondtodisastersthansampleofl/nnGOsnotsupportedby DEPP

IMPA

CT

1.DEPPreducessuffer-inganddeliversbetterhumanitarianservices,throughimprovedpre-paredness,forpeoplearoundtheglobeaf-fectedbydisastersandemergencies

1.Reductioninthenum-berofliveslost,affectedcommunitiesandeco-nomiclossesduetoim-provedpreparednessin10DEPPfocuscountries

In2017thenumberofreporteddeaths,totalpeopleaffected,andeco-nomiclossesduetonat-uraldisastersin10DEPPcountriesisreducedfromthepreviousyear

MoredirectfundingtolocalnGOs,aswellasmorefundingintodisas-terpreparedness,willresultinfasterandmoreefficientresponseontheground.

Itisassumedthatforindicator1thatinlinewithWHSguidelines,allagencieswillself-reportannuallyagainstcommit-ments. Reductionsinoverallnumbersofliveslostandeconomicdamagesinthe10DEPPfocuscountriesareprimarilyduetohu-manitarianandgovern-mentinitiatives,althoughannualandseasonalfluc-tuationsintheseverityandnumberofdisastersalsoplayarole.


2.Humanitarianactors,includingnGOsandgov-ernments,investmoreinemergencypreparednessprogrammingtoensurethatresponsetocrisesarebetteranticipatedandplanned

Thetotalamountofglobalofficialhuman-itarianassistancere-portedascommittedtodisasterpreventionandpreparednessin2017isUSD2500m

note:forimpactindi-cator2thedataisonagloballevel-itwasnotpossibletodisaggregatebyDEPPcountry.Also,theDAccodesforre-portingfundscommittedtodisasterpreventionandpreparednesswillchangeintheupcomingyear,sothatmayaffectdataconsistencyacrossmilestones. note:ThedataforIndi-cator1isavailabledisag-gregatedbycountry.

Table1.2:DEPPprogramme-levellogicalframework(fullyimplementedbyAugust2017)

THE DEPP EXTERNAL EVALUATION Purpose

Thethree-yearexternal,independentevaluationincludedprocessandperformanceaswellasimpactevaluations.Theoverallaimsoftheindependentevaluationare:

1) to improve programme effectiveness and enhance learning

2) to assess the extent to which the DEPP overall has provided an efficient and effective approach to strengthening response capacity.

TheresearchdesignfortheevaluationwasguidedbyasetofquestionsandmethodologicalapproachasoutlinedintheTermsofReference(ToR)(Annex4).TheevaluationusedamixedmethodsdesigninordertocaptureacomprehensivepictureoftheDEPP’seffectiveness.Theevaluationwasconductedinfourphases(inception,formative,interim,andsummativephases).Thedetailedmethodologycanbefoundinchapter2.The evaluation focuses at the programme level, and it is important to note that individualevaluationsofthe14individualprojectsarenotwithintheremitofthisevaluation.Theevaluationservestoprovideaccountabilityandlearningforprogramme,projectandexternalstakeholders.ThesestakeholdersincludeDEPPprojectconsortiamembers,DEPPbeneficiaries,programmeandproject-levelstaff,aswellasexternalstakeholdersfromhumanitariannGOs,internationalorganisations,theUn,andgovernments.

Theevaluation’sinceptionphasewascompletedinJanuary201636.TheinceptionphasereportreviewedtheDEPPtheoryofchange,andpresentedtherefinedevaluationmethodologyanditslimitationsaswellastheprocessforselectingcountriesfortargetedintensivedatacollectionduringtheformative,interim,andsummativephasesoftheevaluation.Theevaluation’sformativephase focusedon relevanceofprogrammeoutputs and theefficiency andeffectivenessofprogramme delivery. The formative phase report, completed in May 201737, assessed theprogrammedesignstage(2011toearly2015),theprogrammeimplementationstage(October

36 EvaluationoftheDisastersandEmergenciesPreparedness(DEPP)Programme:InceptionPhaseReport,feb18,2016.PhamP,SharmaV,Scott,J,Gleason,K,Gibbons,n.

37 Sharma,V,HémonoR,ScottJ,Gibbonsn,ShahV,GleasonK,yangn,HavilandM,PhamP.(2017)TheDisastersandEmergenciesPreparednessProgrammeEvaluation:formativePhaseReport,May3,2017.HarvardHumanitarianInitiative.


2014 toMarch2017),andpresentedbaselinequantitative indicators related toknowledge,attitudes,exposuretotheDEPP,andemergencypreparedness.Theinterimphasereportwascompleted inOctober2017andpresentedperceptionsonprogramme implementationandoutputs(October2014-October2017),short-termoutcomechanges,aswellasprogresssincetheformativephaseevaluationreportandadoptionofformativephaserecommendations.TheinterimreportalsoidentifiedcurrentgapsandemergingsuccessesinprogrammeimplementationandprovidedguidancetoDEPPstakeholdersabouthowtoimprovethepotentialeffectivenessoftheDEPPandevidencetoinformfuturesimilarprogrammes.

Thissummativephasereportpresentsthecumulativefindingsfromallphasesoftheexternalevaluation from november 2015 to May 2018. The aim of the summative phase was toassessintermediateoutcomesandpreliminaryindicatorsoflikelihoodofimpact.findingsareorganisedinfivemainsectionsthatcorrespondtothekeyevaluationquestions:1)RelevanceandValidityofDesign;2)RelevanceandEffectivenessoftheInterventions;3)EffectivenessofManagementArrangements(inrelationtocollaboration);4)EfficiencyandValueforMoney(VfM);5)SustainabilityoftheInterventionandlikelihoodofImpactoftheProgramme.

Evaluation Criteria and Questions

As per the evaluationToR, the evaluationwas designed based on five criteriawhichwereadapted from the Development Assistance committee’s (DAc) principles for EvaluatingDevelopment Assistance: relevance and fulfilment of objectives, efficiency, effectiveness,impactandsustainability38.Theevaluationgatheredandanalyseddatainordertoanswerthefivekeyevaluationquestionsandassociatedsub-questionspresentedbelow.MoredetailedDEPP-specificevaluationcriteriaandquestionsareincludedinAnnex5.

EVALUATION QUESTION 1: RELEVANCE AND VALIDITY OF DESIGN1. To what extent are the objectives of the programme intervention consistent with

stakeholders’ requirements and the programme design logical and coherent?

a. Has the programme targeted the right people in the right places?

b. To what extent does the programme design (theory of change) support the projects’ design (logical framework)?

c. In what ways was the programme design process participatory? Were project beneficiaries adequately engaged before, during and after?

d. To what extent was the programme design logical and coherent?

i. Were the objectives of the programme clear, realistic and likely to be achieved within the established time schedule and with the allocated resources (including human resources)?

e. Have prioritised target groups (people with disabilities, older people) and gender aspects been taken into consideration in the programme design?

38 TheDAcPrinciplesfortheEvaluationofDevelopmentAssistance,OEcD(1991),GlossaryofTermsUsedinEvaluation,in‘MethodsandProceduresinAidEvaluation’,OEcD(1986),andtheGlossaryofEvaluationandResultsbasedManagement(RbM)Terms,OEcD(2000).


EVALUATION QUESTION 2: RELEVANCE AND EFFECTIVENESS OF INTERVENTIONS2. In what ways have DEPP capacity building programmes strengthened emergency

preparedness and response capacity amongst participants?

a. What delivery mechanisms are working effectively and why?

b. To what extent is DEPP contributing to greater preparedness and response among local organisations and communities?

i. Has local capacity to respond to disasters changed since the start of DEPP? If yes, how has it changed? If not, why not?

ii. Has DEPP led to improved knowledge and understanding of best practices relating to disaster and emergency preparedness and re-sponse? If yes, in what ways?

EVALUATION QUESTION 3: EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS3. To what extent was the programme’s theory proven that capacity development is

more effective when undertaken as a multi-agency collaborative approach?

a. Is the collaborative approach of multi-stakeholder platforms an effective delivery mechanism?

b. Focusing on coalitions, partnerships, and connectedness – what can be said about the effects of strengthened networks?

c. What have been the main patterns of collaboration, and the benefits and disadvantages of informal vs. formal collaboration?

d. What unique contribution did collaborative relationships and “multi-stake-holder platforms” make towards deepening cross-programme learning?

EVALUATION QUESTION 4: EFFICIENCY AND VALUE FOR MONEY (VFM)4. How economically have resources and inputs (funds, expertise and time) been con-

verted to results? To what extent does preparedness improve the efficiency of hu-manitarian response?

a. Have resources (funds, human resources, time, expertise, etc.) been allocat-ed strategically to achieve the programme objectives?

b. Have resources been used efficiently? In general, do the results achieved justify the costs? Could the same results be attained with fewer resources?

c. Have the programme funds and activities been delivered in a timely man-ner?


EVALUATION QUESTION 5: SUSTAINABILITY OF THE INTERVENTION AND LIKELIHOOD OF IMPACT OF THE PROGRAMME

5. To what extent and in what ways have the benefits of the programme become embedded?

a. What contribution has the programme made in strengthening national preparedness systems?

b. Has the programme taken into consideration prioritised target groups (people with disabilities, older people, women, children and youth)? What contribution has the programme made in strengthening inclusion of target groups and gender aspects at the level of national and local institutions?

c. In what ways has DEPP influenced institutional and policy environ-ments?

d. What is perceived to be the be the most significant change attributed to the DEPP and why?

Evaluation Timeline

Theexternalevaluationhadfourphases(SeeTable1.3):aninceptionphaseduringwhichtheevaluation framework andmethodologywere designed; a formative phase to evaluate theimplementation of theDEPP by assessing the relevance of outputs and the efficiency andeffectivenessofprogrammedelivery; an interimphase to assess short-termoutcomes; andasummativephasetoassess intermediateoutcomesandpreliminary indicatorsof likelihoodof impact. Amodified timelinewas approved by the evaluation steering committee due todelaysincontractinglocalresearchpartners,securinglocalapprovalsandchallengeswithdatacollectionduetoareasofinsecurity.

PHASE OBJECTIVE DURATION ACTUAL END DATE*

Inception finaliseevaluationmethodology 8Months May2016*

formative Assessrelevanceofprogrammeoutputsandefficiencyandeffectivenessofdelivery 12Months May2017*

InterimAssessshort-termoutcomesdeliveredbytheprogrammeandreflectonprogrammemanage-mentprocess

6Months October2018

Summative Assessintermediateoutcomesandpreliminaryindicatorsoflikelihoodofimpact 7Months May2018

Table1.3:Externalevaluationphases,objectivesandtimeline

* Revised end date

49 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy

figure2:DEPPProjectDescriptions

EVALUATIONMETHODOLOGY

cHAPTER2


EVALUATION DESIGN EVALUATION METHODOLOGY

Theevaluationteam’sapproachusedamixedmethodsdesignwithquantitativeandqualitativedatacollection,includingorganisational-levelassessmentsandcommunity-levelassessments,aneconomicVfMassessment,analysisofhumanitarianpreparednessandresponsenetworks,humanitarianresponseassessmentsandcasestudies,observationandsitevisits,anddocumentreview.findingsfromthesedatasourcesweretriangulatedtoaddressthefivemainevaluationquestionsandtheirsub-questions.AspertheevaluationToR,theevaluationframeworkwasdevelopedbasedontheDAcPrinciplesforEvaluatingDevelopmentAssistance,withtheaimto improve future aid policies and programmes and to provide a basis for accountability.39

TheframeworkwhichoutlinestheevaluationquestionsandtheircorrespondingindicatorsispresentedinTable2.1below.Datawerecollectedthroughoutthefourphasesoftheevaluationagainsttheevaluationframework,andincludedallindicatorsincludedbelow.Theseindicatorsarepresentedthroughoutthisreporttoanswerthecorrespondingevaluationquestions40.

EVALUATION DESIGN MEASURES

Main Evaluation Question

Secondary Evaluation Questions

DEPP Programme Objective / Element of Theory of Change

Indicators

1. RELEVANCE AND VALIDITY OF DESIGN

To what extent are the objectives of the programme intervention consistent with stakeholders’ requirements and the programme design logical and coherent?

•Hastheprogrammetargetedtherightpeopleintherightplaces? •Towhatextentdoestheprogrammedesign(theoryofchange)supporttheprojects’design(logicalframework)? •Inwhatwayswastheprogrammedesignprocessparticipatory?Wereprojectbeneficiariesadequatelyengagedbefore,duringandafter? •Towhatextentwastheprogrammedesignlogicalandcoherent? •Weretheobjectivesoftheprogrammeclear,realisticandlikelytobeachievedwithintheestablishedtimescheduleandwiththeallocatedresources(includinghumanresources)? •Haveprioritisedtargetgroups(peoplewithdisabilities,olderpeople)andgenderaspectsbeentakenintoconsid-erationintheprogrammedesign?

ProblemStatement(Theoryofchange):Inthecontextofrisingneed,insuffi-cientpreparednesssystemsand[few]peoplewiththerightknowledge,attitudesandskillsareavailabletoensureeffectivedeliveryofassistance,particu-larlyatthenationallevel.

•numberofcapacitydevelopmentprogrammesdevelopedandimplemented •numberofnationalstaffandcounter-partstrained •numberofplatformsestablished •numberoflearningeventsheld •numberofreports/casestudiesdevelopedandshared •Increasedadvocacytostrengthenpolicyandpractice •numberofprojectswithgovernmentinvolvement •numberofhazard,risk,andearlywarn-ingsystemsdeveloped/improved

•StakeholderperspectivesonDEPPdesignprocess,objectives,resourcingandprioritisedgroups

39 TheDAcPrinciplesfortheEvaluationofDevelopmentAssistance,OEcD(1991),GlossaryofTermsUsedinEvaluation,in‘MethodsandProceduresinAidEvaluation’,OEcD(1986),andtheGlossaryofEvaluationandResultsbasedManagement(RbM)Terms,OEcD(2000).

40 forexample,evaluationquestiontwointheframeworkbelowisassociatedwithindicatorsonknowledge(improvedknowledgeoncorehumanitariancompetencies,disasterpreparedness,humanitarianresponse,onageing,disabilityandVAWG).Thesearepresentedinchapter4,thechaptercorrespondingtothisevaluationquestion(seefigure4.6).


2. RELEVANCE AND EFFECTIVENESS OF THE INTERVENTIONS

In what ways have DEPP capacity building programmes strengthened response capacity amongst participants?

•Whatdeliverymechanismsareworkingeffectivelyandwhy? •TowhatextentisDEPPcontributingtogreaterpreparednessandresponseamonglocalorganisationsandcommu-nities? •HaslocalcapacitytorespondtodisasterschangedsincethestartofDEPP?Ifyes,howhasitchanged?Ifnot,whynot? •HasDEPPledtoimprovedknowledgeandunderstandingofbestpracticesrelatingtodisasterandemergencypreparednessandresponse?Ifyes,inwhatways?

Objective 1: Toimproveknowledgeandunderstandingofpeopleinthesystemregardingbestpracticeforhumanitarianpreparednessandresponse.

Stakeholder perspectives on effective delivery mechanisms

Improved knowledge and skills of national staff and counterparts in best practices for humanitarian preparedness and response •Improvedknowledgeoncorehumanitar-iancompetencies,disasterpreparedness,humanitarianresponse•Improvedknowledgeonageing,disabili-tyandVAWG

Improved emergency preparedness among organisations and communities•Improvedperceivedlevelofprepared-nessofhumanitarianstafftorespondtoadisaster•Improvedperceivedlevelofprepared-nessoforganisationstorespondtoadisaster•Improvedperceivedlevelofprepared-nessofcommunitytorespondtodisaster•Improvedcommunityaccesstoearlywarningalerts•improvedactualleveloforganisationalpreparednesstorespondtodisasters(Organisationalpreparednessscore)•Improvedactuallevelofcommunitypreparednesstorespondtodisasters(Vulnerabilityindex/communitypreparednessscore)

# and types of disasters organisations responded to in the previous 12 months, and types of response activities

Improved delivery of humanitarian assis-tance during a humanitarian response •Improvedperceptionsofhumanitarianstaffontheirownperformanceduringapreviousresponse•Improvedperceptionsofhumanitarianstaffontheirownabilitytorespondtoadisasterinthefuture•Improvedperceptionsonorganisations’performanceduringapreviousresponse•Improvedperceptionsonorganisations’abilitytorespondtoadisasterinthefuture

•Improvedsatisfactionofcommunitymemberswithpreviousresponsetoadisaster•Increasedspeedofresponse •Increasedcollaborationduringresponse


3. EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS

To what extent was the programme’s the-ory that capacity development is more effective when undertaken as a multi-agency collaborative approach proven?

•Isthe‘collaborative’approachofmulti-stakeholderplatformsaneffectivedeliverymechanism? •focusingoncoalitions,partnershipsandconnectedness–whatcanbesaidabouttheeffectsofstrengthenednetworks? •Whathavebeenthemainpatternsofcollaboration,andthebenefitsanddisadvantagesofinformalversusformalcollaboration? •Whatuniquecontributiondidcollaborativerelationshipsand‘multi-stakeholderplatforms’maketowarddeepeningcross-programmelearning?

Objective 2: Toincreasethenumberofcoalitionsandpartnershipsdeveloped.

Increased number of networks and coalitions formed •Increasednetworksize •Increasedlinkspernode •Increasednetworkdensity •numberofseconddegree,thirddegreeconnections

Increased collaboration and strength of connections•Increasednumberofcollaborationareas •Increasedfrequencyofcollaboration •Increasedstrengthofconnections•Stakeholderperceptionsoncollaboration•Typesofbenefits/disadvantagesofinformalandformalcollaboration

Emerging evidence base for what works in building humanitarian capacity •numberandtypesoflearningdocu-mentsgeneratedandshared•SharinganduseofevidenceandlessonsacrossDEPPprojectsandexternally

4. EFFICIENCY AND VALUE FOR MONEY (VFM)

How, econom-ically, have resources/ inputs (funds, expertise, time etc.) been converted to results? To what extent does preparedness improve the efficiency of humanitarian response?

•Haveresources(funds,humanresources,time,expertise,etc.)beenallocatedstrategicallytoachievetheprogrammeobjectives? •Haveresourcesbeenusedefficiently?Ingeneral,dotheresultsachievedjustifythecosts?couldthesameresultsbeattainedwithfewerresources? •Haveprogrammefundsandactivitiesbeendeliveredinatimelymanner?

All DEPP Objec-tives.

4 ‘E’ Framework •Economy •Effectiveness •Efficiency •Equity•Percentofbudgetallocationtopro-grammemanagementcosts,M&E,nPAcs•Timelinessofdeliveryoffundsandactivities•costperactivityandresultsareas•StakeholderperspectivesonVfM,deliveryoffunds


5. SUSTAINABILITY AND LIKELIHOOD OF IMPACT

To what extent and in what ways have the benefits of the pro-gramme become embedded?

•Whatcontributionhasthepro-grammemadeinstrengtheningnationalpreparednesssystems? •Hastheprogrammetakenintoconsiderationprioritisedtargetgroups(peoplewithdisabilities,olderpeople,women,childrenandyouth)?Whatcontributionhastheprogrammemadeinstrengtheninginclusionoftargetgroupsandgenderaspectsatthelevelofnationalandlocalinstitutions? •InwhatwayshasDEPPinfluencedinstitutionalandpolicyenvironments? •WhatisperceivedtobethemostsignificantchangeattributedtoDEPP,andwhy?

Objective 3: Toimproveinstitutionalandpolicyenviron-mentsforbuildinghumanitariancapacity. Objective 4: Toimproveprepared-nesssystemsforcommunitiesatriskofdisaster. Objective 5: Strengthenedevidencebaseforwhatworkstohelpbuildhumanitariancapacityatscale.

Improved national preparedness systems•Stakeholderperceptionsonstrength-enednationalpreparednesssystems

Inclusion of vulnerable groups•Percentageoforganisationswithinclusionpolicies•Perceivedinclusionduringorganisationalpreparednessactivities•Perceivedinclusionduringorganisationalresponseactivities•Perceivedinclusionincommunitypreparednessplans

Improved institutional environment/ arrangements •Perceivedindividualinfluencewithorganisation•Perceivedinfluenceoforganisationsoninstitutionalandpolicyenvironment•Perceivedinfluenceoforganisationonnationalpreparednesssystems•Inclusivenessoforganisationalpolicytowardsl/nnGOs•changeinorganisationalpolicyonworkingwithl/nnGOs

Most significant change •Stakeholderperspectivesonmostsignif-icantchangeattributedtotheDEPP

Table2.1:Evaluationframeworkcontainingkeyevaluationquestions,correspondingDEPPprogrammeobjectiveandindicators

Duetolimitedresources,theevaluationwasdesignedtoconductin-persondatacollectioninasubsetofDEPPcountries,andremotedatacollectionamongtheremainingDEPPcountries.Theevaluationmethodologythereforeincluded:

1) Anintensivesetofevaluationactivitieswithon-sitequantitativeandqualitativedatacollection in a subset of DEPP focus countries (Kenya, Ethiopia,Myanmar and thePhilippines)

2) Aminimumsetofevaluationactivitiesincludingadeskreviewandremotequantitativeand qualitiatve data collection.Qualitative datawere collected across all tenDEPPcountries, and quantitative data were collected in five non-intensive set countries(SouthSudan,bangladesh,Pakistan,Jordan,andDRc).TherewerenoDEPPactivitiesinMozambiqueatthetimeofdatacollection,thereforedatawerenotcollectedaspartoftheminimumset.

The intensiveandminimumsetofevaluationactivities,aswellas theotherdatacollectionsourcesandtheirrespectivestudylocations,aredescribedinmoredetailbelow.

Datawerecollectedattwotimepointsthatwereroughly12monthsapart–timepoint1(T1)duringtheformativephaseoftheevaluation,andtimepoint2 (T2) inthe interimphase, inordertoassesschangesovertime.AtruebaselinewasnotpossiblegiventhattheevaluationteamwasselectedandcontractedafterthestartoftheDEPPprojects.Assuch,datacollectioninT1,which serves as thefirst datapoint in the evaluation and a comparison forT2datacollection,tookplaceafterDEPPactivitieshadalreadybegunformostprojects.Thepotential


effectsofthislackofatruebaselineassessmentaredescribedingreaterdetailinthelimitationssection.Datafromrelevantcomparisongroupswerecollectedwherepossible.

COUNTRY SELECTION FOR INTENSIVE SET EVALUATION ACTIVITIES

OfthetenDEPPcountries,theexternalevaluationteaminitiallyaimedtoselectthreecountriesfortargeteddatacollectionaspartoftheintensivesetevaluationactivities.Duringthecountryselectionprocess,anumberofkeycriteriawere identifiedandapplied.These included theselectioncriteriaasdefinedinTable2.2.Toensurediversityofgeographicregions,typesofhumanitariancrises,inclusionofbothcDAcandSTARTnetworkprojectsandarangeoftypesof targetedbeneficiaries includingcommunity-leveldirectbeneficiaries, theevaluationteamoptedtoexpandthenumberofcountriesselectedfromthreetofive.Additionalcriteriathatfactoredintothedecision-makingprocessincludedensuringtheinclusionofseveralcountrieswithahighlikelihoodofanupcomingdisasteroremergencytopermitpotentialassessmentofoutcomesrelatedtorealemergencysituations.Inaddition,theteamsoughttoincludecountrieswithvaryingnumbersofDEPPprojectsanddifferentprojectcombinationstopermitstudyofwhetherthesefactorsinfluencednetworkformationandoutcomechanges.

COUNTRY SELECTION CRITERIA CONSIDERATION

Geographic Region EnsuredadequateregionalrepresentationofDEPPprojects.Type of Humanitarian Crisis

consideredtypesofhumanitariancrisesincludingarmedconflict,naturaldisastersfamine,andepidemics.

Inclusion of CDAC and START Network Projects

IncludedcDAcandSTARTnetworkprojects.

DEPP Funds Allocated to Country

DEPPfundsallocatedtoeachcountrywereestimatedbasedonavailableprojectbudgets.countrieswithlessthanandgreaterthantheaverage(10%)allocationwereincluded.

Level of Direct Benefi-ciaries (Household, Community, Organi-sation, etc.)

Assessedthetypesofdirectbeneficariesattheprojectandcountrylevels.GiventhatfewDEPPprojectsdirectlytargetcommunities,itwasimportanttoincludecountrieswithcommunity-leveldirectbeneficiaries.

Likelihood of Upcoming Disaster or Emergency

Whilethiswasdifficulttopredict,theevaluationteamconsideredcountriesthathadahighlikelihoodofadisasteroremergencyoccurringduringtheDEPPtimelineasthiswouldallowforassessmentofoutcomesduringadisasteroremergencysituation.

Contextual Factors Accountedforuniquecontextualfactorssuchassocial,economic,politicalfactorstoensurethatselectedcountriesrepresentedadiverserangeofcontexts.

Number of DEPP Projects and Project Combination in Country

AssessednumberofDEPPprojectsandthespecificcombinationofprojectsineachcountry.ThefinalcohortofselectedcountriesincludedcountrieswithdifferentnumbersofDEPPprojectsanddifferentprojectcombinations.

Table2.2:criteriaforselectingcountriesforintensive-setevaluationactivities

Using the selection criteria, fiveDEPP countrieswere selected for intensive set evaluationactivities.TheseincludedKenya(Africaregion,highnumberofDEPPprojects),Ethiopia(Africaregion,mediumnumberofDEPPprojects), thePhilippines (Asia region,mediumnumberofDEPPprojects),andMyanmar(Asiaregion,lownumberofDEPPprojects)(seeTable2.3).ThefifthcountryselectedwasSouthSudan.However,increasedviolenceinJuly2016significantlyaffectedtheDEPPprojectsoperating inthatcountryandrequiredevacuationoftheirstaffmembers;thus,datafortheintensivesetofevaluationactivitieswerenotcollectedinSouthSudan.Afterdiscussionswith thecDAc-nandProtection inPracticeprojectsoperating inSouthSudan,andclosemonitoringofthesecuritysituation,inOctober2016,theevaluationteamrevisedthemethodologyfordatacollection.Insteadofon-sitedatacollectionasinitiallyplannedandoutlinedintheinceptionreport,dataonSouthSudanwascollectedaspartofthe


minimumsetevaluationactivitiesandthroughanin-depthcasestudy.AsthecDAc-nprojecteventuallyshifteditsprogrammingtothePhilippines,thefinalfourintensivesetcountrieswerestillabletoensurerepresentationofall14DEPPprojectswithintheevaluation(seeTable2.3).

ETHIOPIA KENYA MYANMAR THE PHILIPPINES

DEP

P Pr

ojec

t*

PublicHealthEmergencyPreparednessinGambella

ImprovedEarlyWarning–EarlyActionstoStrengthenDisasterPreparednessinEthiopia

StrengtheningEmergencyPreparednessSystemsinMyanmar

UrbanEarlyWarningEarlyAction learningProject financialEnablers cDAc-n ShiftingthePower TalentDevelopment TransformingSurgecapacity ProtectioninPractice ALERT AgeandDisabilitycapacitybuilding linkingPreparednessResponseandResil-ience

Total Number of Projects (at end of DEPP) 5 7 4 7

Key

Sele

ction

Crit

eria

GeographicRegion Africa Africa Asia Asia

InclusionofcDAcandSTARTnetworkProjects

START START STARTcDAcand

START

Directcommunity-levelbeneficiaries yes yes yes no

DEPPfundsAllocatedtocountry(Estimatedbasedonavailableprojectbudgets)

>10% >10% <10% >10%

numberofDEPPProjectsattimeofcountryselection*

5 7 3 5

Table2.3:countryselectionmatrixfortheintensivesetofevaluationactivities

*There were some changes in the implementation countries over the course of the DEPP

CONTEXTS OF INTENSIVE SET EVALUATION COUNTRIES

Asintended,thefinalfourintensivesetcountriesarediverseandhaveverydifferentdisasterprofiles and contexts as detailed in the boxes below (pages 51 and 52). These contextsinfluencedtheimplementationoftheprojectsand,tosomedegree,whatcouldbeachievedineachsetting.Therefore,analysisofthecountry-levelfindingstakesintoaccountthecountry-specificcontexts.


THE PHILIPPINESThe Philippines is at great risk for natural disasters41 due to its wide-ranging terrain, vast coastline, and position on the “Ring of Fire,” an arc of active volcanoes in the Pacific Ocean basin that make this one of the most active and unpredictable geographical areas in Southeast Asia.42,43 Typhoons, earth-quakes, floods, volcanic eruptions, landslides, and fires, are examples of common natural disasters, with roughly 6-7 tropical cyclones causing significant damage annually, out of an average of 22 occurrences.44,45 Man-made threats include climate change, an overuse of coastal, marine, and forest resource, and forced displacement due to occasional conflicts in Mindanao.46,47,48

There is a strong presence of local NGOs and civil society organisations (CSOs) involved in humanitarian work in the Philippines. OCHA also maintains a sub-office in Cotabato City, in order to monitor different aspects of an emergency, such as the number of internally displaced persons (IDPs) and their evacu-ation and transition sites, as well as connect with various stakeholders, including local government, UN agencies, and NGOs.49 With regards to government disaster risk reduction (DRR), the National Disaster Risk Reduction and Management Council (NDRRMC) functions as the lead agency for DRR preparation and actions.50 Other arms of the government that are directly involved in disaster relief include the Office of Civil Defense (OCD), the Department of Social Welfare and Development (DSWD) (which leads immediate disaster relief effects), and the Armed Forces of the Philippines (AFP).51

MYANMARThe humanitarian situation in Myanmar is both complex and large, with an estimated 863,000 people requiring humanitarian assistance in 2018.52 There is a large range of threats and challenges within the country, including natural disasters, food insecurity, armed conflict, inter-communal tensions, statelessness, displacement, and trafficking.53 Natural disasters that affect Myanmar include floods, cyclones, earthquakes, droughts, fires, and landslides in the mountainous regions.54,55 Other humanitarian challenges include internal displacement, with roughly 241,000 displaced (77% women and children) and living in camp or camp-like situations in Kachin, Kayin, Shan, and Rakhine states.56 In 2012, 120,000 were displaced following inter-communal violence, and in 2016, tens of thousands were displaced due to border post attacks in northern part of the Rakhine State.57 Occasional conflicts also occur in the Kachin and Shan states, which displaced nearly 100,000 people.58 More recently, from August to November 2017, more than 600,000 refugees of the Rohingya ethnic minority left Myanmar to resettle in Bangladesh.

Currently, various INGOs have a presence in Myanmar, though the amount of activity and work conducted by these organisa-tions is lower compared to that of several other DEPP countries, as Myanmar only recently opened to humanitarian actors. The presence and level of engagement of local NGOs are lower than that of INGOs due to the current humanitarian landscape.

41 UnitednationsOcHA.AboutOcHAinthePhilippines[Internet].[placeunknown]:Unitednations;[cited2018April24].42 centerforExcellenceinDisasterManagementandHumanitarianAssistance.PhilippinesDisasterManagementHandbook

[Internet].[Hawaii]:centerforExcellenceinDisasterManagementandHumanitarianAssistance;2018March[cited2018April24],p92.

43 ReliefInternational.Asia–Philippines[Internet].[placeunknown]:ReliefInternational;[cited2018April24].44 UnitednationsOcHA.AboutOcHAinthePhilippines45 centerforExcellenceinDisasterManagementandHumanitarianAssistance.PhilippinesDisasterManagementHandbook.46 UnitednationsOcHA.AboutOcHAinthePhilippines.47 centerforExcellenceinDisasterManagementandHumanitarianAssistance.PhilippinesDisasterManagementHandbook.48 ReliefInternational.Asia–Philippines.49 UnitednationsOcHA.ThePhilippines,withauniquehistoryofembracingrefugees,turnsinwardto

addressIDPs[Internet].[placeunknown]:OcHA;[cited2018April24].50 centerforExcellenceinDisasterManagementandHumanitarianAssistance.PhilippinesDisasterManagementHandbook.51 Ibid.52 MyanmarHumanitariancountryTeam(UnitednationsandPartners).2018HumanitarianneedsOverview

[Internet].[Myanmar]:ReliefWeb;2017Dec[cited2018April24],p28.53 ibid.54 UnitednationsOcHA.MyanmarcountryProfile(12-01-17)[Internet].[placeunknown]:OcHA;2017

January[cited2018April24].55 UnitednationsOcHA.AboutOcHAMyanmar[Internet].[placeunknown]:OcHA;[cited2018April24].56 Ibid.57 UnitednationsOcHA.MyanmarcountryProfile(12-01-17)[Internet].[placeunknown]:OcHA;2017

January[cited2018April24].58 Ibid.


KENYAKenya faces a range of humanitarian challenges, including drought, food insecurity and malnutrition, and outbreaks of diseases such as cholera. The country hosts nearly 500,00 refugees and asylum-seekers.59 As of January 2018, six different counties within Kenya were battling an active Cholera outbreak. Drought conditions persist over large portions of the country leading to inter-communal conflict in certain areas.60 These challenges have impacted communities dependent on agriculture and pastoralism, as the frequent recurrence of natural disasters such as drought result in a loss of crops and/or livestock. This has contributed to food insecurity, with 3.4 million people requiring food assistance, and a rise in food prices. Finally, the continued influx of refugees has also placed a strain on Kenya’s already scarce resources such as water.61

The government of Kenya has had active roles in responding to some of these humanitarian challenges. For instance, they led the drought response at the county and national levels in 2017.62 However, the country’s infrastructure is not generally able to respond to large-scale crises. In late 2017, these challenges were further exacerbated by delays, uncertainty, and civil unrest during the political election.63

INGOs and multi-lateral organisations have a strong presence in Kenya, with several closely collaborating with the government in certain humanitarian responses.

ETHIOPIAEthiopia’s key humanitarian challenges fall under the following categories: displacement of individuals, disease outbreaks, climate-re-lated issues, socio-economic difficulties, malnutrition, and a lack of safe drinking water.64 Ethiopia hosts a large population of refugees, and IDPs due to a combination of internal conflict and climate-related difficulties such as drought and flooding.65 Throughout Ethiopia, resources such as food and safe drinking water are scarce and are worsened by drought, leading to rising levels of food insecurity and malnu-trition.66 UN agencies have indicated that these challenges are contributing to higher rates of violence against women, and limited access to health facilities, particu-larly for those affected by these crises.67

Ethiopia receives support in the form of funding and in-country presence from various multi-lateral organisations and INGOs, in part to increase prevention activities, raise the level of emergency preparedness, and strengthen the system for recovery efforts.68 Many of these organisations have identified priority areas and donor funding, such as improving emergency systems and increasing the capacity of local government.69 Several organisations have also sought to collaborate with the local and national governments, such as to develop the Ethiopia Humanitarian and Disaster Resil-ience Plan (HDRP) and a comprehensive drought response plan.70

59 UnitednationsOcHA.Kenya[Internet].[placeunknown]:OcHA;[updated2018february20;cited2018April24].

60 UnIcEfKenya.KenyaHumanitarianSituationReport[Internet].[Kenya]:UnIcEf;2018January[cited2018April24],p9.

61 Savethechildren.HumanitarianResponse:TheSituationforchildreninKenya[Internet].[England]:Savethechildren;[updated2018;cited2018April24].

62 UnIcEfKenya.KenyaHumanitarianSituationReport.63 Ibid.64 USAID.Ethiopia[Internet].[placeunknown]:USAID;[updated2018April9;cited2018April24].65 Ibid.66 USAID.ActiveUSGProgramsfortheEthiopiaResponse[Internet].[placeunknown]:USAID;[cited2018

April24].67 UnitednationsPopulationfund.EthiopiaHumanitarianEmergency[Internet].[placeunknown]:UnfPA;

[cited2018April24].68 UnitednationsOfficeforthecoordinationofHumanitarianAffairs.Ethiopia:2018Humanitarianand

DisasterResiliencePlanseeksUS$1.6btoassist7.9Mpeople[Internet].[placeunknown]:OcHA;2018March[cited2018April24].

69 USAID.Ethiopia.70 Reliefweb.Ethiopia:Governmentandpartnerslaunchthehumanitariananddisasterresilienceplan

(HDRP)for2018.[placeunknown]:ReliefWeb;2018March[cited2018April24].


DATA COLLECTION APPROACHES AND TOOLS

GiventhecomplexityoftheDEPPanditsrangeofactivitiestargetingdifferentlevelsofaction(includingindividual,organisational,community,andsystems)anditsobjectivestostrengthennetworksand improveefficiencyof thehumanitarianresponse,anumberofdatacollectionapproaches were conceptualised, and corresponding data collection instruments weredeveloped.Overall seven typesofassessmentswereplannedwithmanyof these includingseveraldifferent,complementarydatacollectionelements.ThesevenassessmenttypesalongwiththecorrespondingtoolsarepresentedinTable2.4.AllofthedatacollectioninstrumentsareavailableinAnnex6.

SAM

PLIN

G

GRO

UP

MET

HO

D

STU

DY

TOO

L

DES

CRIP

-TI

ON

RAN

DO

M

SELE

CTIO

N

MIN

IMU

M

SET

KEN

YA

THE

PHIL

IPPI

NES

ETH

IOPI

A

MYA

NM

AR

UK

ORGANISATIONAL ASSESSMENT T1 T2 T1 T2 T1 T2 T1 T2 T1 T2 T1 T2

countrydirector(Seniormanagementifcountrydirectornotavailable)* Q

uantitative

OrganisationalSurvey

Questionsonpreparedness,response,&capacitybuildingactivitiesimple-mented,coordinationstructures,per-ceptionsofinstitutionalenvironment,network&collaboratingpartners,&theoutputs&outcomesoftheDEPP

Organisationalhumanitarianstaff

Quantitative

Knowledge,AttitudesandPractices(KAP)Survey

Questionsonknowledge,skills,&behaviourinhumanitarianprepared-nessandresponse,corehumanitarianconcepts,prioritisedgroups,exposuretocapacity-buildingactivities,networkandcollaboratingpartners,andtheoutputs&outcomesoftheDEPP

✔

Seniormanagement(countrydirectororanotherseniormanagerifthecountrydirectorwasnotavailable)*

Qualitative

In-depthinterviews

Questionsonexperienceswithcapac-itybuilding,collaboration,institutionalenvironmentandpolicies,organisa-tionalpreparedness,improvingpre-parednesssystemsforcommunities,contextualfactors

✔

Mid-levelorSenior-levelstaff

Quantitative

Organisationalchecklist

Questionsonorganisationalemergencypreparednessplans,andorganisationalresponsecapacityandcoordination

Programmestaff/management& projectleads

Qualitative

In-depthInterviews

QuestionsonexperienceswithDEPPactivities,perceptionsofprojectandprogrammeeffectiveness,contextualfactors,numberandtypeofspecificprojectactivities

COMMUNITY LEVEL ASSESSMENT T1 T2 T1 T2 T1 T2 T1 T2 T1 T2 T1 T2

communitymembers Q

uali-

tative community

focusgroupdiscussion

Questionsoncommunityvulnerability,risk,andresiliencetodisastersandemergencies

communityleaders

Quantitative

communitysurvey

Questionsoncommunityprepared-nessplansandactions,structuralmitigationactivities,neworimprovedearlywarningsystems,politicalcom-mitmentoflocalgovernmentandcommunitystructures

communitymembers

Quantitative

Householdsurvey

Questionsonhouseholdprepar-edness,disasterriskknowledge,exposuretohazards,accesstoearlywarningalerts,householdsbehaviourrelatedtopreparedness,perceptionsofofcommunitypreparednessforemergenciesanddisasters

✔

ECONOMIC ASSESSMENT T1 T2 T1 T2 T1 T2 T1 T2 T1 T2 T1 T2

Programmestaff/management& projectleads

"Qualitative/

budget&

financialreview"

Valueformon-eyin-depthinterviews

Questionsonresourceallocation,valuechains&transationcosts,&time&resources

NETWORK ANALYSIS T1 T2 T1 T2 T1 T2 T1 T2 T1 T2 T1 T2

collaboratingpartnersidentifiedinnetworksectionoforganisationalsurveys&KAPsurveysduringorganisational Assessment

Quantitative

networksurvey

Questionsoncollaboratingor-ganisations,trustamongpartners,&frequencyofcollaborationwithinnetworks


HUMANITARIAN RESPONSE ASSESSMENT AND CASE STUDIES T1 T2 T1 T2 T1 T2 T1 T2 T1 T2 T1 T2

DEPP&externalstakeholders

Qualitative/Quantita

-tive/Documentreview/

Observation&sitevisits

In-depthin-terviews,Self-administeredquestionnaires

Questionsondisasters&emergerciesinDEPPcountries,DEPPcontributiontohumanitarianresponses

OTHER ASSESSMENTS T1 T2 T1 T2 T1 T2 T1 T2 T1 T2 T1 T2

Projectstaff,organisationalstaff&externalstake-holders O

bserva

-tion&

sitevisits

n/A

Participationincountry-levelandgloballearningevents,visitstooragnisationalpartnersandexternalstakeholdersduringdatacollection

n/A

Document

review Matrices Reviewofprogrammeandproject

leveldocumentsandresources

Table2.4:Methodology,studytools,samplepopulationanddatacollectionsiteforeachtypeofassessment

ETHICS

Ethicalconsiderationsforthisevaluationwerecarefullyassessedandthedatacollectionwasdesignedtoensureprotectionofindividualsparticipatinginthestudy.HumansubjectsresearchapprovalfortheevaluationprotocolandstudyinstrumentswasobtainedfromtheHarvardT.H.chanSchoolofPublicHealth’sInstitutionalReviewboard(IRb)aswellasfromrelevantbodiesineachoftheintensivesetcountries.Inthefourintensivesetcountries,theevaluationteampartneredwithlocalresearchorganisationswhoimplementedthedatacollection.TheseincludedthebusaracenterforbehavioralEconomics(Kenya),TheEthiopianPublicHealthAssociation(Ethiopia),andInnovationsforPovertyAction(Myanmar,ThePhilippines).Theselocalresearchpartnershiredthestudystaff.TheevaluationteamconductedsitevisitsatT1andT2toparticipateintheenumeratortraining,pilotingofinstrumentsandprotocols,andseveraldaysofdatacollectiontoensuredataquality.followingthis,theevaluationteamwasinclosecontactwiththein-countryresearchteamsinordertoremotelysupervisetheongoingdatacollection.Thisinvolvedregularmeetingsanddebriefingsessionswiththeevaluationteam,aswellasregularandfrequentchecksofincomingdatatoensurequality.QuestionnaireswerepilotedandprogrammedintoKoboToolbox,anelectronicdatacollectionsoftwarewhichpermittedtheevaluationteamtocloselymonitorincomingdatafromthefieldteams.

SAMPLE SIZE

ThesamplesizefortheevaluationisdetailedinTable2.5.Intotal,2675quantitativesurveysand qualitative interviewswere completed during T1, including 422 in theminimum set ofevaluationactivitiesand2253intheintensivesetofevaluationactivities.InT2,atotalof3440quantitativesurveysandqualitativeinterviewswerecompleted,with250intheminimumsetand3190intheintensiveset.

MINIMUM SET INTENSIVE SET NETWORK COMMUNITY VFM CASE STUDY

iDis

Organisational

surveys

KAPSurveys

Total

iDis

Organisational

checklist

Organisational

surveys

KAPsurveys

networksurveys

HouseholdSurveys

VillageSurveys

communityfGDs

VFM

iDis

caseStudyIDIs

Total

T1 51 77 294 422 47 185 185 352 453 1004 39 19 13 3 2253

T2 51 51 148 250 25 157 160 304 334 2072 65 43 8 22 3190 Table2.5:Samplesizeforevaluationactivities


METHODS FOR THE INTENSIVE SET OF EVALUATIONACTIVITIES DATA COLLECTION PROCEDURES

A local researchmanager based in each intensive set country oversaw and coordinated allof thefieldwork.Thestudy instrumentswerewritten inEnglishandtranslatedtothe locallanguageinallcountriesexceptKenya,andadaptedtothelocalcontext,pilotedandmodifiedtoensurethatthemeaningsofquestionsandresponseswerecorrect.UpdatedinstrumentswereapprovedbytheappropriateIRbs.Maleandfemaledatacollectorsaswellassupervisorswererecruited ineachcountrytaking intoaccountrelevantcharacteristicsneededforeachtypeof survey. for example, for the community andhousehold surveys, enumeratorswhocouldspeakthe local languagewererecruitedfromthestudyareas.Theenumeratorsweretrainedforaminimumoffivedays,andconductedfieldpracticeoverthreetofivedayswithorganisationsorcommunitieswhichwerenotpartofthesamplepopulation.Thispermittedenumeratorstopracticeadministeringthesurveysunderreal-worldconditions,andtoassessthetranslatedinstrumentsandaddresserrorsorproblemswiththetranslationpriortothestartofdatacollection.Atleastoneevaluationteammemberwaspresentforandparticipatedinthetraining,thepilotingandthestartofthedatacollection.Afterthis,theevaluationteammemberremained in regular contactwith the in-countrydatacollection team to remotely supervisethefieldwork.Regularmeetingswithin-countryteamswereheld,andin-comingdatawerecheckedregularlytoensurehighquality.

Verbal informedconsentwasobtained fromparticipantsand interviewswereconducted inprivatesettingstoensureconfidentiality.Inordertominimisebiasandtopreventinducementto participate, no compensationwas provided to respondentswho participated. Instead, acertificateofparticipationwasprovided.Questionnaireswereadministeredby local, trainedenumeratorsusingKoboToolboxonAndroidtabletsineitherEnglishorinthelocallanguage.In-depthinterviewslastedapproximatelyonehourandwereaudiorecordedonAndroidtablets.Quantitativeandqualitativedatawereuploadedtoasecureserver,whichisonlyaccessiblebykeyevaluationstaffthathaveethicalclearance.copiesoftheaudiorecordingonthetabletsweredeletedtoprotectprivacyandconfidentiality.Dataweremonitoredonanongoingbasistoensurehighquality.Anyerrorswerecorrectedimmediately.Enumeratorswerecontinuouslyprovidedfeedbacktoimprovetheirperformanceanddataquality.

ORGANISATIONAL-LEVEL ASSESSMENT

figure 2.1 provides an overview of the sampling approaches and survey instruments thatwereusedforeachofthesamplegroupsintheorganisational-levelassessment.first,alistoforganisationsandcommunitiesassociatedwiththeDEPPprojectswascreatedineachintensiveset country. In parallel, a list of comparison organisations were identified in each countrythroughavarietyofmeans.listsofbothnationalandinternationalorganisationsworkingonemergencypreparedness and response in each countrywere constructedwith informationfromvarious sources includingUnOcHA, humanitarianworking groups, andnationalnGOregistries.Organisationsthatweresimilarinsize,scope,andwithsimilargeographicfocusasthoseworkingontheDEPPineachcountrywereselected,andtheproportionofinternationalnGOs to nationalnGOswas kept constant across control andDEPP groups. In the eventthat there were more control organisations than needed that met the selection criteria, asubsetoforganisationswasrandomlyselected.Incountrieswherethereweremorethan50DEPPorganisations,arandomselectionoforganisationswasselectedforparticipationintheevaluation.Incountrieswheretherewerelessthan50DEPPorganisations,allorganisations


wereselectedforthesample.

foreachorganisation,thefollowingsurveyswereconducted:1)oneorganisationalsurvey,2)oneorganisationalchecklist,and3)threeknowledge,attitudes,andpractices(KAP)surveys.Inaddition,arandomselectionof15%organisationsinthesamplewereselectedtoparticipateinanin-depthinterview(IDI)inadditiontothesurveysabove.

ORGANISATIONAL SURVEY, KAP SURVEY, ORGANISATIONAL CHECKLIST, AND IN-DEPTH INTERVIEWS

In-countryresearchpartnerscarriedoutthefollowingprocedures:

1) Email invitations were sent to the country directors, or main contact of eachorganisationinthesampletoinformthemthattheirorganisationwasselectedtoparticipate in theDEPPevaluation,andorganisationswereasked toprovide theresearchpartnerswithalistofhumanitarianstaff.

2) Appointmentsweremadewithcountrydirectors(ormembersofseniormanagementwhencountrydirectorswerenotavailable)toconductanin-personorganisationalsurvey. In addition to completing the organisation survey, the individual wasasked toa)namea staffmemberwith in-depthknowledgeof theorganisationalpreparedness plans, procedures, and systems, to complete the organisationalchecklist;b)recommendanothermemberofseniormanagementtocompletethein-depthinterview(fororganisationsselected);andc)providealistofstaffwithintheorganisationifthiswasnotprovidedbytheorganisationpriortotheinterview.

3) Thestaffmemberidentifiedbythecountrydirectorormemberofseniormanagementwasapproachedtocompletetheorganisationalchecklist.

4) fortheKAPsurvey,threestaffmemberswithineachorganisationwererandomlyselected from the list of humanitarian staff, and appointmentsweremadewithselectedindividualsforanin-personsurvey.

5) Inthe15%oforganisationsthatwererandomlyselectedforthein-depthinterviews,in addition to the quantitative surveys, country directors ormembers of seniormanagementthatdidnotparticipateintheorganisationalsurveyswereselectedtoparticipateinanin-depthinterview.

figure2.1:Samplingapproachandstudyinstrumentsfororganisational-leveldatacollectionineachintensivesetcountry

*Surveys completed by different members of organisational management, depending on availability


Allsurveysconductedbasedontheaboveprocedureswerecarriedout in-person,withtheexceptionofasmallnumberofsurveysconductedoverSkypeorphoneduetoextenuatingcircumstances.ThenumberoforganisationsvariedpercountryasitwasbasedonthenumberofDEPPimplementingpartnersandbeneficiariesineachcountry,andthiswasdependentontheprojectsthemselves.Inmostcases,thisdidnotexceed50DEPPorganisationsand400humanitarianstaffworkingatDEPPorganisationspercountry.Anequalnumberofcomparisonorganisations,humanitarianstaff,communitiesandhouseholdswhichweresimilarinsizeandlocationorother factors to theDEPPstudypopulationwerealso selectedusing the samesamplingapproach.

COMMUNITY-LEVEL ASSESSMENT

A list of all villages whereDEPP activitieswere implemented at the community level wascompiledinMyanmarandEthiopia,.InMyanmar,DEPPvillagesinKayinandShanstatewereselectedforthesample.Togeneratealistofcomparisonvillages,in-countryresearchteamsworkedwiththelocalgovernmentsorotherpartnerstoobtainalistofallvillagesinthearea,includingtheirpopulationsizeandotherdemographiccharacteristics.fromthislist,asetofcomparison villageswhichwere similar in size and demographics, but far enough from theDEPPvillagestominimiseriskofexposuretoDEPPintervention,wereselected.

InEthiopia, theGambella regionwas selected for the community assessmentbecause twoDEPPprojectswereimplementingcommunity-levelactivitiesthere(PHEPGambella,EWEA).DEPP interventionsbenefittedallvillageswithinadistrict (woreda),andthusa listofDEPPworedas71 andnonDEPPworedas, including the villages in eachworeda,was constructed.Anyareasorvillagesinthatlist,deemedinaccessiblebythelocalgovernmentduetoviolence,securityorfloodingwereremovedfromthesample.ArandomsampleofvillagesthatwereaccessibleintheDEPPworedaswereselectedfordatacollection.forthecomparisongroup,arandomsampleofvillagesinthenon-DEPPworedaswasselectedfordatacollection.

figure2.2:Samplingapproachandstudyinstrumentsforcommunity-leveldatacollection

Ineachsampledcommunity,thefollowingsurveyswereconducted:1)onecommunitysurvey,2)~1000householdsurveys3)onecommunityfocusgroupdiscussion.Allsurveysconductedbasedontheaboveprocedureswerecarriedoutin-person.focusgroupdiscussionquestionswereadaptedfromtheGOAlcommunityResilienceToolkit72.

71 Aworedaistheequivalentofadistrict.72 ToolkitforMeasuringcommunityDisasterResilience.GuidanceManual.Goal,2014.


COMMUNITY SURVEY, HOUSEHOLD SURVEY, COMMUNITY FOCUS GROUPDISCUSSION

In-countryresearchpartnerscarriedoutthefollowingprocedures:

1) localresearchpartnerscontactedlocalorganisationsandgovernmentsimplementingactivitiesineachvillage,whothenintroducedtheresearchpartnerstorelevantcommunityleadersandinformants.

2) Thecommunityleaderfromeachvillagewasselectedforthecommunitysurvey.

3) Arandomsampleofhouseholdswasselectedforthehouseholdsurvey.

4) community members involved in disaster preparedness were recruited for thecommunityfocusgroupdiscussion.

METHODS FOR THE MIMINUM SET OF EVALUATIONACTIVITIES IN-DEPTH INTERVIEWS

Thestudypopulationfor theminimumset in-depth interviewscomprisedUK-basedprojectleadership, project leadership in DEPP countries, DEPP programme-level leadership andtargetedstakeholders.Alistofprojectandprogrammeleadershipwasfinalisedandparticipantswererecruitedthroughperson-to-personcontactatDEPPlearningeventsandthroughemailinvitations.foreachDEPPproject,theevaluationteamaimedtosampleboththeUK-basedleadershipandthein-countryleadership.IfaprojectdidnothaveleadershipattheUKlevel,thein-countryleadershipwereinvitedtoparticipate.

fortheDEPPtargetedstakeholders,purposivesamplingwasapplied.Usingalistofcontactsfrom the inceptionand formativephases, targetedstakeholderswere invited toparticipate.Therewereseveralparticipantswhowereinvolvedattheprojectlevelandprogrammelevel;inthiscase,thoseparticipantswereinterviewedonlyonce.SomeofthekeystakeholderswhotookpartwereindividualswhohadpreviouslyworkedontheDEPPbutarenolongerformallyengagedwiththeprogramme.Additionaleffortwasmadetolocateandinvitetheseindividualsforparticipationintheinterviews.

Verbal informedconsentwasobtainedforall interviews. InterviewswereconductedbyoneofthreeevaluationteammembersinEnglish.Theinterviewswereconductedeitherinpersonor by Skype, and lasted approximately 45-75minutes. The interviewswere recorded usingcallnotesoftwareormobileaudiorecordingsoftwareforSkypeinterviewsandonanAndroidtabletusingastockaudiorecordingsoftware.Evaluationteammemberstranscribedtheaudiorecordedinterviews.TranscriptswereimportedintoDedoosequalitativeanalysissoftware.

followingthein-depthinterviewsconductedatT2,all interviewparticipantswereemailedalinktoanoptionalonlinequestionnaireonKoboToolboxtocaptureadditionalperspectivesonprojectdeliveryandimpactthatwerenotcoveredduringtheinterviews.


ONLINE ORGANISATIONAL SURVEY AND KAP SURVEY

The study population for the minimum set online quantitative questionnaires comprised1) organisational leadership and 2) humanitarian programming staff from implementingorganisationsandbeneficiaryorganisationsinDRc,SouthSudan,Pakistan,bangladesh,andJordan.AlistofDEPPorganisationswasfinalisedwiththeUK-basedprojectleadsandRegionallearningAdvisors (RlAs).Participantswererecruitedby theevaluationteamthroughemailinvitation.

Abridged versions of the organisational survey and KAP survey from the intensive setof evaluation activitieswere sent via email invitation through links to online surveys usingKoboToolbox.OnlinequestionnairesforallparticipantswereadministeredinEnglishexceptintheDRcwhereitwastranslatedtofrench.QuestionnaireswereemailedtoDEPPimplementingandbeneficiaryorganisationsandwerecompletedusingKoboToolbox.Participantsprovidedconsentpriortobeginningthequestionnaire.

norewardsorcompensationwereprovidedtoparticipants.Acertificateofstudyparticipationwasprovidedtoparticipantsinprojectcountrieswhocompletedtheonlinequestionnaire.

ECONOMIC ASSESSMENT (VALUE FOR MONEY)

InterviewsfortheVfManalysiswereconductedinpersonandviaSkypebytheevaluationteam’s economistandVfMadviser.Purposivesamplingwasapplied.ThestudypopulationfortheeconomicdatacollectionincludedDEPPportfoliomanagementteammembers,DEPPboardmembersandprojectstaff.

Areviewofbudgets,financialinformationandprojectreports,wasalsoconductedtoassessVfMattheprojectandprogrammelevel.Inaddition,datafromselectquestionsoncostandefficiencyfromtheorganisational-levelassessmentoftheevaluation(organisationalandKAPsurveys)wereanalysed.costperresultindicatorswerecreatedthroughthefinancialreviewprocess.TheextenttowhichVfMrecommendationsfromformativephasewereimplementedbytheprojectswasalsoassessed.

NETWORK ANALYSIS

Thenetworksurveywascarriedoutintwophasesatbothtimepoints.Thefirstphaseincludedadministeringquestionsonnetworksandcollaboratingpartnersthatwereembeddedwithinthe quantitative organisational andKAP surveys. In the network section of these surveys,organisational leadership and humanitarian staff were asked to identify the organisationsthey collaboratedwith over the last sixmonths and to describe the nature or purpose ofthecollaboration.Theyselectedfroma listof32collaborationareas inwhichhumanitarianactors are likely to engage. Respondents were also given the option to name their ownareasofcollaboration.Respondentsthen identifiedtheirmaincontactateachcollaboratingorganisation.Theactors thatwerenamedbyfirstphasesurveyparticipantsareconsideredtobe1stdegreeactors;theyareonedegreeofseparationawayfromthesurveyinformant.Thein-countryresearchpartnersthencalledeachofthe1stdegreeactorsandthestandalonenetworksurveywasconductedwiththat individualoverthephone.These individualswerethenaskedtoidentifytheirmaincontactateachcollaboratingorganisation,andthosecontactsareconsidered2nddegreeactors, as theyare twodegreesof separation from theoriginalsurveyinformant.Thestandalonenetworksurveywasthenconductedwiththe2nddegreeactors,withtheexceptionofthePhilippinesduetolimitedtimeandresources.

HUMANITARIAN RESPONSE ASSESSMENT AND CASE STUDY

In-depthcasestudieswereaddedtotheevaluationmethodologyduringtheformativeandinterim phases to provide further perspectives on emergency preparedness and response


activities in the programme, and to assesswhether improved preparedness translated intomoreefficientandtimelyhumanitarianresponse.SimulationexerciseswereinitiallyproposedduringtheinceptionphaseasamethodtomeasureDEPPbeneficiaries’performanceduringhumanitarianresponseactivitiesinacontrolledmanner.TheintentionwastocollectsuchdataduringsimulationsthatwereplannedbyDEPPprojectsthemselves.However,overthethree-yearprogramme,projectschangedtheirplansregardingsimulations,anditwasalsonotedthatseveraloftheDEPPprojectsrespondedtoactualdisastersintheprojectcountriesprovidinganopportunity to assess contributions to real humanitarian responses.After reflectionanddiscussionwith thelearningProject, theevaluation teamdecided toconduct two in-depthcasestudies.ThefirstcasestudywasconductedinSouthSudantocaptureexperiencesandperspectivesofprojectactivitiesduringtheescalationofviolenceintheformativephaseandhowthis impactedproject implementation.Thesecondcasestudyfocusedonhumanitarianresponses inEthiopiaandKenyaduring thecourseof theDEPPtocapture thequalityandspeedofresponseactivitiesinlieuofsimulation-baseddatacollection.

ThecasestudyapproachallowedfordeeperexplorationandanalysisofactualhumanitarianresponsesthattookplaceduringtheDEPPprogrammecycle.Purposivesamplingwasapplied.Potentialparticipantswerecontactedbyemailwithabriefdescriptionofthecasestudytobeconductedaspartoftheevaluationandinvitedtoparticipate.InKenya,responsecasestudydatacollectionalsoincludedsitevisitsandobservationaldataaswellasinclusionofbeneficiarypopulationinterviews.InEthiopia,duetosecurityconcerns,asitevisitwasnotconductedinMoyaledistrictandasmallersampleofparticipantswereincludedinthedatacollection.

Inaddition,duringthedisseminationandvalidationworkshopsconductedbytheevaluationteamintheintensivesetcountries,aself-administeredquestionnairewascirculatedtodocumentdisastersandhumanitarianresponseactivities.ThequestionnaireaskedparticipantstoidentifyallofthedisastersandemergenciesthathadoccurredintheircountriessincethebeginningoftheDEPP,andtoindicatewhichcrisestheirprojecthadrespondedto.Thesedataallowedtheevaluationteamtoassessthenumberandtypeofdisastersandemergenciesineachintensivesetcountry,andprovidedadescriptionofthetypeandextentoftheresponseactivitiesthattheDEPPorganisationconducted.

OBSERVATION AND SITE VISITS

Membersoftheevaluationteamandlocalresearchpartnershadopportunitiestovisitprojectsand engage with project stakeholders in countries where the intensive set of evaluationactivitiestookplaceandincountrieswhereteammembersattendedlearningevents.Priortodatacollection,theevaluationteamconductedsitevisitsandfacilitatedinductionworkshopsin each country selected for the intensive evaluation activities to introduce the externalevaluationtoDEPPprojectleadershipandstaff,andgatherfeedbackontheproposedplanforthatcountry.Thesesessionsalsoprovidedanopportunitytoengageindialoguewithprojectleadershipandstaff,whichcontributedtotherefinementofthemethodology,studyprotocolanddatacollectioninstruments.

Evaluationteammembersalsoparticipatedinanumberoflearningandcollaborationmeetingsandconferences(Table2.6).TheseeventspermittedengagementwithDEPPprojectteamsandkeystakeholders,andalsomorestructuredassessmentandobservationonthedifferencesinperspectivesofDEPPleadershipandstaffattheUKlevelandatthecountrylevel,contextualfactorsthatimpactprojectimplementation,andthevalueofevidenceanddifferentresearchmethodologies in thehumanitarian field.During the summativephase, the evaluation teamorganisedvalidationworkshopsinthefourintensivesetcountriestodisseminatefindingsandgatherfeedbackatthecountrylevel.Theresultsofthevalidationworkshopsprovidedfurthercontextandinsightintotheevaluationfindingsandhelpedtoinformtherecommendations.

Evaluationteammemberscompilednotesduringobservationandsitevisitsandextractedkeythemes.Observationdatapresentedinthisreportarebasedonthesefieldvisitsandconferenceattendancetoenabletheevaluationteammemberstointeractwithprojects.


OBSERVATION AND SITE VISITS COUNTRY DATEMEl/learningProjectlaunch UK March2015

londoncollaborationday UK October2015

londoncollaborationday UK february2016

InceptionphaseworkshopsEthiopia,Kenya,Myanmar,

thePhilippinesMarch2016

STARTconference UK May2016

launchofT1datacollectionKenya,Myanmar,thePhilip-

pinesSeptember-november2016

londoncollaborationday UK november2016

nairobilearningconference Kenya november2016

launchofT1datacollection Ethiopia May2017

DEPPlearningconference ThePhilippines July2017

Evaluationworkshop UK July2017

launchofT2datacollectionEthiopia,Kenya,Myanmar,

thePhilippinesSeptember-november2017

formativephasedisseminationevents Ethiopia,Kenyanovember2017,January

2018

Summativephasedisseminationandvalidationworkshops

Ethiopia,Kenya,Myanmar,thePhilippines

March2018

PreparingforShockconference Switzerland March2018

Table2.6:EvaluationteammemberparticipationatDEPPevents

DOCUMENT REVIEW

The evaluation team engaged in continuous and comprehensive information collection andreview of all available documentation and materials relevant to the programme and theprojectsthroughouttheprogrammecycle.Documentswereobtaineddirectlyfromprojectandprogrammestaff,aswellasviatheDEPP’sonlineboxfolders,andtheonlineDEPPlearningPlatform73. This included project reports, documents, case studies, presentations and anyotheravailablematerials inadditiontodatacollectedthrougheachproject’smonitoringandevaluation system.Documents and learning captured through theDEPP’s learning Projectactivitieswerealsoobtained.Oncetheexistingdocumentswereobtainedandcompiled,thereviewandanalysisinvolvedassessingeachdocument,categorisingit,andextractingkeydatainamatrixwherekeythemes,suchasgenderconsiderations,couldbecompared.Table2.9presentsasummaryofthedocumentsreviewedaspartofthedocumentreview.

finally, a targeted literature review related to humanitarian capacity building activitieswasconductedandexternaldocumentswerealsoconsultedaspartofthisprocesstoinformtheoverallinterpretationofdata.AcompletelistofdocumentsreviewedisprovidedinAnnex7.

ANALYSIS

Datawere analysedover the three-yearevaluationperiod.Differencesbetween subgroupsandcountrieswereassessed,anddataweretriangulatedfrommultiplesourcestostrengthenandvalidatefindings.context-specific informationandperspectives fromthe local researchteamsandcountry-levelDEPPstakeholders(throughvalidationworkshops)werealsoappliedtocountry-leveldatatoensuresoundinterpretations.

73 ThelearningPlatformcanbefoundhere:https://disasterpreparedness.ngo/.


forthequalitativedata,in-depthinterviewsconductedinotherlanguagesweretranslatedtoEnglish.The analysisprocess included listening to audio recordingsor reviewing transcriptswhichwere thencodedandanalysedusingDedoose software forqualitativedata analysis.Mainthemes,concepts,andquoteswereextractedeitherfromtheaudiorecordingsorfromthetranscriptsandcategorisedinadetailedmatrixtofacilitatecomparisonandidentificationofpatternsoverallandwithinsubgroups.

QuantitativedatawereanalysedusingStatisticalAnalysisSystem(SAS)softwareandSTATA(version 13.1). Descriptive statistics were generated, with comparisons made betweenintensivesetcountriesaswellasacrosstheentiresampleof10countrieswhenpossible,andbetweentheDEPPandcomparisongroups.T-testsandchi-squaredtestswereconductedtoassesswhether differencesbetweenDEPPand control groupswere statistically significant.Difference-in-DifferenceEstimationwasusedtoestimatetheeffectoftheDEPPinterventionsbycomparingthechangesinoutcomesovertimebetweentheDEPPgroupandthecomparisongroup.Datawerealsodisaggregatedbygenderwhererelevant.

TheeconomicandVfManalysisfocusedontheprogrammeandcountrylevels.Theanalysisinvolvedreview,bytheevalutionteam’sVfMadvisor,ofdocumentsincludingquarterlyreports,adhocVfMreports,learningProjectdocuments,theprogramme-levelbudget,andprojectandprogrammeexpendituredata.Interviewdatawerecodedandanalysedandempiricaldataoncostsandefficiency,collectedaspartoftheintensivesetdatacollection,werealsoanalysedaspartoftheVfManalysis.formativephaseVfMrecommendationswerealsoassessedtodeterminetheextenttowhichtheywereimplemented.Theassessmentalsoincludedinternalbenchmarkingofunitcosts,costeffectivenessanalysis(includingcostperresultsassessment),analysisofadherencetoprocurementproceduresandanalysisofdecision-makingandprojectmanagementwithinDfIDandimplementingagencies,andhowVfMwasconsidered.findingsweretriangulatedwiththemultipledatasources.

networkdatawereanalysedbyRootchange,aninstitutionspecialisedinnetworkmappingandnetworkvisualisations,usingORA,acustomisednetworkanalysis toolandvisualisationplatformdevelopedbycASOSatcarnegieMellon.collaborationpatternswereidentifiedandnetworks,groupsandorganistionsfromeachcountrywerecompared.ThestatisticalanalysissoftwareRwasusedforstatisticalsignificancetestsbetweengroups.

Documentreviewincludedin-depthassessmentsofprojectreports,documents,casestudies,presentations,logicalframeworksandotheravailablematerials.

Anydifferencesinopinionswithintheevaluationteamwereresolvedthroughobtainingothersourcesofinformationforfurtherclarification.Thisincludedthecollectionofadditionaldata,furtheranalysisorsubgroupanalysis.findingssupportedbymultipledatasources(includingbothquantitativeandqualitativedata)wereweightedmoreheavily.consensuswasdeterminedthroughthereviewofadditionaldataand internaldiscussion.Therewerenounintendedorunexpectedfindingsinthisphaseoftheevaluation.Ingeneral,whiletherewassomevariationintheperspectivesofstakeholders,therewasgeneralconsensusonmostissues.Whereopinionson issuesweremixed, thesehavebeendescribed inthetext.Therewerenodifferencesofopinionwithintheevaluationteam,noconflictsofinterestandtheevaluationteamwasabletoworkfreelywithoutinterference.

DISSEMINATION AND FEEDBACK TO BENEFICIARIES

Atvarioustimes throughout theevaluation, stakeholderswereengaged toobtain feedbackonthemethodologyandprocessesortopresentfindingsfromtheevaluation.SomeoftheseeventsarepresentedinTable2.6.findingswerealsodisseminatedviasocialmediaandthroughpresentationswithexternalstakeholders.AdisseminationandcommunicationplanforthisfinalsummativereportisavailableinAnnex8.


COUNTRY OWNERSHIP, MANAGEMENT AND CAPACITY BUILDING The evaluation has been implemented in accordance with the Paris Declaration on AidEffectiveness(2015)principles74.TheintensivesetdatacollectionactivitieswereledbylocalresearchpartnersinEthiopia,Kenya,MyanmarandthePhilippines.Thelocalpartnersadaptedfieldprotocolsforthespecificcontext,ensuredsensitivitytolocalcultureandconsideredlocalethicalandlegalconcerns.capacitybuildingactivitiesforin-countrystaffoccurredatvarioustimesandincludedtrainingforeachlocalresearchmanagerinoverseeingthefieldworkandmanagingtheteam,aswellastrainingsfordatacollectorsontheinstrumentsandprotocols.capacitybuildingonevidencegenerationandhowtoappraisethequalityofevidencewasalsoconductedforalargerangeofDEPPstakeholdersataDEPPglobaleventthattookplaceinnairobi,Kenyain2016.

GENERAL CHARACTERISTICS OF THE STUDY POPULATION

Intotal,2,315interviewsorsurveyswereconductedduringT1,ofwhich347wererelatedtotheminimumsetactivitiesand1968werepartoftheintensiveset.InT2,3,440interviewsorsurveyswereconducted.Ofthosecompleted,250werefromtheminimumsetand3,190werefromtheintensiveset.

Tables2.7and2.8belowpresentthedemographiccharacteristicsofrespondentsinterviewedaspartof theevaluation’squantitativemethods:1)organisationalsurvey,2)KAPsurvey,3)householdsurvey,4)communitysurvey.Moredetailed,country-specificdemographicdataareprovidedinAnnex9.

ORGANISATIONAL SURVEY KAP SURVEY T1 T2 T1 T2

VariableDEPP

(N=172)Comparison

(N=78)DEPP

(N=143)Comparison

(N=64)DEPP

(N=479)Comparison

(N=138)DEPP

(N=333)Comparison

(N=119)

Age, mean (SD) 42.8 ± 9.7 47.4 ± 9.2 43.5 ±

10.0 46.6 ± 9.9 36.0 ± 9.1 40.0 ± 10.7 38.5 ± 9.6 40.4 ± 10.3

% Female 48 (27.9%) 19 (24.4%) 37 (25.9%) 16 (25.0%) 175 (36.5%) 40 (29.0%) 103 (30.9%) 36 (30.3%)

Education LevelSecondary School (some/ completed)

1 (0.6%) - 4 (2.8%) - 14 (2.9%) 4 (2.9%) 12 (3.3%) 4 (3.4%)

University (some/completed)

63 (37.8%) 25 (32.1%) 53 (37.1%) 18 (28.1%) 51 (53.0%) 71 (51.5%) 154 (6.6%) 66 (55.5%)

Master’s Degree 93 (54.1) 41 (52.6%) 73 (51.1%) 41 (64.1%) 184 (38.4%) 51 (37.0%) 142 (42.6%) 47 (39.5%)

Professional / Advanced Degree

8 (4.7%) 2 (2.6%) 8 (5.6%) 2 (3.1%) 12 (2.5%) 1 (0.7%) 11 (3.3%) -

Other 6 (3.5%) 10 (12.8%) 5 (3.5%) 3 (4.7%) 9 (1.9%) 5 (3.6%) 7 (2.1%) -

74 ParisDeclarationonAidEffectivenessandtheAccraAgendaforAction(2015).Availableat:http://www.oecd.org/dac/effectiveness/parisdeclarationandaccraagendaforaction.htm.


Type of OrganisationINGO 63 (36.6%) 37 (47.4%) 49 (34.2%) 28 (43.8%) 207 (43.2%) 70 (50.7%) 124 (37.2%) 59 (49.5%)

National NGO 45 (26.2%) 20 (25.6%) 37 (25.9%) 17 (26.6%) 92 (19.2%) 19 (13.8%) 60 (18.0%) 22 (18.5%)

Local NGO 52 (30.2%) 14 (18.0%) 50 (35.0%) 14 (21.9%) 144 (30.1%) 29 (24.4%) 105 (31.5%) 29 (24.4%)

Interna-tional Organi-sation (UN, World Bank etc)

- 3 (3.9%) 1 (0.7%) 1 (1.6%) 7 (1.5%) 4 (2.9%) 8 (2.4%) 3 (2.5%)

Government 4 (2.3%) 2 (2.6%) 4 (2.8%) 2 (3.1%) 15 (3.1%) 3 (2.2%) 18 (5.4%) 4 (3.4%)

Private Sector - 1 (1.3%) - 1 (1.6%) - - 1 (0.3%) -

Academic Institution 2 (1.2%) - 1 (0.7%) - 2 (0.4%) - 4 (1.2%) -

Other 6 (3.5%) 1 (1.3%) 1 (0.7%) 1 (1.6%) 12 (2.5%) 1 (0.7%) 12 (3.6%) 2 (1.7%)

Years Working in the Humanitarian Field<1 year 11 (10.7%) 3 (3.9%) 3 (3.2%) - 82 (11.7%) 8 (5.8%) 31 (9.3%) 1 (0.8%)

1-3 years 10 (9.7%) 4 (6.3%) 3 (3.2%) 4(6.3%) 90 (18.8%) 11 (8.0%) 60 (18.0%) 8 (6.7%)

3-5 years 7 (6.8%) 8 (10.3%) 16 (17.2%) 7 (10.9%) 69 (14.4%) 17 (12.3%) 53 (15.9%) 19 (16.0%)

5-10 years 30 (29.1%) 16 (20.5%) 26 (28.0%) 16 (25.0%) 123 (25.7%) 44 (31.9%) 95 (28.5%) 42 (35.3%)

10-15 years 17 (16.5%) 16 (20.5%) 20 (21.5%) 17 (26.6%) 72 (15.0%) 27 (19.6%) 63 (18.9%) 27 (22.7%)

15+ years 28 (27.2%) 31 (41.0%) 25 (26.9%) 20 (31.3%) 43 (9.0%) 31 (22.5%) 31 (9.3%) 22 (18.5%)

Table2.7:DemographicdatafororganisationalandKAPsurveysacrossallcountriesatT1andT2

HOUSEHOLD SURVEY T1 T2

DEPP Comparison DEPP Comparison

Variable Myanmar (N=589) Myanmar (N=415) Myanmar (N=570) Myanmar (N=433)

Age, mean (SD) 44.8 ± 14.4 43.3 ± 13.9 45.1 ± 14.2 44.0 ± 14.2

% Female 299 (50.8%) 211 (50.8%) 288 (50.5%) 225 (52.0%)

Attended school 452 (76.7%) 306 (73.7%) 447 (79.1%) 329 (76.5%)

Literate 424 (72.0%) 298 (71.8%) 425 (75.0%) 315 (73.4%)

Average #of family members living in dwelling

5.1 ± 2.2 5.2 ± 2.2 5.5 ± 2.3 5.4 ± 2.2

ReligionChristian 347 (58.9%) 294 (70.8%) 368 (64.6%) 311 (71.8%)

Buddhist 219 (37.2%) 117 (28.2%) 200 (35.1%) 121 (27.9%)

Other (Hindu, Islam,

Traditionalist)23 (3.9%) 4 (1.0%) 2 (0.4%) 1 (0.2%)

Community Survey T1 T2

DEPP Comparison DEPP Comparison

Variable Myanmar (N=19) Myanmar (N=20) Myanmar (N=19) Myanmar (N=19)


Age 45.6 ± 11.6 38.2± 10.9 43.4 ± 11.7 38.2 ± 10.6

% Female 1 (5.3%) 1 (5.0%) 1 (5.3%) 0 (0.0%)

Table2.8:DemographicdataforhouseholdandcommunitysurveysinMyanmaratT1andT2

*Data for Ethiopia were only available at T2 and thus were not included in the Difference-in Difference analysis

ORGANISATIONAL-LEVEL ASSESSMENT

Characteristics of Organisational survey respondents at T1

TheresponseratefortheorganisationalsurveyatT1was62.6%.Overall,DEPPstudyparticipantsoftheorganisationalsurveyatT1were27.9%femaleand72.1%maleandaveraged42.8yearsof age.Most participants held a university degree (33.1%) ormaster’s degree (54.1%) andworkedatinternationalnGOs(36.6%),nationalnGOs(26.2%)andlocalnGOs(30.2%).Themajorityofparticipantswereinsenior-levelpositions(90.1%).nearlyhalfoftheparticipantshadbeenworkingattheircurrentorganisationbetween10and15ormoreyears(56.3%)andhadbeenworkinginthehumanitarianfieldfor10to15ormoreyears(72.8%).

Characteristics of Organisational survey respondents at T2

The response rate for the organisational survey at T2 was 72.1%. Overall, DEPP studyparticipantsoftheorganisationalsurveywere25.9%femaleand74.1%maleandaveraged41.9yearsofage.Mostparticipantsheldauniversitydegree(32.9%)ormaster’sdegree(51.0%)andworkedat internationalnGOs(34.3%),nationalnGOs(30.6%)andlocalnGOs(32.6%).Themajorityofparticipantswereinsenior-levelpositions(87.4%),hadbeenworkingattheircurrent organisationbetween5 and10 years (25.8%) or between10 to15ormore years(31.1%).nearlyhalfhadbeenworking in thehumanitarianfield for10to15ormoreyears(48.39%).

Characteristics of KAP survey respondents at T1

TheresponseratefortheKAPsurveyatT1was79.8%.Overall,DEPPstudyparticipantsoftheKAPsurveywere36.5%femaleand63.5%maleandaveraged36yearsofage.Mostparticipantsheldauniversitydegree(49.5%)ormaster’sdegree(38.4%)andworkedatinternationalnGOs(43.2%),nationalnGOs(19.2%)andlocalnGOs(30.0%).Themajorityofparticipantswereinmid-level (40.9%)or senior-levelpositions (44.5%).nearlyhalfof theparticipantshadbeenworkingattheircurrentorganisationbetween1and3years(25.9%)or3and5years(19.0%)andhadbeenworkinginthehumanitarianfieldbetween3and5years(14.4%)or5and10years(25.7%).

Characteristics of KAP survey respondents at T2

The response rate for the KAP survey at T2was 72.1%.Overall, DEPP study participantsof the KAP survey were 30.9% female and 69.1% male and averaged 38.5 years of age.Most participants held a university degree (42.0%) ormaster’s degree (42.6%) andworkedatinternationalnGOs(37.2%),nationalnGOs(18.0%)andlocalnGOs(31.5%).Themajorityofparticipantswere inmid-level (44.4%)orsenior-levelpositions(47.2%).nearlyhalfoftheparticipantshadbeenworkingattheircurrentorganisationbetween1and3years(24.0%)or3and5years(21.0%)andhadbeenworkinginthehumanitarianfieldbetween3and5years(15.9%)or5and10years(28.4%).

COMMUNITY-LEVEL ASSESSMENT

Characteristics of Household Survey Respondents at T1

TheresponserateforthehouseholdsurveyatT1was98.1%.Demographiccharacteristicsofthe1,004respondentswhoparticipatedinthehouseholdsurveyinMyanmararepresentedinTable2.8andAnnex9.Theaverageageofrespondentswas44.2yearsandthesamplewas


roughlyhalfmale(49.2%)andhalffemale(50.8%).Themajorityofrespondentshadattendedsomeschooling(75.5%)andwereliterate(71.9%).Theaveragehouseholdsizewas5.2individualsperdwelling,andapproximately64%ofthesamplereportedtobechristian.

Characteristics of Household Survey Respondents at T2

Theresponserateforthehouseholdsurveyin?MyanmaratT2was99.9%.Theaverageageofrespondentswas44.7yearsandthesamplewasroughlyhalfmale(48.9%)andhalffemale(51.1%).Themajorityofrespondentshadattendedsomeschooling(78.0%)andwereliterate(74.3%).Theaveragehouseholdsizewas5.4individualsperdwelling,andapproximately68%ofthesamplereportedtobechristian.

Characteristics of Community Survey Respondents at T1

TheresponserateforthecommunitysurveyatT1was100%.Thissurveywascompletedwithcommunityleaderswithin19villagesinMyanmar.Theaverageageofrespondentswas41.8years.94.9%ofsamplerespondentsweremale,whileonly5.1%ofrespondentswerefemale.

Characteristics of Community Survey Respondents at T2

TheresponserateforthecommunitysurveyatT2was100%.Thissurveywascompletedwithcommunityleaderswithin20villagesinMyanmar.Theaverageageofrespondentswas40.8years.97.4%ofsamplerespondentsweremale,whileonly2.6%ofrespondentswerefemale.

MINIMUM SET OF EVALUATION ACTIVITIES Qualitative Interviews – Minimum Set of Evaluation Activities at T1

Atotalof51 individualsparticipated in theminimumsetqualitative interviewsatT1witharesponse rateof83.6% (i.e., 61were approached and10people didnot participate in thestudy).Ofthe51studyparticipantsinterviewed,sixwereDEPPstakeholders,28country-levelprojectleadership,and17UK-basedprojectleadership.Overall,studyparticipantswere37.3%femaleand62.7%malewithanaveraged39.8yearsofage.Moststudyparticipantshavebeenworkingattheircurrentorganisationbetween1and3years(49%)and3and5years(19.6%),andhavebeenworkinginthehumanitarianfieldbetween1and3years(15.7%),3and5years(21.6%),5and10years(23.5%),10and15years(13.7%)andmorethan15years(19.6%).

Qualitative Interviews – Minimum Set of Evaluation Activities at T2

Atotalof51 individualsparticipated in theminimumsetqualitative interviewsatT2witharesponserateof88.1%(i.e.,60individualswereapproachedand9individualsrefusedordidnotrespond).Ofthe51studyparticipantsinterviewed,fivewereDEPPstakeholders,31werecountry-levelprojectleadership,and15wereUK-basedprojectleadership.Overall,57.7%ofstudyparticipantswerefemaleand42.3%weremale.Theaverageageofstudyparticipantswas40.1years.Moststudyparticipantshavebeenworkingattheircurrentorganisationbetween1and3years(48.9%)andhavebeenworkinginthehumanitarianfieldbetween5and10years(37.0%),10and15years(13.7%).

Optionalqualitativefollow-upquestionnairesweresentatT2to43projectandprogrammestakeholders following in-depth interviews to provide respondents with an additionalopportunitytoshareinformationandperspectives.Ofthissample,twoparticipantsresponded.TheresponserateforthequalitativequestionnairesatT2is4.7%.

NETWORK SURVEY

Table2.9belowprovidestheoverallnumberoforganisationswhorespondedtothenetworksurvey,thetotalnumberoforganisationstheyidentifiedascollaboratorsandthetotalnumberoflinksbetweenorganisationsidentifiedineachcountry.linkscorrespondtodifferentformsofcollaborationandaredescribedinmoredetailinchapter5.


DuringT1,thetotalnumberoforganisationswhorespondedtothesurveywasnearlythesameinthePhilippinesandKenya(72and73respectively),howeverthenumberoflinksidentifiedwasmorethandoubleinthePhilippines.InT2,thenumberoforganisationswhoparticipatedinthenetworksurveyinKenyaincreasedfrom73into142,withnearlyfourtimesthenumberoftotallinks.OrganisationsinthePhilippineswhotookthesurveyincreasedbyapproximatelyonethird,howevertherewasa48%decreaseinthetotalnumberoflinks.Duringthevalidationworkshop in the Philippines, event participants attributed the decrease in number of linksto survey fatigue. largenumbers of organisational partners increased thetime required tocompletethenetworksurvey.Therefore,aftertheirexperiencesinT1,respondentslimitedthenumberoforganisationstheyreportedascollaboratingpartners,astheyknewthata largernumberofpartnersincreasedthetimeneededtocompletethesurvey.

InMyanmar,fewerorganisationscompletedthenetworksurveyinbothtimepointsbecauseof the small number of DEPP projects active there relative to the other countries. fewercollaborating organisations were identified, which in turn resulted in the identification offewerlinks.InEthiopia,thetimebetweenT1andT2waslessthantheothercountriesduetothedelays incontractingoutlinedinthe limitationssection,andthereforethenumberoforganisationsidentifiedwassimilarinbothtimepoints.However,thenumberoflinksincreasedby44%atT2.Analysesofthenetworksintheintensivesetcountriesaredescribedinfurtherdetailinchapter5.

COUNTRY AND TIME OF DATA COLLECTION

NUMBER OF ORGANSIATIONS

WHO COMPLETED NETWORK

SURVEY

TOTAL NUMBER OF ORGANISA-

TIONS IDENTIFIED

TOTAL NO. OF LINKS

The Philippines

T1 72 401 5622T2 92 254 2754

KenyaT1 73 161 2109T2 142 317 8027

MyanmarT1 33 75 207T2 30 55 226

EthiopiaT1 63 132 2438T2 59 137 4150

Table2.9:numberoforganisationsparticipatinginthenetworksurvey,identifiedorganisationsandlinksbycountry


DOCUMENT REVIEW

Intotal,634documents,includingreports,casestudies,videos,podcasts,articles,handbooksandothermaterialswerereviewed(Table2.10).Thisincluded334learningandevidenceitemsthatareavailablefordownloadontheDEPPlearningPlatform,whichrepresentsabout52.7%ofthetotalitemsreviewed.

TYPE OF RESOURCE TOTAL NUMBER REVIEWED

NUMBER (%) AVAILABLE ON LEARNING PLATFORM

Quarterly Reports 167 N/ABlog 80 77 (96.3%)Report/ Review 76 50 (65.8%)Case Study 68 43 (63.2%)Video/ Twitter Video 60 53 (88.3%)Key Findings/ Key Learnings/ Lessons Learned 38 37 (97.4%)Guidelines/ Minimum Standards 21 15 (71.4%)Article 19 9 (47.4%)Programme and Project-level Logical Frameworks 16 N/APodcast 14 14 (100.0%)Final Project Evaluation 13 0 (0.0%)Final Report Narratives 10 N/ATemplate/ How-to Guide/ Toolkit 9 8(88.9%)Concept Note 8 0 (0.0%)Summary/ Executive Summary/ Report Summary 6 6 (100.0%)Handbook/ Field Guide/ Framework 6 6 (100.0%)Terms of Reference/ Proposal 6 2 (33.3%)Presentation 4 3 (75.0%)Research/ Analysis 3 3 (100.0%)Policy Paper 2 2 (100.0%)Resource Catalogue 2 2 (100.0%)Business Case 1 0 (0.0%)Methodology 1 1 (100.0%)Slides 1 1 (100.0%)Mobile App 1 1 (100.0%)Newsletter 1 0 (0.0%)Reflection Journal 1 1 (100.0%)TOTAL 634 334 (75.7%)*

Table2.10:Summaryofdocumentsreviewed

*Does not include documents that are not applicable


HUMANITARIAN RESPONSE ASSESSMENT AND CASE STUDY

Atotalof22 in-depth interviewswereconducted for theKenyaandEthiopiacasestudies.InKenya,16 in-depth interviewswereconductedandanalysed.Of the16respondents,11(68.8%)weremaleandfive(31.2%)werefemalewithameanageof36.7years.Atotalofsevenorganisationswererepresentedintheinterviews.Respondentsreportedworkingonaverage7.6yearsinthehumanitariansectorand6.2yearsfortherespectiveorganisation.

InEthiopia,six in-depth interviewswereconducted.Ofthesixrespondents,fiveweremale(83.3%)andonewasfemale(16.7%).Themeanageofintervieweeswas42.8years.ThesixorganisationsrepresentedwerelocalbeneficiaryorganisationsaffiliatedwiththeShiftingthePowerproject.Respondentsreportedworkingonaverage7.8yearsinthehumanitariansectorand9.2yearsfortherespectiveorganisation.InSouthSudan,threeinterviewswereconducted.Giventhesmallnumberofinterviews,demographicdataarenotprovidedtoprotecttheidentityofthestudyparticipants.

LIMITATIONS Therearenotablestrengths to theevaluationactivitiesanddesign.These include thewiderangeofstudyinstrumentstocapturedataatdifferentlevelsofactionandadiversesampleofDEPPorganisations,programmeleadership,andprojectleadershipacrossDEPPcountriesandprojectswhowereinterviewedandsurveyed;however,itisimportanttoconsiderthestudy’slimitations.

Theresponseratefortheintensiveandminimumsetofevaluationactivitieswas,ingeneral,very high at both time points andwithin an expected range given the evaluation targetedsample. At T1, the response rate was 79.8% response rate for the KAP, 62.6% for theorganisationalsurvey,98.1%forthehouseholdsurvey,100%forthecommunitysurveyand83.6%forin-depthinterviews.AtT2,theresponseratewas72.1%fortheKAP,72.1%fortheorganisationalsurvey,99.9%forthehouseholdsurvey,100%forthecommunitysurveyand85%forin-depthinterviews.Responseratewaslowestfortheorganisationalsurveyatbothtimepoints.

Theorganisationalsurveywasadministeredtocountrydirectorsoforganisations,whowereoftendifficulttoinvolvebecauseofbusyschedulesandtravel.Tomitigateanypotentialbiasesa low response rate for the organisational surveymight introduce, various strategieswereemployed to increase participation. These included multiple contact attempts, flexibility intermsofinterviewdatesandtimes,andanoptionforaremoteinterviewiftheindividuals’travelschedulesdidnotpermitaface-to-faceinterview.fortheothertypesofsurveysandinterviews,non-responderswereprimarilyorganisationalleadershipandhumanitarianprogrammingstafffromimplementingandbeneficiaryorganisationsthatwereunavailabletoparticipateduetoschedulingconflicts.

HouseholdandcommunitysurveyswereconductedinMyanmarinbothtimepoints,butwereconductedinEthiopiaonlyduringT2duetocontractualdelayswiththelocalresearchpartner.Therefore, theresponseratesforthesecommunity-levelsurveysreflectonlydatacollectedinMyanmar.TheEthiopiadatawerenotusedfortheDifference-in-Differenceanalysisasthisrequirestimeseriesdata.

The majority of non-responders (n=10 at T1; n=9 at T2) for the minimum set qualitativeevaluation activities were primarily project leadership, and on subset analysis, they wererelatively evenly distributed among countries and projects indicating that there was nosystematic difference between responders and non-responders.Within the overall sample,therewerefewer interviewsconductedamongprogramme leadership,as theseparticipantsconstitutedamuchsmallergroupofstaffmembersthanthecombinedprojectleadership.The


evaluationteamalsocontactedkeyprogrammeandprojectstaffwhowerenolongerpartoftheDEPPtoensurethat theirviewswere included.Theviewsof therespondentsmaynotbe representativeofallof theprogrammestaff,fieldstaffandothers;however, therewerecommonthemesamongthepeopleinterviewed,andthesedataweretriangulatedwithotherevaluationdatatoensurevalidityandaccuracy.

It is importanttoconsiderthepotentialsourcesofbiasfortheaforementionedsurveysandinterviews. The lack of a true baseline is an important limitation for the evaluation. Datacollectionatthestartoftheprogramme,priortoanyprojectimplementationwasimpossibleduetothetimingoftheselectionandcontractingoftheevaluationteamwhichoccurredafterprojectshadbegunactivities.Sincethefirstquantitativedatacollectionpoint (T1)occurredafterDEPPimplementation,thefullimpactoftheDEPPmaynotbecaptured.AnychangesinoutcomesthatoccurredpriortoT1werenotbecapturedinthequantitativeanalysis,whichmeasureschangesbetween the twodatacollectiontimepoints.However,non-quantitativecomponentsoftheevaluationsuchasin-depthinterviewsanddocumentreviewwereassessedforevidenceofanyoutputsandchangesthatoccurredpriortoT1.

The interviewers assured respondents that privacy and confidentialitywould be respectedat all times, however social desirability bias could have affected responses. Since manyrespondentsareawareoftheexpectationsoftheDEPPandtheevaluation,theymayhavelimitedtheirresponsestowhattheythoughtDEPPmanagementortheenumeratorwouldwanttohear.OurteamutilisedenumeratorsandstaffwhoareexternaltotheDEPPandensuredthat respondents understood that their responseswould not be shared, andwould remainconfidential. Respondents were encouraged to share their honest opinions and reassuredthattheirresponseswouldhavenoimpactontheircurrentemployment.feedbackfromdatacollectionteamsinallcountriessuggeststhatrespondentswereveryforthcomingwithbothpositiveandnegativefeedbackontheDEPP,andthatsocialdesirabilitybiaswasminimised.Recallbiasmayalsohavebeenan issue,asrespondentsmaynotalwaysbeabletoprovideanaccurateaccountofpastactivitiesorexperiencesdatingfromthebeginningofDEPP. Inmanycases,therecallperiodwaslimitedtooneyearorlessinordertominimisethepotentialimpactofrecallbiasontheevaluation.Exclusionbiasmayalsohaveaffectedthestudysincein some settings such asKenya andMyanmar, security-related challenges limited access tocertainorganisations.Inthesesettings,effortsweremadetoremotelyincludeindividualswhocouldnotbephysicallyaccessedduetosecurityissues.Inaddition,topermittheDifferenceinDifferenceanalysis,manyofthesamequestionswereaskedinthesurveysduringT1andT2.Theremayhavebeenalearningeffectwhererespondentsanticipatedthequestionsaskedandpreparedtheirresponseaccordingly.ThiseffectisminimalbutitsimpactmaybeseeninthenetworkdataforthePhilippines,forexamplewhererespondents limitedthenumberofcollaboratingorganisationsdescribedduringT2sothattheirinterviewswouldbeshorter.

Inmostcases,qualitativeinterviewsfortheminimumsetofevaluationactivitieswereconductedviaSkypeandinEnglish;therefore,itispossiblethattheperspectivesofindividualswhomayhavehad limited Internetconnectivityand/orwhodidnotspeakEnglishwerenot included.However,theevaluationteamsampledindividualssystematicallyandnorespondentsdeclinedtoparticipateduetolanguageissues.Ininstanceswhereinternetconnectivitywasanissue,anoptiontoconductaphoneinterviewwasprovided.Therewereminortechnicalissueswiththeclarityandcompletenessoftheaudiofilesduringthedataextractionduetotheprogrammeplatform used to record the Skype interviews. However, the interviewers took notes andthemajority of files did not have any technical issues.Qualitativeminimum set interviewswere conducted by the same three individuals.noteswere compared and the same threeinterviewersextracteddatafromtheinterviews.ItispossiblethattherewasinterviewerbiasaseachindividualhadvaryingamountsofexposuretoandknowledgeofDEPP.However,theinterviewersfollowedadetailedsemi-structuredinterviewguideandaskedsimilarquestionswithsimilarprobestominimisepotentialinterviewerbias.Recallbiasmayalsohavebeenanissue,asrespondentsmaynotalwaysbeabletoprovideanaccurateaccountofpastactivitiesorexperiences.However,theevaluationteamprovidedthequestionareastotherespondentsaheadoftimeandencouragedthemtoshareadditional informationwiththeteamaftertheinterviewtoaddresstheissueofrecallbias.


It is also possible that individualswho did not speak Englishwere not fully represented inthe quantitative questionnaires. However, the online quantitative questionnaires from theminimumsetweretranslatedtofrenchtoensureinclusionofsampledrespondents.Intensivesetquestionnairesweretranslatedtothelocallanguagesandadministeredbylocalenumeratorstoensureinclusion.

Another limitationwhichmight have affected the evaluation is the high turnover ofDEPPprogramme and project leadership. A number of respondents interviewed during T1 wereunabletospeakdirectlyaboutthedesignprocessoftheDEPPprogrammeandofindividualprojects,astheywerehiredtoreplacestaffwhomayhavebeenengagedinthedesignprocess.HighstaffturnoverisarecurringthemethathasemergedbothwithrespecttothedesignandimplementationoftheDEPPandthathashadnotableeffects.Oneconsequencewithrespectto this evaluationhasbeen the lossof internal knowledgeabout topics suchas thedesignprocess and relevant lessons learned.To counter thispotential bias,we included staffwhowerenolongerpartoftheDEPPinoursample,andmadesubstantialefforttoensurethattheirperspectiveswereincluded.

Anotherpotentialweaknessistheinclusionofcommunity-leveldatainonlytwoofthefourintensive set countries (Myanmar andEthiopia). The inclusionof twocountrieswaspart oftheevaluationteam’smethodologyasoutlined inthe inceptionphaseandwaspartiallyduetobudgetconstraints.However,thisdesignalsotakesintoaccountthefactthatcommunity-leveleffectswerenotexpectedineverycountry,asprojectactivitiestargetingthecommunitylevelwerenotimplementeduniformlyacrosstheprogramme.Thetwocountriesselectedforcommunity-leveldatacollectionwereappropriateforassessingDEPPatthecommunitylevel.

ThedocumentreviewwasdependentonwhatwassharedbythelearningProject,andavailableonthelearningplatformandonbox,anonlineplatformusedtostoreprojectandprogrammedocuments. Documents and resources uploaded to box varied by project and programmeleadership,ascertainstakeholdersdidnotuploaddocumentssystematically.Everyeffortwasmadetolocatecertaindocumentsiftheycouldnotbelocatedinthesharedfolders.

There were unanticipated delays during the in-country data collection for the intensivesetof evaluationactivitieswhichnecessitated additionaltime tomaintain the integrity andcompletenessofthesedatacollectionmethods.Insomecases,thiswasduetodelaysinsettingupcontractualagreements,obtainingrespondentlistsandcontacts,challengeswithsecurityaswellaslimitedaccesstoareaswhereconflictorviolencewasoccurring.forexample,accesstoDEPP communities and households inMyanmarwas limited due to insecurity and datacollectionwasdelayeduntilsafeaccesscouldbeguaranteed.InSouthSudan,plannedintensiveevaluationactivitieswerenotconductedduetoanescalationofviolencethathaltedDEPPactivities in country and instead datawere collected remotely as part of theminimum setevaluationactivities.contractualdelayswereasubstantialissueinEthiopia,whereanagreementcouldnotbereachedbetweentheselectedlocalresearchpartnerandHHI.Ultimatelyanewlocalpartnerwasidentifiedandcontracted,howeverthesedelaysleddatacollectiontobeginlaterinEthiopiaforT1comparedtotheotherintensivesetcountries.ThetimeperiodbetweenEthiopiaT1andT2fortheorganisational,KAPandnetworksurveysinEthiopiaisthereforeshorter than for theothercountries. Inaddition, asmentionedabove,due to thesedelays,datacollectionatthecommunitylevelwasonlycollectedatonetimepointandthereforeaDifference-in-Differenceanalysiswasnotpossiblewiththeseparticulardata.

finally,thestudycouldalsobepotentiallyaffectedbybiasattheleveloftheevaluationteam.However it is importanttonotethattherewerenoconflictsof interest,andtheevaluationteamwasabletoworkfreelyandwithoutinterference.


METHODOLOGICAL CONSIDERATIONS AND APPROACH

TheevaluationmethodologywasdevelopedbasedoncriteriaadaptedfromtheDAcprinciplesfor Evaluating Development Assistance, and in alignment with the modified causal chaindevelopedbytheevaluationteam,ratherthantheprogramme’stheoryofchangepresentedin the business case which was found to be flawed, unnecessarily complex and not fullyrepresentativeofthetypesandscopesofDEPPactivities.

Arandomised-controlledtrial(RcT)wasanimportantelementoftheinitialevaluationproposalbecauseoftherigorousevidenceofimpactthatcanbeobtainedwiththismethod.OneoftheaimsduringtheinceptionphasewastothoroughlyinvestigatethefeasibilityofconductinganRcT.Afterextensiveconsultationsduringtheinceptionphaseandreviewofprojectdocumentsandplans,theevaluationteamconcludedthatanRcTwouldnotbefeasibleforvariousreasons.first, project activities across theDEPPwere extremely complex and broad in scope, andtargetedmultiplelevelsincludingcommunities,organisations,andsystems.Additionally,inmanycasestheprojectactivitieshadalreadybegunandwerebeingdeliveredtobeneficiariesbasedonnon-randomcriteria.Thus, theminimumconditions required for anRcT–a reasonablydiscreteinterventionwithalargenumberofunitsthatcouldberandomised,andtheabilitytomaintainunexposedcontrolgroupsduringthetimeframe–couldnotbemet.Theevaluationteam therefore opted to integrate quasi-experimental designs, which include a pre-postdesignandanappropriatecontrolgroupforcomparison,wherepossible.Thisapproachwassupplementedwiththeothermethods(qualitativedatacollection,documentreview,economicanalysis, network analysis, and in-depth case study of emergency responses) to provide amixedmethodsdesignwithmultiplesourcesofdatacapturedatthedifferentlevelsofaction(individuals,communities,organisationsandsystem).Theresultsweretriangulatedtotesttheprogramme’stheoryofchangeanditsunderlyingassumptionsandtoanswerkeyevaluationquestions.

The evaluation methodology – combining multiple research streams in a mixed methodsapproach–attemptedtocapturethebroadscopeoftheDEPP.Theuseofquasi-experimentalmethodsdrewuponinnovativepublichealthandmedicalresearchmethodologiestorigorouslyevaluatecapacitybuilding,andwascombinedwithqualitativeandparticipatoryapproachestocreateanintegrated,multi-dimensionalviewofDEPPprogramming.SincetheDEPPfocuscountriesdiffersignificantlyintermsofcontextualfactorssuchasgovernancestructures,anddisasterandpreparednessprofiles, itwas important toassesshowthesedifferentcontextsinfluencethetheoryofchange,programmeimplementationandoutcomechanges.contextualfactorsateachsiteweredocumentedtocontributetounderstandinghownon-DEPPfactorsinfluenceprogrammedeliveryandoutcomechanges.

ADDRESSING CROSS-CUTTING THEMES

Theevaluationmethodologywasdevelopedtotakeintoaccountthecross-cuttingissuesofgender, poverty, human rights, environment, anti-corruption, capacity building and powerrelations.HIV/AIDSwasnotaspecificcross-cuttingthemerelatedtotheprogrammeandhencenodatawascapturedonthisissue.notethattheevaluationToRonlyrequiredassessmentofgender,inclusionandviolenceagainstwomenandgirls(VAWG)acrosstheDEPP.

Gender:Documentreview,observationandsitevisits,in-depthinterviews,andorganisationalassessmentfocusedonprojectandprogramme’sactivitiesandgenderstrategyandinclusion.Inthecommunity-levelassessmenthouseholdsurvey,theevaluationteamsoughttoincludeasampleof50%maleand50%femaleparticipantstogatherinformationfrombothgenders.All organisational-level assessment study tools also aimed for equal representation of bothgenders. In-country data collection teamswere formedwith particular attention to genderbalance.

Poverty:Documentreview,observationandsitevisits,in-depthinterviews,andorganisationalassessment explored issues relating to poverty and economic inclusion. Households in thecommunity-levelassessmentwererandomlyselectedtoensurerepresentationfromdifferentsocio-economicclasses,anddatawerecollectedonsocio-economicstatus.Otherstudytoolsassessedfoodsecurity,shelter,educationandaccesstoservices.


Human rights:Document reviewandobservationandsitevisitsexplored issues relating tohumanrightsandensuredtheprojectsactivitiesdidnotdiscriminateagainstrace,sex,nationality,ethnicity,languageorreligion.Theelderlyandpeoplewithdisabilitieswereprioritisedwithinall community-level data collection to ensure that their perspectiveswere captured in theevaluation. Data collection protocols were developed to ensure protection of participants,minimiseanypotentialrisksandensureconfidentiality.Onlythoseparticipantswhoprovidedvoluntary,informedconsentparticipatedinthestudy.

Environment: Environmentalfactorsrelatedtodisastersandemergencieswereconsideredaspartofthecontextualfactorsinfluencingeachsettinganditsdisasterprofile.Atthecommunitylevel, data on environmental factors and sustainable environmentalmanagementwere alsocollected.Allevaluationdatawerecollectedusingelectronicquestionnairesprogrammedontabletstominimisetheevaluation’senvironmentalfootprint.

Anti-corruption:ThelocalresearchpartnerswerecarefullyvettedinaccordancewithHarvardUniversity’santi-fraudandanti-corruptionpolicies.Theevaluationteamworkedverycloselywith in-countrypartnersandmonitoredtheirworkduringon-sitevisitsandthroughregularmeetings.Whenpossible,observationandsitevisitstoDEPPprojectsallowedverificationthatactivitieshadbeenimplemented,includingverificationofpurchasedequipmentinuse.

Capacity Building:Asamainfocusoftheprogramme,allstudytoolsaimedtocapturecapacitybuildingandeffectivenessofactivitiesamongDEPPstakeholders.

Power relations:In-depthinterviewsandobservationandsitevisitsexploredpowerrelationsbetweenInGOsandn/lnGOs,anddifferentlevelsofprojectandprogrammestaffinordertobetterunderstandrelationshipswithintheprogrammeandtheireffectonimplementation.

79 SUMMATIVEPHASEREPORT-RElEVAncEAnDVAlIDITyOfDESIGn


RELEVANCE AND VALIDITY OF DESIGN

cHAPTER3


Evaluation Question #1: To what extent are the objectives of the programme intervention consistent with stakeholders' requirements and the programme design logical and coherent?

Sub-questions:a) Hastheprogrammetargetedtherightpeopleintherightplaces?b) Towhatextentdoestheprogrammedesign(theoryofchange)supporttheprojects’design(logical

framework)?c) In what ways was the programme design process participatory? Were project beneficiaries

adequatelyengagedbefore,duringandafter?d) Towhatextentwastheprogrammedesignlogicalandcoherent?

a. Weretheobjectivesoftheprogrammeclear,realisticandlikelytobeachievedwithintheestablishedtimescheduleandwiththeallocatedresources(includinghumanresources)?

e) Haveprioritisedtargetgroups(peoplewithdisabilities,olderpeople)andgenderaspectsbeentakenintoconsiderationintheprogramdesign?

Data sources: • Inception phase interviews (Qualitative)• Minimum set IDIs with programme and project stakeholders(Qualitative)• KAP and organisational surveys(Quantitative)• Intensive set IDIs(Qualitative)• Document review(Qualitative)Key Findings:• Thefiveresultsareas(objectives)oftheDEPPareappropriate, and the right people in the right

places were targeted• Athighlevel,programme objectives are clear, relevant and aligned with DFID priorities. However,

the three-year time frame was insufficienttoreachprogrammeobjectives• Thecountry selectionprocesswasiterative,basedonappropriatecriteriabut was not optimal

due to lack of strategic direction and objectives at portfolio level• There isgood alignment between the project’s design and the overall programme theory of

change. However, the programme-level logical framework was finalised late in programmeimplementationandreportingagainstthefinalversionbeganinAugust2017,duringthe3rdyearofDEPPimplementation

• Thedesign process was not logical or coherent,especiallyduringthefirstSTARTnetworkdesignphasewhenprojectswereretrofittedtothebusinesscase

• Designattheprogrammelevelwasnotadequatelyconsideredandprojects were designed in isolation with limited local consultation and buy in.Despite theflaws in the designprocess,the selected projects did address country needs once they were contextualised during theirimplementationperiods.However,programmesystemsandprocesses,theprogrammetheoryofchange,definitionofkeyterms,andexpectationsabouthowprojectsareintendedtointerlinkandinteractwerenotdevelopedandarticulatedattheoutset.

• Variation in number and type of projects in each country was a lost opportunitytomaximiseefficienciesandimpact

• Resourcing was sufficient at the programme level, but individual project budgets were leanwithinadequateresourcingforM&E,nPAcsandcollaboration

• All projects considered gender in their design, but the level of consideration varied. Someprojectsconsideredotherprioritisedgroupssuchastheelderlyandpeoplewithdisabilitiesintheirdesign

• Response to emergencies was not adequately considered during the design phase – theprogrammewas designed to improve emergency preparedness butmechanisms to enable orfacilitatel/nnGOcontributiontohumanitarianresponsewerenotexplicitlyintegratedintotheprogramme.Despitethis,DEPP contributed to at least 42 humanitarian responses in 11 countries.


OVERVIEW OF THE DEPPDESIGN PROCESS ThedesignprocessfortheDEPPaswellastheindividualprojectswithintheDEPPportfolioissummarisedbelowbasedonfindingsfromin-depthinterviewsanddocumentreview.figure3.1showsthesequenceofstepsinthedesignprocess.

In response to the Humanitarian Emergency Response Review (HERR) launched in 2011,whichincludedacalltoactionforthebritishGovernmenttosubstantiallymodifyitswayofresponding tohumanitarian crises,DfID launched thebusiness casedevelopmentprocess.Evidence suggests that early in the business case development process,DfIDhad alreadyidentified theconsortiumofbritishHumanitarianAgencies (whichwould laterbecome theSTART network) and the cDAc network as potential partners. Once the business casedevelopmentwasunderway, theSTARTnetworkand thecDAcnetwork,undertook theirown,separatedesignprocessesrelatedtotheDEPP.TheSTARTnetworksecretariatsolicitedproject ideasfromits (then)19members in2013throughageneralcall thatwascirculatedinternallytoitsmembers.Intotal,overfiftypre-generatedideaswereputforwardbySTARTnetworkmembers.AllfiftywereconsideredatanIdeasWorkshopheldinSeptember2013,whichinvolvedprimarilylondon-basedheadquartersstaffwhocametogethertodiscussandconsolidatetheideas.Theideaswerereducedto18bytheendoftheday,buttheprocessinvolvedmergingsomeprojectstogether,leadingto,insomecases,theformationofartificialormandatedconsortia.

“Themashingworkshopput together four streamsof individual projects, projectsthatbuildindividual’scapacity,andmashedthemintosomethingthatperhapsifyouhadablanksheetofpapershouldn’thavelivedtogether…Wewereperhapsforcingpeopletocollaboratewhoshouldn’thavebeen...thisthenaffectedthequalityofthecollaborationbetweenthemembers.”(Programmeleadership)

Theprocessofconsolidatingandmergingprojectswasnotbasedonanyparticularstrategyordefinedcriteriaandlackedinvolvementfromin-countrystaff.Therewasapreferenceforprojects thatwere already being implemented or that agencieswere hoping to implementratherthannew,innovativeideas.Thecreationofsomeartificialconsortiahadlongreachingimpacts.OneoftheauthorsoftheDEPPdesigncasestudy75providedoneexample:“Threeagencies brought ideas for humanitarian training schemes that they were already runningwithgreatsuccess.Thesethreeagencieswerebroughttogetherandtaskedwithcreatingoneconsortiumprojectoutofthethree.Atfacevalue,thisseemedtomakesenseinthenameofcollaborationandefficiency;however,thehightransactioncostsofworkingintheconsortiummayarguablyinitiallyhaveoutweighedanyperceivedadvantageofworkingtogether.”

Thecasestudyalsodiscussessomeoftheconsequencesofthesemandatedcollaborations:

"Someconsortiathatweremandatedtoworktogetherstruggledtoreconciledifferinginterests and opinions. Partners reported that these frustrations undermined thecollaboration,andwhichrequiredsustainedefforttodisentanglelaterintheproject’slife-cycle,affectingefficiencyandeffectivenessduringdelivery.”

75 KletzingM,bevanD.PreparingforPreparedness:lessonsfromDesigningandSetting-uptheDisastersandEmergenciesPreparednessProgramme.


figure3.1:SequenceofstepsintheDEPPdesignprocess

followingtheIdeasWorkshop,teams,whichagainwereprimarilyUK-basedstaff,weregiventhreemonthsbytheSTARTsecretariattoformtheirconsortia,includinggeneratingdetailedbudgetsandproposalsbeforepresentingthemtoapanelofsixjudgesaspartofaninternalpeerreviewprocess.However,atthistime,theDfIDbusinesscasehadnotyetbeenfinalised,norhadthespecificDEPPtargetcountriesbeenselected,whichmeantthatprojectscouldnotbedevelopedwiththoseinmind.Inaddition,therewasnotenoughtimetoinvolvecountryoffices,andlondon-basedstaffdidnotfeelitwasjustifiedtousethetimeandresourcesofcountrystaffwhentherewasnoguaranteeoffunding.Therewasinsufficienttimetosetupconsortiaorfullyconsiderimplementationdetailsorappropriateresourcing.Attheendofthisthree-monthperiod,submissionswerepeer-reviewedandninewereselectedtogoforwardtobepresentedtoDfID.Thepeerreviewprocesstookalow-riskapproach.Allofthisworkwasconductedbeforethebusinesscasewasfinalisedandapproved.Thisprocessofdevelopingprojectsbeforethefinalbusinesscasehadseveraleffectswhicharedescribedlater.

According to the DEPP design case study76: “the decision-making on DfID’s investmentincludedextremelylimitedconsultationwithnGOsandotherexpertstodevelopthebusinesscase,whichmeantthatrelevantlessonsfromthehumanitariansector’sbroaderexperienceofimplementingpreparednessprojectswasnotsystematicallyincludedintheevolvingbusiness

76 KletzingM,bevanD.PreparingforPreparedness:lessonsfromDesigningandSetting-uptheDisastersandEmergenciesPreparednessProgramme.

HERR Launched

DFID launches process for business case development

START Network solicits ideas from 19 members CDAC Network

CDAC Network - submits proposal to DFID Board

2011

Sept 2013

Oct-Dec2013

Feb 2014

April 2014

July 2014

Oct 2014

Nov 2014

Jan 2015

Jan 2015

April 2015

May 2015


casefortheDEPP.”Inaddition,evidencesuggeststhattheDfIDcountryofficesdidnotfeedintothedecision-makingprocess,andthattherewasalsoalackofbroaderconsultation.Thefinalbusinesscasedidnot containbasicdesigndetails suchas focuscountriesandprojectselectioncriteria,andtherewasonlyonemonthbeforethefirstboardmeetingtodevelopthesesubstantialareas.

InApril2014, thebusinesscasewasfinalisedandDfIDannounced£40millionof fundingfortheDEPP,ofwhich£10millionwouldbeallocatedtowardan“innovationwindow”;thus,leaving£30millionforDEPPprojects.Thiswasallocatedtothetwopre-selectednetworks,withtheSTARTnetworkreceiving£27millionandcDAcnetworkreceiving£3million.

ThefirstDEPPboardmeetingwasheldduringthatsamemonthinApril2014.Elevencountries– Kenya, Ethiopia, Democratic Republic of the congo, South Sudan,Mozambique, Jordan,Myanmar, the Philippines, bangladesh, Pakistan, and Indonesia –were selected, andDEPPprojectselectioncriteriaandmonitoringandevaluationplanswerediscussed.

Thesecondboardmeeting,inJuly2014,ledtotheapprovalofthefirstfiveDEPPprojects:TransformingSurgecapacity (TSc),Shifting thePower (STP),TalentDevelopment (TD), theAgeandDisabilitycapacitybuilding(ADcAP)projectandtheMElproject(whichwaslaterrebrandedasthelearningProject).Threemoreprojectswereapprovedduringthe3rdDEPPboardmeetinginOctober2014:linkingPreparednessResilienceandResponse(lPRR),financialEnablers(fE),andProtectioninPractice.ThefinancialEnablersprojectwasconsideredhighriskbyDfIDandwasscaledbacksubstantially.Itwasapprovedforimplementationinonlyonecountryratherthanfourasoriginallyenvisionedandwithabudgetreducedby£3million.TheAlERTprojectwasapprovedinJanuary2015duringthe4thboardMeeting.

IncomparisontotheSTARTnetwork,thecDAcnetworkinstitutedadifferentdesignprocess.OncethebusinesscasewasfinalisedinApril2014,asmallgroupofinterestedcDAcnetworkmembersworkedovereightmonthstodevelopaprojectthatspecificallyfittheDEPPbusinesscaseandselectioncriteria.Thisprojectwassubmittedinnovember2014andapprovedduringthe4thDEPPboardmeetinginJanuary2015.

Afterthe4thDEPPboardmeeting,theSTARTnetworkhadfundsremaining(fromthecutstothefinancialEnablersproject)anddecidedtolaunchasecondwaveofproposaldevelopment,buildingonlessonslearnedduringtheinitialprojectdesignphase.contrarytothefirstroundofprojectdesignduringthedevelopmentofthebusinesscase,andinanefforttoimprovetheprocess, it significantly encouraged and incentivisednational project staff from itsmemberorganisations to take the lead, and to involve local stakeholders.One-pageproposalsweresoughtonprojectsthatfocusedonearlywarningsystemsinordertofillanoticeablegapintheDEPPportfoliowithrespecttoresultareatwofocusedondevelopmentofearlywarningsystems.The shift tovery shortproposalswasdone toenablenational project staff tobesignificantlyinvolved.consortiawerepermittedtodevelopnaturallyaroundsharedinterests,andlocalownershipwasencouraged.Acommitteeoftechnicalexpertsconductedareviewofproposals, and significant involvementof in-country teamswasa selectioncriteria.fourprojectsselectedthroughthisprocessweresubmittedtotheDEPPboardandwereultimatelyapprovedinApril2015.TheexternalevaluationwasapprovedinMay2015.

Thefollowingsectionspresentevaluationfindingsrelatedtotherelevanceandvalidityofthedesignprocess,aswellasitssuccessesandlimitations.


QUESTION 1.A HAS THE PROGRAMME TARGETED THE RIGHT PEOPLE IN THE RIGHT PLACES?

The established five results areas (objectives) of the DEPP are appropriate, and the right people in the right places are being targeted. However, the country selection process, while iterative and based on appropriate criteria, was not optimal due to lack of strategic direction and objectives at the portfolio level.

Theevaluationassessedthisquestionwiththreemainlenses:first,whethertherightcountrieswere chosenandwhether the country-selectioncriteriawere clear; second,howdecisionsweremadeabout thenumberofDEPPprojectspercountry;and third,whether theDEPPdesignisconsistentwithbeneficiaryneedsandrequirementswithincountries.

DEPP Targets and Country Selection Criteria

Ingeneral,thetargetsoftheDEPPareappropriate,andtherightpeopleintherightplacesweretargeted.Thefocusonbuildingnationalcapacity,improvingemergencypreparednessandtargetingvulnerablegroupstoensuretheirinclusionduringhumanitarianresponseactivitiesisconsistentwithneeds identifiedduring theevaluationteam’s literaturereviewconductedduringtheinceptionphase.Thefocusremainsalignedwithnewerglobalcommitmentsrelatedtohumanitariansystemreformandforlocalisationofhumanitarianaid77.

Documents suggest that the country selectionprocesswas iterative andbasedon severalcriteria – humanitarian need and likelihoodof natural disasters using the InfORM Index78,followedby feasibilityandopportunity tomake thegreatest impact.Theseareappropriatecriteria.However,accordingtotheDEPPdesigncasestudy79,agenciesperceivedthatDfIDpreferredimplementationincountrieswhereitalreadydirectlyworkedin,implementationinfewercountriestomaximiseimpactandtoworkinsettingsatriskofnaturaldisastersratherthanmanmadeasitwouldbelessrisky.Inaddition,sincethetwonetworkshadalreadybeenselected, theDEPP couldonly operate in countrieswhere thosenetworks had an existingpresence.Ultimately,however,somerespondentsfeltthatkeycountriesandprioritygeographicareas with strong humanitarian need were not included and attributed this mainly to thelimitationsofoperatingonlyincountrieswheretheSTARTnetworkandthecDAcnetworkmembershadapresence.byoperatingthroughpre-selectednetworks,thereisapossibilitythatstrongerorganisationsandinterventionswerenotincluded.Minimumsetintervieweesalsodescribedalackofstrategicdirectionandobjectivesattheportfoliolevel,whichhinderedthepotentialofselectingcountriesandprojectsbasedonstrategiccriteria(e.g.,contextwithabilitytomakebiggest impact,highestdisaster risk,greatestneed,and/ormostvalue formoney).AnotherintervieweestatedthattheselectionofDEPPcountriesbytheDEPPboardhadbeenaconvolutedprocesswithtoomuchinputfromtheSTARTnetwork,andemphasisplacedonexistingmembercoveragewithinthenetwork.ThisisconsistentwiththeDEPPdesigncasestudywhichpresentedproblemswiththecountryselectionprocess,withoneagencyreferringtothefinalsetoffocuscountries:“analmostrandomlist”80.

Coverage of DEPP Interventions Across Target Countries

Of the 11 target countries that were finally selected, projects were implemented in only10 countries (Indonesia ultimately was not targeted by any DEPP projects). Despite thecommitment towards the five result areas, the portfolio of interventions in each countryalsovariedgreatly,withsomecountriesreceivingveryfewprojectsandothersmanymore.Thisseemedtobeduetominimalconsiderationat theportfolio levelof thedistributionofDEPPinterventionsacrosscountriesandtheoptimallevelofexposuretoDEPPinterventionsneededtomaximisepotentialimpact.Thiscoveragestrategy,asdescribedlaterinthereport,

77 SuchastheGrandbargain-Workstream2andTheSendaiframeworkforDisasterRiskReduction.78 http://www.inform-index.org.79 KletzingM,bevanD.PreparingforPreparedness:lessonsfromDesigningandSetting-uptheDisasters

andEmergenciesPreparednessProgramme.80 KletzingM,bevanD.PreparingforPreparedness:lessonsfromDesigningandSetting-uptheDisasters

andEmergenciesPreparednessProgramme.


diminishedthepotentialeffectivenessoftheprogrammeinsomeoftheDEPPfocuscountrieswithfewernumbersofprojects.Onerespondentfeltthattherewasahigherleveloflearningincountrieswithmoreprojectsduetoagreatercriticalmass.forexample,Pakistanhadsevenprojects,and this led toa lotoforganicactivitiesandcostefficiencies,withpeoplesharingresourcesandstartingnewprojects.ThefindingsonprogrammeimplementationlaterinthisreportsupportthehypothesisthattherewasmorecollaborationinsettingswherethereweremoreDEPPprojects. Insomecases,thiswasfurtherenhancedbythepresenceofRegionallearningAdvisors,orlaterthecountrylearningAdvisors(clAs),thatwereembeddedinmanyoftheDEPPcountriestostimulatecollaborationandsharingoflearning.

ThefindingsfromtheprimarydataarefurthersupportedbytheDEPPdesigncasestudy81.Thereportcitesseveralchallengeswiththecountryselectionprocess,indicatingthatthiswasneitherefficientnoreffectiveintermsofaligningprojectswithneedssystematicallyidentifiedupfront.Thereportstates:

“As a result of setting geographical criteria both after the business case and aftermostof theprojectshadbeendesigned,humanitarianneedforpreparednesswasnottheprimaryincentive.Someprojectssoughtapprovaltoimplementincountriesnotonthe list,andoneoftheofficialcountriesreceivednoinvestmentatall.AndalthoughtheDEPPwasdesignedwiththeaimofachievingfiveresults,thefocusofinterventionsvariedbetweencountries.”

Beneficiary Needs in DEPP Target Countries and Relevance

At the project level, many projects did not conduct needs assessments during the designprocess,whichcallsintoquestionwhetherthemostappropriateindividualsweretargetedandwhethertheirneedsandrequirementswereadequatelyaddressedwithineachcountry.Theprocessofretro-fittingpre-designedprojectstospecificcontextsoncethegeographiclocationswereselectedbytheDEPPboardandlackofinvolvementofthein-countrystakeholders,likelyaffectedtheinitialrelevanceoftheprojects.OncetheprojectsandcountrieswereapprovedbytheDEPPboard,mostprojectsreportedspendingasubstantialamountoftimecontextualisingtheirprojectstoeachcountryorsite.Whilethisledtodelays,itwasaimedatensuringrelevanceofthedesign.However,datasuggestthatthisprocessstilloftendidnotincludeconsultationswithbeneficiariessuchascommunities,households,andl/nnGOs,whichremainsasignificantweaknessoftheapproach.

As part of the quantitativeminimumand intensive set surveys,DEPP respondents (n=245)acrossall10DEPPfocuscountrieswereaskedabouttherelevanceoftheDEPPinterventionsintheircountry(seeTable3.1).Themajorityofrespondentsreportedtheinterventionstobeeitherrelevantorextremelyrelevant(93.9%overall)82.ThehighestreportedrelevancewasinEthiopiawhere98.2%ofintervieweesreportedtheDEPPtoberelevantorextremelyrelevant.

81 Ibid.82 These findings are consistent with the findings of the project final evaluations which find each of the projects to

be relevant to existing needs.


ON A SCALE OF 1-5, HOW RELEVANT ARE THE DEPP INTERVENTIONS FOR THE COUNTRY IN WHICH YOU ARE WORKING?

All DEPP countries

N=245

(%)

Local Organisations

N=152

(%)

Minimum set

N=250

(%)

Ethiopia

N=54

(%)

Kenya

N=63

(%)

Myanmar

N=31

(%)

The Philippines

N=60

(%)

Not at all relevant

0

(0.0)

0

(0.0)

0

(0.0)

0

(0.0)

0

(0.0)

0

(0.0)

0

(0.0)

Irrelevant1

(0.4)

1

(0.7)

1

(2.7)

0

(0.0)

0

(0.0)

0

(0.0)

0

(0.0)

Somewhat relevant

14

(5.7)

10

(6.6)

4

(10.8)

1

(1.85)

2

(3.2)

2

(6.5)

5

(8.3)

Relevant150

(61.2)

94

(61.8)

14

(37.8)

38

(70.4)

41

(65.1)

19

(61.3)

38

(63.3)

Extremely relevant

80

(32.7)

47

(30.9)

18

(48.7)

15

(27.8)

20

(31.8)

10

(32.3)

17

(28.3)

Table3.1:RelevanceoftheDEPPinterventions

QUESTION 1.BTO WHAT EXTENT DOES THE PROGRAMME DESIGN (THEORY OF CHANGE) SUPPORT THE PROJECTS’ DESIGN (LOGICAL FRAMEWORKS)?

There is fairly good alignment between the project’s design and the overall programme theory of change. However, the programme-level logical framework was developed late in programme implementation and reporting against the final version of the logical framework began in August 2017, during the third year of DEPP implementation. TheDEPPbusinesscase includesatheoryofchangefortheDEPPandfocusesonfivekeyresultsareasasdescribedearlier.This theoryofchangewaspresumablyutilisedduring thedesignof someof theprojects.However,evidenceon theextent towhich theprogrammetheoryofchangewasusedtoguideprojectdesignorrefinementofprojectactivitiesislimited.DatafromtheinterviewsanddocumentreviewsupportthefactthatindividualprojectsinthefirstSTARTnetworkdesignphasewerenotdevelopedspecificallytofitthebusinesscaseorprogrammetheoryofchange.However,despitethis,thereisfairlygoodalignmentbetweentheprojects’designsandtheoverallprogrammetheoryofchange.Thiscouldhavebeenstrongerhadthedesignprocessunfoldedinalogicalsequence.OneothersignificantweaknesswasthelackofanoverallDEPPprogramme-levellogicalframeworkaswellasguidanceforindividualprojectsindevelopinglogicalframeworksandM&Esystemsthataligntogetherandwiththeprogramme. A programme-level logical framework was developed very late in programmeimplementationandwasinformedbytheexistingindividualprojectlogicalframeworks–thisprocessshouldhavebeenreversedwiththeprogramme-levellogicalframeworkdevelopedatthebeginningoftheprogramme.Thefinalversionoftheprogramme-levellogicalframeworkwasapprovedandreportingagainst itbegan inAugust2017, inthethirdyearoftheDEPPimplementation.consequencesofthisaredescribedlaterinthereport.


QUESTION 1.CIN WHAT WAYS WAS THE PROGRAMME DESIGN PROCESS PARTICIPATORY? WEREPROJECT BENEFICIARIES ADEQUATELY ENGAGED BEFORE, DURING AND AFTER? The DEPP design process was participatory, but it did not adequately involve in-country beneficiaries and stakeholders, or local needs assessments. Overall, the design process for theDEPPwas participatory in that it involved consultationwithvariousstakeholders.However,beneficiariesatthelocalinstitutionallevelandintargetedcommunities and countries were not adequately engaged in the process, especially withrespecttotheSTARTnetworkprojectsdevelopedduringthefirstphase(SeeTable3.2).TheconsultationsthatdidoccurtendedtobeprimarilyattheUKlevelonlyandwereledbylargeInGOsandlondon-basedstaff.Giventhelackofinvolvementofin-countrypartnersandsmalleragencies,thedesignprocessdidnotfollowcurrentbestpracticesrelatedtoparticipatorydesignapproaches.furthermore,theDEPPdesignprocessfocusedmainlyondevelopingindividualprojectsinisolation,withlimitedconsiderationaboutsystemsattheportfoliolevel,andabouthowprojectsmightaligntogether.

furthermore,interviewdatasuggesttherewaslimitedinvolvementoflocalstakeholdersinthedesignoftheindividualDEPPprojects,aswellasalackofneedsassessmentswithinthetargetcountries83,84.Thisledtoinadequatecontextualisationandunderstandingoftheprojectsatthecountrylevel.Projectsthenhadtousetheinitialprojectimplementationperiodtoadapttheirprojectstothelocalcontextwhichinturncontributedtodelaysinprojectimplementation85.Moretimeandresourcesbuiltintotheprojects’inceptionphasecouldhavehelpedprojectstodevelopamorerealisticprojecttimeline.

ALL DEPP COUNTRIES

N= 51

UK-BASED PROJECT

LEADERSHIP

N = 17

IN-COUNTRY PROJECT

LEADERSHIP

N = 28

PROGRAMME LEADERSHIP

N = 6

The design process was participatory ✔ ✔ ✖/✔ ✔

The design process in-volved local stakeholders ✖ ✖ ✖/✔ ✖

There was sufficient time for collaborations during design process

✖ ✖ ✖ ✖

Table3.2:PerceptionsoftheDEPPdesignprocess

✔Mostrespondentsagreewithstatement ✖Mostrespondentsdisagreewithstatement✖/✔Responsesweremixed

83 Thisfindingissupportedbydocumentreview.forexample,TScfinalEvaluationReport,p17states:“Mostcountryplatformstakeholdersandsomeregionalandinternationalstakeholdersfelttheoriginaldesigndidnotadequatelyreflecttheprioritiesatthecountrylevel(forexample,thefocusonaccesstofundingfornationalorganisationswasastrongpriorityinthePhilippinesthatwasnotreflectedintheoriginaldesign)”.TheADcAPfinalEvaluationReport,p2states:“therewascriticismthatcountryofficesandnnGOswerenotinvolvedorinvolvedquitelateintheprocess.”

84 ThePHEPGambellaprojectwasoneofthefewthatconductedabaselinethatwasusedtoguideprogramimplementation.PHEPfinalEvaluationReport,p4:“Theprojectbenefitedfromabaselineassessmentwhichhelpedtoidentifywherecapacityshouldbetargetedandeffortprioritized.”

85 forexample,seeShiftingthePowerfinalEvaluationReport,p30,orTransformingSurgecapacityfinalEvaluationReport,p31.


These issues in the design of both theDEPP and the individual projectswere due to keyflawsinthedesignprocess–anunfavourablesequenceofstepsintheprocess,theselectionofprojects in tranches, and inadequatetimeand resources toengagesmallerorganisationswith limited resources.As illustrated infigure3.1 anddescribedearlier,mostof theDEPPprojectsandproposalsweredesignedbeforethebusinesscasewasfinalised,andthecountryselectionoccurredafterthebusinesscasewasapproved.Untilthecountryselectionoccurred,it would have been difficult to engage local stakeholders in the project development. Analternativescenariowouldhavebeentoconductaglobalassessmentduringthedevelopmentof thebusinesscaseandbefore thecountry selectionwasmade, to identifycountriesandbeneficiariesingreatestneed.Onerespondentstated:

“ThefirstroundoftheDEPPprojectswereabittoolondoncentric…IthinkalsoithastodowiththefactthatthebusinesscasecameverylatefromDfIDandthatalsothecountryfocuscameverylate.Wedidn’tknowuntiltheverylastminutewhichcountrywasgoingtobethefocus.So,itwasverydifficulttoengagewithourcountryteamsononesideandalsolocalpartners,becauseyoucreateaveryhighexpectationandyou’renotevensurethatthatcountryiseveneligible.“(Projectleadership)

Several exceptions exist; interviewees confirmed that the four projects that were fundedduringthelasttranche(UrbanEarlyWarningEarlyAction(UEWEA),StrengtheningEmergencyPreparednessSystemsinMyanmar(SEPS),EthiopiaEarlyWarningEarlyAction(EWEA),andPublicHealthPreparednessinGambella(PHEP))weredesignedthroughamoreparticipatoryprocessanddidinvolveconsultationwithlocalstakeholders.Thiswaslinkedtosmootherandtimelierprojectimplementation.

“forthesecondroundofprojectsthatweretheearlywarningprojects,therewasacleardirectivethattheyhadtobedevelopedwiththecountryofficedirectingthem,and I think that is possibly one of the reasonswhy these projects havemanagedtocatchupsoquicklyeven though theystartedalmostayear later thansomeofthe other projects. It’s because they’ve been developed in collaborationwith thecountry offices so the country office already had the buy-in and the drive to getgoing.”(Projectleadership)

Asthesefourprojectsunderwentamorestreamlineddesignprocessthattookintoaccountsomeoftheshortfallsofthefirstphase,thisfindingisnotsurprising.

QUESTION 1.DTO WHAT EXTENT WAS THE PROGRAMME DESIGN LOGICAL AND COHERENT? WERE THE OBJECTIVES OF THE PROGRAMME CLEAR, REALISTIC AND LIKELY TO BE ACHIEVED WITHIN THE ESTABLISHED TIME SCHEDULE AND WITH THE ALLOCATED RESOURCES (INCLUDING HUMAN RESOURCES)? The design process was not logical or coherent, especially during the first START Network design phase when projects were retrofitted to the business case. At the macro level, programme objectives are clear, relevant and aligned with DFID priorities but the three-year time frame was insufficient to reach programme objectives. Resourcing was sufficient at the programme level, but individual project budgets were lean with inadequate resourcing for M&E, Non-project Attributable Costs (NPACs) and collaboration. Toalargeextent,theDEPPprogrammedesignprocessdidnotfollowalogicalandcoherentapproach.Asdescribedearlier,mostprojectsundertheSTARTnetworkweredesignedpriorto thefinalisationof theDfIDbusinesscase, logical frameworkand theselectionof targetcountries.Uponfinalisationof thebusiness case,nineof thesepre-designedprojectswereretrofittedtoalignwiththebusinesscase,presentedtoDfIDandapprovedintranches.ThecDAcnetworkadoptedadifferent,morelogicalapproachwithrespecttothedesignprocess.Projectsweredevelopedoveralongerperiodoftimeandweredesignedtofitthebusinesscaseandselectioncriteria.nationalprojectstaffandlocalstakeholderstookalargerroleinthedesignofthesecondroundofSTARTnetworkprojectsandconsortiaweredevelopedbased


onsharedareasoffocus.Thus,theevidencesupportsthatthesequenceofstepsinthedesignprocess,particularlyforthoseprojectsdesignedduringthefirstSTARTnetworkphase,werenottakeninalogicalandstrategicorder.

Were objectives clear and achievable within the timeframe?

Theevaluationteamreviewedtheobjectivesof theDEPPasawholeandof the individualprojects.Atamacrolevel,theobjectivesareclear,relevant,alignedwithDfID’shumanitarianpriorities86,87 andaimedtofillacleargapinhumanitariancapacitytoprepareforandrespondtodisasters.Asdescribedintheliterature,increasedfundingisneededforpreparednessactivitiesandforstrengtheningcapacityofnationallevelactorsratherthaninternationalactors88.Thereis a shortageofpeopleandsystemswith sufficientcapacity toprepare forand respond todisasters incountriesathighriskfordisasters,particularlyatthenational level89.TheDEPPobjectivestostrengthennationalcapacitytorespondtodisastersandincreasepreparednesslevelsincountriesatriskfordisastersarerelevantforthetenDEPPfocuscountries.However,the three-year time frame to achieve the selected objectives was unrealistic. A five-yeartimeframewould have beenmore appropriate for these specific objectives 90. In fact, thebusiness case had included a conditional commitment of two years additional yearswhichwouldhaveextendedtheprogrammedurationtofiveyears.Datafromthedocumentreviewsupportstheneedforalongertimeframe91,92,93.

These findings related to the design process, objectives and timeframe are supported byinterviewdataandotherdatacollectedacrossalltenDEPPcountriesduringtheformativephase.IntervieweeswerealsoaskedaboutthevalidityoftheDEPPdesigninordertounderstandhowspecificactivitiesrelatedtoDEPPobjectives.Overall,mostagreedthattheDEPPobjectivesareclearormostlyclear(SeeTable3.3).However,someindividualsfeltthattheobjectivesandfiveresultsareasthatprojectactivitiestargetweretoobroadandwouldhavebenefittedfromincreased specificity and refinement.However, thebroadnessof the results areaswas alsoconsideredapositivepointasitpermittedthediversityofdifferentprojectsthatexistedwithintheDEPPportfolio.Theremayhavebeensomeambiguityaroundtheprecisemeaningofthedifferentresultsareas,suchasresultsareafour(Improvedinstitutionalarrangementsandpolicyenvironmentssothatnationalsystemsforhumanitarianresponseandpreparednessarebettersupportedandmoresuitable).Differentagenciesappear tohave interpretedkey termsandphrasesdifferently.Theprogrammewouldhavebenefittedfromclearlydefiningkeyconceptssuchascollaborationandinstitutionalarrangementsearlyon.Thisinturnwouldhaveenabledprojectstousestandardisedmeasurementsanddevelopstandardisedindicators.

86 https://www.gov.uk/government/speeches/uk-aid-in-2015-the-progress-so-far-and-the-priorities-ahead.87 https://www.gov.uk/government/publications/dfid-single-departmental-plan-2015-to-2020/single-depart-

mental-plan-2015-to-2020.88 HolmesJ.HumanitarianResponseinthe21stcentury.ThebrownJournalofWorldAffairs,vol.17,no.2,

2011,p113–121.,http://www.jstor.org/stable/24590801.89 HumanitarianResponseReview.AnindependentreportcommissionedbytheUnitednationsEmergency

ReliefcoordinatorandUnder-Secretary-GeneralforHumanitarianAffairs,OfficeforthecoordinationofHumanitarianAffairs(OcHA).Adinolfietal,August2005.

90 basedonintervieweeresponsesandevaluationteamexpertopinion.91 linkingPreparednessResilienceandResponsefinalEvaluationReport,p10.92 ShiftingthePowerfinalEvaluationreport,p30.“ProgresswasachievedbySTPinwhatwaseffectivelya

projectimplementationtimeframeof2to2.5years.However,inordertofullyachievewhattheprojectsetouttodoatimeframeofdoublethelengthwouldhavebeenmorerealistic.”

93 TransformingSurgecapacityfinalEvaluationReport,p27.“Overalltheprojecttimeframewaslargelyconsideredtooshorttoachieveeverythingitplannedtoachieve.”


ALL DEPP COUNTRIES

N=51

UK-BASED PROJECT

LEADERSHIP

N=17

IN-COUNTRY PROJECT

LEADERSHIP

N=28

PROGRAMME LEADERSHIP

N=6

Objectives are clear ✔ ✔ ✔ ✔

Objectives feasible within the timeframe ✖ ✖ ✖ ✖

Collaboration adequately included in project design

✖ ✖ ✖ ✖

Resources (human and financial) were adequate ✖/✔ ✔ ✖/✔ ✖

Table3.3:DesignvalidityoftheDEPP

✔Mostrespondentsagreewithstatement ✖Mostrespondentsdisagreewithstatement✖/✔Responsesweremixed

Whenmappinghowtheobjectivesandactivitiesofeachproject feed intoeachof thefiveresultsareas,as illustratedonpage16of theDEPPlearningReport201594, it isclear thatwhilesomeprojectsonlytargetedafewofthefiveresultsareas,asagroupofprojects,allfiveresultsareaswereaddressedineverycountryexceptJordanandMozambique.Thisis,inpart,duetotheadditionofthefourearlywarningprojectswhich,asdescribedearlier,weredevelopedthroughaseconddesignprocessthatwasinitiatedwhenstakeholdersidentifiedalargegapinprojectstargetingresultareatwo(improvepreparednesssystemsforearlyactionwithcommunitiesatriskofdisasters)afterthefirsttenprojectshadbeenapproved.Whilemostrespondentsconsideredtheobjectivestobeclear,themajorityfelttheywerenotlikelytobeachievedwithintheestablishedtimeframe.

Were resources (both human and financial) adequate?

DEPP stakeholder and respondent perspectives on this issuewere assessed across all tencountries,andfiveVfMprojectcasestudieswerealsoconducted.Overall,respondentsfeltthat resourcing for theDEPPat theportfolio levelwas sufficient (seeTable3.3).However,ingeneral,most interviewees felt thatprojectbudgetswere stretched, and resourceswereinsufficient to undertake activities to their full potential. There was a need for in-countryresourcesratherthanforUKstaff.non-projectattributablecosts(rangingfrom7%to12%ofthetotalprogrammebudget)wereconsideredinsufficientbyalmostallinterviewees.Therewerealsodiscussionsabout insufficientallocationof funds forcollaborativeactivities.MonitoringandEvaluationbudgetallocationsvaried,asdidthequalityof individualM&Eplans95.TherewasaneedtoincreaseM&Eresourcesforcertainprojectsandrecommendationsweremadeintheformativephaseevaluationreport.Inmanycases,respondentsreportedthatfinancialchallengeswerealsolinkedtoimplementationdelays.Asdiscussedinmoredepthlaterinthereport,theimplementationdelayswereprimarilyduetounderestimatingthetimerequiredforcollaborationandsettinguptheconsortiaandcontracts,aswellastheadditionaltimerequiredtocontextualiseprojectswithinthecountries.

94 DEPP2015learningReport.Accessedon13January2017athttps://startnetwork.org/resource/depp-2015-learning-report.

95 forexample,thefinancialEnablersfinalEvaluationreportsan:“overallunder-developmentofprojectMEl.nooverarchingM&Eframeworkwasdeveloped.Anearlylogframewasneverembraced”,andtheTalentDevelopmentfinalEvaluationreported(p77-81)thatitsM&Eframeworksreliedtoomuchonselfassessments,lackedqualitativeinformationandthattheproject:“lackedasinglecoordinatedM&Esystem.”


Do the projects align as a portfolio?

The individualprojectsweredesignedwithoutdirect communicationbetweenprojects andhencewerenotdesigned to complement eachother. for this reason, theDEPP’sdesign islimitedintermsofitsabilitytofunctionasaportfolio.Onestakeholderstated:

“WewereunderalotofpressurefromDfIDtobasicallyjigsawtogethertheprojectstoeachother…They[DEPPprojects]weren’tdesignedtositalongsideeachotherandcomplementeachother.So,thereisabitofadesignflawinthatsenseaswewereundera lotofpressure tomake theprojects interoperable,more interlinkedwhenactuallytheyweredesignedtobeindividualprojects.Theyjusthappenedtositinthesamepot.“(Programmeleadership)

Portfolio-levelsystemsshouldhavebeendevelopedearlyon,potentiallythroughaseparateinceptionphase,toensurethattheDEPPcouldfunctionwellasaprogramme.forexample,thelackofanoverarchingDEPPlogicalframeworkattheprogrammelevelwasasubstantialweakness.Hadsuchalogicalframeworkbeendevelopedearlyontogetherwithothersystemsand processes, it might have led tomore cohesion and alignment across the projects andperhapsgreatervalidityofDEPPasawhole.

Was collaboration at the programme level addressed during the design phase?

DEPP’s emphasis on collaboration emerged relatively late during the design process andexpectations and objectives related to collaboration, especially inter-project collaboration,wereunclearformanystakeholders.Therewasanassumptionthatcollaborationwouldoccurorganically,however,therewasanunderestimationofthegeneralknowledgeandskillsamongmembersonhowtocollaborateeffectively.further,collaborationwasinterpreteddifferentlybydifferentagencies,andtherewas insufficienttimeandresourcestoeffectivelyengage incollaborativeprocessesduringthedesignphase.InthecaseoftheprojectsdevelopedduringSTART network’s first design phase, some collaborations were mandated, and this led totensionsandlowerqualitycollaborationswheresubstantialtimewasspentintryingtoreconciledifferingopinions,underminingtheefficiencyofdelivery.

QUESTION 1.EHAVE PRIORITISED TARGET GROUPS (PEOPLE WITH DISABILITIES, OLDER PEOPLE) AND GENDER ASPECTS BEEN TAKEN INTO CONSIDERATION IN THE PROGRAMME DESIGN? All projects considered gender in their design, but the level of consideration varied. Some projects considered other prioritised groups such as the elderly and people with disabilities but this was not an explicit requirement for project selection. The DEPP Programme board criteria for project selection required inclusion of a genderstatement,butdidnotprovidefurtherguidelineswithrespecttogender inclusion.Assuch,all DEPP projects considered gender in their proposals and outlined their planned genderapproachintheconceptnotessubmittedtotheSTARTandcDAcnetworks.However,sincetherequirementwasonlytoincludeagenderstatement,thelevelofdetailprovidedandtheproposed level of inclusion varied substantially across projects96. Some project proposalsdescribed disaggregating data by gender, others proposed ensuring gender equality inparticipation,andothersdescribedincorporatingastrongfocusongenderwithinthecontentoftrainingmaterialsandcurricula.Othersweremorevagueandindicatedthatgenderdimensionswouldbeconsideredwithoutanyspecificsabouthow,orstatedthattheapproachwouldbefurtherdevelopedintheprojectinceptionphase,thoughthisdidnotoccurformostprojects.

96 forexample,theTransformingSurgecapacityfinalEvaluationReport,p38states:“Theprojectdesigndidnothaveastrongfocusonwomen’srights.”TheUrbanEarlyWarningEarlyActionfinalEvaluationReport,p16reports:“thoughprojectwasdesignedtoaddressthemostvul-nerable,itdidnotexplicitlyoutlinecross-cuttingissueslikegendermainstreaming,VAWGinitsdesign.”


Prioritised targetgroupswerealsonotexplicitly includedasa requirement in thedesignoftheprojectsandwerenottargetedinmostprojects’activities.However,whilenotaprojectrequirement, prioritised groups, including the elderly and people with disabilities, wereconsideredandincludedinmanyprojectsacrossDEPP.forexample,theAgeandDisabilitycapacityProgramme’sinclusioneffortsaredescribedlaterinthisreporttobeleadingthefieldandextremelyinfluentialamongstotherDEPPprojects.

Design Considerations with Respect to Emergency Preparedness and Response

Responsetoemergencieswasnotadequatelyconsideredduringthedesignphase–theprogrammewasdesignedtoimproveemergencypreparednessbutmechanismstoenableorfacilitatecontribution(especiallyofl/nnGOs)tohumanitarianresponseeffortswerenotexplicitlyintegratedintotheprogramme.forexample,accesstoresources,especiallyresponsefundswerenotsystematicallybuiltintotheprogramme.ThismeantthatinmanycaseswhileDEPPbeneficiaryorganisationsfelttheyhadthecapacitytorespond,theywereunabletosecureadditionalresponsefunds.Thelackofconsiderationtothelinkbetweenpreparednessandresponsecouldhavepotentiallyledtoreducedeffectsoftheprogramme.Despitethis,DEPPcontributedtoatleast42humanitarianresponsesin11countries(Seechapter4andAnnex10forfurtherdetailsonDEPPcontributiontohumanitarianresponseefforts).

93 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS


RELEVANCE AND EFFECTIVENESS OF INTERVENTIONS

cHAPTER4


Evaluation Question #2: In what ways have DEPP capacity building programmes strenghtened preparedness and response capacity amongst participants?Sub-questions:a) Whatdeliverymechanismsareworkingeffectivelyandwhy?b) To what extent is DEPP contributing to greater preparedness and response among local

organisationsandcommunities?a. HaslocalcapacitytorespondtodisasterschangedsincethestartofDEPP?Ifyes,howhas

itchanged?Ifnot,whynot?b. Has DEPP led to improved knowledge and understanding of best practices relating to

disasterandemergencypreparednessandresponse?Ifyes,inwhatways?Data sources: • Minimum set IDIs with programme and project stakeholders (Qualitative)• KAP and organisational surveys (Quantitative)• Organisational checklist(Quantitative)• Community and household survey(Quantitative)• Intensive set IDIs(Qualitative)• Document review (Qualitative)Key Findings:• Implementation delays were universalandduetothreemainfactors:

o limitedcontextualisationandinvolvementofin-countryteamsindesignphaseo Underestimationoftimerequiredforcollaborationo Administrativeandcontractualbottlenecks

• Delays led to increased costs and budget reallocations, and reduced period for activities and potential impact

• Overall,33,388 individuals were exposed to DEPP capacity building exceeding the ultimateprogrammetargetoftraining4,200individualsby795%.However,thegendertargetwasmissedby5%(45%oftraineesarewomen).

• Both DEPP and non-DEPP participants from comparison organisations were exposed to more didactic individual-level capacity building activitiesrelatedtopreparednessandresponsethantootheractiveskillandcompetenciesbuildingexercisessuchdrillsandsimulationsacrossallfourintensivesetcountries.ExposuretootherDEPPactivitieswashigherinDEPPthannon-DEPPbeneficiaries

• DEPP has contributed to the four output areas (corresponding to the causal chain); it hascontributedtoincreasedcapacitybuildingefforts,collaboration,earlywarningsystemdevelopmentandlearninginbeneficiaryorganisations,but success has been variable

• Themosteffectiveindividual-levelcapacitybuildingapproachreportedbybeneficiariesinvolvedin-persontrainingcombinedwithastrategytoreinforceknowledge(simulations,mentoring,coaching)

• Flexible funding approaches,thoughinitiallyconsideredhighrisk,have high potential for effectiveness• Outcome-level changes were variable.Overall,therehavebeenno significant improvements in

knowledge oncorehumanitariancompetenciesofhumanitarianstaff,onself-reportedknowledgeondisasterpreparedness,responsetodisastersandemergencesorageordisability-relatedissuesindisasters

• Therewasnochange inperceived levelof individualpreparedness,but thereweresignificantimprovements inpreparednessofDEPPorganisationsacrossall intensivesetcountries,and inparticularforl/nnGOs.AccesstoearlywarningsysteminformationwassignificantlyhigherinDEPPcommunitiesthaninnon-DEPPcommunities,theperceivedlevelofcommunitypreparednesssignificantlydecreasedinbothDEPPandnon-DEPPcommunities.

• Qualitative data suggests improvements in preparedness thatmay be difficult to detectwithquantitativedata

• DEPP contributed to at least 42 humanitarian responses in 11 countries;16oftheseresponsesoccurred in the Philippines (and 14 of these 16 responseswere driven by one project – thefinancialEnablersproject)

• QualitativedataindicatecontributionofDEPP beneficiaries towards improved responseinthePhilippines,KenyaandEthiopia,butmanyotherDEPPbeneficiarieswere unable to respond due to lack of response funds

• Individualprojectevaluationssuggestsomepositiveoutcomes,butthishasnottranslatedtoglobalprogrammeeffectslikelyduetoshortprogrammeperiodandlackofprojectalignment


QUESTION 2.A: WHAT DELIVERY MECHANISMS ARE WORKING EFFECTIVELY ANDWHY? 1. PROJECT AND PROGRAMME IMPLEMENTATION

A. What was Implemented?

A diverse range of activities of varying quality targeting all five results areas was implemented.

Toexplore interventiondelivery,keyactivitiesthatwereconductedbytheDEPPprojectshavebeengroupedtogetherbyeachofthefiveDEPPresultsareas(Table4.1).

RESULT AREA EXAMPLES OF KEY ACTIVITIES IMPLEMENTED

1 – Improve knowledge and understanding of individuals by sharing

best practice of humani-tarian preparedness and

response

• Targetedcapacitystrengtheningplans• Trainingschemesforalllevelsofhumanitarianaidworkersthrough

jobplacementsandextendedcourses• Sharingofgoodpracticesandlearningonsurgemechanismsacross

humanitarianactorsandDEPPprojects• Trainingsandworkshopstoincreasemulti-sectorknowledgeand

barrierstoageanddisabilityinclusioninpreparingandrespondingtodisasters

• betweenandwithinsectorconsultationstoimproveknowledgeandbestpractices

• Trainingstoimprovethecapacityofhealthcareworkerstodetectandrespondtoemergencies

• communicationandtrainingstargetedatnon-specialistandnon-hu-manitarianstoinformabroaderunderstandingofhumanitariansys-tems

2 – Improve prepared-ness systems for early

action with communities at risk for disasters

• Toolkitdevelopedwithguidelinesforsettingupanurbanearlywarn-ingsysteminothersettings

• Participatoryapproachesarebeingusedtodevelopnewprepared-nessandresponsesystemsattheorganisationallevelutilisingfeed-backfromusers

• Improvedearlywarningandpreparednesseffortstargetedathealthfacilitiestoensureadequateresponseduringanoutbreak

• Pursuitofcommunity-ledandownedpreparednessandresponseactivitiesthroughidentificationofcommunity-specificrisksandvul-nerabilities

• Useofbottom-upapproachtoensuresuccessfuladoptionofearlywarningsystems


3 – Increased number of coalitions, partnerships

and networks which, working together, are

able to address human-itarian needs in a wide range of emergency

situations.

• Initiationofdisasterresponseandmanagementtrainingsatthegov-ernmentlevelincollaborationwithorganisations

• Sharingofrostersandevaluationofsatisfactionforsurgepracticesatnational,regionalandinternationallevels

• EncouragementofthematicprogrammingoutsideofDEPPthroughnetworksformedatconferencesandothernon-DEPPevents

• cross-projectandcross-agencycollaborationtoimproveanden-hanceprojectprototypes,goalsandobjectivesformaximumsuccess

• Transferofprojectownershiptothelocallevelacrossdifferentsec-torsusingaflexiblefundingmechanism

• creationoflocalconsortiatoimproveownership,sustainability,andcapacityatthenationallevelthroughlocallydistributedgrants

4 – Improved institu-tional arrangements and policy environments so

that national systems for humanitarian response are better supported

and more suitable

• Supportinglocalandunderrepresentedactorstobebetterrepresent-edandheardintheirrelevantplatformsandnetworks

• Organisationalcreationandadaptationoftoolsandpoliciesforageanddisabilityinclusion

• Utilisationofaparticipatoryapproachtotransferownershipofhealthcentrepreparednessandresponseplanstogovernment

• Directadvocacyatthelocalandnationallevelforadoptionofanurbanearlywarningmechanismatthegovernmentlevel

5 – Strengthened ev-idence base for what

works to help build hu-manitarian capacity at

scale

• Assessmentofcapacityofindividualsparticipatinginhumanitarianresponserostersandtrainingsatbaselineandendline

• DEPP/non-DEPPworkshops,conferencesandlearningevents• Evidencesharinganddissemination• Individualassessmentsandtrainingscateredtowardgapsincapacity• Utilisationofcasestudiesandcommunitymember’sfeedbackand

perspectivesforinformingprogrammaticpreparednessandresponserecommendations

• creationofaDEPPlearningplatform(https://disasterpreparedness.ngo/)

Table4.1:KeyDEPPactivitiesimplementedbyDEPPresultarea

basedonthedocumentreview,itisevidentthatarangeofdiverseactivitiestargetingeachofthefiveresultsareaswasimplementedacrosstheprogramme.Examplesoftheseactivitiesincludesin-persontrainings,developmentoftrainingpackagesandguidelines,developmentofearlywarningsystems, collaborationactivities suchas learningeventsandconferences,developmentof casestudiesandevidencesharinganddissemination.feweractivitieswereimplementedaroundresultfive.Thequalityoftheseactivitieswasvariable–somewerenotablyhighinquality(forexampletheyutilisedquantitativemethodswitha largesample size),but inothercasesquality is lowerand/ormaybedifficultto independentlyverifydueto inadequatemonitoringandevaluationordocumentation97.

Ofthe336respondentsacrosstencountriesthatweresurveyedaspartofminimumsetorintensive set evaluation activities, 70% reported that their organisations had implementedcapacitybuildingactivities,37.8%reported thatearlywarning systemshadbeendevelopedand32.2% reported implementationofdrills and simulations (Seefigure4.1below).fewerrespondents reporteddrillsandsimulations inEthiopia, andparticipants in resultsvalidationworkshops attributed this to the disaster and emergency profile in Ethiopiawhere droughtis one of themost common emergencies. close to 68% reported training of national staffand44.4%reportedcommunitypreparednessactivities.OnlyaboutaquarterofrespondentsreportedthatevaluationorresearchactivitieswereundertakenaspartoftheirDEPPproject.

97 forexample,thefinancialEnablersfinalEvaluationReport,p19states:“Verylittleisknownaboutthequalityortheusefulnessofthetrainingsduetoalackofmeasurementbyconsortiaortheproject.”


figure4.1:KeyDEPPactivitiesimplementedbetween2015and2018byorganisationssurveyedduringintensiveandminimumset(all10DEPPcountries)evaluationactivities

B. Whose Capacity was Built? (Who Benefitted from DEPP Activities?)

Both DEPP and non-DEPP comparison participants in the four intensive countries were ex-posed to individual-level capacity building related to emergency preparedness and response, but both groups were less exposed to drills and simulations. Exposure to other types of DEPP activities was higher in DEPP participants.

Intermsofexposuretoindividual-levelcapacitybuildingactivities,89%ofhumanitarianstaffatDEPPorganisationssurveyedduringtheKAPsurveyhadparticipatedinacapacitybuildingactivityinthelast12monthscomparedto76%forhumanitarianstaffwithinthecomparisongrouporganisations.further,46%ofDEPPrespondentshadparticipatedinadisasterdrillorsimulationwithin the sametimeperiod compared to approximately30% in the comparisongroup(figures4.2and4.3).ThesefiguresdemonstratethatcomparisongroupparticipantsintheKAPsurveywereexposedtoafairlyhighlevelofcapacitybuildingonemergencyprepared-nessoverthesametimeframe,thoughexposurewasstillhigherby13percentagepointsintheDEPPgroup.Inaddition,exposuretosimulationswashigherby16percentagepointsintheDEPPgroup.Thissuggeststhatiftheindividual-levelcapacitybuildingeffortsinbothDEPPandcomparisongroupsweresimilarlyeffective,thentheremaybeimprovementsinsomeoutcomessuchasknowledgechangeinbothgroups.However,thiswouldnotapplytootheroutcomesre-latedtoactivitiessuchasearlywarningsystemdevelopment,ororganisationalcapacitychangewhereexposurewassignificantlyhigherinDEPPgroups98.Inthosecases,significantoutcomechangesinthecomparisongrouparenotexpected.Overall,theexposuredatadodemonstratehighpenetrationoftheDEPPwithinsampledorganisations,whichisapositivefinding.

98 forexample,figure4.9showsthatDEPPcommunitiesinMyanmarweresignificantlymoreexposedtoearlywarningsystemsthannon-DEPPcomparisoncommunities.

0

20

40

60

80

100

RESEARCH

EVALUATiONS

DRiLLS/SiMULATiONS

PREPAREDNESS ACTiViTiES

PREPAREDNESS TRAiNiNG

CAPACiTY DEVELOPMENT PROGRAMS DEVELOPED AND iMPLEMENTEDEARLY WARNiNG SYSTEMS DEVELOPED

ALL DEPP

KENYA

THE PHiLiPPiNES

ETHiOPiA

MYANMAR

DEPP ACTIVITIES IMPLEMENTEDRE

SPO

ND

ENTS

(%)

70.0

71.4

70.0

88.9

60.0

37.8

57.1

40.0

22.2

50.0

67.8 71

.4 75.0

72.2

70.0

44.4

42.9 45

.0 50.0

45.0

32.2

57.1

10.0

44.4

30.0

24.4

14.3

25.0

33.3

30.0

24.4

13.3

30.0 33

.310

.0


figure4.2:DEPPandnon-DEPPcomparisonparticipationinindividual-levelcapacitybuildingactivitiesoverpast12months

figure4.3:DEPPandnon-DEPPcomparisonparticipation in individual-level drills and/or simulationsoverpast12months

C. Timeliness

Project-level implementation was variable in terms of timeliness and delays led to increased costs, and reduced potential impact.

Universally, there were significant delays in implementation due primarily to three factors.Deficienciesinthedesignprocess, includinglimitedcontextualisationofprojectsandlowin-volvement of in-country teams, required significant time to address during project start-upand contributed to implementationdelays. for example, the lack of in-country involvementduringthedesignprocessmeantthatonceprojectswereapprovedandcontracted,timehadtobetakenfor in-countryteamstounderstandtheprojectactivities,contextualisethemforthespecificsettingandtargetaudienceandensurebuy-inofkeylocalstakeholders99,100,101 .The

99 ShiftingThePowerfinalEvaluationReport,p30.100 DEPPlearningReport,2016:https://disasterpreparedness.ngo/wp-content/uploads/2017/05 DEPP-learning-Report-fInAl-1.pdf.101 TransformingSurgecapacityfinalEvaluationReport,p31.

70%

30%54%

46%

HAVE NOT PARTiCiPATED

HAVE PARTiCiPATED

N = 329

DEPP PARTICIPATION IN DRILLS/SIMULATIONS

N = 119

NON-DEPP PARTICIPATION IN DRILLS/SIMULATIONS

11%

89%

24%

76%

HAVE NOT PARTiCiPATED

HAVE PARTiCiPATED

N = 329

DEPP PARTICIPATION IN CAPACITY BUILDING ACTIVITIES

N = 119

NON-DEPP PARTICIPATION IN CAPACITY BUILDING ACTIVITIES


timerequiredfortheseprocesseswasalsounderestimatedandfurthercontributedtodelays

inmanyprojects.Thiswascompoundedbythefactthatsomeprojectshadnot includedaninceptionphase102,whichwouldhavereservedtimespecificallyfortheseprocesses.Thosethathadincludedaninceptionphasestatedthattheamountoftimewasinsufficient.

Thesecondfactorcontributingtodelayswasunderestimationofthetimeandresourcesre-quiredforcollaborationincludingsettingupconsortia103,104,105,106. finally,lengthyadministrativeandcontractualprocessescontributedsignificantlyto implementationdelays107,108,109.Duetothese delays, the 14 projects across theDEPP portfolio had shorter project timelines thaninitiallyplannedreducing thepotential formeasurable impactwithin theprogrammeperiod.The delays also contributed to increased costs and reduced efficiencies.As a result of thedelays,projectsalsoconfrontedchallengesincompletingactivitiesontime.Sixof14projectshaverequestedandobtainedno-costextensionstolengthenprojectperiods(Seefigure4.4).Accordingto intervieweesanddocumentreview, implementationdidbecomemoreefficientovertimeasprojectsintensifiedefforts.

YEAR 2014 2015 2017 2018

MONTH

January

february

March

April

May

JuneJuly

August

September

October

november

December

January

february

March

April

May

JuneJuly

August

September

October

november

December

January

february

March

April

May

JuneJuly

August

September

October

november

December

January

february

March

April

PROJECT TIMELINESADcAPAlertcDAc-nFEEWEAlearningProjectLPRRPiPPHEPGambellaSTPSEPSTDTScUEWEA

Sub-agreementsigned/ProjectstartdateProjectimplementationstartdate(iflaterthanprojectstartdate)Projectenddateno-costextension

figure4.4:DEPPprojecttimelines

D. Perspectives on Implementation

Despite significant widespread project delays, stakeholders consistently reported effective implementation of the DEPP (particularly at the project level). In fact, 82.8% of stakeholders interviewed (N=232) across all 10 DEPP countries reported that delivery of the DEPP has either been effective or very effective. However, despite these positive perspectives,

102 forexample,figure4.9showsthatDEPPcommunitiesinMyanmarweresignificantlymoreexposedtoearlywarningsystemsthannon-DEPPcomparisoncommunities.

103 financialEnablersfinalEvaluationReport,p18.104 StrengtheningEmergencyPreparednessSystemsinMyanmar,finalEvaluationReport,p19.105 TransformingSurgecapacityfinalEvaluationreport,p31.106 UrbanEarlyWarningEarlyActionfinalEvaluationreport,p14.107 financialEnablersfinalEvaluationReport,p2:“fEPwasthelastoftheDEPPprojectstobesignedoffby

DfID,roughlyayearafterdiscussionsbeganaboutit,eatingintothetimeframeforimplementation.”108 UrbanEarlyWarningEarlyActionfinalEvaluationReport,p6:“Ittookalmost6monthstoconcludethe

project.implementationplanandcloseto9monthstofinalisesub-grantagreementsandoverarchingconsortiumagreement.This‘ate’intothetimethatwastobeusedtoimplementtheprojectactivities.”

109 cDAc-nfinalEvaluationreport,p16.


evidence demonstrates that programme implementation was fraught with significant challenges.

Manyofthechallengesrelatedtoprogrammeimplementationwerepreviouslyoutlinedintheformativephase report.110 These includea lackof systemsandprocessesat theprogrammelevel,theabsenceofafinalisedprogramme-levellogicalframework(untilAugust2017)againstwhichprogresscouldbetrackedaswellasinadequatehumanresourcesattheprogrammeleveltoeffectivelymanage theprogramme.Thenumberandtypeofprojectsbeing implementedwithin each country also varied. In countries with fewer projects implemented, such as inMozambique,Jordan,andMyanmar,therewerefeweropportunitiesforcollectiveaction,loweroverall investmentand therefore, a lower likelihoodofembedding long-termchange.Again,theseproblemscanbe tracedback to theweaknesses in thedesign including the fact thatintervention intensity at the country levelwas not adequately and strategically considered.Impact could have beenmaximised by ensuring a suite of complementary projects in eachcountry that fully addressedall levelsof theprogramme-level theoryof change.Asnonewprojectswere funded after the evaluation formative phase, andwith the exception of thecDAc-n project which shifted country locations from South Sudan to the Philippines, nochangesininterventionintensityattheprogrammeleveloccurred.Unfortunately,thedesignflawscouldnotbeundoneandcontinuedtohaverepercussionsthroughoutthecourseoftheprogramme’sthree-yearduration.

2. ACHIEVEMENT OF THE FOUR OUTPUT AREAS

DEPP contributed to increased capacity building efforts, collaboration, early warning system development and learning in beneficiary organisations, but achievements were variable.

Toassessachievementsof theDEPP,asystematicassessmentofdirectprogrammeoutputsdrawingonprimarydatacollectedviathein-depthinterviews,quantitativedataandsecondarydatafromthedocumentreview,andreferringbacktothesimplifiedcausalchaindevelopedbytheevaluationteam(seeboxbelow)wasconducted.Thefourkeyoutputareasassessedinclude:1)collaboration,2)capacitybuilding,3)generatingandsharinglearningand4)earlywarningsystemdevelopment.notethatthesecorrespondwiththeevaluationsteam’ssimplifiedcausalchain(andthustheprogramme-leveltheoryofchain)andaredistinctfromthefiveDEPPresultsareas.Alatersectionwillpresentoutcome-levelfindings(improvedemergencypreparednessand improvedresponse)oftheDEPP.notethatDEPPimpacts (reducedmortality,morbidityand economic impacts)were considered too far downstream from theDEPP activities andassessmentoftheseindictorswasnotwithinthescopeoftheevaluation.

110 forexampleseep67-68oftheformativePhaseEvaluationReport.


DEPP THEORY OF CHANGEDEPP Impact (Not being assessed by evaluation): • Reduced mortality, morbidity and economic impacts from disasters and emergencies

DEPP Outcomes:• Improved preparedness of individuals, organisations, communities• Improved response by individuals, organisations, communities

DEPP Outputs:• Improved collaboration and strengthened networks related to preparedness and response• Improved capacity to respond to disasters and emergencies (individual, organisation, com-

munity)• Improved early warning systems • Improved learning and strengthened evidence base on preparedness and response

Source: DEPP programme causal chain (see Figure 1.1)

1. Evaluation Output Area 1: Collaboration

The DEPP has contributed to increased collaborations and strengthened networks in several settings, but numerous issues impeded potential effectiveness.

TheactivitiesconductedbyDEPPprojectswereexpectedto, if thetheoryofchangeholds,leadtoenhancedcollaboration,andultimatelystrengthenednetworks,coalitionsandpartner-ships.ThecollaborativeapproachwasessentiallythecornerstoneoftheDEPP;eachofthe14projectswasdeliveredviaconsortiaandintotal45primaryorganisationsmadeuptheDEPP.collaboration within project consortia, but also between DEPP projects and with externalstakeholderswasanexpectedoutputof theDEPP, thoughasdescribedpreviouslythetimerequiredforthislevelofextensivecollaborationwasnotwellbuiltintothedesign,workplansorbudgets.Interviewdatarevealthatthecollaborativeapproach,includingcollaborationwithinprojectconsortiaandamongorganisations,wasideologicallyembracedbyallprojectsandwasperceivedtobeaneffectiveandappropriateapproachtocapacitybuildingaswellasresponse.However,asdescribedbelowtherewerevaryingdegreesofimplementationsuccess.

While the emphasiswas on localisation, and ensuring amore equal balance and role of l/nnGOpartners,thisobjectivewasnotfullymet.forexample,inadditiontothe45primaryDEPPorganisations,therewere222uniqueorganisationalpartners,butmostofthesewereInGOpartners.InternationalnGOsrepresentthelargestsharewithintheDEPPandthatwhileeffortstobemoreinclusivetowardsl/nnGOshavecertainlyincreasedinvolvement,theyhavenottranslatedtoequalopportunitiestocollaborate,engageandparticipateindecision-making.Eveninthoseprojectswithhighersharesofl/nnGOpartnerships,primarilyInGOsmakeuptheprojectconsortiaandstillplaythe“gatekeeper”rolesuggestingthatthepowerdynamicsremaininfavourofInGOs.ThereareafewnotableexceptionstothissuchasthefinancialEnablersandtheShiftingthePowerprojectswherel/nnGOswereabletotakeamoreequalrole.bothoftheseprojectsaimedtodirectlyalterpowerimbalances–financialEnablersthroughahands-offapproachwhichpermittedlocalorganisationstosteertheircapacitybuildingactivities,andShiftingthePowerbystrengtheningthecapacityoflocalorganisationsusingaprocessofself-assessments.

Collaboration within DEPP Project Consortia

Evaluationdatademonstratethestrongperceivedvalueofcollaborationandworkingthroughconsortia,butalsohighlightthechallengesofthisapproach.Theseincludethesubstantialtime,effortandresourcestosetupconsortiaandlackofclearexpectations,objectivesandprotocolsrelatedtocollaboration.forexample,theAgeandDisabilitycapacityProgrammestakeholders


expressedfrustrationaboutthelackofconsortiadecision-makingprotocolsandlackofclarityonwhohadfinalauthority111.Thesechallengeswereamplifiedincaseswherecollaborationsweremandatedorforcedratherthanorganic,andgloballyaroseduetolackofadequateplanningduringthedesignphase.Thesechallenges,inturn,areoneofthethreemainfactorsthathavecontributedtoimplementationdelays112.

Additional challenges of working in consortia included varying governance structures,administrativeprocessesandindividualandorganisationalaims113.Staffturnover,andthelackof incentivestoparticipate incollaborationsonavolunteerbasisplacedadditionalstressonprojectimplementation.Inaddition,thelargenumberofformalorganisationalpartnersthatmakeupeachconsortiabecameahindranceintermsofspeedofdecision-makingandeffectivenessofprojectimplementation114.Projectdeliverywouldhavebeenmorestreamlinedwithfewer,morestrategicconsortiamembersandalsopotentiallydifferentgovernancearrangements.Inaddition,insomeprojects,theconsortiapartnersattheUKlevelweredifferentorfunctionedindependentlyfromin-countryconsortiapartners.ThissetupwasreportedtobeproblematicespeciallywhentherewaslimitedcommunicationbetweenUKandin-countryconsortiapartnersresultinginconfusion,lackofengagementandimplementationdelays.IntheAlERTproject,theconsortiamemberswereneverformalisedwhichledtoalossofefficiencyandlimitedsupportfromotherpartnersduringtheearlyphasesoftheproject115.

collaborationsthatbuiltonestablishedpartnerships,andaroundcommongoalsandwaysofworkingwerethemostsuccessful.newercollaborationsorconsortia,includingthosethatfeltpressuredtoworktogether(suchasthosewhoweremandatedtoworktogetherduringthefirstSTARTdesignphase)tendedtostrugglethemostandwereleastefficient.

Collaboration across DEPP Projects

cross-projectcollaborationwasnotwellarticulatedasaprogrammeobjectiveandactivityduringthedesignphase116,leadingtodifferentinterpretationsofcollaboration,andunderestimationof the time and resources needed. cross-project collaboration was expected to occurorganicallybutsinceprojectsweredesignedasstandaloneprojectsratherthananintegratedportfoliothistypeofcollaborationrequiredconcertedeffort.GeographicimbalancebetweentheUK and target countries in termsof collaboration intensity across projectswas evidentsincethebeginning,thougheffortstostimulatein-countrycollaborationsthroughin-countryand international learning events, international conferences (Kenya-December 2016117, thePhilippines-July2017118)andbyembeddingcountry-levellearningadvisors(clAs119)intensifiedduringthesecondhalfofDEPPandinsomecasesweresuccessful.

cross-project and cross-country exchangevisits to learndirectly fromotherprojects and indifferentcontextswereoneoutcomeofthelearningconferencesthatwerereportedtobeparticularlyeffective.Othereffortstoengagelocalstakeholderstoparticipateinlocallearningeventsandothertypesofcross-countrycollaborationhavehadvaryinglevelsofsuccess.forexample,intervieweesstatedthatinPakistan,despitesubstantialeffortstoengagestakeholders,participationat learningeventsremained low, leadingtothedecisiontoreallocatefundsforfourremaininglearningeventstoanotheractivity.

Inordertofurtherfacilitatesharingoflearningandevidence,theDEPP’slearningProjectalso

111 ADcAPfinalEvaluationReport,p29.112 Theothertwofactorscontributingtodelaysinimplementationwerelengthyadministrativeandcontrac-

tualprocesses,aswellasretrofittingofpre-designedprojectstothebusinesscasewhichthenrequiredsubstantialtimeattheprojectstart-uptocontextualiseandsecurebuy-infromlocalstakeholders.

113 linkingPreparednessResilienceandResponsefinalEvaluationReport:“differentorganisationalcultures,timecommitment,andtheshortdurationofactual‘implementation’heldthecollaborationbackfromflourishing;assuch,theprojectmissedopportunitiesintermsofseizingnewopportunities,particularlyrelatedtojointfundingordevelopingnewprojectinitiatives.”

114 Thisfindingisalsosupportedbydocumentreview.forexampleseeADcAPfinalEvaluationReport,p5.115 AlERTfinalEvaluationReport,p5.116 Thisfindingisalsosupportedbydocumentreview.forexample,seeShiftingthePowerfinalEvaluation

Report,p24.117 102DEPPandnon-DEPPstakeholdersfrom13countriesattendedtheKenyalearningconference.118 127stakeholdersfrom9countriesattendedthePhilippineslearningconference.119 countrylearningAdvisorswereassignedtothePhilippines,Kenya,DRcandbangladesh.


setupanonlinecollaborationplatform120inMay2017.Theplatformcontainsacompilationofblogposts,reports,videosandothermaterialsgeneratedbytheprojects,aswellasaforumfordiscussions.Theplatformanditsusewillbeassessedinmoredetailunderevaluationoutputarea3:generatingandsharinglearning.

Inmanycasescross-projectcollaborationsdidnotgobeyondsharingofinformationandlearning.ThereislimitedevidenceofprojectsworkingjointlywithotherprojectswithinacountrytowardsthecommongoalsandobjectivesoftheDEPP,andtherewasminimaltonojointadvocacyorcommunicationefforts.Thisrepresentsa lostopportunityandlikelydiminishedthepotentialimpactoftheDEPPatthecountrylevel.

Collaboration with External Partners

The external evaluation assessed linkages of theDEPP to other key external partners.TheDEPPbusinesscaseoutlinesexpectationsfortheDEPPtoworkcloselywiththeHumanitarianleadershipAcademy (HlA), a global learning initiative set up to facilitate partnerships andcollaborativeopportunitiesandtoenableindividualstobetterrespondtocrisesintheirowncountries.currentdatademonstratethattheconnectionbetweenDEPPandHlAwaslimited,resultinginalostopportunity.OnlythreeintervieweesdiscussedHlA,andthesewereprimarilyindividualsattheprogrammelevel.OnerespondentdescribedajointDEPPprojectandHlAoutput,andanotherdescribedtheinvolvementofHlAinthePhilippineslearningconferenceasapartnerwhoprovidedlivestreamingoftheeventsessionsthroughtheironlineplatform.ThisconferencevideoisstillavailableontheHlAfacebookpagewhereithasreceived334views.Intotal,thelearningProjectestimatesthattherewereover10,000viewersonlineduringtheevent.Asidefromthesetwoexamples,theinterconnectednessbetweentheDEPPandHlAhasbeenminimal.SeveralotherintervieweesdescribedanintentiontodeveloplinkageswithHlAbutthatthesewerenotpursuedduetolackoftimeandresources.furthereffortstobuildaconnectionbetweenHlAandDEPPprojectswouldhavereinforcedDEPP’sactivitiesandalsoallowedforcoordinationofcapacitybuildingefforts.

TheSTARTfund,whichprovidesrapidfinancingtounderfundedsmalltomediumscalecrises,alsohashadminimallinkagestotheDEPP,despitebeinganinitiativewithintheSTARTnetwork.Atthetimeofthisreport,therewasatleastoneexamplewhereDEPPprojectpartnersinthePhilippineswereabletoaccessSTARTfundresourcestorespondtoacrisis.AccessofSTARTfundresourceswillbediscussedfurtherintheresponsesectionbelow.

A number of projects initiated collaborations with partners external to DEPP includinggovernments, ministries, and the Un. These types of collaborations and partnershipswereviewedasessentialforprojectsuccessandimportantforsustainability,butwerechallengingtodevelop,formaliseandmaintain.notabledifficultiesincludedlengthybureaucraticprocesseswithingovernments,staffturnoverwithintheUnandchangingprioritiesordirectionswithinthegovernment thatweredifficult tonavigate toensure continuityover a longerperiodoftime.Inaddition,thechallengesofworkingwithgovernmentswhentheyarepoorlyfunctioningorrestrictivewerealsodescribed.Thefewsolutionsproposedtoovercomethesechallengesincludedensuringadequatetimeandresourcestoengagewithgovernments,andbuildingintimewithinprojecttimelinestodeveloptheserelationshipsasearlyaspossible.

Strengthened Networks

Analysisoforganisationalnetworkswithinfourcountries(Myanmar,thePhilippines,KenyaandEthiopia)providessomeevidencethattheDEPPhascontributedtoincreasedcollaborationsandstrongernetworks.Pleaseseechapter5forfurtherdetailsonthefindingsfromtheanalysisofhumanitarianandresponsenetworks.

2. Evaluation Output Area 2: Capacity Building

Overall, DEPP exceeded its numeric targets for capacity building, though gender targets were not achieved. The most effective individual-level capacity building approach involved in-person training combined with a strategy to reinforce knowledge (i.e., simulations, mentoring, or coaching). At the organisational level,

120 https://disasterpreparedness.ngo/.


approaches that permit organisational self-assessments, and capacity building approaches tailored to organisations’ needs were most effective. Supporting the development of administrative policies, procedures and systems especially among L/NNGOs was also reported to lead to improvements. Eleven of the 13121 projects (not including the learning Project) implemented a total of 13differentcapacitybuildinginterventions.Theseinterventionswereimplementedin9oftheDEPPfocuscountries(Pakistan,bangladesh,Kenya,Myanmar,Ethiopia,Jordan,DRc,thePhilippines,SouthSudan),butalsoin5non-DEPPcountriesincludingIndonesia,lebanon,Turkey,Thailand,UK,andcolombia.nocapacitybuildingactivitieswereimplementedinMozambique,aDEPPfocal country. Projects conducted capacity building interventions that targeted individual,organisational,communityandsystemslevels(Seefigure4.5).122

INDIVIDUAL ORGANISATION COMMUNITY SYSTEM

ShortcourseslongcoursesTrainingoftrainersMentoringWebinarscoachingE-learningKnowledgeExchangeSimulationsOrganisationalActionPlansResourceStrengtheningTrainingandTrainingPackagesPreparednessPlansGuidelines,ToolsandResourcescommunitycommitteeesandforumscommunityAwarenessRaisingWorkshopsforumscoordinationadnPreparednessMechanismSoftware

figure4.5:DEPPcapacitybuildingstrategiesatindividual,organisational,communityandsystemlevels

Theevaluationteamcollectedindependentdatafromeachprojectregardingcapacitybuildingindicators and triangulated these against project quarterly reporting and programme logicalframeworkreporting(seeTable4.2below).Ascanbeseeninthetable,allprojectsdescribed

121 WhiletheDEPPincludedintotal14projects,thelearningProjectwasdesignedtocapturelearningonly,andthusisnotincludedinthistotal.

122 DevelopedusinglearningProject’sDEPPcapacitybuildingMap(https://kumu.io/Eamsmith/depp-capac-ity-building-activities-map#depp-capacity-building-activities-map/financial-enablers)andprojectreportsanddata.


implementing short courses (represented by green shaded cells in the first row). The nextmost common strategy was training of trainers with 10 projects employing this approach.cumulatively,acrosstheprogramme,thetriangulateddatashowthat33,388individuals(55%male,45%female)wereexposed toDEPPhumanitariancapacitybuildingactivitiesacrossanumberofkeysubjectareas.notethisisahigherfigurethanreportedbytheprogrammelogicalframeworkdata(n=24,304;seeTable4.2).Thisislikelyduetoinaccuraciesinlogicalframeworkreportingthatweredescribedintheinterimphaseevaluationreport.123ThekeysubjectareasforDEPPcapacitybuildingeffortsinclude:

1. Ageanddisabilityinclusion2. communicationwithcommunities3. Disasterandconflictresilienceandpreparedness4. Earlywarning,earlyaction/response5. Protection6. Organisationalcapacitybuilding7. Surge8. Healthsurveillanceinemergencies9. corehumanitariancompetencies10. Managementandleadership11. Research,datacollectionskills12. Wellbeinginresponse13. logistics,finance,administration

ThenumberofindividualsexposedtoDEPPcapacitybuildingactivitiesexceedstheultimatetargetoftheprogrammeoftraining4,200individualsby795%.However,thegoalthathalfofthetraineesshouldbewomenhasnotbeenmet.Only45%oftraineeswerewomen,whichrepresentsanimprovementfromtheinterimreportwhereonly41%oftraineeswerewomen.Thislikelyrepresentssomeofthecorrectiveactionstakenbyprojectstoincreaseaccessibilityforwomen,butunfortunatelybecauseofthelateimplementationofprogrammelogicalframeworkreporting,correctiveactiononlybeganinthelastquarterof2017,towardstheendofprojecttimelines.Atthatpoint,itwasdifficulttocatchupwithintheremainingtime.Itisalsoimportanttonote thatcollectionofgenderdisaggregateddatawasweakandsomeprojectswerenotabletoprovidefullygenderdisaggregateddataforall trainings. Infutureprogrammeand/orprojectiterationsinclusionofwomenshouldbestrengthened.Itisalsoimportanttonotethatthesecapacitybuildingdatawereself-reported,andmostprojectswerealsounabletoprovidedatadisaggregatedbytrainingapproach.Thetotal(n=33,388),therefore,includesalltypesofcapacitybuildingactivities,someofwhichinvolveminimalexposure(webinars,forums,shorttrainings).ItisthusasuboptimalindicatorasitmaynotaccuratelyreflectthetruenumberofindividualswhosecapacityhasbeenbuiltbytheDEPP.Thismayalsoexplainthelargeovershootofthetrainingtargetnumber.Infact,datapresentedlaterinthischapteronlackofsignificantchangeinknowledgeattheindividuallevelalsosupportsthis.

123 Sharma,V,HémonoR,Jean-francoisJ,ScottJ,Gibbonsn,PhamP.(2017)TheDisastersandEmergen-ciesPreparednessProgrammeEvaluation:InterimPhaseReport,October312017.HarvardHumanitarianInitiative,p9.


AD

CAP

ALE

RT

CDAC

-N

EWEA

FE LPRR

PHEP

G

ambe

lla

PIP

SEPS

STP

TD TSC

UEW

EA

TOTA

L

1.Shortcourses(<1month) 2,050 163 1,251 848 8,302 1,604 779 465 6,585 4,029 843 133 27,052

2.longcourses(>1month) 1,426 1,426

3.TrainingofTrainers 39 556 29 30 654

4.Mentoring 15 180 195

5.coaching -

6.Onlinelearning 2,159 3 2,162

7.Webinars -

8.KnowledgeExchange(Exchanges/placements/buddying/secondments)

-

9.Simulations 1,899 1,899

# of Capacity Building Strategies Employed

5 1 5 3 1 2 3 3 7 6 9 9 4

TOTAL # Trained based on triangulated data from document review

4,209 202 3,706 877 8,302 1,619 779 465 6,585 4,032 1,426 843 343 33,388

TOTAL # Trained as per Programme Logical framework reporting

2,050 192 3,706 1,026 5,142 1,614 782 241 5,133 4,032 1,426 624 386 24,304

Table4.2:TypesofcapacitybuildingstrategiesemployedbyDEPPprojectsandnumberofindividualstrained

* Yellow shaded cells indicate that the project implemented that particular capacity building strategy. (e.g., LPRR implemented short courses and mentoring).

Intermsofindividual-levelcapacitybuilding,universally,amulti-prongedapproachemployingseveralstrategiestogetherwasperceivedtobethemosteffectivebyDEPPstakeholders. Ingeneral,thisinvolvedanin-persontrainingcombinedwithanotherstrategysuchasmentorship,coachingorsimulations.Distanceor remote learningwasnot reportedtobeaseffectiveasin-personapproaches.Asdescribedlaterinthischapter,astherewerenomeasurablechangesinknowledge,itisnotpossibletoquantitativelyassesswhichcapacitybuildingapproachwasmosteffective.Inaddition,anumberofintervieweesdescribedinstitutionalbarrierstoapplyingknowledge or skills gained through individual capacity building and this was supported byfindingsfromthedocumentreview.124

Organisationalcapacitybuilding,especiallyofl/nnGOswasalsoseenascriticalforstrengtheninghumanitarian response and the greatest perceived organisational-level changes appear tobeoccurringwhenseveralstrategiesarecombinedandtailoredtothespecificgapsofeachorganisation.The greatest successes described by project stakeholders have often involvedprojects such as Shifting the Power and financial Enablerswhere beneficiary organisations

124 forexample,TalentDevelopmentfinalEvaluationReport,p45.


are empowered to self-assess gaps in capacity and identify their own strategies and plans.ThefinancialEnablersproject,inparticular,alsotookthestepofre-imaginingthefundingandreportingmechanismsof its localgrantees inorder tomake the fundsmoreaccessible.TheProtectioninPracticeprojectprovidedcashawardsratherthangrantstominimisereportingburdenonl/nnGOsand increaseflexibilityofhow fundswereutilised. Inboth cases, thisapproachseemstohaveworkedwell.TheShiftingthePowerprojectreportedthatstrengtheningtheadministrativepolicies,proceduresandprocesses(suchasfinancesystems,humanresourcesandprocurementprocesses)greatlybenefittedorganisationsandimprovedtheiroveralllevelofprofessionalism.125

3. Evaluation Output Area 3: Generating and Sharing Learning

Generation and sharing of learning was high across the programme but this learning was not based on empirical evidence. In addition, there is no quantitative evidence that the learning translated to behaviour change. Accessibility to learning products especially for local stakeholders was an issue. Thethirdevaluationoutputareafocusesonlearningandisdirectly linkedtoresultareafivewhich aims to strengthen theevidencebase forwhatworks tobuildhumanitarian capacityatscale.TheDEPPwasuniqueinthatoneofitsprojects,thelearningProject,wasdevotedentirelytothegoalofgeneratingandsharinglearningandevidence.Overall,findingsrelatedto the learning Project were mixed and have been documented extensively in previousreports.forexample,certainaspectssuchasthein-countrylearningadvisorswereeffectiveinanumberofcontexts,andeventssuchasthe internationalconferencesstimulatedcross-projectcollaboration.Ontheotherhand,thein-countrylearningadvisorswereineffectiveinothersettings,andtheemphasisingeneralofgeneratinglargequantitiesoflearningwithlessemphasison rigoroushighqualityevidencewasproblematic.However, thelearningProjectincorporatedpreviousrecommendationsmadebytheexternalevaluation,whichinturnledtosomeimprovementsintermsofthequantityofevidencegenerated.

Inaddition,severalpiecesofresearchwerecommissionedbythelearningProjectincludingareturnoninvestment(ROI)study,aswellasacasestudyonlocalisationduringthehumanitarianresponse to theRohingyacrisis inMyanmarandanother focusedonwomen’s leadership.126 TheROIanalysis inparticularmaynothavebeenundertakenattheappropriatetimeduringtheprogrammecycle,and likelywouldhavebeenmoreappropriatepriortothestartoftheprogramme. Given that it was implemented as the DEPPwas coming to a close, it couldpotentiallyhaveusedactualoutcomedatabutinsteadreliedonhypothetical,projectedchangesinoutcomes.ThislimitationoftheROIanalysisisimportant,especiallygiventheROIfindingshaveledtoconfusionamongstakeholderswithrespecttotheevaluationofoverallprogrammeoutcomesversusprojectedorhypotheticaloutcomes.Thisbrings intoquestiontheutilityofconductinganROIanalysispost-programmeversusbeforetheprogrammewasimplemented.WhilethisparticularROIstudywasneverintendedtoproducetheactualrateofreturn,suchanapproachwouldhavebeenavaluableexerciseattheendoftheprogramme.

125 ShiftingthePowerfinalEvaluationReport,p9.126 Thefinalproductsfromthesethreestudieswerenotavailableatthetimeofthedocumentreview.


QuarterlyReports

blog

Report/Review

caseStudy

Video

Keyfindings/Keylearnings/lessonslearnt

Guidelines/MinimumStandards

Article

ProgrammeandProjectlevellogicalframeworks

Podcast

finalProjectEvaluations

finalReportnarratives

Template/How-toGuide/Toolkit

conceptnote

Summary/ExecutiveSummary/ReportSummary

Handbook/fieldGuide/framework

TOR

Presentation/Slides

PolicyPaper

Research

Resourcecatalogue

Methodology

MobileApp

ReflectionJournal

newsletter

businesscase

Anal

ysis

TOTA

L

TOTA

L O

N T

HE

LEA

RNIN

G P

LATF

ORM

ADcAP 14 3 3 1 1 1 1 1 1 1 27(4.3%)

13(3.9%)

ALERT 12 1 1 1 1 1 17(2.7%)

1 (0.3%)

cDAc-n 12 1 2 1 1 1 12 1 1 1 1 34(5.4%)

19(5.7%)

FE 11 2 1 1 1 1 1 1 1 20(3.1%)

5(1.5%)

EWEA 10 2 1 1 1 15(2.4%)

2 (0.6%)

learningProject 10 45 19 1 37 12 10 2 3 1 2 1 1 5 2 2 153

(24.1%)118 (35.3%)

LPRR 13 1 4 12 2 1 1 1 1 5 1 1 43(6.8%)

25(7.5%)

PHEP Gambella 11 2 1 1 1 1 1 1 19

(3.0%)4

(1.2%)

PiP 13 1 1 1 1 6 1 24(3.8%)

8 (2.4%)

SEPS 11 3 1 1 1 1 18 (2.8%)

0(0.0%)

STP 14 5 14 21 6 4 2 1 1 1 1 1 2 73(11.5%)

29(8.7%)

TD 14 11 7 5 4 8 1 1 1 1 1 1 55(8.7%)

38 (11.4%)

TSc 14 3 20 8 6 4 6 10 1 1 1 1 2 1 1 2 81 (12.8%)

42 (12.6%)

UEWEA 9 4 1 1 1 1 1 18 (2.8%)

6 (1.8%)

non-DEPP 3 8 15 1 3 3 2 1 1 1 38(6.0%)

24(7.2%)

TOTAL 168 80 76 68 60 38 21 19 16 14 13 10 9 8 6 6 6 5 2 2 2 1 1 1 1 1 1 635 334

Table4.3:Typesofdocumentsproducedbyprojectsandreviewedaspartofthedocumentreview

Table 4.3 presents the type and number of documents that the evaluation team consultedaspartofthedocumentreviewprocess.Thenumberandavailabilityofdocumentscertainlyincreased sinceprevious reports, andaccessibility todocumentswasenhanced through theonlinelearningPlatform.However,themajorityofavailableoutputsdonotconstituterigorousevidence (Table4.4).Thegeneratedcontent isbasedonstoriesandperspectives related tokey DEPP concepts including collaboration, localisation, and capacity building, which werevaluableforprojectimplementationandcouldalsopotentiallyhelpprovidelessonsforfutureprojectdesign.Ofthe334documentsandreportsonthelearningPlatform,28%didincludeempiricalfindingsbuttheywereofvariablequality.Overall,thedocumentreviewrevealedthatthequalityofevidencegeneratedacrosstheDEPPbythelearningProjectandtheindividualprojectswaslow,asitwasprimarilyanecdotalandlackedscientificrigor.Partofthisstemsfromthelimitedguidanceprovidedtoprojectsduringthedesignandinceptionphasesaboutwhattypeoflearningandevidencearerequired,andpartisduetothelimitedcapacityofprojectstaffandpartnersrelatedtoM&E.TheTransformingSurgecapacityfinalevaluationalsonotedtheemphasisonquantityoverqualityandreportedthat:“fewer,bettertargetedproductsmayhavebeenmoreeffective.”127

127 TransformingSurgecapacityfinalEvaluationReport,p48.


EMPIRICAL EVIDENCE INCLUDED

(METHODOLOGY AND RESULTS)

DEPP PROJECT ACTIVITIES

PRESENTED OR DESCRIBED

LEARNINGS FROM DEPP DESCRIBED

TOTAL DOCUMENTS

ON THE LEARNING PLATFORM

yES NO yES NO yES NO

ADCAP1 2 2 1 2 1

3(33.3%) (66.7%) (66.7%) (33.3%) (66.7%) (33.3%)

ALERT1 0 1 0 1 0

1(100.0%) (0.0%) (100.0%) (0.0%) (100.0%) (0.0%)

CDAC-N3 16 17 2 16 3

19(15.8%) (84.2%) (89.5%) (10.5%) (84.2%) (15.8%)

FE1 4 4 1 4 1

5(20.0%) (80.0%) (80.0%) (20.0%) (80.0%) (20.0%)

EWEA 1 1 2 0 0 2

2(50.0%) (50.0%) (100.0%) (0.0%) (0.0%) (100.0%)

LEARNING PROJECT

20 108 85 43 81 47128

(15.6%) (84.4%) (66.4%) (33.6%) (63.3%) (36.7%)

LPRR15 10 13 12 6 19

25(60.0%) (40.0%) (52.0%) (48.0%) (24.0%) (76.0%)

PIP0 8 3 5 1 7

8(0.0%) (100.0%) (37.5%) (62.5%) (12.5%) (87.5%)

PHEP GAMBELLA

2 2 4 0 3 14

(50.0%) (50.0%) (100.0%) (0.0%) (75.0%) (25.0%)

STP9 20 26 3 24 5

29(31.0%) (69.0%) (89.7%) (10.3%) (82.8%) (17.2%)

SEPS0 0 0 0 0 0

0(0.00%) (0.00%) (0.00%) (0.00%) (0.00%) (0.00%)

TD13 25 35 3 33 5

38(34.2%) (65.8%) (92.1%) (7.9%) (86.8%) (13.2%)

TSC22 20 33 9 23 19

42(52.4%) (47.6%) (78.6%) (21.4%) (54.8%) (45.2%)

UEWEA0 6 4 2 2 4

6(0.0%) (100.0%) (66.7%) (33.3%) (33.3%) (66.7%)

NON-DEPP6 18 1 23 0 24

24(25.0%) (75.0%) (4.2%) (95.8%) (0.0%) (100.0%)

TOTAL94 240 230 104 196 138

334(28.1%) (71.9%) (68.9%) (31.1%) (58.7%) (41.3%)

Table4.4:TypesofevidencesharedonthelearningPlatform

AccessibilityofdocumentsforprojectandprogrammestakeholdersimprovedbutthebulkoflearningproductsproducedareavailableonlyinEnglish.basedonpreviousrecommendationsoftheexternalevaluation,technicalsupportwasprovidedtostrengthenindividualprojectM&Eand all projectswere required to undertake an end of project evaluation.All projects havefulfilledthisrequirement,thoughagainthequalityhasbeenvariable.


Therewere some improvements in inclusion of gender disaggregated data, andVfM data,buttheseremainedbelowexpectations.InanyfutureimplementationoftheDEPPorsimilarprogrammes,guidanceandsupportforVfMassessmentstrategiesshouldbeprovidedearlyandconsistentlythroughoutthedesignandimplementationphases.

4. Evaluation Output Area 4: Early Warning System Development

Six diverse early warning systems were developed in three countries and all six are at least partially operational, though the use and testing of each system was limited.

EarlywarningsystemdevelopmentwasakeytargetareafortheDEPP.fivedifferentprojectsinanumberofdifferentcontextsfocusedonthisparticularobjective.Theseincluded:

1. ALERT2. ImprovedEarlyWarningEarlyActionEthiopia(EWEA)3. PublicHealthPreparednessinGambella(PHEPGambella)4. StrengtheningEmergencyPreparednessSystemsinMyanmar(SEPS)5. UrbanEarlyWarningEarlyAction(UEWEA)

Thesefiveprojectsaimedtodevelopsixdifferentearlywarningsystems(SeeTable4.5below)in order to enable early response towards natural disasters such as flooding and drought,urban emergencies such as food insecurity and household shocks, aswell as public healthemergencies including disease outbreaks.Oneof the projects, PHEPGambella, contributedto two systems.As described in previous evaluation reports, most of these projects facedsignificant implementation challenges and delays. In many cases, these delays were linkedtochallenges inprocurementofequipmentsuchasweatherstations128,delays insettingupagreementswiththegovernment129,outbreakofviolenceintargetcommunities130orindelaysindevelopingthesystem131.Infact,delaysweresevereenoughthattherewasanevidentriskthatnotallsystemswouldbeoperationalbeforetheendoftheprogramme.Theinterimreportrecommendedaccelerationofprojectactivitiestoavoidthispotentialoutcome.

PRO

JECT

TYPE

OF

SYST

EM

COU

NTR

Y AN

D

LOCA

TIO

N

SYST

EM

DEV

ELO

PMEN

T

GO

VERN

MEN

T U

SIN

G

SYST

EMN

GO

S, C

SOS

USI

NG

SY

STEM

SIM

ULA

TIO

NS

CON

DU

CTED

CAPA

CITY

BU

ILD

ING

AC

TIVI

TIES

TO

TRA

IN

ON

SYS

TEM

CON

TIN

GEN

C F

UN

DIN

GSY

STEM

USE

D IN

RE

SPO

NSE

ALERTPreparednesssystem(software)

KenyaPakistan,ThePhilippines,bangladesh,Mozambique

✔ ✔ ✔ ✔Trainings,Workshops

✖ ✖

EWEAMonitoringandearlywarningsystem

Ethiopia ✔ ✖* ✖** ✖

Trainings,Trainingoftrainers,Workshops

✖ ✖

128 ImprovedEarlyWarningEarlyActionEthiopiafinalEvaluationReport,p17.129 StrengtheningEmergencyPreparednessSystemsinMyanmarfinalEvaluationReport,p19.130 Ibid.131 AlERTfinalEvaluationReport,p37.


PHEP Gambella

Monitoring,earlywarningandpreparednessplansandsystems

Ethiopia ✔ ✔ ✖ ✖Trainings,Workshops, coaching

✖*** ✔

SEPS

communitypreparednessplansandearlywarning

Myanmar ✔ ✔ ✔ ✔Trainings,Workshops

✖ ✖

UEWEAMonitoringandearlywarningsystem

Kenya(nairobicity) ✔ ✔ ✔ ✔

Trainings,Workshops,Reports,Toolkits,Workshops

✔ ✔

Table4.5:EarlywarningsystemsdevelopedbytheDEPP

*Governments are expected to adopt the system when implementation is complete**NGOs/Civil Society Organisations (CSOs) are expected to adopt the system when implementation is complete*** Contingency funding in the process of being secured

Atthetimeofthissummativereport,allsixearlywarningsystemshadbeenfullydeveloped,fivewereoperationalandtwohadbeenusedinaresponse.However,becauseoftheimplementationchallenges,manyofthesystemshavebeenoperationalforaveryshortperiodoftimewhichmeansthattheyhavenothadtheopportunitytobefullytestedorusedinameaningfulway132.It alsoprevents assessmentsof the effectivenessof the systems.TheUrbanEarlyWarningEarlyActionsysteminnairobicitywasoperationalthelongest,andthereforehadthelongestperiodoftestinganduse.Themainfactorcontributingtoitsquick(relativetoothersystems)deploymentwasthatthesystemwasconceptualisedtobuildonapreviousfive-yearproject133 whereintheurbanearlywarning indicatorsweredevelopedandtested,andthesurveillancemethodologyandsystemprocesseswerearticulatedandrefined.Giventhatitwasabletobuildonthepreviousworkthathadbeenconducted, thesystemwasnotbuilt fromscratch,andthusthegoalofdevelopingaworkingsystemwithinashortprojectperiodwasachieved. Infact,thissystemwasabletodetectthreeemergenciesandtriggerresponseactions.Theseareexploredinfurtherdetailintheboxeslaterinthischapter.Thiswasnotthecasefortheotherearlywarningsystems.Itwasoptimistictosetupandtestanearlywarningsystemfromscratchwithinatwo-tothree-yearproject,andfutureprojectsofthisnatureshouldsetmorerealistictimeframes.

132 forexample,theImprovedEarlyWarningEarlyActionsysteminEthiopiawasreportedtobeonlypartiallyfunctional:TheAutomaticWeatherStationswere“notinstalledasplannedandstillnotfunctionalduetointernetunavailabilityintheareas”,ImprovedEarlyWarningEarlyActionEthiopiafinalEvaluationReport,p6.Also,theAlERTfinalEvaluation,p4reports:“noorganisationhadfullydeployedtheAlERTplatformatthetimeofthisevaluation.”

133 IndicatorDevelopmentforSurveillanceofUrbanEmergencies(IDSUE).ImprovedEarlyWarningEarlyActionEthiopia,finalEvaluationReport,p1.


QUESTION 2.B, 2.C AND 2.D: TO WHAT EXTENT IS DEPP CONTRIBUTING TO GREATER PREPAREDNESS AND RESPONSE AMONG LOCAL ORGANISATIONS AND COMMUNITIES? HAS LOCAL CAPACITY TO RESPOND TO DISASTERS CHANGED SINCE THE START OF DEPP? IF YES, HOW HAS IT CHANGED? IF NOT, WHY NOT? HAS DEPP LED TO IMPROVED KNOWLEDGE AND UNDERSTANDING OF BEST PRACTICES RELATING TO DISASTER AND EMERGENCY PREPAREDNESS AND RESPONSE? IF YES,IN WHAT WAYS?

1. Knowledge Change

Overall, there have been no significant improvements in knowledge on core humanitarian competencies of humanitarian staff, on self-reported knowledge on disaster preparedness, response to disasters and emergences or age or disability-related issues in disasters.

AkeyareaofassessmenttoevaluatetheextenttowhichcapacityhasbeenbuiltasaresultoftheDEPPtraining, isknowledgechangeamonghumanitarianstaffinseveralkeythematicareasrelatedtodisasterandemergencypreparednessandresponse.figure4.6presentsdataonknowledgechangeacrosssixkeyareasusing thepooleddata fromall four intensivesetcountriesaswellastheminimumsetdata.All10DEPPcountriesarethereforerepresentedinthisanalysis.Therewasnosignificantchangeintheoverallcompositeknowledgescorerelatingto the core humanitarian competencies, or in any of the self-reported indicators related toknowledgeonspecificthemes(disasterpreparedness,humanitarianresponse,orage,disabilityor gender issues in a disaster) within DEPP or non-DEPP comparison humanitarian staff.furthermore,therewasnosignificanteffectoftheDEPPonanyoftheseknowledgeindicators,globallyorwhenrestrictedtoindividualcountriesortolocalorganisation(Seefigure4.7).Inaddition,therewasnosignificanteffectoftheDEPPwhenthesamplewasrestrictedonlytothoserespondentswhoreportedparticipatingincapacitybuildingwithinthelast12months.

DEPP DIFFERENCE

(T2 - T1)

CONTROL DIFFERENCE

(T2 - T1)NET DEPP EFFECT

Core Humanitarian Competencies + 0 . 5% - 0 . 5% +1 . 0 %

Disaster Preparedness +1 . 8 % + 4 . 2 % -2 . 4%

Response to disasters and emergencies + 0 . 6% +3 . 4% -2 . 8 %

Age related issues in a disaster +1 . 0 % +1 . 0 % 0 . 0 %

Disability related issues in a disaster + 0 . 6% + 0 . 5% +1 .1%

Issues related to women in a disaster + 0 . 8 % - 0 . 5% +1 . 3 %

figure4.6:Individualknowledgechangeofhumanitarianstaffonkeythematicareas

* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant

ThesefindingssuggestthatwhilemanyindividualshavebeenexposedtotheDEPPcapacitybuilding, according to both logical framework data and independent data collected by theevaluationteam(figure4.6),significantknowledgechange(bothviaobjectivemeasuressuchasthecorehumanitariancompetenciesindicatorandthroughself-reportedmeasures)inseveralexpectedthematicareashasnotoccurred.


DEPP DIFFERENCE

(T2 - T1)

CONTROL DIFFERENCE


Ethiopia +2 . 2 % -1 .7 % +3 .9 %

Kenya 0 . 0 % 0 . 0 % 0 . 0 %

The Philippines +1 .9 % + 0 .9 % +2 . 8 %

Myanmar -1 . 2 % +1 . 6% -2 . 8 %

Local + 0 .7 % +2 . 8 % -2 .1%

figure4.7:Individual-levelknowledgechangeofhumanitarianstaffoncorehumanitariancompetenciesbycountry

* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.

2. Emergency Preparedness

There were no significant quantitative improvements in perceived individual emergency preparedness levels in the DEPP group, but there were significant improvements in preparedness of DEPP organisations across all intensive set countries, and in particular for L/NNGOs. While access to early warning system information was significantly higher in DEPP communities than in non-DEPP comparison communities, the perceived level of community preparedness significantly decreased in both DEPP and non-DEPP comparison communities in Myanmar.

Emergencypreparednesswasassessedat the individual,organisationalandcommunity levelthrough a series of indicators using perceived (self-reported) measures or more objectiveassessments (See box below). The self-reported or perceived measures assess data fromhumanitarianstaffwhoratedtheirownortheirorganisationslevelofpreparednessaswellashouseholdswho rated their community’s level of preparedness to respond to disasters andemergencies.Atthecommunity level,preparednesswasalsoassessedthroughself-reportedexistenceofanearlywarningsystem.

Asself-reporteddatabasedonperceptionsareknowntobeproblematicandoftenbiased,moreobjectivemeasureswerealso included.Atthetimeofthedevelopmentofthemethodologyfor this evaluation, therewas an absence of standardised, objective indicators and tools toassessorganisationalandcommunityemergencypreparedness.Theevaluationteamdevelopedanorganisationalchecklist spanningsevendomains tomore rigorouslyassessorganisationalpreparedness.Theevaluation teamalsoadapted theGOAl toolkit for assessing communityresilience134 comprising a seriesofquestions across30differentdomains inorder to assesscommunitypreparedness(Seeboxbelow).Thepreparednessindicatorsthereforeincludeamixofperceivedandalsomoreobjectivemeasures.

KEY PREPAREDNESS INDICATOR DATA COLLECTED AT 2 TIMEPOINTSIndividual level:

• Perceived: KAP survey with humanitarian staff. As a responder, how would you rate your level of preparedness to respond to a disaster? Where 1=Very unprepared, 2=Unprepared, 3=Somewhat unprepared, 4=Prepared, 5=Very prepared

134 ToolkitforMeasuringcommunityDisasterResilience.GuidanceManual.Goal,2014.


Organisational level:

• Perceived: KAP survey with humanitarian staff. At what level would you rate your organisa-tion’s preparedness to respond to disasters and emergencies? Where 1=Very unprepared, 2=Unprepared, 3=Somewhat unprepared, 4=Prepared, 5=Very prepared

• Actual: Organisational Checklist conducted with each organisation and assessing seven domains: 1-Emergency preparedness plans, 2-human resources and capacity, 3-logistics and supplies, 4-Legal, documentation and policy, 5-Organisation and system processes, 6-Emer-gency response team and staffing, 7-Communication and media. Aggregate score out of 100 generated for each organisation.

Community level:

• Perceived: Household survey: How prepared do you feel your community is to respond to disasters in the future? Where 1=Not prepared at all, 2=Slightly prepared, 3=Somewhat prepared, 4=Prepared, 5=Very prepared

• Perceived (early warning system): Household survey: Does your area have an early warning system?

• Actual: Community focus group using a preparedness assessment tool conducted in each community, assessing 30 domains including emergency response and recovery, contingency planning, early warning systems, capacities in preparedness and response, access to health-care in emergencies, and hazard assessment.

figure4.8presentsthefindingsrelatedtoeachofthepreparednessindicatorsusingthepooleddatafromallcountries.Attheindividuallevel,therewerenosignificantchangesinperceivedlevelsofownpreparednessintheDEPPgrouporthecontrolgroup,andtherewasnoneteffectoftheprogrammeonindividualpreparedness.Attheorganisationallevel,ontheotherhand,significantimprovementsinorganisationalpreparedness,bothwiththeself-reportedindicator,aswellaswiththemoreobjectiveassessmentusingtheorganisationalchecklistwithintheDEPPgrouponly have occurred.However, theDifference-in-Difference analysis showsnooverallsignificanteffectoftheprogrammeintermsoforganisationalpreparedness,likelyduetosimilarchangesinthecomparisongroupoverthesameperiod.figure4.9presentstheorganisationalchecklist data (the rigorous indicator of organisational preparedness) for each country andfor local organisations. Therewere significant improvements in organisational preparednessspecificallywithinKenya(anincreaseof11.1%)andalsowithinlocalorganisations(anincreaseof10.1%)acrossallintensivecountries.Again,however,theDifference-in-Differenceanalysis,which takes into account changes in theDEPP and in the control group over time, showsno statistically significant net effect of the programmeon organisational preparedness.ThemeasuredimprovementsinorganisationalpreparednessintheDEPPorganisations(especiallylocalorganisations)arealsosupportedbyin-depthinterviews,documentreviewandthecasestudy(chapter8).forexample,inEthiopia,localbeneficiaryorganisationsoftheShiftingthePowerprojectnoted the improvements in theirorganisationspreparedness levels, andcitedchangedattitudesandmindsets,inter-departmentalcoordination,improvedprocurementplansandstrengthenedorganisationalproceduresincludingcreatingemergencyresponseteams.135

DEPP DIFFERENCE

(T2 - T1)

CONTROL DIFFERENCE


Individual Preparedness (Perceived)

+2 . 2 % - 0 . 5% +1 .7 %

135 Seecasestudy#2inchapter8.AlsotheShiftingthePowerfinalEvaluationReport,p9,alsodescribesthesechangesinorganizationalpreparedness.


Organisational Preparedness (Perceived)

+2 . 0 %* + 0 . 5% -1 . 3 %

Organisational Preparedness (Actual) +9. 3 %* +3 . 0 % + 6 . 3 %

Community Preparedness (Perceived)

-13 . 4%* -16 . 8 %* +3 . 4%

Community Preparedness (has early warning system)

+ 4 0 . 5%* + 0 .1% + 4 0 . 4%*

Community Preparedness (Actual) + 4 .9 % -7. 2 % -2 . 2 %

figure4.8:change inperceivedandactualpreparednessat individual,organisational,andcommunitylevels


Withrespecttocommunitypreparedness,therewasasignificant41%increaseinthecommunityaccess toearlywarningsystems inDEPPcommunities inMyanmarcompared tonochangein non-DEPP comparison communities in the same area.With theDifference-in-Differenceanalysis,theneteffectoftheDEPPprogrammewasa40%increaseincommunityearlywarningsystems.However,surprisingly,thissubstantialeffectdidnottranslatetoimprovedperceptionsofcommunitymembersonthepreparedness leveloftheircommunities. Infact,perceptionsonlevelofcommunitypreparednesssignificantlydecreasedby13%inDEPPcommunitiesand17% in non-DEPP communities.Attitudes related to thevalue of preparedness activities inimproving responsewerenotpositiveanddidnotchangeovertimewhichmayexplain thisfinding. furthermore, findings during the validation workshop suggest that changes in thegovernment andother contextual factorsmayhave led to increasedpessimismon thepartofcommunitymembers.finally,theobjectiveassessmentofcommunitypreparednesswhichaggregatedindicatorsacross30domainsshowednosignificantchangeinDEPP,controlorviatheDifference-in-Differenceanalysis.

DEPP DIFFERENCE

(T2 - T1)

CONTROL DIFFERENCE


Ethiopia +13 .9 % - 6 .9 % +2 0 . 8 %

Kenya +11 .1%* +7. 6% +3 . 4%

The Philippines +5 . 0 % + 4 . 6% + 0 . 4%

Myanmar + 6 . 2 % + 4 . 6% +1 . 6%

Local +10 .1%* + 6 . 8 % + 8 . 8 %

figure4.9:changeinactualorganisationalpreparednessinEthiopia,Kenya,thePhilippines,Myanmar,andforlocalorganisationsacrossthefullsample



3. Humanitarian ResponseDEPP contributed to at least 42 different humanitarian responses in 11 countries; 16 of these responses occurred in the Philippines (and 14 of these 16 responses were driven by one project – the Financial Enablers project). Qualitative data indicate contribution of DEPP beneficiaries towards improved response in the Philippines, Kenya and Ethiopia, but many other DEPP beneficiaries were unable to respond due to lack of response funds.HumanitarianresponsewasnotconsideredbymostprojectsinthedesignphaseastheDEPPwasintendedtofunctionasadedicatedpreparednessprogramme.Assuchlinkswithresponseactivities were not explicitly developed into the design of the programme. Many projectstakeholders,infact,nowconsideranyDEPPcontributiontowardshumanitarianresponsetobeanunintendedbutpositiveconsequenceofDEPPactivities.Humanitarian response activities and outcomeswere assessed in a number ofways in thisevaluation. first, descriptive primary data were collected at the individual, organisationand community level on the types of disasters and emergencies that have occurred in theprevious12months,aswellastypesofresponseactivities implemented.Second,anumberof quantitative indicators related to performance during past humanitarian responses andabilitytorespondinthefutureweremeasuredatthetwotimepoints(Seeboxbelow).Third,the document review allowed systematic mapping of all responses that DEPP reportedlycontributedtoduringtheprogrammeperiod.fourth,additionalresponsedatawascollectedduring theevaluationvalidationworkshops inKenya,Ethiopia,MyanmarandthePhilippinesinMarch2018,tobetterunderstandwhichdisastersandemergenciesstakeholders inthosecountrieswere able to respond to and tounderstand factors related toboth response andnon-response. Inaddition, responsedatawerecollected fromeachprojectbyemail.finally,additionalqualitativeinterviewswereconductedinKenyaandtheEthiopiatocontributetoamoredetailedhumanitarianresponsefocusedcasestudy(seechapter8).figure 4.10 below presents data on the percentage of DEPP and non-DEPP comparisonorganisationsthatrespondedtoadisasteroremergencyinthepast12months.Ofnote,DEPPorganisationswereaskedtoreportonanyresponseactivityregardlessofcontributionoftheDEPP.Asdepictedinthefigure,thevastmajorityofbothDEPPandnon-DEPPorganisationsimplementedhumanitarian responseactivities in theprevious12months,andtherewasnostatisticaldifferencebetweenT1andT2ineithergroup.furthermore,localnGOswhowereDEPPbeneficiarieswerenotmore likelytorespondtoadisasterafterexposuretoDEPPorwhen compared to non-DEPP local organisations.The types of disasters did vary betweenDEPPandnon-DEPPorganisations,withflooding,droughtanddisplacementbeingthemostcommonacrosstheDEPPorganisations,anddrought,floodingandinfectiousdiseaseepidemicsbeingmostcommonamongthenon-DEPPcomparisonorganisations(figure4.11).Thetypesofresponseactivitiesaresummarisedinfigure4.12highlightingthediverserangeofactivities,though food security and water, sanitation and hygiene (WASH) activities were the mostcommoninbothDEPPandnon-DEPPorganisations.

78.8

0

20

40

60

80

100

ORGANISATION RESPONSES TO DISASTER OR EMERGENCY IN PREVIOUS 12 MONTHS

ORG

ANIS

ATIO

NS

(%)

DEPP COMPARiSON DEPP COMPARiSON DEPP COMPARiSON DEPP COMPARiSON DEPP COMPARiSON DEPP COMPARiSON

LOCAL NGOS KENYATHE PHiLiPPiNESETHiOPiAMYANMARALL

T2

T1

79.5

71.8 79

.2 84.1

69.6

47.8

85.0

68.8

95.2 10

0

100

100

100

81.082

.6

64.368

.4

92.9

78.8

66.7

76.9

67.7 73

.7 77.1

figure4.10:Percentageoforganisations that responded toadisasteroremergency in theprevious12


figure4.11:Types of disasters and emergencies that organisations responded to in theprevious12months

figure4.12:Typesofresponseactivitiesimplementedbyorganisationsintheprevious12months

Quantitative indicators to assess perceived performance during a previous response duringthe last12months, andaswell as futureability to respond toadisasteroremergencyaresummarisedinfigure4.13.Attheindividuallevel,humanitarianstaffatDEPPandnon-DEPPcomparisonorganisationswereaskedtoratetheirpastresponseperformanceandtheirabilitytorespondinthefuture(seeboxbelowfordescriptionoftheindicators).

LANDSLiDES

DISASTER AND EMERGENCY RESPONSES

COMPARiSON

DEPP

0

10

20

30

40

50

60

TYPHOONS

FLOODiNG

VOLCANO ERUPTiONS

STORM SURGE

DROUGHT

WAR AND CONFLiCT

iNFECTiOUS DiSEASE EPiDEMiC

EARTHQUAKES

MONSOON

MASS EXPOSURE TO TOXiC CHEMiCALS

DiSPLACEMENT

9.0

3.4

44.1

33.6

9.6

10.1

40.8

52.1

5.7

5.9 3.

3

1.7 8.4 3.4

16.2

29.4 30.6

22.7

34.2

26.9

0.3 0.8 9.

3

2.5

RESP

ON

DEN

TS (%

)

ORGANISATION RESPONSE ACTIVITIES

COMPARiSON

DEPP

0

10

20

30

40

50

60

70

80

12.6 16

.1

62.2

68.8

62.9

54.7

33.6

46.9

25.9 28

.1

26.6

43.8

31.5 35

.9

28.0

42.2

30.1

39.1

20.3

29.7

21.0

34.4

8.4

15.6

11.2 14

.1 16.8

14.1

20.4

31.3

ORG

ANIS

ATIO

NS

(%)

WASH

FOOD SECURiTY

NONE

EDUCATiON

SHELTER

HEALTH

MEL

NUTRiTiON

CHiLD PROTECTiON

AWARDS

FUNDRAiSiNG

LOGiSTiCS

MANAGEMENT

SURGE

TELECOMMUNiCATiONS


KEY RESPONSE INDICATOR DATA (RELATED TO RESPONSE IN THE PREVIOUS 12 MONTHS) COLLECTED AT 2 TIMEPOINTS

Individual level:

• Perceived past performance: KAP survey with humanitarian staff. How would you rate your performance during your last response? Where 1=Poor, 2=Fair, 3=Good, 4=Very good, 5=Ex-cellent

• Perceived future ability: KAP survey with humanitarian staff. How would you rate your ability to respond to a disaster in the future? Where 1=Poor, 2=Fair, 3=Good, 4=Very good, 5=Excellent


• Perceived past performance: Organisational survey with senior leadership. How would you rate your organisation’s overall response to the disaster or emergency (in the last 12 months)? Where 1=Poor, 2=Fair, 3=Good, 4=Very good, 5=Excellent

• Perceived future ability: Organisational survey with senior leadership. How would you rate your organisation’s ability to respond to a disaster in the future? Where 1=Poor, 2=Fair, 3=Good, 4=Very good, 5=Excellent

Community level:

• Satisfaction with previous response: Household survey: How satisfied were you with pre-vious response to disaster? Where 1=Not satisfied at all, 2=Slightly satisfied, 3=Somewhat satisfied, 4= Satisfied, 5= Very Satisfied

ThereisadownwardtrendamongstbothDEPPandnon-DEPPhumanitarianstaffreflectingdeclining perceptions of own performance and future ability to respond, though neither ofthesedifferenceswerestatistically significantexcept forpastperformance in thenon-DEPPcomparison group. Overall there was no net DEPP programme effect for either of theseindicators using the pooled data from the four intensive countries. Therewas also no netprogrammeeffectwhenrestrictingthesampletospecificintensivesetcountriesortoonlylocalorganisations.However,casestudyqualitativedatafromKenyaandEthiopiademonstratesomeindividual-levelchanges inpreparedness, includingknowledgeonprocedures,resources,andimprovedconfidencetorespond(Seechapter8).ThecDAc-nprojectfinalevaluationreportalsoreportedthatindividualsbenefittingfromtheirprojectinSouthSudanandbangladeshfeelmoreequippedandpreparedtoimplementresponseactivities,especiallythoserelatedtotwo-waycommunication136.Thisdemonstratesthattherehavebeendocumentedimprovementsinqualitativeindicatorsrelatedtoabilitytorespond,butthatthesehavenotyettranslatedintowidespreadquantitativechangeacrosstheprogramme.

DEPP DIFFERENCE

(T2 - T1)

CONTROL DIFFERENCE


Individual Response (Perceived Past Performance)

- 0 . 6% - 4 .9 %* + 4 .1%

Individual Response (Perceived Future Ability)

-1 . 4%* - 6 . 2 % + 4 . 8 %

Organisational Response (Perceived Past Performance)

-1 . 8 %* +3 . 4% -5 .1%

136 cDAc-nfinalEvaluationReport,p47.


Organisational Response (Perceived Future Ability)

+2 . 8 %* + 4 . 5%* -2 . 4%

Community Response (Satisfaction with Past Performance)

-15 . 2 % -16 . 6% +1 .1%

figure 4.13: change in perceived response performance (both previous and future) at individual,organizationalandcommunitylevels


Attheorganisationallevel,therewerenosignificantchangesintheperceivedpastperformanceduringaresponse,orinperceivedfutureabilitytorespondinboththeDEPPandnon-DEPPcomparison groups in the four intensive set countries, and therewas no net effect of theprogramme on either of these indicators with the pooled or restricted samples. However,as described inmore detail below, qualitative data, and data extracted from the documentreviewdodemonstratesomeexamplesofimprovedorganisationalresponseactivities.Atthecommunity level, drawing on household data from Myanmar, beneficiary satisfaction withhumanitarianresponseintheprevious12monthssignificantlydecreasedby15%intheDEPPcommunitiesand17%inthecontrolcommunities.OveralltherewasnosignificantneteffectoftheprogrammeonbeneficiarysatisfactioninMyanmar.Discussionduringvalidationworkshopssuggestedthismighthavebeenduetoworseningcontextualfactorsandpoliticalclimatewithinthecountry.

Documentreviewandin-depthinterviewsdemonstratecontributionoftheDEPPtoatleast42differenthumanitarianresponsesin11countries(seeTable4.6below,andAnnex10).Therewerenodocumentedcontributions to responses inMozambique,oneof thecountrieswiththefewestDEPPprojects(oneproject).Thelargestnumberofresponsecontributions(n=16)occurredinthePhilippineswhichalsohadthehighestnumberofDEPPprojectsimplemented(n=6).notethatKenyaandPakistanalsohad6DEPPprojectsbutinthesespecificcontexts,DEPPcontributedtosubstantiallyfewerresponsesthaninthePhilippines.Thisdifferencewasalsonotedwith independentresponsedatacollecteddirectlyfromprojectsduringvalidationworkshops. In fact, according to data collected during these workshops, there was DEPPcontribution to 33 responses in the Philippines (versus 16 responses extracted from thedocument review).furtherdiscussiononwhyDEPPcontributed tomore responses than inother countries canbe foundbelow.Only three projects provided response informationbyemail (Table4.6).This is likelybecause theprojectswerewrappingup andproject staffdidnothavetimetocontributetothiscomponentoftheevaluation.completedetailsofthe42responseeffortsareinAnnex10.

COUNTRY PROJECTS

IMPLEMENTED (#)*

DOCU-MENT

REVIEW

INTENSIVE SET COUNTRY

WORKSHOPSSUBMITTED VIA EMAIL**

NUMBER

OF DEPP

RESPONSES

NUMBER

OF EVENTS

NUMBER

OF DEPP

RESPONSES

NUMBER

OF EVENTS

NUMBER

OF DEPP

RESPONSES

NUMBER

NOT RE-

SPONDED

TO

REASONS

FOR NOT

RESPOND-

ING

1The Philippines

PIP, FEP, TSC, CDAC-N, LPRR, ALERT (6)

16 33 33 1 1 0 n/A

2 MyanmarPIP, LPRR, SEPS (3)

2 2 2 4 3 1

OutoftheDEPPregionalscope


3 EthiopiaSTP, PHEP Gambella, EWEA, TD, (4)

4 12 12 3 1 2

ResponsewasnotfundedbytheDEPP

4 Kenya

STP, ADCAP, LPRR, TD, UEWEA, ALERT (6)

3 15 15 4 4 0 n/A

5 PakistanPIP, STP, TSC, ADCAP, LPRR, ALERT (6)

4 n/A n/A 6 DRC

PIP, STP, LPRR, TD (4)

2 n/A n/A 7 Bangladesh

STP, CDAC-N, LPRR, TD, ALERT(5)

6 1 1 7 7 0 n/A

8 South Sudan PIP, CDAC (2) 4 n/A n/A 1 1 0 n/A

9Jordan- Lebanon- Syria***

TD (1) 1 1 10 Mozambique ALERT (1) 0 n/A n/A 11 India**** TSC (1) 1 n/A n/A TOTAL 42 64 64 20 17 3 Table4.6.contributionofDEPPtohumanitarianresponseactivitiesinDEPPcountries

*The Learning Project was implemented in all DEPP countries and is not included, **Only four projects submitted data, ***One response reported by Talent Development in Kenya, ****TSC deployment

findings based on the document review, qualitative interviews and validation workshopssuggests thatprojects in thePhilippinesoutperformed theotherDEPP focal countrieswithrespecttocontributiontohumanitarianresponses.notethatuniversally,thesecontributionstoresponseswerenotnecessarilydesignedoutcomesfromtheoutsetbutbecameobjectivesasprojectsandplanningevolved.Somestakeholdersexpressed these responseactivities tobeunplannedsideeffects.QualitativedatasuggeststhatDEPPcontributiontosomeoftheseresponses led tomore timely, locally driven responses. In other cases, it is not possible toassesstimelinessoreffectivenessofaparticularresponseeffortwithavailabledatasources.Incontrasttothequalitativeindicators,quantitativedataonperceptionsof improvedspeedandeffectivenessof responseactivitiesdidnot showasignificantmeasurableeffectof theDEPP.furtheranalysisontheperformanceoftheDEPPtowardshumanitarianresponsesinthePhilippinesrevealsthefollowing:

• ThehighperformanceinthePhilippinesintermsofabsolutenumberofhumanitarianresponseeffortswaslargelydrivenbythefinancialEnablersprojectwhichcumulativelycontributed to14different responses (outof16DEPP responses in thePhilippinesduringtheDEPPprogramme)

• SeveralfactorscontributedtotheoverallperformanceinthePhilippinesvis-à-visothercountries.Theseinclude:

o The fact that the Philippines has a high frequency of disasters/emergencieseach year to which projects could theoretically respond. This has createdgreater incentive forgovernmental andnon-governmentpartners todeveloppreparednessprojectsandmoreopportunityfortheseactorstobuildupontheir


experiencefromeachresponse.

o networks in thePhilippinesarehighly locallydriven (seechapter5),andthegovernmentandotherkeyactorsprovidespaceforcivilsocietyaction.

o flexiblefundingmechanisms(viathefinancialEnablersproject,andtoalesserdegreeProtection inPractice)allowed localactors to identify relevantneeds,and implement accordingly. This included using funding towards responseactivities(viamechanismssuchasaquickresponsefund).ThiswasnotthecaseinnumerousothersettingswhereDEPPbeneficiarieswhowantedtorespond(andnowbelieved theyhad thecapacity to)were limited inwhat theycouldcontributeduetoabsenceofdesignatedresponsefunds.DEPPbeneficiariesinthePhilippineswerealsosuccessfulatsecuringresponsefundsexternaltotheDEPPsuchasviatheSTARTfund,orthroughotherdonors.

o The large number of DEPP projects being implemented in the Philippines(n=6)providedagreateropportunityforcollectiveaction,andalargeroverallinvestment.

o TheDEPPprojectsinthePhilippinesshowedasubstantiallyhighlevelofinter-projectcollaborationandwereable to jointlycontribute tocertain responses(i.e.,Marawiresponse).Overall, theDEPPprojects inthePhilippines, inmanyways,operatedmorelikeaprogramme(andtheinitialvisionoftheDEPP)thaninother focuscountries,albeitdespite thekeyprogramme-levelsystemsandprocessesthatwereessentialforportfoliofunctioning.

The contributions to humanitarian responses in the Philippines, Kenya and Ethiopia wereexamined in further detail and arepresented in theboxesbelow.All three settingsprovidesomeevidenceofimprovedorganisationalresponseactivities,andthePhilippinesandKenyademonstratethecontributionofthecollaborativeapproachandworkingthroughconsortiatoimprovedhumanitarianresponseactivities.

Humanitarian Response: Focus on the Marawi Crisis

What:

TheMarawicrisisbeganonMay23,2017,whenthegovernmentofthePhilippineslaunchedamilitaryoperationagainstpro-ISISmilitantsinthelanaodelSurprovinceofMarawicity.Thisfive-monthlongincursionresultedinthedisplacementofroughly98%ofthepopulation,oralmost360,000people137.

Who:

AtotalofsixDEPPprojectswereimplementedinthePhilippines:ProtectioninPractice,financialEnablers,TransformingSurgecapacity,lPRR,AlERTandthecDAc-ncommunicatingwithcommunitiesProject.

Objective:

TounderstandhowtheseprojectscontributedtoMarawiresponseefforts

Data sources:

• MinimumsetIDIswithprogrammeandprojectstakeholders(Qualitative)

• IntensivesetIDIs(Qualitative)

137 https://reliefweb.int/report/philippines/unicef-philippines-humanitarian-situation-report-marawi-re-sponse-10-november-2017;http://www.unhcr.org/ph/marawi-crisis.


• networkanalysis(Quantitative)

• Documentreview(Qualitative)

Key Findings:

Intotal,atleast16disastersandemergencies(includingflooding,typhoons,earthquakes,fires)inthePhilippineshadresponsecontributionbytheDEPP,asubstantiallyhigherfigurethanany other DEPP country. Self-reported data indicate successful inter-project collaborationandcoordination,wherecapacitybuildingoflocalorganisationsworkingtogetherinconsortiacontributedtoafaster,morelocallydrivenresponseeffort.ThemostprominentexampleofsignificantcontributionwastotheMarawicrisis.fourof thesixprojectsrespondedtotheMarawicrisesincludingthefollowingresponseactivities:

• Aprotectionnetworkof600localrespondersinMindanaoabletoimplementinclusiveprogrammingwasestablished

• localconsortiawhosecapacitywasstrengthenedbytheDEPPwerethefirsttorespondtothecrisis(anddidsoindependentlywithoutaleadInGO).TheywererecognisedforthefirsttimebyUnOcHAoncrisisresponsemapsdemonstratingincreasedpresenceandrecognitionoflocalactors

• Variousconsortiacoordinatedtheirresponsestogether,allowingforefficientuseofhumanresourcesandsharingofknowledge,contactsandresources

• Examplesdemonstratebeneficiary involvement indesignand implementationofsomeresponseactivities

• localactorsgainedconfidenceandskillsastheytookonmoreresponsibilityduringtheresponse

factorsthatenabledafastermorelocallydrivenresponseincludedthefollowing:

• Incontrasttoothercountryexamples,consortiapartnersinthePhilippineswereabletoaccessquickresponsefundsthroughanumberofmechanismswhichenabledresponseactivitiestobeimplementedrapidly

• ThePhilippines isadisaster-pronecontext,withahumanitariannetwork that ishighlydrivenby localactors.This isreflectiveofacontextwherethegovernmentandotheractorsprovidespaceforcivilsociety,which isdeeplyengrained insocietyandcultureandconsideredtobethestrongestinAsia138.TheprominenceoflocalactorswithinthePhilippinesallowscollaborativeactionatthelocalleveltooccurmoreorganicallythaninothercountrycontexts

• AkeyenablingfactorwasthemandatebythegovernmentthatonlylocalnGOscouldrespondtothiscrisis.Thisprovidedthespaceandopportunityforlocalactorstoactivelyrespond

Conclusion:

Self-reported data suggest strong inter-project collaboration, coordination between DEPPorganisationsandunifiedresponseactivitiesdeliveredbylocalconsortiaratherthanindividualorganisations.Self-reporteddatasuggestthattheresponsewasfaster,moreefficient,moreinclusiveandinvolvedbeneficiaries,andthiswasdrivenbytheimmediateavailabilityofflexiblefundsandacontextsupportiveoflocalactors.However,thereisnowaytoobjectivelyassesshowtheresponsemighthavedifferedintheabsenceoftheDEPPinthePhilippines.

138 USAID’scSOSustainabilityIndexranksfilipinocSOshighestinAsiaacross7dimensions:legalenviron-ment,organisationalcapacity,financialviability,advocacy,serviceprovision,infrastructure,andpublicimage.


Humanitarian Response: Focus on urban emergencies in Kenya

What:

• Anurbanearlywarningsurveillancesystemininformalsettlementsinnairobicountytomonitorfoodsecurity,householdshocksandpublichealthindicatorsonaroutinebasis(everytwomonths)detectedthreeemergenciesandtriggeredresponseactivities.

1. Diarrhoealoutbreak:february2017toJune20172. choleraoutbreak:April2017tofebruary20183. foodinsecurity:OctobertoMarch2018

Who:

TheDEPP’sUrbanEarlyWarningEarlyAction(UEWEA)ProjectdevelopedandimplementedtheurbanearlywarningsystemthroughitsconsortiummembersconcernWorldwide,Oxfam,andKenyaRedcrossSociety, in seven sub-counties (Kibera,Kasarani,Ruaraka,Makadara,Starehe,KamukunjiandEmbakasiEast).ThesystemaimedtoidentifyslowonsetorchronicemergenciesinurbancontextsthroughroutinemonitoringofasetofsensitiveindicatorsthatweredevelopedalongwithspecificthresholdspriortothestartoftheprojectandtosetupacoordinatedresponsemechanismwithinthenairobicitycountryGovernment.

Objective:

TounderstandhowDEPPprojectscontributedtoresponseeffortsininformalsettlements

Data sources:

• MinimumsetIDIswithprogrammeandprojectstakeholders(Qualitative)• IntensivesetIDIs(Qualitative)• Documentreview(Qualitative)• Responsecasestudydatacollection:16IDIs,sitevisitsandobservations(Qualitativeand

Observational)Key Findings:

• DEPPcontributionstoresponsesinurbansettlementsinnairobithroughearlydetectionagainstasetofsensitiveindicatorswithclearthresholds,standoutasbeingparticularlyinnovativeintheirsound,data-drivenapproachtoidentifyingurbanemergenciesandfortheirstrongcollaborationwithgovernment.This isoneofthefewsystemsspecificallytargetingurbancontextsandfillsanimportantneed,especiallyasthepopulationsinurbansettingscontinuetogrow.Thefollowingemergencyresponseactivitiesweretriggered:

o Diarrhealoutbreak:healtheducation,massscreeningofchildrenunder5years(8,338screenedinKorogocho,13,284inEmbakasi),anddistributionofwaterpurificationproducts

o cholera outbreak: health promotion (171,139 households and 2630 studentssensitised),solidandliquidwastemanagement,andwaterqualityandfoodsafetymonitoring

o foodsecurity:Massscreeningofchildrenunder5years(44,825)andcashtransfersof2000KSH(roughly20USD)providedto3,034households(2,085women,949men)whowereseverelyfoodinsecureweremadeviamobilemoneyinKorogochoandKibera

• Thefirstresponsewastheslowest–noresponsepackageexisted,andlackofimmediatefundingsource led todelays.The responseoccurred inApril2017approximately twomonthsafterinitialdetection.


• Althoughthesurveillancesystemwasnotthefirsttoidentifythecholeraoutbreak,theuseoftheconsortiummodelenabledconsortiummemberstocollaborateinthecholeraresponseoutsideoftheirDEPPactivities.

• Datacollectedsuggestthatresponsetimeimprovedsignificantlybetweenthefirstresponsetodiarrhoeatothemostrecentresponsetofoodinsecuritywhichwasundertakenwithinafewweeksofdetection.

• factors enabling success include: conducting a needs assessment and research toinformthesystem,anddevelopingthesystemalongsidegovernmentactors toensurecoordination,buy-in,alignmentwithgovernmentprioritiesandsustainability

Conclusion:

Datacollectedsuggeststhattheconsortiummanagementmodelaswellascollaborationwithgovernmentwerecriticaltosuccessfulimplementationofthissystemandpermittedcoordinatedhumanitarian responses that improved with each subsequent emergency. Availability ofimmediateresponsefundsandpreviouslyagreeduponresponseactionswerekeytoensuringmorerapidresponses.

Humanitarian Response: Focus on drought in Ethiopia

What:

Ethiopiahasbeenaffectedbylargescaledroughtssince2015.Initiallyinducedbythe2015/2016Elnino,andfurtherexacerbatedbylow2016Autumnrains,andbelowaverage2017Springrains,atleast8.5millionpeoplewereinneedofrelieffoodassistancebymid-2017inSouthernandSouth-easternEthiopia.Theeffectsof thedroughthavebeen furtherexacerbatedbydiseaseoutbreaks,largescalelossoflivelihoodassets,anddisplacement.conflictintheOromiaandSomaliregionsbeginninginSeptember2017furthercontributedtothedisplacementofhundredsofthousandsofpeopleoftenindrought-affectedareas.

Who:

AtotaloffourDEPPprojectswereimplementedinEthiopia:ShiftingthePower,PublicHealthPreparedness in Gambella, Talent Development and Improved Early Warning Early Action–Ethiopia.

Objective:

TounderstandhowDEPPprojectscontributedtoresponseeffortstodroughtinEthiopia

Data sources:

• MinimumsetIDIswithprogrammeandprojectstakeholders(Qualitative)

• IntensivesetIDIs(Qualitative)

• Documentreview(Qualitative)

• Responsecasestudydatacollection:7IDIs

Key Findings:

Self-reporteddatasuggest6outof10localpartnersoftheShiftingthePowerproject,whichhadaimedtobuildl/nnGOcapacityandpreparedness,respondedtothedroughtintheareaswheretheymanagedevelopmentprograms.Theseincluded:communityInitiativefacilitationAssistance(cIfA),communityDevelopmentInitiative(cDI),RiftValleychildrenandWomenDevelopmentOrganisation (RVcWO), Action forDevelopment, SOS Sahel, andHUnDEE.Specificresponseactivitiesledbythesel/nnGOsin2017-2018included:


• cIfA targeted35,060households in theMoyale,MiyoandGuchiworedas to reducelivestock mortality and implemented a livestock vaccination campaign which reached263,187cattle

• cDIsecureda£10,000responsegrantfromOxfamandShiftingthePowerwhichenableddistributionofseedsto200householdsincommunitiesaffectedbythedroughtinWestArsiZone,Oromia

• RcVWOobtaineda£60,000grantfromActionAidtoprovidelivestockfeedsupporttoatargeted1,030households

• Action for Development provided £60,000 in cash transfers to drought affectedcommunitiesinborena

• SOSSahelsecured$530,000directly fromUnOcHA’sEmergencyResponsefundtoprovideanimalfeedinthreeworedas(Wachille,DubuluqandDire),indirectlybenefitting28,256individuals.This isoneoftheonly instanceswherea localEthiopiannGOhasbeenabletosecurefundsthroughthismechanismwithoutanintermediarypartner

• HUnDEE obtained $410,000 from Kindernothilfe (KnH) to provide emergency foodassistanceto10,000peoplein4KebelesoftwoworedasintheborenaZone

There was no available evidence as to whether speed, costliness or effectiveness of theresponsewereaffected.factorsthatenabledthesel/nnGOstorespondinclude:

• ShiftingthePowersetupa£50,000responsefundaccessibletol/nnGOpartnersviaapeer-reviewedapplication

• Supportwithproposalwriting,andtrainingonfundraisingstrategies includingadonormapping exercise facilitated access to direct response fundingwhich in turn enabledresponseactivities

• ThemagnitudeofthedroughtcrisisinEthiopiacreatedanopportunitytopromotelearningbydoing;thus,creatingastronglinkbetweenemergencypreparednessandresponse.forexample,l/nnGOpartnersobtainedsupportandpracticeinapplyingforresponsefundsviaShiftingthePower’sresponsefundpeerreviewprocess,andonepartnerwasabletodirectlyimplementaresponseprojectasaresult.

Conclusion:

Self-reporteddatasuggestthatexperientiallearningrelatedtofundraisingandimplementationofhumanitarianresponseactivities,alongwithaccesstoflexibleresponsefundingmechanismscontributedtosixdroughtresponsesinEthiopia.

4. Linking Emergency Preparedness Capacity to Disaster Response

Despitethecontributionstowardshumanitarianresponsesthathavebeendocumentedprimarilyvia qualitative documents and through self-reported data, it is difficult to objectively (andquantitatively)assessincreasedspeed,coverage,orefficiencyofresponseinthe42responseexamplesandaswelltotracethesebacktoincreasedpreparednessevenwiththediverserangeofdatasources.Infact,acrossthefourintensivesetcountries,self-reportedquantitativedataonthespeed,efficiency,andcostlinessofresponseactivitieswerenotdifferentbetweenDEPPandnon-DEPPcomparisonorganisationsacrossthefullsample.However,thesedatapertaintoanyresponseactivitiesintheprevious12monthsandmayormaynothaveincludedthe42responseexamplesidentifiedbytheevaluationteam.further,theevaluationteamisunabletocomparetheseindicatorsamongDEPPandnon-DEPPcomparisonorganisationsthatdirectlyrespondedinthe42emergencyresponsesdescribedaboveasthereis limitedavailabledataaboutnon-DEPPcontributiontothesesameresponses.ThiswouldhaveprovidedmoreconcreteevidenceonDEPPcontributiontoimproveddeliveryofhumanitarianresponse.nevertheless,thebestevidenceacrosstheDEPPforstrengthenedresponseisinthePhilippineswherethere


issomeself-reportedlinkbetweenimprovedcapacity,improvedpreparednessandfollowedbyearlier,moretimelyresponse.Thequantitativedatadoshowstatisticallysignificantincreasedpreparedness (using the actualmeasure) amongDEPPorganisations in the full sample, andespeciallyamonglocalorganisations.However,thiswasnotsignificantinthePhilippines,likelyduetosmallsamplesize.

AnimportantfindingfromthequalitativedataaswellasthecasestudyonEthiopiaandKenyainchapter8,highlightthatinmanycasesemergencypreparednessandresponseactivitiesweredisjointed.This isnotsurprisingasmostprojectsweredesignedasstandalonepreparednessprojectswithoutclearlyarticulatingthelinktoresponseorincorporatingactivitiesthatmightbetterenablel/nnGOstorespond.Intervieweesemphasisedtheneedtostrengthenthislinkbetweenemergencypreparednessandresponseactivities,andalsodescribedtheneedtoseekseparatefundingforresponseactivitiesasasignificantbarrier.

PROJECT VERSUS PROGRAMME OUTCOMES AND ADDED VALUE OF BEING PART OFTHE DEPP

Individual project evaluations suggest some positive outcomes, but this has not translated to global programme effects likely due to short programme period and lack of project alignment.Astheexternalevaluationfocusedonglobalprogramme-leveloutputsandoutcomesaswellascountry-levelprogrammeresultsusingpooleddatafromeachcountry,individualprojecteffectscannot be assessed. Previous evaluation reports, therefore, recommended strengthening ofthe individual projectM&E systems and the implementation of end of project evaluations.Theexternalevaluationteamprovidedadetailedevaluationchecklistaswellasguidanceonindicatorstoassessinanefforttoharmonisemeasurementacrossprojects.Allprojectshavecompletedindividualevaluations,whichhavebeenreviewedaspartofthedocumentreview,thoughthequalityandrigorofeachevaluationvaries.

Anumberofindividualprojectevaluationsreportpositiveoutcomeswhichshouldbeinterpretedobjectively,inlightofthequalityofthespecificevaluationmethodologyandapproachanditslimitations.Withthisinmind,astherewaslimitedalignmentofprojectsattheportfoliolevel,eachprojectworkedtowardsitsownobjectivesandowntheoryofchange.Itisconceivablethatanindividualprojectmighthavecontributedtopositiveoutcomechangeonspecificindicators,butthatgloballyattheprogrammelevel,cumulativechangeinthisindicatorwasnotachieved.Infact,positiveoutcomesthathavebeensuggestedbyprojectevaluationshavenottranslatedintocumulativeglobalprogrammeeffectsandthisislikelyduetothelackofprojectalignmentand/ortheshortprogrammeperiods.

Added Value of Being Part of the DEPP

Giventhattheprojectsweredesignedindependentlyandwithlimitedconsiderationtoportfoliofunctioning,andevaluationdatademonstratedlimitedcohesionasaportfolio,theaddedvalueofbeingpartoftheDEPPprogrammehasbeenlimited.ThemostbeneficialaspectofworkingundertheDEPPhasbeentheabilitytosharecontactsandknowledgeandleveragepreviouslybuiltconnectionsandnetworks.However,thedatasuggestthatthelackofcohesionacrosstheprojectportfolioledtolimitedawarenessandunderstandingoftheDEPP,andlackofasenseofbelongingorconnectiontotheDEPPatthecountrylevel.Duringin-depthinterviews,mostintervieweescouldspeakabouttheirindividualprojectsbutinmostcases,wereunabletoreflectontheDEPPasawholeanddiscussitsobjectivesandfunctioningwithintheircountry.Thislowprogrammeawarenessandidentityaffectedprogramme-leveladvocacy,stakeholderbuy-inandengagementattheprogrammelevel,coordinatedprojectimplementationandlonger-termsustainabilityoftheprogramme.

DOSE –RESPONSE ASSESSMENTAsdescribedintheinceptionphasereport,theexternalevaluationaimedtoconductadose-responseassessmentwhereinprogramme-leveloutcomescouldbecomparedacrosscountriestakingintoconsiderationthenumberofprojects,sizeofinvestmentsandhumanitariancontext.Thehypothesistobetestedwasthatcountrieswithahigherinvestment,withmoreprojectsworkingcollaborativelywoulddisplaylargereffects.Infact,thefourcountriesselectedfortheintensivesetdatacollectionwereselectedtorepresentarangeofDEPP“doses”.Unfortunately,


as there were very few quantitative programme-level outcome changes, a dose-responseanalysisisoflimitedvalue.ItisworthwhiletohighlightthatthePhilippines,thecountrywiththehighestlevelofDEPPprojectsandinvestment,hadthelargestcontributiontohumanitarianresponse.

NUMBER OF DEPP PROJECTS (AT THE END OF DEPP)Mozambique 1Jordan 1South Sudan 2*Myanmar 3**Ethiopia 4DRC 5Bangladesh 5Pakistan 6Kenya 6The Philippines 6*

Table4.7:numberofDEPPprojectsbycountry(notincludingthelearningProjectwhichwasimplementedinall10countries)

*CDAC Network’s project moved from South Sudan to the Philippines, thus altering the project numbers in those two countries

**Myanmar originally had 2 projects, but LPRR decided to implement activities in Myanmar raising the number of projects to 3


EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS

cHAPTER5

129 SUMMATIVEPHASEREPORT-EffEcTIVEnESSOfMAnAGEMEnTARRAnGEMEnTS

Evaluation Question #3: To what extent was the programme's theory that capacity development is more effective when undertaken as a multi-agency collaborative approach proven?Sub-questions:a) Isthe‘collaborative’approachofmulti-stakeholderplatformsaneffectivedeliverymechanism?b) focusingoncoalitions,partnershipsandconnectedness–whatcanbesaidabouttheeffectsof

strengthenednetworks?c) Whathavebeenthemainpatternsofcollaboration,andthebenefitsanddisadvantagesofinformal

versusformalcollaboration?d) Whatuniquecontributiondidcollaborativerelationshipsand‘multi-stakeholderplatforms’make

towarddeepeningcross-programmelearning?Data sources: • Minimum set IDIs with programme and project stakeholders(Qualitative)• KAP, organisational and network surveys(Quantitative)• Intensive set IDIs(Qualitative)• Document review (Qualitative)Key Findings:• Despite the original vision, the DEPP focused more on consortia-based collaborative

approaches rather than multi-stakeholder platforms at the national, regional and international levels.collaborativeapproacheswereconsistentlyreportedtobethemostappropriate,preferredapproach to deliver humanitarian capacity building, but respondents identified challenges inengagement,communicationandcoordination.

• As a delivery mechanism, collaborative and consortia-based approaches were particularly effective when the following conditions were met:o Existenceofjointobjectives,valuesandcommonwaysofworkingo collaborationswereorganicnotforcedo Sufficienttime,space,resources,capacityandwilltofosterhealthycollaborativerelationshipso Streamlineddecision-makingandcontractualprocessesandfewerconsortiapartnerso buildingonexistingrelationships

• Thetopthreecollaborationareas(areaswherehumanitarianorganisationsmayworktogether)inthefourintensivedatacollectioncountries(ThePhilippines,Kenya,Ethiopia,andMyanmar)wereadvocacy,communitycapacitybuildingandprojectimplementation.

• There was some evidence of strengthened humanitarian response networks among three of the four intensive set countries (The Philippines, Kenya and Ethiopia, while the network in Myanmar remained unchanged).

• The Philippines network is highly locally driven (70% links arewith l/nnGOs); network sizeincreasedbutthenetworkbecamelessdensewithfewerconnections.The Ethiopia network is internationally driven with little input from local organisations,whilein Kenya, the network is balanced between local and international actors.Therewasevidenceofnetworkgrowthinbothsettings.TheMyanmarnetworkisverysmall,isolatedanddominatedbyinternationalactors.

• Collaboration findings were mixed.collaborationfrequency(aproxyindicatoroftrust)andqualityofrelationshipsincreasedinKenyaandthePhilippinesbutdecreasedinMyanmarandEthiopiabetweenT1andT2.

• Collaboration within the DEPP cohort increased in Myanmar and the Philippines, but both did not increase engagement with non-DEPP actors. Ethiopia and Kenya had increases in relationships between DEPP and non-DEPP organisations.

• Overall,therewasnoquantitativeevidenceofincreasedlocalisation(noincreaseinproportionofrelationshipswithlocalactorsbetweenT1andT2)inanyintensivecountrywithinthenetworkdata.Infact,theMyanmarnetworkbecameevenmoredominatedbyInGOsbyT2.Howeverqualitativedatasuggestschangesinattitudesaroundlocalisationareoccurring(seecH7)

• Not enough time passed between the data collection points to sufficiently document and test the hypothesis that strengthened networks and greater collaboration lead to better preparedness. DataatT1pickedupnetworkeffectscausedbyever-shiftingstrategiescommontofirstyearimplementation.T2likelycapturedsomedistinctiveprogrammeimpactsofDEPP,butsufficienttimetofollownetworkchangerequiresatleastanadditional12months.

• betweenprojectcollaborationcontributedtoimprovedsharingoflearningandevidencebutthisdidnotnecessarilytranslatetobehaviourchange.


OVERVIEW OF THE APPROACH TO MEASURING AND ASSESSING COLLABORATION AND MANAGEMENTARRANGEMENTS collaboration is considered to be one of the defining features of theDEPP andwas usedbyallprojects inanumberofmanners.This includedcollaborationwithinprojectconsortia,collaborationamongDEPPprojectsandcollaborationwithexternalstakeholders.Programmeoutputsrelatedtocollaborationaredetailedinchapter4inthesectiononoutputarea1.Thischapterwillfurtherexaminecollaborationatthesevariouslevelsincludingthroughstrengthenednetworks. The analysis brings together several streams of data, including qualitative andquantitativedatafromtheminimumandintensivesetcountries,aswellasdocumentreviewandthemeasurementandanalysisofhumanitarianpreparednessandresponsenetworksinthefourintensivesetcountries.

NETWORK ANALYSIS: UNDERSTANDING NETWORK STRUCTURES AND TOP COLLABORATION AREAS

AcriticalfocusofthenetworkanalyseswastoidentifypatternsofcollaborationbetweenDEPPandnon-DEPPactorsandbetweenlocalandinternationalactorsinthefourintensivecountriesandtodetectanychanges inthesecollaborationpatternsduetotheDEPP. Inaddition,thenetworkanalysisaimedtoassesstheextenttowhichstrongorganisationaltiesorrelationshipsare associatedwith higher performance. Overall, 32 collaboration areas, or potential areaswhereorganisationsmightworktogetherwithinhumanitarianpreparednessandresponse,wereassessed inorder todetermine thesizeandstructureofhumanitariannetworks inEthiopia,Kenya,MyanmarandthePhilippines.

Thenetworksurveywasadministeredintwophases.ThefirstphaseinvolveddatacollectionwithanestablishedsetofactorsineachcountrywhorepresentedDEPPprogrammepartnersandbeneficiaries,aswellasotherswhowerenotpartofDEPP(whichservedasthecomparisongroup).Theactorsidentifiedbyphaseoneparticipantsascollaboratingpartnersareconsideredtobe1st degreeactorsastheyareonedegreeofseparationawayfromthesurveyrespondent(Seefigure5.1).The1stdegreeactorsidentifiedinphaseoneweretheninvitedtoparticipateinthenetworksurvey.Theactorstheyidentifiedareconsideredas2nddegreeactors,astheyaretwodegreesofseparationfromtheinitialsurveyinformant.Thesurveywasdesignedtoincludeactorsuptothreedegreesofseparationfromtheinformantforallcountries.Giventimeconstraints,only2nddegreeactorsweresurveyed in thePhilippines forbothdatacollectionperiods.

ACTiON AiD CARE KENYA1ST DEGREE ACTOR

iSLAMiC RELiEF2ND DEGREE ACTOR

EXAMPLE OF DEGREES OF SEPARATIONIn this example, Action Aid took the first phase survey and named CARE Kenya. CARE Kenya took the survey in the

second phase and named Islamic Relief. Islamic Relif is 2 degrees of separation away from Action Aid.

figure5.1:Degreesofseparationinnetworkanalysis


All network survey participants were asked to identify the organisations they collaboratedwithovertheprevioussixmonths,andtodescribethenatureorpurposeofthecollaboration.Participantsselectedfromalistof32collaborationareasandwerealsoprovidedtheoptiontonametheirownareasofcollaboration.Thisreportconcentratesprimarilyonthefullcountrynetwork, which is the combination of all collaboration areas, but also looks at individualcollaborationareanetworkstopaintalargerpictureofcountrytrends.ThenetworkdatawereanalysedwithORA, a network analysis tool developed bycASOS atcarnegieMellon.ThestatisticalsoftwarepackageRwasusedfornetworkstatisticalsignificancetests.fullnetworkanalysisdetailsandfindingsareprovidedinthereportinAnnex12.

QUESTION 3.A: IS THE ‘COLLABORATIVE’ APPROACH OF MULTI-STAKEHOLDER PLATFORMS AN EFFECTIVE DELIVERY MECHANISM? QUESTION 3.C: WHAT HAVE BEEN THE MAIN PATTERNS OF COLLABORATION, AND THE BENEFITS ANDDISADVANTAGES OF INFORMAL VS. FORMAL COLLABORATION?

Despite the original vision, the DEPP programme focused more on consortia-based collaborative approaches rather than multi-stakeholder platforms at the national, regional and international levels. Nevertheless, collaborative approaches were consistently reported to be the most appropriate, preferred approach to deliver humanitarian capacity building and response activities, while some challenges in finding efficient processes for engagement, communication and coordination remained.

Multi-stakeholder Platforms

“Multi-stakeholderplatforms”comprisealargepartoftheoriginalDEPPprogrammetheoryofchangeandintheDEPPbusinesscaseweredescribedas“platformswithabroadmembershipe.g.,InGOs,nationalnGOs,localgovernment,nationalgovernmentacademia,privatesectorcompanies,theRedcrossRedcrescentMovement,Unagencies”whichifsuccessful,should“improvepolicyatnationalandinternationallevel”.Thebusinesscasedescribedimplementinga singleplatformper countryor regionwhere theDEPPprogramme is active aswell as aninternational platform. Possible products of the platforms that were envisioned included:synthesised lessons identified, joint evaluations and common position papers for advocacy.TheplatformswerehypothesisedtobethemechanismacrosstheDEPPthroughwhichpolicychange,evidencegenerationanddisseminationwouldoccur.

Despitethisemphasisonmulti-stakeholderplatforms,onlyafewprojectssuchasTransformingSurgecapacity andcDAc-nactually implemented this approach.TheDEPP focusedmuchmore on consortia- based collaborative models rather than on larger multi-stakeholderplatforms. Those projects that did employ platforms reported challenges in setting up andsustainingtheplatformsbutdidreportsomeorganisationalchangeasaresult.Theplatformsweremoreeffectiveincontextswhereexistinghumanitarianstructuresweremorematureandwithsupportivegovernments.ThecDAc-nfinalevaluationnotesthatthethreecountry-levelplatformsitestablishedorstrengthenedwere“relevanttotheircontextandaneffectivewayof implementing joint preparedness activities” and effective in creating new links betweenlocal,nationalandinternationalactors139.However,itwasnoted,thatastrongerlinktoglobalpolicyandmoreadvocacyeffortsshouldbeintegratedintotheseapproaches,andthatfurther,systematicinclusionoflocalandnationalactorsisneededtoensurecontextualisationandlocalownership.TransformingSurgecapacityestablishedtwonationalmulti-stakeholderplatforms,anAsiaregionalplatformandaglobalplatformforcollaborativeandlocalisedsurgemechanisms.TheTransformingSurgecapacityfinalevaluationfoundthatthenationalplatformsweremosteffectiveandthatregionalandinternationalplatformsshouldtakeamoresupportiveorenablingrolethatisdefinedbythenationalplatforms140.Itwasalsoreportedthatthenationalplatformsdidnotadequatelyreflectcountry-levelprioritiesastheyweredesignedatheadquarterslevel,andthatfutureplatformsshouldinvolveanationally-leddesignprocessandimplementation.

Consortia-based Collaborative Approaches

All14DEPPprojectswereimplementedbyconsortiaoforganisationsworkingtogether,withawide rangeof typesand sizesof consortiawithin theDEPP.DatacollectedamongDEPPorganisationsandkeystakeholderssuggeststhattheconsortiaapproachwasappropriateinthe

139 cDAc-nfinalEvaluationReport,p4.140 TransformingSurgecapacityfinalEvaluationReport,p8.


DEPPfocalcountrycontexts,andapreferredapproachtodeliverhumanitariancapacitybuildingactivities.Self-reporteddatafromconsortiamemberssuggestthatoverallconsortiafunctionedrelativelywell (seefigure5.1below),despite anumberof identifiedchallenges including inengagement,communicationandcoordination.Thesedatasuggestthatconsortiafunctionalityalsoimprovedbetweenthefirst(T1)andsecond(T2)datacollectionpoints.Therewerealsoseveral exampleswhere the consortia approach facilitated the delivery of capacity buildingactivitiesandalsoresponseefforts.forexample,thecasestudyinchapter8highlightshowtheconsortiumstructureoftheUrbanEarlyWarningEarlyActionprojectinKenya,includingtheestablishedproceduresandmanagementstructure,facilitatedatimelyresponsetoacholeraoutbreak.ThenextsectionwillfocusonexaminingthedegreetowhichcollaborationoccurredacrosstheDEPP,themainpatternsofcollaborationandthereportedchallengesandbenefitsofinformalandformalcollaboration.

0

10

20

30

40

50

60

70

80

T2T1T2T1T2T1T2T1T2T1T2T1

NOT WELL AT ALL

VERY WELL

WELL

SOMEWHAT WELL

NOT WELL

DEPP CONSORTIA FUNCTIONALITY

MiN SETKENYATHE PHiLiPPiNESETHiOPiAMYANMARFULL SAMPLE

figure5.2:DEPPconsortiafunctionalityattimepoint1(T1)andtimepoint2(T2)

Is Collaboration Occurring and What Types of Organisations are Involved?

Datafromthenetworkanalysis,in-depthintervieweesandintensivesetsurveyssuggestthatorganisationsinboththeDEPPandcomparisongroupscollaboratedwithoneanotherduringcapacitybuildingandhumanitarianpreparednessand responseactivities.furthermore,bothDEPPandcontrol grouporganisationsand staffstronglyvalued theapproachofworking inconsortiaandthroughnetworks.forexample,approximately94%ofrespondentsinbothDEPPandcontrolgroupintensivesetsurveysstatedthattheywouldapproachotherorganisationstocollaborateduringhumanitarianresponseactivities.SeveralchangesincollaborationpatternsbetweenT1andT2havebeenidentifiedfromthenetworkanalysisincluding:

• Increasedcollaborationfrequency(aproxyindicatoroftrustandqualityofrelationships)141 inKenyaandthePhilippines,whilebothindicatorsdecreasedinMyanmarandEthiopia

• Increasedcollaborationwithin theDEPPcohort inMyanmarand thePhilippines, butdifficultyengagingwithnon-DEPPactorsinbothsettings

• IncreasedcollaborationbetweenDEPPandnon-DEPPactorsinEthiopiaandKenya

Thesechangeswillbeexploredmorefullyinthesectiononstrengthenednetworkslaterinthischapter.

141 Whenanactorindicatesfrequentcollaboration,weassumethereishightrustandperceivedvalueincon-tinuedengagement.


What Have Been the Benefits and Disadvantages of Formal Versus Informal Collaboration?

both formal collaborations and informal collaborations occurredwithin DEPP projects, andinformalcollaborationsoccurredacrossDEPPprojectsandwithexternalstakeholders.formalcollaborationwasdefinedasanycollaborationinvolvingasignedcontractualagreementdefiningrolesandresponsibilitiesofcollaboratingentities.Informalcollaborationsweredefinedasthosecollaborations that occured more spontaneously and without formal agreements in place.Acrossallcountries,humanitarianstafffromDEPPorganisationsidentifiedtheadvantagesanddisadvantagesofformalandinformalcollaborations(seeTable5.1)

FORMAL COLLABORATION

Advantages % Disadvantages %

Sharingofresources 52.4Difficulttocollaborateeffectivelywhenorganisationshavedifferentpoliciesandprocesses

29.8

Improvednetworking 51.2 Slowandcumbersomedecision-makingprocesses 28.6

Exchangeofideas 48.4 Settingupaformalcollaborationisex-tremelytimeconsuming 20.2

Abilitytoaccessothersourcesoffund-ingwhenapplyingasaconsortium 40.5 Difficulttomanagedisagreements 20.2

Programmedeliveryanddesignmoreeffective 40.5 Unclearobjectives 13.1

INFORMAL COLLABORATION

Advantages % Disadvantages %

Exchangeofideas 48.8Difficulttocollaborateeffectivelywhenorganisationshavedifferentpoliciesandprocesses

28.8

Improvednetworking 46.4 Difficulttomanagedisagreements 18.9

Sharingofresources 41.7 Slowandcumbersomedecision-makingprocess 18.9

Informalcollaborationsleadtoformalcollaborations 25.0 Unclearobjectives 16.4

learningfromthecollaboration 21.4 Maytakealotoftimewithoutyieldinganyconcreteoutputs 9.4

Table5.1:AdvantagesanddisadvantagesofformalandinformalcollaborationRespondentscouldselectmorethanoneadvantageordisadvantageforeachtypeofcollaboration

Despite the identifiedchallenges, formal collaborationswerehighlyvalued,withadvantagessuchasnetworking,exchangeofideasandincreasedeffectivenessofprogrammedesignanddelivery. Ingeneral,many respondentshighlighted theutilityof informal collaborations, andemphasisedthatinformalcollaborationscanstimulatenewideasandprojects,whichultimatelycanturnintoformalcollaborations.PatternsofcollaborationbetweenlocalandinternationalactorsandbetweenDEPPandnon-DEPPactorswillbefurtherexploredinQuestion3.busingdatafromthenetworkanalysis.

As a delivery mechanism, collaborative and consortia-based approaches were particularlyeffectivewhenthefollowingconditionsweremet:

• Existenceofjointobjectives,valuesandcommonwaysofworking• collaborationswereorganicnotforced• Sufficient time, space, resources, capacity and will to foster healthy collaborative


relationships• Streamlined decision-making and contractual processes as well as fewer consortia

partners• buildingonexistingrelationships

QUESTION 3.B: FOCUSING ON COALITIONS, PARTNERSHIPS AND CONNECTEDNESS – WHAT CAN BE SAID ABOUT THE EFFECTS OF STRENGTHENED NETWORKS?

There was some evidence of strengthened humanitarian response networks in the Philippines, Kenya and Ethiopia, while the network in Myanmar remained unchanged. Not enough time passed between the data collection points in these four countries to sufficiently document and test the hypothesis that strengthened networks and greater collaboration lead to better preparedness.

ThissectionaimstoexaminewhethernetworksinfourDEPPoperationalcountrieshavebeenstrengthened,andanyeffectsofstrengthenednetworksinthesesettingsusingnetworkdatacollectedattwotimepointsinKenya,Ethiopia,MyanmarandthePhilippines.

Collaboration Areas

looking across all four countries, advocacy, community capacity building, and projectimplementationwereconsistentlyamongthemostactivecollaborationareasduringbothdatacollectionperiods(seeTables5.2and5.3).142

THE PHILIPPINES KENYA MYANMAR ETHIOPIA

Collaboration

Areas# of

LinksCollaboration

Areas# of

LinksCollaboration

Areas# of

LinksCollaboration

Areas# of

Links

1Advocacy 418 Advocacy 179

Communi-ty Capacity Building

29 Project Imple-mentation 207

2

Communi-ty Capacity Building

341Communi-ty Capacity Building

151 funding 20Communi-ty Capacity Building

181

3Project Imple-mentation 283 Project Imple-

mentation 123 Advocacy 14 ProjectDesign 163

4communityconnections 270 facilitation 118 Project Imple-

mentation 14 funding 153

5

communityPlanning 255 community

Planning 101 InformationSharing 11

communi-ty-basedRiskAnalysis

147

6 facilitation 249 ProjectDesign 96 WaSH 10 WaSH 139

7


234 communityconnections 95 Education 10 ProposalWrit-

ing 131

8Dataresources 223

conflictMiti-gationExper-tise

94 communityPlanning 9

climatechangeandAdaptation

110

9Education 220


90EarlyWarningSystemsEx-pertise


106

10


208communi-ty-basedRiskAnalysis

90climatechangeandAdaptation

7 facilitation 91

142 notethatforEthiopia,advocacywasranked18thinthefirstdatacollectionand13thintheseconddatacollection.


Table5.2:Top10collaborationareasbycountryatfirsttimepoint(T1)

THE PHILIPPINES KENYA MYANMAR ETHIOPIA

Collaboration

Areas# of

LinksCollaboration

Areas# of

LinksCollaboration

Areas# of

LinksCollaboration

Areas# of

Links

1 Advocacy 271 Community Ca-pacity Building 612 Project Imple-

mentation 26 Project Imple-mentation 298

2Project Imple-mentation 202 Advocacy 512 Community Ca-

pacity Building 25 ProjectDesign 255

3

Community Capacity Build-ing

194EarlyWarningSystemsExper-tise

409 Advocacy 17 ProposalWrit-ing 237

4communityPlanning 121 community

connections 396 communityPlanning 15 funding 229

5

communityconnections 108


387 funding 14Communi-ty Capacity Building

213

6 funding 105 facilitation 372 ProjectDesign 12 WaSH 205

7facilitation 101 community

Planning 349EarlyWarningSystemsExper-tise


198

8

DataResourcesincludingdatasets,collectionand analysis

98 ProjectImple-mentation 345 climatechange

andAdaptation 9 facilitation 167

9

VulnerableGroups 97 conflictMitiga-

tionExpertise 311communi-ty-basedRiskAnalysis

9 VulnerableGroups 155

ProjectDesign 96 climatechangeandAdaptation 284 ProposalWrit-

ing 8communi-ty-basedRiskAnalysis

151

Table5.3:Top10collaborationareasbycountryatsecondtimepoint(T2)

In the Philippines, the top three collaboration areas (advocacy, project implementation andcommunity capacity building) remained consistent at both data collection points, thoughtheorderdiffered.Anotable change in this countrywas the jump innumberofties in thefundingnetwork,anditsjumpfromrank11torank6betweenthetwotimeperiods.Thetopcollaborationareasinthiscountrywerealsothemostdistributednetworks143.

InKenya,twoofthetopthreecollaborationareasremainedthesame–communitycapacitybuildingandadvocacy.EarlywarningsystemsexpertisewasthethirdmostpopularcollaborationareaatT2(upfromrank11inT1).InMyanmar,thefundingnetworkdroppeddownthelist,andthecommunityplanningnetworkmovedupthelist.Otherwise,thetopfourcollaborationareasinthisnetworkremainedthesamebutwithadifferentorder.InEthiopia,projectimplementationandprojectdesignwereinthetopthreeatbothT1andT2,butproposalwritingjumpedfromrank7atT1torank3atT2.Topcollaborationareaswerealsothemostdistributednetworks.

143 A distributed networkisonewithmanylinksbetweenmultipleactorsinthenetwork,allowingformoreequalflowofinformationtoallactorswithinthenetwork.


Part 1: Have the Networks in the DEPP Intensive set Countries Been Strengthened?

The DEPP theory of change is built around the premise that strengthened networks andincreasedcollaborationarecriticaltoimprovingpreparednessandresponseandshiftingtoamorelocalisedapproach.Thenetworkanalysisenabledassessmentofchangesinthehumanitarianpreparednessandresponsenetworksovertimeandprovidesevidenceofstrengthandqualityofcollaborationsbetweenvariousinstitutions.Specifically,thissectionwillexaminechangesinthenumberoflinksbetweenorganisations,thefrequencyofcollaborationaswellasthequalityofcollaborativerelationships.

A. Organisational Links

The charts below summarise the relationship data thatwas collected at both timeperiods.Theaveragenumberofcollaborationareasperuniquelinkcaptureshowmanycollaborationareasonaveragebetweentwoorganisations.Theaveragenumberoftotallinkspersurveyedorganisationrepresentsthecontributionofeachorganisationtothefullnetwork.finally,theaveragenumberofuniquelinkspersurveyedorganisationrepresentstheaveragenumberoforganisationstowhomthatorganisationistied.Overall,theDEPPcohortacrossallcountriesrepresentedonlyasmallproportionoftheentirehumanitariannetworksmappedaspartofthisanalysis(between9and24%).

Country Time Point

ORGANISATIONS LINKS

Tota

l #

Surv

eyed

Tota

l #

Iden

tified

Tota

l # o

f DEP

P O

rgan

isatio

ns

Iden

tified

Tota

l # o

f Lin

ks

# of

Uni

que

Link

s

Aver

age

# of

Co

llabo

ratio

n A

reas

per

Lin

k

Aver

age

# of

To

tal L

inks

per

O

rgan

isatio

n

Aver

age

# of

U

niqu

e Lin

ks p

er

Org

anisa

tion

The Philippines

T1 72 401 36(9%) 5622 679 8.3 78.3 9.4

T2 92 254 33(13%) 2754 489 5.6 29.9 5.3

KenyaT1 73 161 30

(19%) 2109 315 6.7 28.9 4.3

T2 142 317 28(9%) 8027 807 9.9 56.5 5.7

MyanmarT1 33 75 17

(23%) 207 103 2 6.3 3.1

T2 30 55 13(24%) 226 76 2.3 7.5 2.5

EthiopiaT1 63 132 22

(17%) 2438 329 7.4 38.7 5.2

T2 59 137 26 (19%) 4150 434 9.6 70.3 7.4

Table5.4:Organisationsandlinksbycountry

InthePhilippines,therewasanincreaseinboththenumberoforganisationssurveyedandthenumberofuniqueorganisationsonthemap,thusoverallalargernetworksize.However,therewasasubstantialreductioninthenumberofrelationshipsbetweenorganisations(theaveragenumber of links per organisation decreased from9.4 to 5.3),which suggests a decrease innetworkdensity.However,thisfindingislikelyduetobiasasaresultofthelearningeffectandsurveyfatigue.DuringthePhilippinesresultsvalidationworkshop,respondentsindicatedthatbasedontheirexperiencewiththefirstroundofnetworkdatacollection,theywereawarethatinterviewwouldtakelongeriftheyidentifiedmorecollaboratingorganisations,thustheyonlylistedmaincollaborators.


InKenya,thenumberoflinkswithinthenetworkalmostquadrupledsignifyingalargegrowthduring the time period. In addition to twice as many surveyed organisations at T2, eachorganisation also reported relationships that spanned across more collaboration areas andmoreorganisations.ValidationworkshopparticipantsinKenyasuggestedtheyhadintensifiedcollaboration efforts, which might explain the observed network growth. The Myanmarcountrynetworkremainedthesameintermsofoverallsize,thoughitisimportanttohighlightit is substantially smaller than the three other country networks. In addition, in Myanmar,organisations reported relationships with fewer organisations at T2, though relationshipsspannedmorecollaborationareassignifyingincreasednetworkdensity.

ThedatafromEthiopiashowedaslightdecreaseinthenumberoforganisationscompletingthesurveybutasmallincreaseinnumberoforganisationsinthemap.Inaddition,therewerealsoalmosttwiceasmanylinksorconnectionsbetweenorganisationsatT2–someoftheseweredueto relationshipswithmoreorganisations,butsomeweredueto increasedcollaborationacrossmorecollaborationareaswiththesameorganisations.

B. Quality of Collaboration

Twoindicatorswereusedtoassessthequalityofrelationshipswithinthenetwork.Thefirstwasfrequencyofcollaborationwitheachactoridentified(Seetable5.5)andthesecondwaslikelihoodofrecommendingthatorganisationtoothersonascaleofoneto10(alsoknownasnetpromoterscale) (See table5.6).frequentcollaboration (more thansixtimesover theprevioussixmonths)suggestsahighleveloftrustandperceivedvalueincontinuedengagement.Whenanactorindicatesahighlikelihood(scoreof9to10)ofrecommendingtheotheractor,itsuggestshightrustandreverence.Sincethismeasurecapturesbothperceptionsandactions,thismetricbetterassessestruefeelingsoneactorhasofanother.Thenetpromoterscale isadjustedforbiasbycategorisingonlythehighestscoresaslikelytotakeaction;theothersarelikelytobepassiveordetractors.144

Collaboration Frequency over Last 6

Months

THE PHILIPPINES* KENYA ETHIOPIA MYANMAR*

T1 (N=5622)

T2 (N=2754)

T1 (N=2109)

T2 (N=8027)

T1 (N=2438)

T2 (N=4150)

T1 (N=207)

T2 (N=226)

Collaborate Rarely (1-2 times)

558 (10%)

198 (7%)

95 (5%)

100 (1%)

91 (4%)

297 (7%)

18 (9%)

26 (12%)

Collaborate Often (3-4 times)

1371(24%)

449(16%)

596(28%)

889 (11%)

484(20%)

815(20%)

43 (21%)

52 (23%)

Collaborate Frequently (>6 times)

3693(66%)

2107(77%)

1418(67%)

7038(88%)

1863(76%)

3038(73%)

146(70%)

148(65%)

Table5.5:frequencyofcollaborationwithinthepast6monthsbycountry*2X2 Chi-square Test indicates significant difference between T1 and T2 at the p<0.05 level

Inallcountriesacrossbothtimeperiods, theactors interactedwithoneanother frequently.InMyanmarandEthiopiafrequencyofcollaborationdecreasedbetweenthetwotimepoints.Kenyasawthelargestincreaseinfrequencyofinteractionwithanincreaseof21%oftiesthatreportedfrequentcollaboration.fortraceablerelationships,therewasastatisticallysignificantchangeinthefrequencyofinteractionbetweenorganisationsinthePhilippines(p=0.009)andinMyanmar (p=0.028). However, the difference in the change for DEPP versus non-DEPPcomparisongroupswasnot significant in either country.This suggests thatbothDEPPandnon-DEPPorganisations’levelsofcollaborationchangedinsimilarwaysovertime.

144 Thescaleisbasedonbain’snetPromoterscale.furtherdetailsareavailablehere:http://www.netpromot-ersystem.com/about/measuring-your-net-promoter-score.aspx


Likely to Recommend

Partner (scale of 1-10)

THE PHILIPPINES* KENYA* + ETHIOPIA* + MYANMAR

T1 (N=5622)

T2 (N=2754)

T1 (N=2109)

T2 (N=8027)

T1 (N=2438)

T2 (N=4150)

T1 (N=207)

T2 (N=226)

Not Likely (Score 1-6)

812 (14%)

258 (9%)

310(15%)

735 (9%)

266 (11%)

503(12%)

91 (44%)

125(55%)

Somewhat Likely (Score 7-8)

2189(39%)

1028(37%)

814(39%)

2233(28%)

902(37%)

1555(37%)

74 (36%)

63 (28%)

Very Likely (Score 9-10)

2621 (47%)

1468(53%)

984(47%)

5059(63%)

1270(52%)

2092(50%)

41 (20%)

38 (17%)

Table5.6:likelihoodtorecommendorganisationstoothers(netpromoterscale)bycountry*2X2 Chi-square Test indicates significant difference between T1 and T2 at the p<0.05 level + Statistically significant difference (p<0.05) between DEPP and non-DEPP organisations in this country

Myanmarwastheonlycountrytoshifttowardsbeinglesslikelytorecommendothersbetweentimeperiodoneandtwo.Overhalfofitsrelationshipswerereportedasnotlikelytorecommendtheotheractorbythesecondtimepoint.Inallothercountries,overhalfoftheirrelationshipswerereportedasverylikelytorecommendtheotheractor.Aswithfrequencyofinteraction,Kenyasawthelargestincreaseinlikelihoodtorecommendotherswithanincreaseof16%oflinksreportedasverylikelytorecommend.ThisagainsignifiesstrengthenedrelationshipsandislikelyduetotheincreasedcollaborationandrelationshipbuildingconductedbyKenyaDEPPprojects.

forallcountriesexceptMyanmar,theincreaseinlikelihoodtorecommendotherswasstatisticallysignificant.145InbothKenyaandEthiopia,thechangeinlikelihoodtorecommendotherswithintheDEPPcohortwassignificantlyhigherthanthechangeinthenon-DEPPcohort,providingevidencethatDEPPhascontributedtoincreasedqualityofrelationshipsinthesesettings.

Taken together, thesedata show thatnetworks inEthiopia,Kenyaand thePhilippineshavebecomestrengthenedandingeneralqualityofrelationshipshasimprovedacrossthenetworks.ThereisevidenceinKenyaandEthiopiathatDEPPhascontributedtostrengthenedrelationshipquality.

Part 2: Is there Evidence of Improved Localisation?Overall, there was no quantitative evidence of increased localisation (no increase in proportion of relationships with local actors between T1 and T2) in any intensive country within the network data. In fact, the Myanmar network became even more dominated by INGOs by T2. However qualitative data suggests changes in attitudes around localisation are occurring (see CH 7).

A. Patterns of Collaboration Between National and Local NGOs

The DEPP has emphasised localisation and promoted a more locally-driven approach toemergencypreparednessandresponse.Theevaluationassessedpatternsofcollaborationwithnational and localorganisations, andwhether thosepatterns changedovertime.figure5.3presentsdatafromtheintensivesetdatacollectiononself-reportedchangeinthenumberofpartnershipswith localornationalorganisations.As illustrated, approximately70%ofDEPPorganisationsreportedanincreaseinpartnershipswithl/nnGOsatT1comparedwith65%atT2,representingasmall,non-significantdecrease.Onlyabout50%ofnon-DEPPcomparisonorganisations reported an increase in partnershipswith l/nnGOs atT1 but this increasedto59%atT2.Thesedata suggest no significant self-reported increase in partnershipswithl/nnGOs in either group. However, as discussed earlier, perceptions may be biased andpotentially inaccurate.Aspartof thenetworkanalysis,partnershipsandcollaborationsweredirectlymeasuredtomoreobjectivelyassessthenumberofrelationshipswithl/nnGOsandwhetherthesehavesignificantlychangedovertimeineachofthefourintensivesetcountries.

145 ThePhilippines:p<0.001;Kenya:p=0.047;Ethiopia:p=0.004.

7


figure5.3:ReportedchangeinnumberofpartnershipswithlocalandnationalnGOs

figures5.4and5.5comparetheoverallnetworkstructuresinthePhilippines,Kenya,EthiopiaandMyanmaratT1andatT2.Inthesefigures,eachorganisationisrepresentedbyanode(diamondsrepresentDEPPorganisations,andcirclesrepresentnon-DEPPorganisations),whileactivecollaborations146betweenorganisationsarerepresentedbythelinesconnectingthenodes(orlinks).Arrowsrepresentthedirectionofthecollaboration.Thesizeofthenodeisakeyelementofthesefiguresasitrepresentsthetotaldegreecentrality,orthenumberoflinksonenodehastoothernodes,whichisameasureofinfluence.Themorelinksorconnectionsanorganisationhas,thelargerthesizeofthenodeinthenetwork.Given the DEPP theory of change, a distributed network where there is greater cohesion and links between multiple actors in the network, more equal flow of information to all actors in the network, and minimal bottlenecks, promotes sustainability and is more desirable than a centralised and isolated network.Whenweexaminedallthenetworksinallcollaborationareas,inallintensivesetcountriesexceptMyanmar,thetopthreecollaborationareaslistedinTable5.2and5.3werethemostdistributednetworks.Table5.7complementsthecountrynetworkvisualisationsbypresentingthepercentageofrelationshipsortiesthattarget147l/nnGOsfromvarioussourceorganisations(InGO,l/nnGOorall).Tiestargetinglocalactorsrepresentlocalisationofthehumanitarianresponsewithinthecountry.Iflocalisationincreasedbetweenthetwotimepoints,weexpecttheshareoflinkstol/nnGOsfromanysourcetoincrease.

146 Activecollaborationsarethosethathaveinvolvedinteractionwithinthelast6months.147 Thesourcegroupisthegroupfromwhichthetieiscoming,andthetargetgroupisthegrouptowhichthe

tieisgoing.

0

10

20

30

40

50

60

70

80

MORE

THE SAME

FEWERPARTNERSHIPS WITH L/NNGOS

DEPP

T2T1T2T1

COMPARiSON

7.1

23.8

69.1

8.8

26.4

64.8

5.0

45.050.0

11.1

30.2

58.7


THE PHILIPPINES T1 THE PHILIPPINES T2

KENYA T1 KENYA T2

figure 5.4: network visualisations for the Philippines andKenyaattimepoint1andtimepoint2


MYANMAR T1 MYANMAR T2

ETHIOPIA T1 ETHIOPIA T2

figure5.5:networkvisualisationsforEthiopiaandMyanmarattimepoint1andtimepoint2


Data suggest that networks in the Philippines were highly locally driven, with more than 70% of links occurring with L/NNGOs (Figure 5.4 and Table 5.7). Thisisconsistentwithothersourcesthatsuggestthatcivilsocietyisdeeplyentrenchedinthiscontextandconsideredtobeoneof thestrongest inAsia148.However, therewasnochange intheproportionof linksto l/nnGOs betweenT1 andT2, suggesting no change in the level of localisation. In the Kenyan humanitarian preparedness and response network, local actors were active, but the relationships evenly targeted international and local actors(figure5.4andTable5.7).TherewasnochangeinthisdistributionbetweenT1andT2suggestingnoincreaseinprominenceoflocalactorswithinthenetwork.In Ethiopia as well, there was no change over time(figure5.5andTable5.7).However,unlikethePhilippinesandKenya,the Ethiopian network has high dominance of the INGOs with little input from local organisations. finally, thenetwork in Myanmar was substantially different from the other country networks. It was significantly smaller, and highly isolated (figure 5.5 andTable 5.7).This networkmoved fromone thathadanevensplitofrelationshipsbetweeninternationalandlocalactors,toonedominatedbyInGOs.ThismaybeduetothepoliticalsituationinMyanmar.DuringthevalidationworkshopinMyanmar, informedstakeholdersreportedthat thegovernment increasedrestrictionsandthishadaffectednGOregistration.Inaddition,localorganisationsmightbecoordinatingandcollaboratingmoreduringtimesofdisasterandresponsebutnotinbetween.

TARGET ORGANISATION: L/NNGO

Source Organisation

THE PHILIPPINES KENYA* ETHIOPIA* MYANMAR*

T1 T2 T1 T2 T1 T2 T1 T2INGO 1056

(67%)580(65%)

422(50%)

1658(47%)

411(27%)

837(29%)

41 (39%)

72 (47%)

L/NNGO 2893(70%)

1259(68%)

659(52%)

2144(48%)

202(21%)

213(17%)

39 (39%)

12 (17%)

All Organisations

3949(70%)

1839(67%)

1081(51%)

3802(47%)

613(25%)

1050(25%)

80 (39%)

84 (37%)

Table5.7:numberandpercentagesoflinkstargetingl/nnGOsfromvarioustypesofsourceorganisationsNote: The source group is the group from which the tie is coming, and the target group is the group to which the tie is going.*2X 2 Chi-square Test indicates significant at the p<0.01 level

B. Patterns of Collaboration Between DEPP and non-DEPP Cohorts

InordertounderstandcollaborationwithinoracrossDEPPandnon-DEPPgroups,theanalysisalsoexaminedthetiesorlinksthattargetoraredirectedtowardsDEPPgroupsfromvarioussources (see Table 5.8). In the Philippines, collaborationwithin the DEPP cohort increased(DEPPtoDEPPconnectionsincreasedfrom17%atT1to29%atT2).However,overallwhilebothDEPPandnon-DEPPgroupsmostlytargetednon-DEPPorganisationsatbothtimes,itwasanongoingchallengetogetthenon-DEPPcohortinvolvedandreachingouttoDEPPactors.non-DEPPactorswerenomorelikelytoreachouttoDEPPorganisationsinT2thantheywereinT1(non-DEPPtoDEPPtiesatT2was20%).

148 USAID’s CSO Sustainability Index ranks CSOs in the Philippines as the highest across 7 dimensions: legal environment, organisational capacity, financial viability, advocacy, service provision, infrastructure, and public image.


TARGET ORGANISATION: DEPP

Source Organisation

THE PHILIPPINES* KENYA ETHIOPIA* MYANMAR*

T1 T2 T1 T2 T1 T2 T1 T2

DEPP 334(17%)

379(29%)

460(59%)

1773(51%)

548(48%)

709 (41%)

42(40%)

78(75%)

Non-DEPP 562(15%)

287(20%)

470(35%)

1620(36%)

368(28%)

856 (35%)

19(19%)

8 (7%)

All Organisations

5622(16%)

666 (27%)

930(44%)

3802(47%)

916(38%)

1565(38%)

61 (30%)

86 (38%)

Table5.8:numberandpercentagesoflinkstargetingDEPPorganisationsfromvarioustypesofsourceorganisationsNote: The source group is the group from which the tie is coming, and the target group is the group to which the tie is going.*2X 2 Chi-square Test indicates significant at the p<0.01 level

Kenya’sfullcountrynetworkwasmadeupofmanyrelationshipsfromnon-DEPPactors,thoughthisdeclinedattheseconddatacollectionpoint.AtT2,theabsolutenumberofrelationshipsfromDEPPactorswasapproximately3.9timeshigherthanatT1,whereasabsolutenumberofrelationshipsfromnon-DEPPactorsatT2wasabout3.5timesthatofT1.Theselargeincreasesagain highlight the substantial overall network growth that occurred in Kenya. In terms ofcollaboration in Kenya, a slight decline in the percentage of relationships between DEPPorganisationsoccurredfromT1toT2 (from59%to51%).Also, therewasnochange inthepercentageofnon-DEPPtoDEPPlinks.However,thesedatashowhighlevelsofcollaborationbetweennon-DEPPandDEPPactors,with about35%ofnon-DEPP relationships targetingDEPPactors.KenyaisonesettingwheretheDEPPcohortseemstohavebeenabletoformrelationshipswithactorsoutsideofitsgroup,despiteasmallcohortsizerelativetotheotheractorsinthenetwork.

InMyanmar,ontheotherhand,theDEPPcohortwasnotverysuccessful inreachingthoseactors outside of the DEPP programme. In fact, non-DEPP actors actually decreased theircollaborationwithDEPPactorsfrom19%to9%.collaborationbetweenDEPPorganisationsincreasedby35%fromT1toT2with40%ofDEPPlinkstargetingotherDEPPorganisationsatT1to75%atT2.“Exclusivity,”oronlyengagingwithone’sowngroup,seemstobealargefactorinthiscountry,thoughithastheleastdisparitybetweengroupsizesofanycountrywithaboutone-quarterofallactorsinthenetworkintheDEPPcohort.

In Ethiopia, a fairly even split of network relationships came fromDEPP versus non-DEPPactors.Thenon-DEPPgroupbecameslightlylessexclusiveafterthesecondtimepoint,asitformedmorerelationshipswiththeDEPPgroup.comparedtoothercountries,thiscountry’sDEPPcohortwasfairlysuccessful(alongwithKenya)informingrelationshipswithnon-DEPPactors,as35%ofnon-DEPPtiestargetedtheDEPPgroup.

C. Influence of DEPP Actors Within the Networks

Table5.9 presents the top50organisations in each country assessed through total degreecentrality,ametricofthetotalnumberoftiesofeachorganisation.notethatthismetricdoesnottakeintoaccountthedirectionorstrengthofindividualties,ratheronlythetotalnumber.Thetableshowstheproportionofthetop50organisationsthatbelongtotheDEPPcohort,aswell as the share that are national/local compared to international. for example, in thePhilippinesatT1,14ofthetop50organisationsareDEPP(28%),whiletheremaining36(72%)werenon-DEPPorganisationsand35ofthetop50 (70%)werenational/localorganisationscomparedtotheremaining15(30%)whichwereinternational.


T1 T2

Country National / Local International All

OrganisationsNational /

Local International All Organisations

THE PHILIPPINES

DEPP 9(18%) 5(10%) 14(28%) 14(28%) 6(12%) 20(40%)

non-DEPP 26(52%) 10(20%) 36(72%) 22(44%) 8(16%) 30(60%)All Organisations 35(70%) 15(30%) 50 36(72%) 14(28%) 50

KENYA DEPP 8(16%) 11(22%) 14(28%) 6(12%) 13(26%) 19(38%)non-DEPP 12(24%) 19(38%) 36(72%) 19(38%) 12(24%) 31(62%)All Organisa-tions 20(40%) 30(60%) 50 25(50%) 25(50%) 50

ETHIOPIA DEPP 6(12%) 13(26%) 19(38%) 9(18%) 10(20%) 19(38%)non-DEPP 13(26%) 18(36%) 31(62%) 10(20%) 21(42%) 31(62%)All Organisations 19(38%) 31(62%) 50 19(38%) 31(62%) 50

MYANMAR DEPP 8(16%) 8(16%) 16(32%) 7(14%) 6(12%) 13(26%)non-DEPP 16(32%) 18(36%) 34(68%) 13(26%) 24(48%) 37(74%)All Organisations 24(48%) 26(52%) 50 20(40%) 30(60%) 50%

Table5.9:Top50influentialorganisationsbyDEPPversusnon-DEPPandinternationalversusnational/localNote: Influence is measured by total degree centrality (the total number of ties of the organisation). Statistical tests to determine statistically significant change between DEPP versus non-DEPP and International versus national/local were conducted but no significant differences were found at the p<.0.05 level.

comparingT1toT2inthePhilippinesshowsthatthetop50organisationsremainedlargelylocal(around70%ofactorsatbothtimepoints),buttherewasa12-percentagepointincreaseintheproportionoftopactorsthatwerefromtheDEPPcohort(from28%to40%).Inaddition,a 10- percentage point increase in the proportion of the top 50 organisations that werespecificallyDEPPnationalorlocalorganisationisevident(from18%to28%).Thischangewasstatisticallysignificant(p=0.046).InKenya,theproportionofnationalorlocalorganisationsinthetop50increasedfrom40%to50%byT2,representinganevensplitbetweeninternationalandnational/local.TherewasnochangeinthepercentageofDEPPorganisationsinthetop50organisationsovertime,oroverallinthetopactorsbetweenT1andT2(p=0.066).WhiletheoverallpercentageofDEPPorganisationswithinthetop50inEthiopiadidnotchange,thedistributionofnationalandinternationalDEPPorganisationsshifted.AtT112%ofthetop50wereDEPPnationalandthisincreasedto18%byT2.Therewassignificant(p=0.016forT1,p=0.005forT2)prominenceofinternationalnon-DEPPactorsinthetop50organisationsforbothtimeperiods.InMyanmar,therewasadecreasefrom32%to26%inthepercentageofDEPPactorsinthetop50.Theprominenceofnon-DEPPinternationalorganisationsisevident,asalmosthalfofthetop50actorsfallintothiscategory.


D. Do Strengthened Networks and Greater Collaboration Lead to Improved Preparedness?

Toexaminethequestionofwhetherstrengthenednetworksandgreatercollaborationleadtoimprovedpreparedness,theanalysesassessedchangeinnetworkingtogetherwithorganisationalpreparednessovertime.firsta‘networkingscore’forDEPPandnon-DEPPcohortswascreated,combiningin-degree,out-degree,betweennessandeigenvectorcentralitymeasures.TheT1andT2averagenetworkingscoreswereassessedusingapairedt-testtoanalysewhethertherehadbeenanysignificantchangeinacrosstimeperiods.ThechangeinnetworkingscoreswerealsocomparedbetweenDEPPandnon-DEPPgroupsusingt-testsforindependentsamples.next,correlationsbetween thenetworking score and the aggregatedorganisational preparednessscore149acrosssevendomains(seechapter4)wereassessed.Thiswascomparedwithadisasterpreparednessscoreforbothgroups.TheonlycountrywithanysignificantchangeinnetworkingscorewasEthiopia,wheretherewasasignificantdecrease(p=0.015).nocountrywasfoundtohaveasignificantcorrelationbetweenchangeinnetworkingscoresandchangeindisasterpreparednessscores.

However,notenoughtimepassedbetweenT1andT2tosufficientlydocumentandtestthehypothesis that strengthened networks and greater collaboration leads to better disasterpreparednessandultimatelyresponse.In2016,DEPPfieldoperationswerestilladaptingtolocalfeedbackandinterventionswerestillevolving.DatacollectioninT1inevitablypickedupnetworkeffectscausedbyshiftingstrategiesthatarecommontofirstyearprojectimplementation.T2more likelycapturedsomeof thedistinctiveprogramme impactsofDEPP interventions,butsufficienttimetofollownetworkchangerequiresatleastanadditional12months,ifnotmore.

QUESTION 3.D: WHAT UNIQUE CONTRIBUTION DID COLLABORATIVE RELATIONSHIPS AND ‘MULTI-STAKEHOLDER PLATFORMS’ MAKE TOWARD DEEPENING CROSS-PROGRAMME LEARNING? Cross-project collaboration contributed to improved sharing of learning and evidence at the programme level but this did not necessarily translate to behaviour change.As described in chapter 4, there was emphasis on enhancing cross-programme learningthrough a number of approaches led primarily by the learning Project. This included thedeploymentoftwoRegionallearningAdvisors,oneinEthiopiatocovertheAfricaregion,andtheotherinPakistantocovertheAsiaregion.Asthesewerereportedtobehelpfulintermsof stimulatingcollaborationandcross-programme learning, theevaluation’s formativephasereportrecommendedtoadditionallyembedlearningAdvisorsatthecountryleveltofurtheramplifycollaborationandlearning.Asreportedintheinterimreport,countrylearningAdvisorswere assigned to the Philippines, Kenya, DRc and bangladesh, in addition to Ethiopia andPakistan.countrylearningAdvisorsappearedtohavecontributedtoincreasedcollaboration.One intervieweenoted: “Initially,whenwe [didnothavea]DEPP learningadvisorhere,wewereworkingdifferently…. therewas…astandardapproach.butafterherarrival,westartedcollaboratingwitheachotherandtooktheopportunitiesformbenefitingfromeachother…wesharedlearningswitheachotherandincreasedoutreach.”

149 Organisationalpreparednessscore:Organisationalchecklistconductedwitheachorganisationandassess-ingsevendomains:1-Emergencypreparednessplans,2-humanresourcesandcapacity,3-logisticsandsupplies,4-legal,documentationandpolicy,5-Organisationandsystemprocesses,6-Emergencyresponseteamandstaffing,7-communicationandmedia.Aggregatescoreoutof100generatedforeachorganisa-tion.


figure5.6:UsefulnessoflearningProjectatT2

QuantitativedatafromKAPsurveys(figure5.6)showthatatT2,59%ofintervieweesreportedthatthelearningProjectwaseitherveryusefulorextremelyusefulinrelationtotheinterviewees’work.Approximately11%reported that thelearningProjectwaseithernotuseful at all,orslightlyuseful.Intermsofevidencesharing,89%ofrespondentsreportedsharingofevidencewithintheirproject,and72%reportedsharingofevidenceacrossprojectswhichsuggestshighlevelsofsharingbothwithinconsortiaandattheprogrammelevel.Thisprovidesevidenceofcoursecorrection,ascross-projectsharingwasreportedlylowerearlierintheimplementationoftheDEPP.

Qualitative interview data also supports the fact that collaboration and evidence sharingbetweenandacrosstheDEPPhasincreased.ThiswascatalysedtosomedegreebythelearningconferencesinKenya(December2016)andthePhilippines(July2017).botheventsbroughttogetherlargenumbersoflocalpartnersandstakeholdersatthecountrylevelovermultipledaystoshareprojectlearningsandchallenges,andstimulatecollaborationandsharing.TheintentionwastoshifttheactionfromtheUKtothecountry levelbymaximisingengagementof localstakeholders.At theconference inKenya,102DEPPandnon-DEPPstakeholders attended,representing13 countries,while at the conference in thePhilippines, 127people attendedfrom9countries,whichisinlinewiththeobjectiveofincludingadiverserangeofpartners.Thelearningconferenceswereperceivedbyintervieweestobeeffectiveinsharinglearning,creatingbridgesacrossprojectsandimprovingprogrammecohesion.forexample,onerespondentstated:

“[The]Internationallearningconference[is]wherewesharedourachievementsandbarriers…attheendoftheconference,therewere6-7projectswhoagreedtoworkwithADcAP…afterthat,someofthemevenfollowedupwithmehere.So,workingintheDEPPasawholehasadifferentkindofimpact,becauseithelpsusshareourlearningwithothercolleaguesaswellandhelpsusincreaseouroutreach.”

Another interestingoutcomeoftheseconferenceshasbeencross-projectandcross-countryexchangevisits to learn directly fromother projects and in different contexts. for example,MyanmarprojectstaffvisitedthelinkingPreparednessResilienceandResponseprojectinKenyafollowingthenairobiconference,andthePhilippinesconferenceagendaintentionallyincludedprojectsitevisits.Inaddition,UrbanEarlyWarningEarlyActionprojectstakeholdersdescribedplanstovisitaprojectinnepal,andthebeginningofacollaborationwiththeAlERTproject.nepalisoneofseveralnon-DEPPcountriesinwhichDEPPprojectsareimplementingactivities.Stafffromthebangladeshcommunicatingwithcommunities(cWc)workinggroupcompletedan exchangevisit to thePhilippinescWcgroup and this has led to on-going collaborationbetween the twoworkinggroups.Therehavealsobeen reportsof theTalentDevelopmentprojectcollaboratingwiththeProtectioninPracticeproject,andofmanyprojectsworkingwithorincludingAgeandDisabilitycapacityProgramme’sMinimumStandardsforAgeandDisabilityInclusion.

While therewasevidencethatcollaborativerelationshipscontributedtosharingof learning,therewaslessevidenceonhowlearningwasutilisedandwhetherthistranslatedintoactualbehaviourchange.

7%

42%

N = 154

USEFULNESS OF LEARNING PROJECT4%

30%

17%

NOT USEFUL AT ALL

SLiGHTLY USEFUL

SOMEWHAT USEFUL

VERY USEFUL

EXTREMELY USEFUL

147 SUMMATIVEPHASEREPORT-EffIcIEncyAnDVAlUEfORMOnEy

EFFICIENCY AND VALUE FOR MONEY

cHAPTER6


Evaluation Question #4: How economically have resources/ inputs (funds, expertise, time etc.) been converted to results? To what extent does preparedness improve the efficiency of humanitarian response?

Sub-questions:

a) Haveresources(funds,humanresources,time,expertise,etc.)beenallocatedstrategicallytoachievetheprogrammeobjectives?

b) Haveresourcesbeenusedefficiently?Ingeneral,dotheresultsachievedjustifythecosts?couldthesameresultsbeattainedwithfewerresources?

c) Haveprogrammefundsandactivitiesbeendeliveredinatimelymanner?

Data sources:

• Organisational surveys (Quantitative)

• Intensive set IDIs(Qualitative)

• Document review including financial information(Qualitative)

• IDIs on VFM(Qualitative)

Key Findings:

• focusingoneconomyandefficiency,VFM shows good potential for the programme with areas for improvement

• The collaborative model necessarily lends itself to a degree of inefficiency, high transactions costs and slow information flows. This model had substantial benefitshowever,and for some projects the benefits have likely outweighed the costs.forotherprojects thesamebenefitsof themodelcouldhavebeenachievedat lowercostandgreaterefficiencyifthecontracting,M&Esystemandcostingofactivitieshadbeenmoreconsciouslyplannedfrominception.Thecollaborativemodelthatwastiedtohierarchicallongdeliverychainsandmultipletiersofcontractsandfundingflowlikelycompromisedeconomyandefficiencytoadegree.

• Goodindicatorsofcosteconomywereevident,thoughsome budgets were lean with insufficient resources for portfolio management and collaboration activities.

• There were shortfalls in terms of systems and resources set up for efficiency in governance and strategy, portfolio management, decision-making and consortium arrangements.The spendingonManagement functionsat theprogramme levelweretooeconomicalandwould have benefitted from some funds and time being redeployed from the Learning Project to the DEPP management team in the form of M&E expertise, bothattheprogrammeandprojectlevels.If,however,independencewasdesired,thenmoredirectlinkageandcommunicationbetweenthemanagementteamandthelearningProjectteamwherebydatafromM&Eflowsregularlyanddirectlytothemanagementteamthroughbothautomaticreportingordirectdataaccessthroughadashboardfollowedbyregularlybilateralmeetingscouldhavebeenconsidered.

• Positive findings in terms of VFM reporting, adaptive management and collaborative waysofworkingwereidentified.

• Thereare no strong quantitative VFM findings to suggest that preparedness improved the efficiency of humanitarian response. TheempiricaldatafoundnosignificantdifferencebetweenDEPPandcomparisonorganisationswithregardstoperceivedimpactofDEPPoninstitutionalspeedandcostofresponse,andtheextenttowhichinstitutionalandpolicyenvironmentaffectsthespeedandcostofresponse.

• Cost per result indicator analyses demonstrated good VFM per result actually achieved.


OVERVIEW OF THE APPROACH TO ASSESSING EFFICIENCY AND VALUE FOR MONEY ThischapterreportsfindingsfromtheeconomicassessmentoftheDEPPprogrammewhichwasledbytheteam’sEconomicAdvisor(SeeAnnex2forbiography).Theeconomicanalysisusedthe4Eframework(economy,efficiency,effectiveness,andequity)toevaluatetheDEPP’sValue forMoney (VfM). conclusions on effectiveness (cost of outcomes and impacts) andequity (distributionof impacts) are presented inchapter 4 (Relevance andEffectiveness ofInterventions)andchapter7(sectiononinclusionofvulnerablegroups).Tolimitduplication,thischapterfocusesonefficiencyandeconomy,butallofthefindingsaredrawntogetherwiththeVfMworktobuildacompleteVfMpicture,withparticularfocusattheprogrammeandthecountrylevel.Themethodologyisdetailedinchapter2,butbrieflyitcomprisedquantitativeandqualitativemethods,inparticular,budgetandexpenditureanalysis,internalbenchmarkingofunitcosts,costeffectivenessanalysis(includingassessmentofcostperresultindicators),keyinformantinterviewswithintheDEPPmanagementteam,DEPPboardandwithintheselectedcasestudies,documentreview,analysisofadherencetoprocurementprocedures,andanalysisof how decision-makers and project managementwithin DfID and implementing agenciesconsiderVfM.DatacollectioninstrumentsarefoundinAnnex7.

TheformativephaseoftheeconomicevaluationcoveredportfoliolevelVfM–economyandefficiency.Thecountry-levelVfManalysisiscoveredinthesummativereport,largelybecauseitreliesontheoutcomesanalysisconductedbythewiderevaluationteaminthesummativephase.

QUESTION 4.A: HAVE RESOURCES (FUNDS, HUMAN RESOURCES, TIME, EXPERTISE, ETC.) BEEN ALLOCATED STRATEGICALLY TO ACHIEVE THE PROGRAMME OBJECTIVES?

Programme Budget

Table6.1belowshowsthetotalprogrammebudgetfortheDEPP.Theconclusionshavenotchangedsincetheformativephasereport–managementcosts (lessthan5%ofDEPPtotalbudget)aretooleanforaprogrammeofthissizegiventhehighresponsibilitiesofmonitoring,reporting,coursecorrection,strategicdirectionandgeneralsteeringof£25.5millionofprojectfunds(excludingthelearningProjectbudget).ThelearningProjecthadahealthybudget.Someof thiscouldperhapshavebeenredeployed intothemanagement functions.Seebelowforfurther discussion on this issue.Alternatively, further investment could have beenmade indeveloping and implementing an electronicM&E informationmanagement system to trackprogress inreal-timeandenablemoreagilecoursecorrection.Thissystemcouldhavebeenindependently set-up by the learning Projectwith themanagement teamhaving direct ondemandviewingprivileges.

TYPE OF COST AMOUNT (£) % OF TOTAL BUDGET

STARTnetworkManagementcosts (includes12.6%nPAcrateandProgrammeboardconsultancycosts)

936,222 3.1

cDAc-nManagementcosts (includes7%nPAcrate)

245,186 0.8


STARTnetworkProjectfunds (excludingthelearningProject)

22,720,403 75.7

learningProject (includes£1millionfortheindependentevaluation) 3,343,375 11.1

cDAc-nProject 2,754,814 9.2

Total 30,000,000 100

Table6.1:TotalDEPPprogrammebudgetbasedonactualspendinguntilDec2017andforecastbudgetuptoJune2018

Value Chains and Transactions Costs within Projects TheDEPPbusinesscaseemphasizesatvariouspointsthattheDEPPwouldbeimplementedusingcollaborativemodelthroughmulti-stakeholderplatforms.ThebusinesscasestatesthattheDEPPwill“supporthumanitariancapacitythroughthecollective,collaborativeactionofInternationalnon-GovernmentalOrganisationsandtheirpartners....Supportwillnotbelimitedtoskillsbuilding,butwillpromotepathwaysbetweenlocalandinternationalactorsthroughthecreationofjointplatformsandnetworks,andbuildcommunitypreparednesssystems.PartnersmayincludeotherInGOs,localnGOs,localgovernment,nationalgovernment,academia,privatesectorcompanies,theRedcross/RedcrescentMovement,Unagenciesandothercivilsocietyorganisations.”Inaddition,thebusinesscasehighlightsmulti-stakeholderplatformsasameanstobuildcapacity:“Analysissuggestsitisimportanttosupportnetworksandmulti-stakeholderplatformstodevelopcapacity,workingwithexistingoneswhereeverfeasible.”Itdescribesmulti-stakholderplatformsas:“a‘multiplicityoforganisationsatdifferentscalesofgovernanceworkingtowardsmorecoordinatedandintegratedactions.”150

ThebusinesscasearticulatedthatDEPPinterventionswouldbedeliveredthroughmulti-stakeholderplatformsthatwouldbeimplemented:“fromlocaltointernationallevelsthatenablecollectiveactionandcaptureandsharelessonsandgoodpractice.Theseplatformswillhaveabroadmembership-e.g.,InGOs,nationalnGOs,localgovernment,nationalgovernmentacademia,privatesectorcompanies,theRedcrossRedcrescentMovement,Unagencies.Successfulplatformsshouldimprovepolicyatnationalandinternationallevel.Weenvisageasingleplatformpercountryorregionwheretheprogrammeisactiveandaninternationalplatform.Possibleproductsoftheplatformsare:synthesisedlessonsidentified,jointevaluationsandcommonpositionpapersforadvocacy.”

Thereareclearlyrisksandcostsoftakingthisapproach.However,intheVfMsectionofthebusinesscase,therewasnodescriptionoftherisksofthisapproachintermsoflayersofmanagementoverheadsandhightransactionscosts.Itwouldbedifficulttoimplementsuchanapproachwithoutincurringsomedegreeofextratransactionscosts,compliancecostsandriskmanagementateverylevel.Overheadsarenecessaryforagentstomanagerisk,andpreventfraudandcorruption.Moreover,sincethebusinesscasewaswrittencomplianceassuranceswerestrengthenedbyDfID;inDecember2016,DfIDissuedalettertoallsuppliers,statingthatsuppliershouldmeetevengreaterrequirements.

Given that the collaborative approachwas agreed upfront, itwould have been difficult forprojectstoavoidalotofthesecosts,whichariseinnecessarymanagementoverheadsateverylevelofthedeliverysupplychain.Insomecountries,regulatorybodiesrequireagenciestobeabletomaintaincertainduediligenceprocessesthatarecoveredbyoverheadcosts.WhetherthischoiceofmodelreflectsgoodVfMdependsonwhetherthebenefitsofthemulti-sectorcollaborative approach outweigh these extra transactions costs. This differs depending onprojects,butasdescribed inchapters4,and5,themajorityofDEPPstakeholdersdescribestrong value of collaboration andworking through consortia and consistently describe thecollaborativemodeltothebethemostappropriateapproachtodeliverhumanitariancapacitybuildingandresponseactivities.Therewasatleastoneproject(theTalentDevelopmentproject)wherethebenefitsofthecollaborativemodelwerenotfelttooutweighthecostsbyanumber

150 Djalante,R.Adaptivegovernanceandresilience:theroleofmulti-stakeholderplatformsindisasterriskreduction.nat.HazardsEarthSyst.Sci.2012;12:2923-2942.


of stakeholders151.This particularprojectwas amore traditional individual capacitybuildingprogram that directly trained humanitarian staff in a number ofDEPP countries. It did notincludeotherDEPPcomponents(suchasearlywarningsystemdevelopment,andpolicyefforts)thatperhapsmaybebettersuitedforthecollaborativeapproachthantraditionaltraining.

DuringVfMinterviews,severalprojectsalsoexpressedthatthecollaborativeapproachworkedwell for their projects. Exampleswere provided of specialisation in someprojectswith thisstructureleadingtosavings.forexample,intheTransformingSurgecapacityproject,cAfODdonatedtimeandintegratedprojecttasksintotheirownwork.Thistypeofstructureisbeneficialasitcanalsocanleadtoincreasedsustainabilityandownershipandallowcontinuationoftheworkbeyondtheendoftheprojects.

Therearetwokeyissuestoconsiderwithrespecttothecollaborativemodel:1)thecollaborativemodelwillnecessarilyresultinhighertransactionscostscomparedtoanalternativemodelwithlesslayers,and2)theremaybeacasetosuggestthatthecollaborativeapproachcouldhavebeenachievedatgreaterefficiencyandlowercost.Thelatterisconsideredindetailbelow.notably,therearesomeexamplesofprojectsthatappearednottohaveoverlyhighoverheadsateverylevel.forexample,expendituredatafromtheProtectioninPracticeprojectdemonstratethatoverheadswerenotbeingappliedateverytier.Theoriginalprojectbudgetwas£804,634andatJune2017thiswasthebudgetandthedeliverysupplychainthatDfIDreviewedaspartoftheirinternalreviewofDEPP.ThenPAcsetforthiswas£89,594whichisroughly11.13%.Theprojectgainedafurther£66,998foratotalbudgetof£870,137.WhilethisdidhaveannPAcchargedtoit,agranttocEDIER(alocalnGO)bytheProtectioninPracticeprojectwasthenexecutedandnooverheadsweretaken.

Table6.2showsthetiersoffunding,onecouldmisleadinglyconcludethatthedifferencebetweenTier2and3(whichis£224,068attherevisedJune2018budget)couldbesolelyduetooverheads.Thisisinfactnottrueastheactualoverheadfigurebetweenthesetierswasactually£96,846.IntermsoftheoverheadstakenatTier3,unfortunately,thefinancialmanagementsystemwasnotsetuptocheckwhetheroverheadsweretakenornot,andtowhatextentifso.

furtherinvestigationoftheProtectioninPracticeexample,showsthatthecostsrelatedtoprojectstaffwerelistedasacombinationofdirectandindirectcosts,butinrealityallthestaffwereeitherprotectiontechnicalspecialistsorfinancestaff.Theprotectiontechnicalspecialistswerehybridprotectionadvisor-managers(theywererequiredtoprojectmanagetheirpiecesofworkbutalsotoadviseatworkshopsandontoolkitdevelopment).Inaddition,thereweresmallerpercentagesoftwofinancestafftimeincludedinthebudget.ThisincludedonefinancestaffpersonatOxfamandoneatInternationalRescuecommittee(IRc).Thesepositionswereessentialtocoverthemixtureofregularinternalaccountsmanagementalongsidetheflexiblefundinggrantsthatwentthroughbothagencies.Accordingtothedeliverysupplychainthisincludedover30differentl/nnGOorganisationsreceivinggrants.Thechaindoesnotgiveanyindicationofhowmuchworkbehindthescenesthiswas,orwhethertheSouthSudanconflictcreatedmoreworkforthefinancialmanagementoftheproject.

151 TheTalentDevelopmentProjectfinalEvaluationReport,p12,notes:“AttheendofTDP,thereweremixedfeelingsabouttheconsortium’sadded-value.HalfofrespondentsquestionedtheneedforhavingaconsortiumbecausetheythoughtthecostofrunningtheconsortiumoutweigheditsbenefitsandwouldnotwanttocommittoaTDP2,shouldtheybegiventheopportunity.”


Tier2 Tier3 Tier4 Tier5

STARTnetworkSub-granttoOxfam

OxfamSub-grantto6l/nnGOsand1InGO(IRc)

Agencies

IRccontractorAgreementwithWorldVisionforWorkshops;Sub-grantto32l/

NNGOs

WorldVisioncontractorAgreement

(ThisAmountisaStraightPassThroughoffunds)

June2017budget £804,634 £571,129 £72,364 £10,439

RevisedJune2018budget £870,137 £646,069 £61,924 £10,439

Table6.2:TiersoffundingforProtectioninPractice

It’sworthnotingthatintheJune2017budget,£250,684wasspentoncapacitybuildingandpilots(PIPbudgetline12)andbasedonanalysisofactualsexpenditure,noneofitwasspentintheUKonoverheads.Rather,itwasspentincountryonacombinationofelementsincludingsalariesoftheprotectiontechnicalspecialists,costsrelatedtothel/nnGOparticipants,costsrelatedtotrainingoftrainerworkshops,andpilotcosts.Thisagainemphasisesthatprojectcontentandactivitiesarethemajorcostateachtier,andnotoverheads.

fortheProtectioninPracticeexample,thereareanumberofcoststhatsitacrossTier2andTier3.ThisoccurredbecauseOxfamdirectlyprovidedfundstoIRcandtosixl/nnGOsatthesametier.Thismeansthatmoneyhitfrontlinel/nnGOsatTier3,notattheendofalongchain.ThisisagoodexampleintheDEPPofatleastsomel/nnGOsnotsittingdowntheendofanunnecessarilylongdeliverysupplychain.

A Reconsideration of the Collaborative Multi-stakeholder Approach and VFM?

Itisclearthatthecollaborativeapproachwasmoreappropriateforsomeprojectsandinthosecases,thebenefitslikelyoutweighedthecosts.forexample,insomelocationsandprojectsitwasnecessarytoworkwiththespecialisedorganisations,especiallywhenkeyrelationshipswerenotwiththeleadagency.Asanexample,inGambella,Ethiopia,christianAidcouldnotdirectlycontractwiththeMinistryofHealthanditwasessentialtoworkwithlocalandnationalactorswiththerightlanguageskillsandcontextualisedknowledge,andexistingpartnershiproutestomanagethecollaborativeaction.Allworkwithnationalorlocalgovernments,andpotentiallynon-stateactorsinconflictsettingsneededin-countrystakeholderstodothis.

butinotherprojectsthismayhavebeenlessofanissue.Theprojectswerelikelyunnecessarilyburdenedwithahierarchyofflowoffunds,hightransactionscostswithcontracting,andcompliancecostsateverylevel.Anotherproblemrelatedtothishasbeenaroundinformationflow–longsupplychainsprolongedthetimetakenforreporting,theapprovalsprocessateverylevelwaslikelyunnecessaryandinformationflowwasslowasaconsequence.

Ifthelongsupplychainissuehadbeenconsciouslythoughtthroughattheinceptionphase,forexamplebymappingoutthechainusingtheDfIDapproachforstakeholdersandadaptingitbasedonneeds,itislikelythatprocessescouldhavebeenstreamlinedresultinginlowercostsoverall,greaterefficiencyandbetterVfM.Ifthereportingapprovalsdidnotneedtobetiedtothesupplychain,amoreproactiveMElsystemcouldhavebeensetupwhichbypassedcumbersomeapprovalsateverylevel.Separatingoutthelayersofthesupplychainwiththereportingrequirementsandinformationflowscouldhavefacilitatedthis.Also,developingbettercostmodelsonspecificactivitieswouldhaveimprovedefficiencyandloweredcosts.

Inconclusion,thecollaborativemodelnecessarilylendsitselftoadegreeofinefficiency,high


transactionscostsandslowinformationflows.Thismodelhadsubstantialbenefitshowever,andforsomeprojectsthebenefitshavelikelyoutweighedthecosts.forotherprojectsthesamebenefitsofthemodelcouldhavebeenachievedatlowercostandgreaterefficiencyifthecontracting,M&Esystemandcostingofactivitieshadbeenmoreconsciouslythoughtthroughfrominception.Thecollaborativemodelthatwastiedtohierarchicalandlongdeliverychainshaslikelycompromisedeconomyandefficiencytoadegree.

Allocation of Time and Resources for DEPP Management, Governance, Learning and M&E Functions

In terms of resource allocation to the management functions of DEPP, as explored in theformativephase,thiswasnotconsideredtobeoptimalasresourcesweretooleantoundertakefull portfoliomanagement functionsadequately, asonewouldexpect for aportfolioof thissize.nosignificantchangeshavebeenmadetoresourcesallocatedtotheDEPPmanagementteamsincetheformativephasereport.AsseeninTable6.1themanagementcostscompriseover4%of thebudget. In-depth interviews indicated thatmanagement resourceswerenotincreasedbecauseSTARTnetworkmembersfeltthatfundingacentralfunctionwashighcostandwouldtakecostsawayfromfrontlineactivities.budgettofundafull-timeadvocacypostwithinthemanagementteamwasprovidedfromthelearningteambudgetafteritwasrealised(and ‘demanded’byDEPPprojects) that thiswasneeded.Theprogrammemanagementalsoused consultancy forVfM advice –which is costed into the 4% figure of STARTnetworkadministrationcosts.TheDEPPteam,therefore,diddrawuponexternalexpertiseasandwhenrequired.

However,whilemanyprogrammesareaimingtoreducehighadministrativecost,thiswastoolean for such a complexprogramme tobemaximally effective.basedonfindings from IDIsandthesheervolumeofreportingfromafundofthissize,extraresourcesinthemanagementteamwouldhavebeenwellspent.SuchresourcescouldhavebeenspentontechnicalM&Eexpertisethatwouldhavebetterenabledthemanagementteamtoscrutiniseprojectprogressviaquarterlyreportsandothermeans,soastobetterreportresultstotheprogrammeboard.Thiswouldhavebetterenabledtheprogrammeboardtounderstandhowresults(outputsandoutcomes)wereunfolding,andtomakestrategiccoursecorrectiondecisions.

Programme-level Resource Allocation

TheVfM review explored resource allocation at the programme/DEPP level, through IDIsanddocumentreview.Inparticular,questionswereaskedaboutwhethertheportfoliomixofprojectswasconsideredoptimal.

EvidencesuggeststhatDEPPdidnotfunctionasaportfolio,andeachprojectfunctionedasastandaloneproject.ThiswasduetothedesignhistoryoftheDEPP–thesetof individualprojectswasnotdesignedasabalancedportfoliointermsofrisk,typesofactivities,size,andcost.Ideally,inaportfolio,arangeofexperimentsisneededinordertoassesswhatworksandthisshould leadtoadditional funds formoresuccessfulprojectsas implementationunfolds.Thispracticallywouldtranslateintoaseriesofpilots,followedbyscaleupandcoursecorrectionwhereneeded.Insteadofthismodel,projectsweredesignedinisolation,andretrofittedtothebusinesscaseortotheselectioncriteriaandfocalcountriesusingatop-downapproach.In-depthinterviewshaveindicatedthattheProgrammeboardwasnotabletoadequatelysteerandcoursecorrectthefundandthattheydidnothaveagoodhandleonresultsofprojects.

Theothermainvehicle for steering theDEPPProgrammewas the learningProject.At thestartofDEPPtherewasastrategicdecisiontoallocatesignificantresources(£3million)tothisproject.Thereweresomesuccesses in termsof thegenerationof learning fromtheproject(presented in chapter 4); the move of the DEPP learning Project team to have countrylearningAdvisors,andregionalandnationaleventswasgreatlyappreciatedbycountrystaff,andgavemoreopportunitiesandspacestodiscusscollaborationandshare learning.ProjectstaffindicatedthattheyhadusefulsupportfromthelearningProjecttogathercasestudiesfortheproject.forexample,theTransformingSurgecapacityteamindicatedthattheydidnothavethecapacitytogathercontentforthefutureforSurgeeventandthelearningProjectteamwasabletosupportthem.TheDEPPonlinelearningplatformwasconsideredusefulasamoreflexiblespacetoshowcaselearningfromtheproject.Thisalsosupportsthesustainability


ofprojects.

Some interviewees indicated thatwhile thereweresuccesseswith thelearningProject, thedemandofprojectresourcessuchastimeandeffortsweretoohighandhenceplacedtoomuchburdenonprojects. for example, interviewees stated that therewas toomuch informationrequestedfromprojects,andsomeduplicationwithinformationthatwasrequestedfromregularreportsandfinalevaluations.RespondentsalsostatedthatwhileUKlearningeventshadbeenuseful,thefinalgloballearningeventinGenevainMarch2018wasdemandingandconfusing.

Itisquestionableastowhethertheset-upofthelearningProjectandstrategicresourceallocationwastherightone. In-depth interviews indicatedthat theDEPPwouldhavebenefittedfrommuchmoreresultsmonitoring,reporting,analysinganddata-drivendecision-making.Arguablyaprogrammeneedsgoodmonitoring,reportingandevaluationbeforeitcanengageinlearningactivities.Strong,harmonisedM&Ehasbeendistinctlymissingattheprojectandprogrammelevel.HavingsomemoneyallocatedtoM&Ewouldhavereapedbenefitsintermsofgreaterabilityofprojectstomeasureresults,andfeedtheseuptotheProgrammeboard,soitcouldbemorestrategicinitsactionsandmoreefficientlycoursecorrecttheDEPP.StrongerM&EwouldalsohavebeencomplementarytolearningProjectactivitiesandwouldhaveservedtoenhancethefluidityoflearning.

In-depthinterviewsalsoindicatedthatthelearningProjectactivitiescouldperhapshavebeenbetterplacedattheprogrammemanagementteamlevel,ratherthanasacompetingprojectwithitsownagendaandobjectives.learningobjectivesshouldnotbeseparatetoprogrammemanagement objectives, as they were in the existing DEPP structure. In-depth interviewsindicated that the learning Project team could have communicated their findings better tothemanagementteamandProgrammeboard. Ingeneral, therealsocouldhavebeenbettercommunicationbetweenthemanagementteamandtheProgrammeboardintermsofobjectivesandexpectationsrelatedtothelearningProjectactivities.Programmeboardmembersarenotlikelytoreadreportsandinsteadneededmoretargetedandconcisecommunicationsmaterials.Alternativeinnovativemodelofinformationandlessonslearnedcommunicationchannelshouldhavebeenexploredandimplemented.

The Use of VFM Indicators and Methods

TheuseofVfMindicatorshavemainlybeenwithinthequarterlyreports.Aftertheformativephaserecommendationto increasetheuseofVfMindicators, theDEPPmanagementteammadeastrongefforttoincreasetheprojectstaffcapacityaroundVfMconceptsandreportingthroughtheimplementationoftwohalf-daytrainingevents.Thistraininghadseveraloutcomes.first,someprojectscarriedoutadhocreportsonVfM.Second,mostprojectsincorporatedVfMasacomponentoftheirfinalevaluations.Duringthetraining,aVfMcompletiontemplateforprojectswasintroducedtorecordVfMqualitativeand/orquantitativefindingsatcompletion.Atthetimeofwritingthisreport,thecompletedtemplateswerenotyetavailableandthushavenotbeenreviewedbytheEvaluationteam.

ReviewofavailabledocumentsprovidedseveralgoodexamplesofadhocVfMreports.TheTalentDevelopmentprojectgeneratedsomeusefulVfMfindingsintheirfinalevaluationreportwhichfocusedonthecosteffectivenessofdifferenttrainingmodels.Inaddition,theTransformingSurgecapacityprojectcreated tworeports,oneon theVfMrelated tocollaborationwhichhighlighteditshighcostandoneonthefinancialsustainabilityofsurgecapacity.However,itisnotclearhowthesereportswillbeusedgoingforward.

Table6.3summarisestheVfMindicatorsreportedintheMay2017projectquarterlyreportswhichwerethelatestavailablereportatthetimethetablewasdeveloped.AtleasttwoofthereportshadnoVfMsections.However,theevaluationteamacknowledgesthatsomeprojectteamshaveintegratedVfMthinkingthroughoutdifferentsectionsoftheirreports.Whilethereare some useful indicators, to have hadmoremeaningful reporting ofVfM, itwould havebeennecessarytousemorespecificandindepthVfMindicatorsforeachproject.Acrosstheprojects,acoresetofcross-cuttingindicatorswhichcanbeaggregatedandcomparedwitheachother,asdescribedinformativephaserecommendationswouldhavebeenusefultoadequatelycaptureVfM.Theremainingindicatorsshouldhavebeenprojectspecificandtailoredtoeachproject’sneeds.Indicatorsshouldhavebeenbothquantitativeandqualitative.


HIGH-LEVEL SUMMARY OF VFM REPORTING IN MAY 2017 QUARTERLY REPORTS

Economy

Cost savings, benefits in kind, Oxfam procurement procedures, use of local mate-rials, pro bono work from London School of Economics, pro bono support for web-site development, discounted flights

Leveraged other donor funding in Lebanon, DRC and South Sudan

OFDA-funded consultation, leveraging and sharing of resources, working with oth-er projects and sharing resources

Efficiency

Efficiencies with integrated programming, where partners are already working, joint training, collaboration with partners to provide efficient support for confer-ences, Learning Advisors hosted by DEPP partners in country, proactive use of processes and tools, using data, prepositioning of goods

Equity Targeting the right people, partners and beneficiaries have been fully involved

Effectiveness Community-level institutions, good coordination and communication with Govern-ment. Ensuring capacity and sustainability of in country staff so programme can be more locally oriented in the future.

Table6.3:SummaryofVfMreportingfromMay2017QuarterlyReports

TheDEPP could have benefitted from the addition of indicators as it matured in order toprovideamorecomprehensivepictureofVfM.Ingeneral,themostimportantdriversofVfMareeffectivenessandequity152 andsuchdatashouldbecollectedasprojectsandprogrammesmature. Some suggested indicators are shownbelow inTable 6.3.These are not necessaryquantitativewhichmakesdirectcomparisonslessstraightforward,butneverthelesstheyofferusefuldatatoinformdecisionsaboutstrategyandfunding.Suchdataareusefulforassessingtheprogrammeasawhole,for learningandalsotomakechangesdynamically.forexample,those projects that perform particularly high on these indicators are those that likely showhigherreturnsandthuscouldbepromotedmorethanthosethatdonot.

Suggested VFM indicator Comments

Economy/Efficiency

Evidence of economies of scope and scale within operations and budget

Economy: Economies of scope through shared overheads across projects – measured by cost savings, or just narrative

Number of operational adaptations resulting in better cost effectiveness or value generation

Efficiency: Semi-qualitative, to demonstrate that operations were changed on the grounds of VFM; actual number of changes and supporting narra-tive

Effectiveness and equity (more important for strategic decision making)

Rating of the quality of partnerships Efficiency/Effectiveness: Partnership quality a key attribute of efficiency –a rating system

152 HighlyeconomicalandefficientprogrammeswillnotoffermuchVfMiftheyaretargetingthewrongthings/people(equity),ordoingthewrongthings(effectiveness).


Does the programme continue to meet needs identified upfront?

Useful to revisit what the project is actually aim-ing to achieve at key break points. If the answer to the question is no, this is a breach of effec-tiveness and resources are not being allocated to where they are needed. Narrative.

Is there an exit strategy or other measures to en-sure sustainability?

Effectiveness: Without sustainability of the research uptake the interventions are compro-mised; i.e., what measures are in place to ensure that the interventions will continue post project completion? Explanatory narrative.

In terms of the methodology, is there a trade-off between cost and working with hard-to-reach populations, and if so, is this explicitly accounted for in the budget?

Equity: A key equity point – it is often more ex-pensive to reach vulnerable groups, so this needs conscious budgeting to maintain VFM. Costs and narrative.

Table6.4:ExtrasuggestedVfMindicatorsforQuarterlyReports

Insummary,intermsofstrategicresourceallocation,theevidenceshowsthattheinitialprocessofselectingprojectswasnotstrategic,andthismayhaveimpactedontheeffectivenessofthewholeprogramme.WhilethelearningProject interventionswereuseful, itmighthavebeenmorestrategictoredeploysomeofitsresourcesintothemanagementteamforexpertisefortechnicalprogrammaticandproject-levelmonitoringandreportingofoutputsandoutcomes.ThiswasnotpartofthelearningProjectteam’sremit.ThiswouldhavepermittedtheProgrammeboard to havemore regular and direct results to ensure on-time and on-demand decisionmaking,andhenceenablecoursecorrectionasneeded.

Itisnotedthatasthelogicalframeworkdatastartedtobecomeavailableandregularlyreportedonaquarterlybasisbythelearningteamthemanagementteamwasthenabletousethisforreporting toDfID, board and take course correctionmeasures such as increasing focus ongender considerations. However, thiswas insufficient in terms ofmonitoring reporting andcoursecorrection.

QUESTION 4.B: HAVE RESOURCES BEEN USED EFFICIENTLY? IN GENERAL, DO THE RESULTS ACHIEVED JUSTIFY THE COSTS? COULD THE SAME RESULTS BE ATTAINEDWITH FEWER RESOURCES?

Thediscussionaboveonvaluechainsandtransactionalcostsdemonstratesomeinefficiencieswithin the programme and help to partially respond to this question. further assessmentof empirical data related tohowDEPPpreparedness activities affected the cost of disasterresponse,andthecostperactivitiesandcostperresultsachievedwilladdressthequestiononwhetherresultsachievedjustifythecostoftheprogramme.

Empirical Findings on Disaster CostTo assess cost effectiveness of theDEPP projects, several lines of analysis come together.first,surveyquestionswereincludedintheT2organisationaldatacollection(seeboxbelow).ThesequantitativequestionnaireswerecompletedwithseniormanagementatbothDEPPandcomparisonorganisationsinthefourintensivesetcountries(Ethiopia,Kenya,MyanmarandthePhilippines),andthequestionscapturetheirperceptionsaboutwhetherDEPPpreparednessactivitieshaveimprovedthespeedandcostofresponse.Asthisdatawasonlycollectedatthesecondtimepoint,itisnotpossibletoestimatechangesovertime.Instead,theanalysisfocuseson assessing differences between theDEPP and comparison organisations in each countryandoverallwiththepooleddataset.StatisticaltestsareusedtoanalysewhetheranyobserveddifferencesbetweentheDEPPandcomparisongrouparestatisticallysignificant(p<0.05).


KEY COST EFFECTIVENESS DATA COLLECTED AT TIMEPOINT 2Organisational level: Organisational survey with senior management.

• Perceptions on preparedness and speed of response: On a scale of 1-5, to what extent have the preparedness activities helped improve the speed of response?

• Perceptions on preparedness and cost of response: On a scale of 1-5, to what extent have the preparedness activities helped reduce the cost of response?

• Perceptions on timeliness of previous response: How would you rate the timeliness of your organisation’s response to the disaster? Where 1=Very untimely, 2=Untimely, 3=Untimely, 4=Somewhat timely, 5=Timely

• Perceptions on cost of previous response: How would you rate the costliness of your or-ganisation's response to the disaster? Where 1=Very expensive, 2=Expensive, 3=Somewhat expensive, 4=Inexpensive, 5=Very inexpensive

• Perceptions on institutional and policy environment and speed of response: On a scale from 1-5, how much does the institutional and policy environment impact the speed of your response?

• Perceptions on institutional and policy environment and cost of response: On a scale from 1-5, how much does the institutional and policy environment impact the cost of your response?

Overall,noneoftheindicatorsassessedshowanystatisticallysignificantdifference153betweentheDEPPandcomparisonorganisations (i.e.,anydifferencesobservedarebychanceratherthananyrealdifference). Table6.5belowpresentsperceptionsonpreparednessandspeedandcostofresponsebycountryandforthepooledsample.MostrespondentsinbothDEPPandcomparisonorganisationsreported3to4outofthe5-pointscale154whenaskedwhetherpreparednessactivitieshelpedtoimprovethespeedofresponse.Asimilartrendisseenwhenaskedwhetherpreparednessactivitieshelpedtoimprovethecostofresponse.Overall,therewasnosignificantdifferenceinanyofthecountriesbetweentheDEPPandcomparisongrouporinthepooleddata.ThefindingssuggestthatDEPPpreparednessactivitieshavenotimprovedtheefficiencyofresponse.However,itmustbenotedthatthecomparisongrouphadbeenexposedtosomecapacitybuildingactivities(thoughtoalesserdegreethanDEPPorganisations)andthismightexplainthelackoffindings.

On a scale of 1-5, to what extent have preparedness activities helped improved the speed of response?

1 = no improvement

5 = very strong improvement

SCORETHE PHILIPPINES KENYA ETHIOPIA MYANMAR FULL SAMPLE

DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison

10

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)1

(5.3%)0

(0.0%)1

(1.2%)0

(0.0%)

20

(0.0%)0

(0.0%)1

(3.7%)0

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)2

(16.7%)1

(1.2%)2

(3.7%)

33

(16.7%)4

(40.0%)15

(55.6%)8

(57.1%)8

(40.0%)8

(44.4%)7

(36.8%)3

(25.0%)33

(39.3%)23

(42.6%)

411

(61.1%)3

(30.0%)8

(29.6%)3

(21.4%)11

(55.0%)9

(50.0%)11

(57.9%)6

(50.0%)41

(48.8%)21

(38.9%)

153 Inprinciple,astatisticallysignificantdifferenceisaresultthatisnotattributedtochance.Moretechnical-ly,itmeansthatifthenullhypothesisistrue(whichmeanstherereallyisnodifferencebetweenthetwogroups),thenthere’salowprobabilityofgettingaresultthatlargeorlarger.

154 Ahigherscoreindicatesahigherlevelofperceivedpreparedness.


54

(22.2%)3

(30.0%)3

(11.1%)3

(21.4%)1

(5.0%)1

(5.6%)0

(0.0%)1

(8.3%)8

(9.5%)8

(14.8%)

On a scale of 1-5, to what extent have preparedness activities helped reduce the cost of response?



10

(0.0%)0

(0.0%)0

(0.0%)2

(14.3%)1

(5.0%)0

(0.0%)0

(0.0%)1

(8.3%)1

(1.2%)3

(5.7%)

20

(0.0%)0

(0.0%)4

(15.4%)0

(0.0%)0

(0.0%)2

(11.8%)3

(15.8%)3

(25.0%)7

(8.4%)5

(9.4%)

34

(22.2%)4

(40.0%)9

(34.6%)7

(50.0%)8

(40.0%)6

(35.3%)8

(42.1%)3

(25.0%)29

(34.9%)20

(37.7%)

412

(66.7%)4

(40.0%)10

(38.5%)3

(21.4%)9

(45.0%)8

(47.1%)7

(36.8%)5

(41.7%)38

(45.8%)20

(37.7%)

52

(11.1%)2

(20.0%)3

(11.5%)2

(14.3%)2

(10.0%)1

(5.9%)1

(5.3%)0

(0.0%)8

(9.6%)5

(9.4%)

Table6.5:contributionofpreparednessactivitiestospeedandcostofresponse

Table6.6presentstheperceivedcostlinessandtimelinessoforganisationalresponseactivitiescarriedoutintheprevious12months.Intheory,ifDEPPpreparednessactivitiesdidaffectthespeedandcostlinessof responseactivitieswewouldexpectDEPPorganisation’s responsesto be less costly andmore timely compared to the non-DEPP group. However, therewasno significant difference between DEPP and comparison organisations for either of theseindicatorsinanyofthecountriesorinthepooledsample.Thesefindingsareinlinewithotherquantitativeindicatorspresentedinchapter4whichshownosignificantdifferenceinperceivedperformanceduringapastresponseforDEPP,comparisongroupandoverall.

contrary to these quantitative findings, qualitative data and case study data presented inchapter4suggestsimprovementsinspeedandefficiencyofresponseincertaincontexts(seechapter4).Whilequalitativechangehasstartedtoappear,quantitativechangerequireslongerandmoresustainedeffortstoachieve.Itmaybetooearlytoassesswhethertheresultsachievedjustifythecost.

How would you rate the costliness of your organisation’s response to the disaster?

1= not costly at all

5=very costly



11

(6.7%)0

(0.0%)2

(7.7%)2

(14.3%)1

(5.3%)0

(0.0%)0

(0.0%)1

(10.0%)4

(5.6%)3

(5.9%)

21

(6.7%)2

(22.2%)9

(34.6%)8

(57.1%)4

(21.1%)3

(16.7%)2

(18.2%)0

(0.0%)16

(22.5%)13

(25.5%)

32

(13.3%)1

(11.1%)7

(26.9%)2

(14.3%)1

(5.3%)6

(33.3%)5

(45.5%)2

(20.0%)15

(21.1%)11

(21.6%)

48

(53.3%)6

(66.7%)7

(26.9%)2

(14.3%)13

(68.4%)9

(50.0%)4

(36.4%)7

(70.0%)32

(45.1%)24

(47.1%)

53

(20.0%)0

(0.0%)1

(3.9%)0

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)4

(5.6%)0

(0.0%)


How would you rate the timeliness of your organisation’s response to the disaster?

1= not timely at all

5= very timely

ScoreTHE PHILIPPINES KENYA ETHIOPIA MYANMAR FULL SAMPLE


10

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)

20

(0.0%)0

(0.0%)2

(7.7%)1

(7.1%)1

(5.3%)0

(0.00%)1

(9.1%)1

(10.0%)4

(5.6%)2

(3.9%)

30

(0.0%)0

(0.0%)6

(23.1%)1

(7.1%)4

(21.1%)6

(33.3%)4

(36.4%)1

(10.0%)14

(19.7%)8

(15.7%)

411

(73.3%)6

(66.7%)16

(61.5%)8

(57.1%)11

(57.9%)9

(50.0%)6

(54.6%)8

(80.0%)44

(62.0%)31

(60.8%)

57

(29.2%)3

(33.3%)2

(7.7%)4

(28.6%)3

(15.8%)3

(16.7%)0

(0.0%)0

(0.0%)9

(12.7%)10

(19.6%)

Table6.6:Perceivedcostlinessandtimelinessoforganisationalresponseactivities

Table6.7presentsfindingsrelatedtotheeffectoftheinstitutionalandpolicyenvironmentonthe speedof response.Again,most respondents confirmed that the institutional andpolicyenvironment influences the speed and costliness of their ownorganisation’s response.Thishighlightsthefactthatimprovedpreparednessalonemaynotbesufficienttoimproveefficiencyofresponse.Institutionalandpolicyissueswouldalsoneedtobeaddressedinordertoimproveresponse performance. This is important to consider during future project and programmedesignprocesses.

On a scale of 1-5, how much does the institutional and policy environment impact the speed of your response?

1 = no impact

5= very strong impact



11

(5.3%)0

(0.0%)1

(3.6%)1

(7.1%)0

(0.0%)0

(0.0%)0

(0.0%)1

(6.7%)2

(2.27%)2

(3.5%)

20

(0.0%)0

(0.0%)0

(0.0%)0

(0.0%)1

(5.0%)2

(11.1%)1

(4.8%)2

(13.3%)2

(2.27%)4

(7.0%)

33

(15.8%)2

(20.0%)9

(32.1%)3

(21.4%)7

(35.0%)4

(22.2%)9

(42.9%)6

(40.0)%28

(31.8%)15

(26.3%)

413

(68.4%)7

(70.0%)11

(39.3%)7

(50.0%)9

(45.0%)7

(38.9%)8

(38.0%)5

(33.3%)41

(46.6%)26

(45.6%)

52

(10.5%)1

(10.0%)7

(25.0%)3

(21.4%)3

(15.0%)5

(27.8%)3

(14.3%)1

(6.7%)15

(17.1%)10

(17.5%)

On a scale of 1-5, how much does the institutional and policy environment impact the cost of your response?

1 = no impact

5= very strong impact



11

(5.3%)0

(0.0%)1

(3.6%)1

(7.1%)0

(0.0%)0

(0.0%)0

(0.0%)1

(6.7%)2

(2.4%)2

(3.5%)


20

(0.0%)1

(10.0%)0

(0.0%)0

(0.0%)0

(0.0%)2

(11.1%)4

(22.2%)3

(20.0%)4

(4.7%)6

(10.5%)

37

(36.8%)3

(30.0%)12

(42.9%)5

(35.7%)11

(55.0%)4

(22.2%)7

(38.9%)3

(20.0%)37

(43.5%)15

(26.3%)

410

(52.6%)4

(40.0%)10

(35.7%)5

(35.7%)7

(35.0%)8

(44.4%)7

(38.9%)8

(53.3%)34

(40.0%)25

(43.9%)

51

(5.3%)2

(20.0%)5

(17.9%)3

(21.4%)2

(10.0%)4

(22.2%)0

(0.0%)0

(0.0%)8

(9.4%)9

(15.8%)

Table6.7:Perceptionsabout the influenceof the institutional andpolicyenvironmenton speedandcostlinessofresponse

ThelackofsignificantdifferencesbetweenDEPPandcomparisongroupswithintheempiricalfindingswouldsuggestthattheresourcesinvestedinDEPPhavenotledtoaprogramme-wideimprovementinefficiencyofthehumanitarianresponse.However,itmustbenotedthatitmaywellbetoosoontomeasureimpact,andthesefindingsarethereforenotindicativeofpotentialimpact.Moreover, thesample size is relatively small and thisparticularanalysis isbasedonperceivedtimelinessandefficiencyofresponseratherthanactual,andperceptionsdatacanbebiased.ThefactthatsomequalitativedatasuggestsimprovedspeedofresponseprovidepreliminaryindicatorsthatDEPPhasgoodpotentialtoleadtoimprovementinefficiencyofthehumanitarianresponse.Ingeneral,someofthein-depthinterviewsalsoreiteratedthatitistoosoontomonitorresultsandimpact(e.g.,cDAc-n).

Cost Per Result Indicators

Asecondlineofenquirytoassesscosteffectivenessoftheprogrammeistoassessthecostper activity and cost per result achieved. Projectswere asked to provide expenditure databrokendownbyspecificactivities,toenableestimationoftheexpenditurethatwasincurredtoachieveemergencypreparednessandcapacity-relatedoutcomes.Projectswereaskedtobreakdowntheircostsaccordingtotheactivitieslistedbelow:

• Individualcapacitybuilding• Organisationalcapacitybuilding• Systems-levelcapacitybuilding• Individualpreparedness• Organisationalpreparedness• communitypreparedness

This enables an analysis of the actual cost to achieve changes in preparedness levels andcapacityindifferentcontexts.Theunitcostscanthenbecomparedovertime(iftime-seriesdataareavailable,whichisnotthecaseforthisevaluation),oracrosscountries.Thegoalofthecountrycomparisonistoassesswhichcountryoutcomescostmore,andtounderstandwhy.

ThemethodologyinvolveslinkingthefinancialdatawiththeoutcomesfromthewiderfindingsfromtheKAPandorganisationalresponsesurveys.Asdiscussedinchapter4ofthereport,there were few significant differences in the degree of change over time between DEPPandcomparisongroupsformostofthequantitativeindicators.becauseofthenatureofthehumanitariansystem,andthefactthattherewassomeexposuretocapacitybuildingwithinthecomparisongroup(thoughatalowerlevelthantheDEPPgroup),theremighthavebeensomeimprovementsinthecomparisongroupovertime.forthisreason,thereissomemeritinassessingthechangeinoutcomesfromT1andT2intheDEPPorganisationsonlyineachoftheintensivesetcountries,andthenestimatingthecostperunitchangefortheseoutcomes.Toconduct this typeofanalysis,a fullfinancialdataset isneededpercountrywhichhadastatisticallysignificantfinding.

Table6.8summarisesthefinancialdatawhichwereceivedfromprojects.Theevaluationteamacknowledgethatthiswasnotastraightforwardtaskforprojects,whichexplainswhysomeprojectswerenotabletoprovidethedataintime,giventheburdenofothertasks.


PROJECT

FINANCIAL DATA RECEIVED FROM PROJECTS

Costs fully disaggregated by each activity for all countries combined

Costs fully disaggregated by each activity and by country

PiP Yes Yes

ALERT Yes Yes

SEPS no no

LPRR yes no

Surge yes no

UEWEA yes no

STP Yes Yes

ADcAP Yes Yes

FE no no

TD no no

EWEA no no

PHEPGambella yes no

cDAc-n Yes Yes

Table6.8:Summaryoffinancialdataprovidedbyeachproject

Withoutafullfinancialdatasetthatmatcheduptoanyoftherelevantquestionsandcountries,missingdatawereestimatedfromprojectquarterlyfinancialreportsastheyprovidedfiguresforthecostsperthefiveDEPPresultsareasforeachproject.155

Table 6.9 below summarises the results with statistically significant findings from the KAPsurveys,andthecorrespondingcostperunitchange.Asthetableillustrates,thehighestVfMisforMyanmar,whichyieldedacostperpercentagepointincreaseinperceivedorganisationalpreparednessofonly£1,138comparedtothePhilippineswherethecostperpercentagepointincrease inthesame indicatorwas£71,287.QualitativedatasuggestthatSEPSwasabletoimplementactivitiesatalowercostduetothesmallernumberofconsortiapartnersandthefactthattheeconomiccostoflivingandoperatinginMyanmarislowerthanothercountries.Generally, it isnotadvisable tomakedirect comparisonsbetweencountriesbecauseof thedifferencesincostbaseswithincountries,andthemanyothercontextualfactorsthatcomeintoplay.KenyaalsohadaverylowunitcostcomparedtothePhilippines.Thismethodologywouldbemoremeaningfulifmoredatapointsovertimewereavailableand,inmorecountries,sothatcomparisonsovertimeandacrosscountriestounderstandreasonsforthefindingscouldbemade.

155 ThefiveDEPPresultsareasare:1)Improvedknowledgeandunderstandingofindividualsbysharingbestpracticeforhumanitarianpreparednessandresponse;2)Improvedpreparednesssystemsforearlyactionwithcommunitiesatriskofdisasters;3)Increasednumberofcoalitions,partnershipsandnetworkswhich,workingtogether,areabletoaddresshumanitarianneedsinawidevarietyofemergencysituations;4)Improvedinstitutionalarrangementsandpolicyenvironmentssothatnationalsystemsforhumanitarianresponseandpreparednessarebettersupportedandmoresustainable;and5)Strengthenedevidencebaseforwhatworkstohelpbuildhumanitariancapacityatscale.


INDICATOR WITH STATISTICALLY SIGNIFICANT

CHANGE BETWEEN T1 & T2 IN DEPP GROUP

COUNTRY

WHICH COST EFFECTIVENESS

INDICATOR IS RELEVANT?

DEPP

TOTAL COST INCURRED TO MAKE

REPORTED CHANGE

COST PER

1. Perceived Organisational Preparedness: Atwhatlevelwouldyourateyourorganisation’spreparednessresponsetodisastersand

emergencies?

MyanmarOrganisationalpreparedness

+8.9% £10,129 £1,138

2. Perceived Organisational Preparedness: Atwhatlevelwouldyourateyourorganisation’spreparednessresponsetodisastersand

emergencies?

ThePhilippinesOrganisationalpreparedness

+5.5% £392,078 £71,287

3. Perceived Individual Response – Performance

in Previous Response: Howwouldyourateyourperformanceduringyourlast

response?

ThePhilippinesIndividual

capacitybuilding+6.8%

£253,784

£37,321

4. Perceived Organisational Response – Performance

in Previous Response: Howwouldyourateyouroverallresponsetothedisaster?

KenyaOrganisationalcapacitybuilding

+15.8% £220,057 £13,928

5. Actual Organisational Preparedness: compositeindicatorassessingsevendomains:1-Emergency

preparednessplans,2-Humanresourcesandcapacity,3-logisticsandsupplies,4-legal,documentationandpolicy,5-Organisationandsystemprocesses,

6-Emergency responseteamandstaffing,7-communication

andmedia

KenyaOrganisationalcapacity

buildingandorganisationalpreparedness

+11% £258,670 £23,515

Table6.9:Summaryofcostperresultindicators

fortheindividualperformanceindicatorinthePhilippines(itispossibletomatchupthedataonthenumberofpeopletrainedinthePhilippinesbyDEPP,dividedbythetotalcost.Thenumberofpeopletrainedis8,404anddividingthisfigureby£253,784givesanannualunitcostfigureof£30.20.Itisdifficulttomakeavaluejudgmentonthisfigurebecausethereisnobasisforcomparison;however,giventhefactthatweseeasignificantimprovementinthesatisfactionrating,andthatfromourexperienceofotherprojects,£30.20isnotoutsidethereasonablerangeandappearstobeagoodVfMfinding.


QUESTION 4.C: HAVE PROGRAMME FUNDS AND ACTIVITIES BEEN DELIVERED IN A TIMELY MANNER?

Use of Resources during Set-up and Lags Asexplainedatlengthintheformativephasereport,resourcesandtimenecessaryforsetupwereunderestimated,andtheprocessofsettingupandcontractingwasinefficient.ThiscontinuedtohaverepercussionsthroughoutthecourseoftheDEPP,astherewaslesstimeforimplementationgiventhattimewasusedforsetup.Seechapter4forfurtherdiscussiononimplementationdelaysandno-costextensions.Projectsattheoutsetdidnothavecleardecision-makingprocesses,whichcausedfurtherdelays.Therewaslackofstrongownershipofdecisionswhichcausedadditionaldelays.Greaterclaritywasneededonstructuresandaccountabilitywithintheprojects.Also,projectswereunclearwhatthefunctionoftheProgrammeboardwas,andsomefeltitcameacrossasan“Examboard,”ratherthanafunctiontohelpimprovetheprogramme.

Underspending due to Delays Thedelaysdescribedabovecontributedtoongoingunderspendingacrosstheprogramme.Table6.10showsthevariancebetweenbudgetandactualexpenditurefromaselectionofprojectsusingdatafromtheMarch2017quarterlyreports.Ascanbeseen,thereishighvarianceinthelevelofunderspendingrangingfrom73%tolessthan10%.

However,datasuggestthatafter18monthsthespendratepickedup,andsincemid-2017thevariancebetweenbudgetandexpenditurehasbeenmuchmorereasonable.Thevariancesatthetimecouldpartlybeexplainedbytimelagsinbigexpenditureitems(suchassoftwaresystemsorprocurementofpublicequipment)whichoccurredearlyon,butactualexpenditureislagged.Thishassincebeenharmonised.

PROJECTUNDERSPEND IN MARCH 2017 QUARTERLY REPORT

COMMENTS

UEWEA 73%

PHEPGambella 35%Askedforano-costextensiontoallowfor underspend.DelaysinsigningMOUswith

government,andrecruitmentwasachallenge.

SEPS 23%

EWEA 64%

STP 41% Grantsdelayed

FE 41%

TD lessthan10%

ADcAP lessthan10%

learningProject lessthan10%

Table 6.10: Variance in spending against budgets for selection of projects reported in March 2017QuarterlyReport



SUSTAINABILITY & LIKELIHOOD OF IMPACT

cHAPTER7

165 SUMMATIVEPHASEREPORT-SUSTAInAbIlITyAnDlIKElIHOODOfIMPAcT

Evaluation Question #5: To what extent and in what ways have the benefits of the programme become embedded?Sub-questions:a) Whatcontributionhastheprogrammemadeinstrengtheningnationalpreparednesssystems?b) Hastheprogrammetaken intoconsiderationprioritisedtargetgroups (peoplewithdisabilities,

older people, women, children and youth)? What contribution has the programme made instrengthening inclusionof targetgroupsandgenderaspectsat the levelofnational and localinstitutions?

c) InwhatwayshasDEPPinfluencedinstitutionalandpolicyenvironments?d) Whatisperceived(qualitativeanalysis)tobethemostsignificantchangeattributedtoDEPP,and

why?Data sources: • Minimum set IDIs with programme and project stakeholders (Qualitative)• KAP and organisational surveys (Quantitative)• Intensive set IDIs(Qualitative)• Document review(Qualitative)Key Findings:• Insomesettings,DEPP has contributed to strengthening national preparedness systemsbut

typicallyonasmallgeographicscale,andbyindividualprojectconsortia• Thelack of mainstreaming of inclusion of gender and prioritised groups across the programme

resulted in a fragmented programme wide approach and uneven results.Therewasnoevidenceof individual knowledge changeon inclusion; there was evidence of organisational change on inclusion, in particular in Ethiopia and among local organisations across the intensive set countries

• Monitoring of gender data improvedovertheprogramme;reporting on other prioritised groups remained low.

• TherehavebeenseveralconcreteexampleswhereDEPP has impacted government policies or systems or increased political commitment:o UEWEAproject’sadvocacyandsensitisationeffortsledtoanamendmentoftheDisaster

andEmergencyManagementAct(DEMAct),2015toincludefoodsecurity(Kenya)o lPRRproject demonstrated changes at the institutional level (christianAid adopted the

updatedresilienceframework)o financial Enablers’ cSO partnerswere able to secure seats in the national government

DisasterRiskReductionManagementcouncil(Philippines)• Therewasno statistically significant change in policy-related quantitative indicatorsinthefour

intensivesetcountries• changesinquantitativeindicatorsrelatedtolocalisationhavenotyetoccurredwithinthreeout

offourintensivesetcountries.In Ethiopia, however, the percentage of DEPP organisations with policies inclusive to L/NNGOs increasedby16%betweenT1andT2

• Attitudinal changes towards localisation have also occurred, including in how INGOs consider, address, and involve local actors and communities.ThisadoptionofthelocalisationapproachisconsideredtobeDEPP’smostsignificantchangebykeyinformants

• There is some evidence of benefits being embedded within organisations and systems. Thepotentialforlonger-termeffectsisdependentontheextenttowhichprogrammecomponentscancontinuebeyondtheendofDEPP

• Thelikelihoodofsustainabilitywasenhancedforprojectsorprojectcomponentsthatdemonstratedthefollowingcharacteristics:o builtonexistingworkandexistingpartnerships/consortiao Partneredwiththenationalandlocalgovernmento Had/developedapolicyoradvocacyelemento Workedtowardssystemslevelchangeo Paidgreaterattentiontoexitplansearlierintheprojectcycleo Developedtools,guidelines,orsystemsthatfillanimportantgapo Involvedbeneficiariesand/orlocalstakeholderso ExhibitedgoodVfMo builtlinkageswithotherentitiesandotherDEPPprojectso Implementedincontextsthatweremorefertileforchange


QUESTION 5.A: WHAT CONTRIBUTION HAS THE PROGRAMME MADE IN STRENGTHENING NATIONAL PREPAREDNESS SYSTEMS?

In a few settings, DEPP contributed to strengthening national preparedness systems though typically on a small geographic scale, and by individual project consortia rather than by DEPP as a whole.

Documentreview,logicalframeworkdataandstakeholderinterviewsindicatesomeevidenceofinfluenceoftheDEPPonnationalpreparednesssystems.forexample,theUrbanEarlyWarningEarlyActionsystemtodetecturbanemergenciesandenablerapidresponsehasbeenadoptedbythenairobicountygovernmentinKenya,who’vealsocommittedtoaddressingurbanfoodsecurityissues.ThePublicHealthPreparednessprojectinGambella,Ethiopia,workedwiththegovernment to improve disease and outbreak surveillance, leading to improved surveillanceandreportingatvariouslevelsofthehealthsysteminseveraldistrictsandinthestrengthenedcapacitytotestforpathogensattheregionallaboratory.InPakistan,managementofthesurgeplatformdevelopedundertheTransformingSurgecapacityprojecthasbeentakenoverbythenationalHumanitariannetwork(nHn),withanMOUinplacewiththelocalgovernmentbodytoensureimplementationaftertheprojectends.TheMarsabitcountyGovernmentinKenyaadopted linking Preparedness Resilience and Response’s conflict-sensitive approach duringhumanitarianresponse(includingduringthe2016-2017droughts)thusprovidingevidenceofDEPP’scontributionstonationalpreparednesssystems.Inothersettings,however,thedegreetowhich theDEPPhas influencednational preparedness systems and government disasterplanshasbeenlessclear.

These positive influences are important and represent significant investment and efforts.However,itshouldbenotedthatgenerallytheseeffectshavebeenonasmallergeographicscale–oftenatthedistrictorcountylevelandinmostcases,eachpositiveexampleofchangewasdirectlyattributabletoanindividualproject.bydesign,theDEPPprogrammedidnotimplementall14projectsineachofthe10focalcountriesandanindividualprojectmaynotnecessarilyhavebeenimplementedacrossanentirecountry.Inaddition,notallprojectstargetedsystems-levelchange.Infutureprogramming,moresystematicconsiderationtocumulativeresultswithinageographicscalearenecessarytomaximiseimpacts.

Theshort3-yearprogrammeperiodandthewaythatDEPPprojectswereunevenlyprogrammedinthe10countries,impededtheabilitytoachievesignificantsystem-widechanges.Influencinggovernment systems is a complex task that requires sustained and evolving contextualisedefforts at multiple levels, alongwith adequate time and flexibility. Interviewees noted thatwhilepartnershipswithgovernmentactorshavebeeninstrumentalininfluencingnationalandsystems-levelchange,buildingthesetypesofrelationshipscanbeaslowandlengthyprocessduetobureaucraticandadministrativechallengesincludingchanginggovernmentsandshiftingpriorities. Incertain fragileandconflict-affectedcontextssuchasSouthSudan,partnershipswiththegovernmentmaybeevenmorechallengingorpotentiallyposeriskstotheproject.DEPPstakeholdersinSouthSudandescribedthesechallenges(seechapter8).furthermore,duetodelaysincontracting,theexternalevaluationcouldonlybegindatacollectionaftersomeprojectshadalreadybeganimplementation.furthermore,itwasrequiredtocompleteitsscopeofworkataboutthesametimeDEPPprojectsclosed.ThisledtoashortertimeperiodbetweendatacollectionpointsandfurtherreducedtheabilitytocaptureanypotentialimpactDEPPmayhavehadatthesystemlevel.

furtherassessmentofDEPPeffectsoninstitutionalarrangementsandthepolicyenvironmentaredetailedlaterinthischapter.

QUESTION 5.B: HAS THE PROGRAMME TAKEN INTO CONSIDERATION PRIORITISED TARGET GROUPS (PEOPLE WITH DISABILITIES, OLDER PEOPLE, WOMEN, CHILDREN AND YOUTH)? WHAT CONTRIBUTION HAS THE PROGRAMME MADE IN STRENGTHENING INCLUSION OF TARGET GROUPS AND GENDER ASPECTS AT THE LEVEL OF NATIONAL AND LOCAL INSTITUTIONS? The lack of mainstreaming of inclusion of gender and prioritised groups across the programme resulted in a fragmented programme wide inclusion approach and uneven results. While there was no evidence of individual knowledge change, there was evidence of organisational change with respect to inclusion, in particular in Ethiopia and among local organisations across the intensive set countries.


While the DEPP business case underscored the disproportionate impact of disasters andhumanitarianemergenciesonwomeninparticular,andemphasisedthattheprogrammewouldstrategicallyaddressinclusion–atleastinregardtogenderandViolenceAgainstWomenandGirls(VAWG)–suchafocuswaslargelyabsentfromtheprogramme.Thebusinesscasestatedthatthemanagementteamwouldberesponsiblefor“ensuringpriorityareasincludinggender,violence against women and girls…are addressed throughout the programme managementcycleprocess”,andthat“projectswillbeencouragedtobuildthecapacityoftheirpartnersonViolenceAgainstWomenandGirlsandthiswillbeakeycriteriaforthedesignedcomponenttodeliveragainst.”Whilethismayhavebeentheintention,onlyoneofthe14projects(ProtectioninPractice)focusedonprotectionandtosomedegreeVAWG.cDAc-n’sprojectinSouthSudanandlinkingPreparednessResilienceandResponse’sworkinMyanmaralsohadsmallVAWG-focusedelements.Itisnoteworthythatthebusinesscasedidnotarticulatewellinclusionofother prioritised target groups including peoplewith disabilities, older people, children andyouthorhighlightthisapriorityareafortheDEPP.

Thisabsenceofdefinitionsanddetailsinthebusinesscaseandsubsequentlackofemphasisoninclusionofprioritisedgroupsduringtheconceptualisationoftheprogrammehadseveraldownstreameffects.first,whiletheprojectselectioncriteriadidrequiregenderconsiderationstobeoutlinedinprojectproposals,projectswerenotmandatedtoprovidedetailedinclusionplans.Withoutdefinitionsandclearguidance,differingapproaches to inclusionwere incorporatedinprojectdesignsdrivenmainlybyprojectororganisational leadership.Thisyieldedvaryingdegreestowhichinclusionwasintegratedintoprojectactivitiesandultimatelyafragmentedprogrammewideapproach.ViolenceagainstWomenandGirlswasnottakenonasakeycross-cuttingthemeandassuchtheprogrammedidnotdeliverwellagainstany intendedVAWGcriteria.

Acrosstheprogramme,inadditiontoProtectioninPractice’sfocusonprotection,theAgeandDisabilitycapacityProgrammefocusedoninclusionoftheelderlyandpeoplewithdisabilities.bothof theseprojectsmadestrides tobuildprotectionand inclusioncapacityand in somecasesinfluencedtheworkofotherprojects.ADcAPinparticularstronglychampionedinclusioneffortsincludingatprogramme-leveleventsandhelpedshapeprogramme-leveldialogueonthisissue.However,theextenttowhichprioritygroups(women,children,theelderly,andpeoplewith disabilities and other vulnerable/marginalised groups)were holistically included acrosstheprogrammeand inclusionefforts adequatelymonitoredwereminimal.Arguably, amoreintegratedapproachwhereinclusionandprotectionweremainstreamedacrosstheprogrammeratherthanthespecificfocusofoneortwoprojectsmighthaveyieldedbetterresults.

Intermsofmonitoringandreporting,previousDEPPevaluationreportshighlightedaconsistentlackofdisaggregateddataandminimallydocumentedgenderconsiderationsacrossprojects.Recommendationstostrengthengenderreportingwithinprojectdataweretakenupaftertheformativephaseevaluationreport.Therewassomeevidenceofcorrectiveactionsleadingtoan increase indatadisaggregationbygenderover the courseof theprogramme.However,reportingonotherprioritisedgroupsremainedlow,makingitdifficulttoassessthetrueextenttowhichthesegroupshavebeenreachedbycapacitybuildingorotheractivities.


KEY INCLUSION INDICATOR DATA COLLECTED AT TWO TIMEPOINTSIndividual level:

• Perceived knowledge on inclusion: KAP survey with humanitarian staff. How knowledgeable do you feel you are regarding [Age related issues in a disaster; Disability related issues in a disaster; Issues related to women in a disaster]? *

• Perceived knowledge on VAWG: KAP survey with humanitarian staff. How knowledgeable do you feel you are regarding Gender-based Violence? *


• Perceived inclusion during preparedness: Organisational survey with senior leadership. Were members of any of the following groups involved in the design and implementation of the preparedness activities/programming? [Women, the elderly, people with disabilities]

• Perceived inclusion during response: Organisational survey with senior leadership. How well did this response take into account the needs of vulnerable groups including the elderly, wom-en and people with disabilities? [Note this indicator was only measured at T2.]*

• Inclusion policy: KAP survey with humanitarian staff and organisational survey with senior leadership. Does your organisation/country office have a policy about inclusion of vulnerable groups such as women, children, people with disabilities, the elderly?

Community level:

• Inclusion in community preparedness plans: Household survey. On a scale of 1-5, tell me whether you agree or disagree, with the following statements: a) Community preparedness plans and actions have taken into account the needs of women, b) Community preparedness plans and actions have taken into account the needs of the elderly, c) Community preparedness plans and actions have taken into account the needs of the people with disabilities.*

*Indicatorsweremeasuredbasedona5-PointlikertScalewhere1=Poor,2=fair,3=Good,4=Verygood,5=Excellent.

Analysis of intensive set data collected in the four intensive set countries (Kenya, Ethiopia,Myanmar and the Philippines) provides additional evidence on how programme activitiesactuallyincorporatedandinfluencedinclusionattheindividual,organisationalandcommunitylevel.Theboxabovedescribesthespecificinclusion-relatedindicatorsthatwereassessedatT1andT2amongbothDEPPandcomparisongroupsandanalysedforchangeovertime.

Perceivedknowledgechangeswithrespecttoinclusion(alsopresentedinchapter4)andVAWGispresented infigure7.1.Ascanbeseen, therewasnoperceivedchange in self-reportedknowledgelevelsacrossanyofthefourindicatorsamongrandomlyselectedhumanitarianstaffatDEPPorganisations,comparisonorganisationsoroverallasaresultoftheDEPP.Inaddition,therewerenosignificantchangesinanyoftheintensivesetcountriesorwhenthesamplewasrestrictedonlytolocalorganisations.ThisfindingsuggeststhatDEPPdidnotimpactindividual-levelknowledgeoninclusion.

DEPP DIFFERENCE

(T2 - T1)

CONTROL DIFFERENCE


Age Related Issues in a Disaster +1 . 0 % +1 . 0 % 0 . 0 %

Disability Related Issues in a Disaster + 0 . 6% + 0 . 5% +1 .1%

Issues Related to Women in a Disaster + 0 . 8 % - 0 . 5% +1 . 3 %

Gender Based Violence +1 . 2 % -1 . 4% +2 . 6%

figure7.1:PerceivedindividualknowledgerelatedtoinclusionandGender-basedviolence* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant


At the organisational level, using the pooled data from all four countries, there were nosignificantchangesintheinclusionofprioritisedgroupsinpreparednessprogrammingintheDEPP group or in the comparison group (See figure 7.2).However,DEPPorganisations inEthiopiashowedastatisticallysignificantincreaseininclusionoftheelderlyinpreparednessprogramming(+26.8%,p<0.05),andininclusionofpeoplewithdisabilities(+30.9%,p<0.05)betweenT1andT2(figure7.3).TherewasnosimilarincreaseinthecomparisonorganisationsinEthiopia.TherewasinfactastatisticallysignificantnetDEPPeffectforinclusionofpeoplewithdisabilities(+52.2%,p<0.05).Thisispositive,butsomewhatsurprisingfindingasEthiopiawasnotoneofthecountrieswheretheAgeandDisabilitycapacityProgrammewasdirectlyactive.InKenyawhereADcAPinclusionadvisorswereactive,therewasnosignificantchangeindisabilityinclusionduringpreparednessactivitiesinDEPPorganisationsoroverallasaresultoftheDEPP.ImprovementsinEthiopiacouldhavebeenduetoprogrammeleadership’sincreasingemphasis on the inclusionof prioritised groups as best practice, disseminationofADcAP’sMinimumStandardsforAgeandDisabilityInclusioninHumanitarianActionthroughtheirowneffortsand/orthroughglobalDEPPevents,ortheinfluenceoftheShiftingthePowerprojectwhichincludedsomeinclusionefforts,oracombinationofthese.

DEPP DIFFERENCE

(T2 - T1)

CONTROL DIFFERENCE


Women involved in design and implementation of preparedness programming

+ 0 .7 % + 6 .7 % - 6 . 0 %

Elderly involved in design and implementation of preparedness planning

0 . 0 % + 0 . 3 % - 0 . 3 %

People with disabilities involved in design and implementation of preparedness planning

- 0 . 2 % - 0 .9 % + 0 . 8 %

figure7.2:Perceivedinclusionofprioritisedgroupsduringdesignandimplementationofpreparednessprogramming* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.

ADEPP

DIFFERENCE (T2 - T1)

CONTROL DIFFERENCE


Ethiopia +28 . 8 %* + 6 . 5% +2 0 . 3 %

Kenya - 8 .1% +14 . 3 % -2 2 . 4%

The Philippines -1 . 4% -2 . 3 % + 0 .9 %

Myanmar -11 . 2 % -23 .9 % +1 2 . 6%

Local +1 .7 % 0 . 0 % +1 .7 %


BDEPP

DIFFERENCE (T2 - T1)

CONTROL DIFFERENCE


Ethiopia +3 0 .9 %* -21 . 2 % +52 . 2 %*

Kenya -<11 .7 % + 4 . 8 % -16 . 4%

The Philippines -5 . 6% +18 . 8 % -24 . 4%

Myanmar -7. 5% -7.7 % + 0 . 2 %

Local - 0 . 8 % -5 .9 % +5 .1%

figure7.3. Perceived inclusionof the elderly (PanelA) andpeoplewithdisabilities (Panelb) duringdesignandimplementationofpreparednessprogrammingbycountry.* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.

Inclusionofvulnerablegroupsduring responseactivitieswasassessedonlyatT2,and thusaDifference-in-Differenceanalysis isnotpossible.T2data is insteadanalysedforstatisticalsignificanceusingchisquaretests.Table7.1presentsthefindings.Overall,therearenostatisticallysignificantdifferencesbetweenDEPPandcomparisonorganisationsinanycountryorinthefullsample.However,Ethiopiadatashowatrendofincreasedinclusionduringresponseactivities(70%inEthiopiaDEPPorganisationsreportedverygoodorexcellentcomparedto61.1%inthecomparisonorganisations).This trend is consistentwith thesignificant improvements ininclusionduringpreparednessinEthiopiadescribedabove.

How well did this response take into account the needs of vulnerable groups including the elderly, women and people with disabilities?

SCORETHE PHILIPPINES KENYA ETHIOPIA MYANMAR FULL SAMPLEDEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison

Poor 0(0.0%)

0(0.0%)

0(0.0%)

0(0.0%)

0(0.0%)

0(0.0%)

0(0.0%)

0(0.0%)

0(0.0%)

0(0.0%)

Fair 1 (5.9%)

1 (11.1%)

2 (7.7%)

0(0.0%)

3(15.0%)

2 (11.1%)

3(27.3%)

4(36.4%)

9(8.8%)

7(13.2%)

Good 7(41.2%)

2 (22.2%)

10(38.5%)

5(33.3%)

3(15.0%)

5(27.8%)

5(45.5%)

5(45.5%)

37(36.3%)

17(32.1%)

Very Good

7(41.2%)

4(44.4%)

12 (46.2%)

7(46.7%)

13(65.0%)

8 (44.4%)

3(27.3%)

2 (18.2)%

49(48.0%)

21 (40.0%)

Excellent2

(11.8%)2

(22.2%)2

(7.7%)3

(20.0%)1

(5.0%)3

(16.7%)0

(0.0%)0

(0.0%)7

(6.86%)8

(15.1%)

Table7.1:PerceivedinclusionofprioritisedgroupsduringhumanitarianresponseatT2* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.

Ethiopiaalsoshowspositivefindingswithrespecttochangesinorganisationalpolicyontheinclusionofprioritisedgroups.Asshown infigure7.4, therewasanon-significant increaseof3.5%ofDEPPorganisationsinEthiopiawithinclusionpoliciesrelatedtoprioritisedgroupsbetweenT1andT2.However,overallcomparedtothecomparisongroup,therewasasignificantnetDEPPeffect of+9.9% (p<0.05).This alignswellwith theprevious findings forEthiopiaandwhentakentogetherdemonstrateconsistently,acrossaseriesofindicators,evidenceofimprovedinclusionattheorganisationallevelinEthiopiathatcanbeattributedtotheDEPP.Withrespecttoorganisationalpolicy,strongpositivefindingsarealsoevidentwhenthesampleisrestrictedtolocalorganisations(Seefigure7.4).AmongalllocalDEPPorganisations,therewasastatisticallysignificantincreaseof+6.5%(p<0.05)inorganisationswithpoliciesonthe


inclusionofprioritisedgroups,andthis translated intoastatisticallysignificantoverallDEPPeffectof+14.0%(p<0.05).ThisisanotablefindingdemonstratingtheDEPPeffortsculminatedinstrengthenedinclusionpoliciesinlocalorganisationsacrosstheprogramme.

DEPP DIFFERENCE

(T2 - T1)

CONTROL DIFFERENCE


Full Sample +3 .1% -2 .7 % +5 . 8 %

Ethiopia +3 . 5% - 6 . 4%* +9.9 %*

Kenya - 0 .7 % + 4 . 2 % -3 . 4%

The Philippines +1 .9 % + 0 . 8 % +1 .1%

Myanmar -1 2 . 2 % -7.9 % +2 0 .1%

Local + 6 . 5%* -7. 5% +14 . 0 %*

figure7.4:Organisationswithpoliciesontheinclusionofprioritisedgroups* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.

Inclusionwithincommunitieswasalsoanalysed.Atthecommunitylevel,householddatafromMyanmarwasusedtoconstructindicatorsontheperceivedinclusionofwomen,theelderlyandpeoplewithdisabilitieswithincommunitypreparednessplans.Asillustratedinfigure7.5,perceptionsonthelevelofinclusivenessofcommunitypreparednessplansactuallysignificantlydecreasedbetweenT2andT1inbothDEPPandnon-DEPPcomparisoncommunities.TherewasnostatisticallysignificantoveralleffectoftheDEPP.Thesedeclinesinperceivedinclusionmayhavebeenlinkedtochangesincontextualfactorssuchaspoliticalclimate,insecurityorotherfactors,orthetendencyofpeopletojudgethemselvesortheircommunitiesmorerealisticallyonceexposedtocapacitybuildingorotheractivities.SinceperceptionsoninclusiondecreasedstronglyinbothDEPPandnon-DEPPcomparisoncommunitiesatroughlythesamelevel,thereisnoevidencethatDEPPimprovedinclusionwithincommunitypreparednessplans.


DEPP DIFFERENCE

(T2 - T1)

CONTROL DIFFERENCE


Myanmar: Inclusion of women in community preparedness plans

- 6 . 2 %* -5 . 0 %* -1 . 2 %

Myanmar: Inclusion of the elderly in community preparedness plans

-5 . 4%* - 4 . 4%* -1 .1%

Myanmar: Inclusion of people with disabilities in community preparedness plans

- 6 . 0 %* - 4 . 8 %* -1 . 2 %

figure7.5:Perceivedinclusionofprioritisedgroupsincommunitypreparednessplans* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.

Overall,thelackofintegrationofactivitiestopromoteinclusionacrosstheprogrammeresultedinafragmentedprogrammewideapproachandunevenresults.Whiletherewasnoevidenceofindividualknowledgechange,therewasevidenceoforganisationalchange,inparticularinEthiopia and among local organisations across the intensive countries.Whilemonitoring ofgenderdataimprovedoverthecourseoftheprogramme,reportingonotherprioritisedgroupsremainedlow.

QUESTION 5.C: IN WHAT WAYS HAS DEPP INFLUENCED INSTITUTIONAL AND POLICY ENVIRONMENTS?

There have been several concrete examples where DEPP has impacted government policies or systems or increased political commitment but there was no statistically significant change across the DEPP in policy-related quantitative indicators.

Institutionalandpolicyenvironmentchangeswerecapturedattheprogrammelevelthroughlogicalframeworkdataandthroughindicatordataprovidedtotheevaluationteambyprojects.Thiswastriangulatedwithdocumentsreviewedaspartofthedocumentreview.Accordingtologicalframeworkdata,anumberofprojectshavecreatedpolicybriefs,recommendationsandreports to influencepolicyon a rangeof topics suchas collaborativeplatforms, localisationofaidandageanddisability inclusion.Engagementwiththegovernment, theUnandotherinfluential actors alongwith advocacy eventswere also undertaken as strategies to createchangeininstitutionalandpolicyenvironments.

Thereareseveralexampleswheretheseeffortshaveimpactedgovernmentpoliciesorsystemsorincreasedpoliticalcommitment.forexample,theUrbanEarlyWarningEarlyActionproject’sadvocacy and sensitisation efforts led to an amendment of the Disaster and EmergencyManagementAct (DEMAct), 2015 to include food security and anMOUwith thenairobicitycountyGovernmenttoensurethegovernment’srole inaddressingurbanfoodsecurityissue.156 ThelinkingPreparednessResilienceandResponseprojectdemonstratedchangesattheinstitutionallevel.forexample,christianAidadoptedtheupdatedresilienceframework,157

andaViolencetoPeacestrategy.158financialEnablers’reportedexampleswherecSOpartnerswereable tosecureseats in thenationalgovernmentDisasterRiskReductionManagementcouncilsandtechnicalworkinggroupsprovidingopportunitiesforengagementandcollaboration.ThecDAcnetworkProjectadvocatedfor inclusionofcommunicatingwithcommunities inthebangladeshGovernment’snationalStandingOrdersonDisasters(SOD),whichiscurrentlyunderrevision(seeSection9).ThefinalversionoftheSODhasyettobereleasedbutisexpected

156 UrbanEarlyWarningEarlyActionfinalEvaluationReport.157 https://www.christianaid.org.uk/sites/default/files/2017-05/resilience-framework.pdf.158 https://www.christianaid.org.uk/resources/about-us/tackling-violence-building-peace-global-strate-

gy-2016.


toincludethedesiredchanges.

InordertofurtherexamineinfluenceoftheDEPPoninstitutionalandpolicyenvironments,anumberofindicatorscollectedwithinthefourintensivesetcountrieswereanalysed(seeboxbelow).

KEY POLICY INDICATOR DATA COLLECTED AT TWO TIMEPOINTSIndividual level:

• Perceived individual influence within organisation: How would you rate YOUR ability to influence the institutional and policy environment within your organisation? Where 1=No Voice or Influence, 2=Minimal Influence, 3=Some Influence, 4=Significant Influence, 5=Strong Influence, seen as equal partner


• Perceived influence of organisation on institutional and policy environment: Organisational survey with senior leadership. How would you rate your organisation’s ability to influence the institutional and policy environment? Where 1=No Voice or Influence, 2=Minimal Influence, 3=Some Influence, 4=Significant Influence, 5=Strong Influence, seen as equal partner

• Perceived influence of organisation on national preparedness systems: Organisational survey with senior leadership. How well is your organisation or country office able to influence gov-ernment policies related to national preparedness systems [in country of survey]? Where 1=No Voice or Influence, 2=Minimal Influence, 3=Some Influence, 4=Significant Influence, 5=Strong Influence, seen as equal partner

• Inclusiveness of organisational policy towards L/NNGOs: Organisational survey with senior leadership. On a scale of 1 to 5, how inclusive is your organisation’s policy / approach to work-ing with local / national NGOs? Where 1=Very Exclusive, 2=Exclusive, 3=Somewhat Inclusive 4=Inclusive, 5=Very inclusive

• Change in organisational policy on working with L/NNGOs: Has there been any change in your organisation’s policy and/or approach to working with local or national NGOs in the last 1 year? Where 1=More inclusive, 2= No change, 3=Less inclusive

The indicators focus on individual influence within one’s own organisation, as well as theorganisation’s influence on the policy environment in general or specifically related tonationalDisasterManagementPlans.Toassesschangesrelatedtolocalisationpolicy,dataoninclusivenessoforganisationalpoliciestowardsl/nnGOsaswellaschangesinthesepolicieswerecollectedandanalysed.

DEPP DIFFERENCE

(T2 - T1)

CONTROL DIFFERENCE


Individual Influence Within Organisation +1 . 6% -2 . 2 % +3 . 8 %

Organisational Influence on Institutional & Policy Environment

- 4 .7 % +1 .1% -5 . 8 %

Organisational Influence on Policies related to National Preparedness Systems

+2 . 4% +3 .1% - 0 . 6%


Organisational Practice with Respect to Government Disaster Management Plans

+1 . 4% +1 . 8 % - 0 . 4%

Inclusiveness of Organisational Policy to L/NNGOs

+1 . 3 % +1 . 4% - 0 .1%

Change in Organisational Policy on working with L/NNGOs

- 0 . 6% + 0 . 2 % - 0 .9 %

figure7.6:changeinIndividualandorganisationalpolicy-relatedindicators

* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.

Table7.5providesthefindingsacrossallcountries.Ascanbeseen,despitesomeofthepositivepolicy changeexamples thathaveemerged from thedocument reviewprocesses, therearenoobserved significant changes in anyof the six policy-related indicators overtime.Theseindicatorsmeasurebothperceptions,aswellasthelevelofinfluencewithinacomplexpolicyenvironment.The lattermaytakesignificanttimetochangeandmaynotfullycaptureothertypesofpolicychangeorshiftsinthelandscapethataremoredifficulttomeasurequantitatively.

Changes in quantitative indicators related to localisation have not yet occurred except in Ethiopia where the percentage of DEPP organisations with policies inclusive to L/NNGOs increased by 16% between T1 and T2.

Ontheissueoflocalisation,anareathathasbeenastrongfocusoftheDEPPsinceitsconception,qualitativedatacollectedaspartoftheevaluationandduringevidencevalidationworkshopsineachofthefour intensivesetcountriessuggestslowbutsteadychanges inattitudesandmindset favouring amore localised approach.The speed andextentof this typeof changeseemed to depend to some degree on the specific contextual setting, political climate andotherexternalfactors.InKenya,forexample,severalrespondentsdescribedananti-localisationeffectoccurringasInGOssetuplocalbranchofficesandindoingsopushedoutl/nnGOs.Therepercussionsofthishavealreadybeenfeltasonel/nnGOdescribedtheirannualbudgetdropping by 90% due to the influx of InGOs into local areas. Despite this,more positiveinstancesofchangesinpowerdynamics,greateraccessofl/nnGOstoleadershippositionsandfundinghavebeendocumentedinseveralcontexts.Theseareexploredinmoredetailinthenextsection.

The question of whether this perceived shift in mindset and attitudes contributed to anymeaningfulchangeintermsoforganisationalpolicyornumberofpartnershipswithl/nnGOswasalsoassessed.Withrespectto inclusivenessoforganisationalpoliciestol/nnGOsandchange in policies related toworkingwith l/nnGOs (figure7.5), therewas no statisticallysignificantchangeovertimeduetotheDEPPinthepooleddataorinanyofthecountry-leveldataexcept forEthiopia.Respondents fromDEPPorganisations inEthiopia reporteda16%increaseinthelevelofinclusivenessoftheirorganisation’spolicestowardl/nnGOs,andthiswassignificantatthe0.05level.However,duetosmallincreasesininclusivenessofEthiopiancomparisonorganisations, thenetDEPPeffectwasnotstatisticallysignificant inEthiopia. Inothersettings,thereisnoevidencethatlocalisation-relatedpolicieshavechangedasofyet.

Inaddition,furtherdatarelatedto localisationarepresented inchapter5.networkanalysisshowsnochangeovertimeintheproportionoflocal/nationalnGOswithinthehumanitariannetworksintheintensivesetcountriesandlimitedchangeintermsofthenumberofrelationshipswithl/nnGOs.Thosefindingsareconsistentwiththefindingabovethatorganisationalpolicyrelatedtol/nnGOshasnotchanged.

Takentogetherthesefindingssuggestthatwhilechange inquantitative indicatorsrelatedtolocalisationhavenotyetoccurred(exceptinEthiopia),therehasbeensomeprogresstowardsqualitativeattitudinalchangethatisnecessarytoeventuallyleadtomoresystem-levelchanges.


Itisnotclearhowclosewearetoatippingpointwherechangesoccurandcanbestatisticallydetected,butmomentumisgrowing,especiallyincertaincontexts,anditiscriticalthatthesegainsnotbelost.PerceivedqualitativechangearoundlocalisationwillbefurtherdiscussedinthenextsectionandsustainabilityandleveragingDEPP’sinvestmentsintolonger-termchangewillbeexploredlaterinthischapter.

QUESTION 5.D: WHAT IS PERCEIVED TO BE THE MOST SIGNIFICANT CHANGE ATTRIBUTED TO DEPP, AND WHY? Qualitative data captured attitudinal change towards the localisation approach including in how INGOs consider, address and involve local actors and communities, which is considered to be DEPP’s most significant change.

consistently, project and programme level stakeholders identified the adoption of the localisation approachas themostsignificantchangeattributable to theDEPP.Stakeholdersnoted observing a shift in the way InGOs consider, address and involve local actors andcommunities,andattributedittotheDEPP.forexample,oneintervieweestated:

“[DEPP]haspushedthelocalisationagendaatsuperspeedandit’sreallyhappening.Probablyalloftheelementswerethere,butasasystemtheyreallyneededjustatinypushtomakeithappen.TheDEPPallowedthat.”

Theoften-citedtendencytoconsiderlocalorganisationsassub-contractorsorimplementers,rather than as decisionmakers or equal partnerswas also perceived as changing in certaincountries.Anin-countryrespondentinPakistandescribedthischange:

“The most significant change among international organisations, I believe, is therealisation that local organisations are not only the implementing agent but theyarepartnersandtheyneedtobetreatedaspartners.ThisissomethingthatIhaveobservedinthebehaviour,thechangethatIhaveobservedinthementororganisation…revisiting theirpartnershipmodalitiesandprocess throughwhich theycankindofreducethesubvertingapproach.Thatkindofapproachisnowdiscouragedandnewmodels are coming in by these international organisations.before this, itwas justkindofadonorandreceiverkindofrelationship,likeagrantpartnership.butnowit’sbeyondthat.Internationalorganisationsarecommittingtoinvestingmorewhilethelocalactorsarehavingalong-termengagementwiththem.”

Otherexamplesofthistypeofchangeweredescribedinbangladeshwherelocalactorsbecameinvolved indecision-makingprocesseswithin a largenationalpreparednessplatform, and inthePhilippineswherelocalorganisationswerethefirsttomobiliseandrespondtothecrisisinMarawi.Thesetypesofshiftscontributedtoincreasedconfidenceamonglocalorganisationsandnotonlybecameanopportunityforexperientiallearningbutalsoprovidedmomentumtothecause.Additionally,theopeningoftheUnPooledfundtol/nnGOs(forexample,ShiftingthePowerlocalpartnersinDRcandPakistanbecameeligibleforthesefunds)andthegreateraccessofl/nnGOstotheInter-AgencyStandingcommitteeclustersystemsignallednotableprogresstowardsthelocalisationagendawithinahumanitarianarchitecturetypicallyknownforitsslowpaceofchange.

Itwas noted that further progress in this domain towards a concrete “re-balancing” of thehumanitariansystemwouldrequireenhancedunderstandingofthebarrierstothelocalisationapproachindifferentcontexts.forexample,inEthiopia,wheretherearegovernmentrestrictionson civil society organisations, advocacy effortsmay need to take a different shape than inthePhilippineswhere civil society is engrained in the cultural fabric andprovided spacebythegovernment.nevertheless, acrossprogrammecountries,DEPPstakeholdersagreed thatideologically, a localisation approach is critical to improving preparedness and humanitarianresponseandthatincrementalchangesarebeginningtooccurontheground.Theseviewsandtheobservedchangesthathavebeendocumentedabovealignwellwiththeglobaldiscoursearound localisationofhumanitarian aidwhichhas gainedground, especially since the2016WorldHumanitarian Summit. To ensure further progress beyond the end of theDEPPwillrequiresustainedeffortstobuilduponthechangesthathavebeguntoemerge.


CONCLUSION: TO WHAT EXTENT AND IN WHAT WAYS HAVE THE BENEFITS OF THE PROGRAMME BECOME EMBEDDED?

There is some evidence of embedded benefits and the potential for some longer-term effects but this is dependent on the extent to which different programme components are able to obtain support to continue beyond the end of DEPP.

DEPP’sthree-yearprogrammecycletoestablishandimplement14projectsin10countrieswithdifferingdisasterprofilesthroughaconsortia-focusedapproachendsinJune2018;allprojectsexceptthelearningProjectcompletedtheiractivitiesbytheendofMarch2018.Toassesstheextenttowhichbenefitsoftheprogrammehavebecomeembeddedandhavethepotentialtoachievelonger-termeffects,documents,projectendevaluationsandqualitativedatawereanalysedacrossseveralareas:sustainability/exitplanningoverthecourseoftheDEPP,takeupof sustainability-focused recommendations, the sustainability of programme/project outputsandoutcomes,andlikelihoodoffutureimpact.Itisimportanttonotethatmostprojectshadapproximately twoyearsor less to implement their activitiesdue to approval of projects inwavesaswellasthedelaysinstart-up.Thisprovidedlimitedtimetosupportsustainability.

Sustainability / Exit Planning

Theinterimreporthighlightedanumberofweaknessesrelatedtosustainabilityplanningattheprojectandprogrammelevels.Individualprojectswererequiredtoprovidesustainabilityplansintheirproposals;however,thesegenerallywerenotverydetailedwithrespecttotheprojects’sustainabilitymodels,didnot includesufficient informationregardingcostsofsustainingtheproject, potential revenue sources, integration intocommunitiesor local systemsor specificactionstobetakenforhandover,andinmostcasestheywerenotfurtherdevelopedthroughoutthecourseoftheproject.Anoverallprogramme-levelsustainabilityplanwasnotadequatelyoutlinedduringthedesignoftheprogramme,whichwasaconsiderableweakness.basedonstakeholder interviews, therewas little direction provided from the programme level to theprojectsontheissueofsustainabilitywhichhasbeenproblematicforprojectsandcontributedtouncertainty.

The lack of adequate sustainability planningmay have also been influenced by the explicitdescription of a contingent allocation of £60million for a future phase of the programmeby DfID in the business case: “An extension to the programme, possibly for 2 years, willbe considered subject to the findings of amid-term review supported by the independentevaluation(in2015).ThepreviousSecretaryofState(AndrewMitchell)hasgivenin-principleagreementforupto£100millionover5years.”159Despitethisstatement,furtherclarificationonwhensuchadecisionwouldbemadeortheprocesswasnotmadetransparenttoDEPPstakeholdersoverthecourseoftheprogramme,butneverthelesssethighexpectations.Thisreportedly influencedhow intensivelyDEPP stakeholders pursuedother funding sources aswellaslikelydelayedthedevelopmentofanexitplan.forexample,onerespondentstated:

“Ithink[sustainability]hasbeenachallengebecauseIthinkifpeopleknewthat[morefundingfromDfID]wasn’tanoptiontheycouldhavethoughtfromtheverybeginninghowtostrategicallyplanforthat.”

In October 2017, DfID announced that it would not be funding a second phase of theprogramme.This announcement cameonlyfivemonthsbefore theprogrammeclose ratherthanat themid-pointof theprogrammeas laidout in thebusinesscase.At this late stage,littletimewasleftforboththeoverallprogrammemanagementteamandindividualprojectstoseekalternativesourcesoffunding.However,fromariskmanagementperspective,DEPPprogrammemanagement and leadership shouldhaveattempted to raise alternative fundingordiversified its fundingsourcesandstrengthenedotherprocessesofsustainabilitysuchasembeddingDEPPwithinlocalsystemsearlieronintheprogrammecycle.

The interimphase evaluation report provided strong recommendations formore deliberate,strategicsustainabilityplanningandtheneedforguidanceonsustainabilityplanningandopendiscussion initiatedwith projects about the availability of continued funding to extend the

159 businesscaseInterventionSummary:DisastersandEmergenciesPreparednessProgramme,DfID.


DEPP.currentdatashowsthattheserecommendationsweretakenonboardandanumberofprojectshavemadeprogresswithrespecttosustainabilityplanning.Allprojectendevaluationsandfinalprojectnarrativereportsalsoincludedasustainabilitysection.Inaddition,theSTARTnetworkrecruitedaconsultanttoevaluateoptionsforcarryingtheDEPPforward includingfundingoptions.

Current status of exit planning

Atthetimeofthisreport,anumberofprojectsdemonstratedsubstantialprogressintermsofexitplanning.forexample,theUrbanEarlyWarningEarlyActionproject’ssurveillancesystemhasbeenadoptedbythenairobicountygovernment,andmanagementofthePakistanSurgerosterhasbeentakenoverbythenationalHumanitariannetwork(nHn),withanMOUinplacewiththelocalgovernmentbodytoensureimplementationaftertheprojectends.Inaddition,local consortia that received support from the financial Enablers project havemade planstocontinuetocollaborate.Ontheotherhand,otherprojects facedisruptiondueto lackofsustainedfundingandlossordispersionofprojectstaff.Thismaylimitpotentialforsustaininganypositiveeffectsandbuildingoncurrentaccomplishments.There isalso thepotential tolosesomeofthegainsthatweremadebytheDEPP.Staffmemberswhocontinuewithinthesameorganisations,togetherwiththelearningproductsgeneratedacrosstheDEPPandbythelearningProject,willconstitutethelargestpartofDEPP’s institutionalmemory.WhilesomeprojectsandprojectcomponentsshowindicationsofbecomingembeddedandmaycontinueonaftertheDEPPinsomeform,itisclear,thattheprogrammeasawholewillnotcontinueinitscurrentconfigurationandwouldneedtoberemodelledforanyfuturephases.

Sustainability of programme/project outputs and outcomes

Atthetimeoftheprogrammeclose,therehavebeenpositivesignsofemergingsystem-levelchanges, including shifts in attitudes towards localisation and changes in organisational andgovernmental policies as reported earlier in this chapter, aswell as strengthenednetworks.Mostintervieweesexpressedthatmoretimewasneededtoembedthesechanges.Datafromproject end evaluations, narrative reports and in-depth interviewswere analysed to furtherassesstheextenttowhichDEPPoutputsandoutcomeshavebecomeembedded.

Sustainable Outputs – Achievements and Contributions to the Four Output Areas

Attheoutputlevel,anumberofachievementsoftheprojectscontributetothesustainabilityanddemonstratethatbenefitshavebecomeembedded.

Collaboration

collaboration and partnerships are important contributors to achieving sustainability.RelationshipsbuiltthroughcollaborationandconsortiawereperceivedtobeoneofthemostenduringelementsoftheDEPP,howeverthesewerecontingentonthestructureandset-upofconsortiaandalsowereinfluencedbyfactorssuchasstrongleadership,politicalwill,competingallegiances, and availability of funding. for example, the financial Enabler’s final evaluationsuggeststhatseveraloftheconsortiafundedbytheprojectinthePhilippineswilllikelybreakdownorstruggletocontinuetoworktogetherduetoreasonsincludingincompatibleplansandlackofstrongcommitmentamongstleadershiptoprioritisefuturecollectiveaction160.Oneoftheconsortiareportedlystoodoutashavingamuchhigherlikelihoodofsustainabilityduetoasharedcommitmenttolocation,priorexperienceofjointaction,complementarycompetencies,demonstratedlearningandcommittedleadership161.Stakeholdersalsoemphasisedthatbuildingsustainablecollaborationandpartnershipswaschallengingtoaccomplishwithintheprovidedtimeframe.forexample,oneintervieweestated:

“Idon’tthinkwhatwearetryingtoachieve,itcouldbeachievedinthistimeframe.I thinkbecause it’snotonlywithdeliveringprojectactivitiesanddone, itsactuallydoingmorethandeliveringactivitiesbecauseit’spairedwithchangingorganisationalculture,changingorganisationalbehaviour,bringingchangeinthemindsofdisasterpractitioners,soIthinkitrequiresmoretimethanit’sdedicatedfor.”

160 financialEnablersfinalEvaluationReport.161 Ibid.


Capacity Building

buildingcapacityhasaninherentpotentialforsustainabilityatalllevelsofaction(i.e.,individual,organisational, community capacity). However, these capacities need to be maintainedor reinforced or risk being lost. likelihood of sustainability is higher among capacities builtbeyondknowledgechangesuchasthosethat involveskillbuildingandbehaviourchangeattheindividuallevel.Ifretained,improvedskillsandbehaviourscancontinuetoeffectchangeandcanalsobetransferredtoothersbeyondthelifeoftheproject.Attheorganisationallevel,improvedcapacityintheformofchangedpoliciesorgovernanceorputtinginplacesystems(humanresources,finance,procurement)tooperationalisecapacitybuildingwerealsonotedtocontributetoincreasedsustainability162.Itwasalsonotedthatsustainabilityattheorganisationallevelwouldbedependenton internal approvalof thenewstrategiesandpoliciesand theirimplementation,butthatthiswouldbeinfluencedbythenature,sizeofoperationsandscaleof humanitarian activities of the organisations.163Therewere also examples of Shifting thePowerbeneficiaryorganisationsinPakistanwithimprovedgovernance,whoarenoweligibleforadditionalfundingbecauseoftheseimprovedinternalsystems–demonstratinghowimprovedgovernancecanhelptoensurefuturesustainability.

Generating and Sharing Learning

Thegenerationoflearning,anotherkeyDEPPfocusarea,alsocontributedtowardsthegoalofsustainability.ThedevelopmentandcreationofDEPPprojectinfrastructuresuchasprotocols,tools,proceduresandguidelinesaresustainableinthattheywillcontinuetoexistaftertheendoftheprojectperiod.forexample,theAgeandDisabilitycapacityProgramme’stheGoodpracticeguide;embeddinginclusionofolderpeopleandpeoplewithdisabilitiesinhumanitarianaction,TransformingSurgecapacity’sThefutureofHumanitarianSurge,andProtectioninPractice’sGPcProtectionMainstreamingToolkit,areexamplesofsomeofthetools, learningproductsandguidesthatwillhaveacontinuedeffectaftertheDEPPends.notably,anumberoftheseareavailableinmultiplelanguagesandthroughdifferentchannelsincreasingtheirpotentialforcontinueduse.forexample,theProtectioninPracticeGPcProtectionMainstreamingToolkitisavailableinUrdu,bengali,burmese,french,Arabicandavailablethroughawebsiteandasamobileapp through theGoogleplayand iTunesAppStore.Toensure that these learningproducts and tools becomemore embedded into the humanitarian sectorwill also requiresomedegreeofcontinuedadvocacy,awarenessbuildingandpromotionaleffortswhichmaybedependentonavailabilityofadditionalfundsandorganisationalcommitment.AstheleadofthelearningProject,AcfhasmadethecommitmenttokeepthedigitallearningplatformforanadditionaltwoyearsaftertheclosingofDEPPandwillcontinueoutreachanddisseminationefforts.

Early Warning System Development

functioningearlywarningsystemsalsoincreasethepotentialofprogrammesustainabilityastheyarelikelytocontinueaftertheprogrammeperiod.However,sustainingthesesystemsisdependentonsecuringorganisationalandgovernmentbuy-in,ensuringtheadoptionanduseofthesystems,andhavingfundingearmarkedformaintenance.Severaloftheearlywarningprojects,includingtheUrbanEarlyWarningEarlyAction,ImprovedEarlyWarning–EarlyActionstoStrengthenDisasterPreparednessinEthiopiaandPublicHealthPreparednessinGambella,haveahigherlikelihoodofsystemschangeandhavebecomeembeddedwithinlocalandnationalstructureswhichwillensurelonger-termsustainability.Thesethreeprojectshaveincommontheapproachofworkingwithinestablishedsystemsandstructuresandinvolvingkeyexternalstakeholders such as the local governments and local health systems.These three projectshave confirmed sustainability plans with Kenyan and Ethiopian governments, respectively,whohavebegunincorporatingprojectactivitiesintotheirworkplansandbudgets.WithsomeinfluencefromtheUrbanEarlyWarningEarlyActionproject,theKenyangovernmentpassedlegislation requiringcounties toestablishemergencyanddisastercommitteesat thecountyandsub-county level.Thishashelpedtoensuresustainability forprojectactivitiesafter theDEPPfundingcyclefinishes.InEthiopia,theMinistryofHealthandRegionalHealthbureaus

162 ShiftingthePowerfinalEvaluationReport.163 Ibid.


providedexitstrategiesandguidelinestoensuresustainabilityofactivities.Thesethreeprojectsestablishedconcrete, feasiblestrategiestoensure long-termsustainabilityandcanthereforeprovide important lessonsfortheotherprojects intheportfolio.However, it is importanttonotethatunlessfundingissecuredtoensurecontinuedfunctioningofthesystems,thereisapotentialriskthattheimpactoftheinvestmentinthesesystemswillbediminishedorlost.

Table7.2providesexamplesofachievementsandcontributionstothefouroutputareasfromthefinalprojectnarrativereports,finalevaluationsandinterviewsthatarelikelytobesustainedbeyondDEPPwithsufficientmaintenanceandfollow-up.

PROJECT EXAMPLES OF ACHIEVEMENT/CONTRIBUTION OUTPUT AREA

ADCAP

• Developed and finalised version of the humanitarian inclusion standards for older people and people with disabilities, which are available to agencies throughout the world in various accessible formats and languages

• Developed complementary training resources and guidance on good practice relating to organisational change

Capacity building; Learning

ALERT• Created ALERT early warning system platform

• Contributed to creation or revision of organisational preparedness plans

Early Warning System Development; Capacity building

CDAC-N

• Created message libraries for use during disasters and emergencies

• Developed toolkits and resources for two-way communication with disaster-affected communities

• Established multi-stakeholder platforms

• Strengthened capacities to support two-way communication

Capacity building; Collaboration; Learning

EWEA

• Established early warning system in Ethiopia

• Strengthened capacity of district (woreda) experts and the community

• Developed/strengthened community contingency plans

Capacity Building; Early Warning System Development

FE

• Strengthened local coordination mechanisms and networks

• Empowered and strengthened confidence and experience of local actors to respond to disaster

• Strengthened technical competencies of partners

• CSO partners were able to secure seats in the national government Disaster Risk Reduction Management councils

Capacity Building; Collaboration

LPRR

• Changed attitudes on community involvement in resilience

• Kenya Marsabit County Disaster Risk Management draft Bill has recognised the role of local organisations in coordination of emergency/disaster-related events

• Tools and strategies were integrated by INGOs and LNGOs

• Platform of NGOs and peace dialogue at village level was established

• Integration of LPRR’s conflict-sensitive approach into Marsabit County Government response

Capacity Building; Collaboration


PHEP Gambella

• Provided equipment to health facilities

• Trained existing health staff to implement government strategy

• Improved surveillance and detection of disease and outbreaks

• Developed toolkits


PIP

• Local partners attending Oxfam’s Global Protection Peer Group meeting shared learning with international staff and other local partners

• Developed GBP Protection Mainstreaming Toolkit

• Developed infrastructure to support protection initiatives

• Contributed to changes in organisational policy


SEPS• Established early warning systems at community level

• Supported the development of community preparedness plans


STP

• Contributed to changes in organisational policies, systems and capacities

• Supported development of Emergency Preparedness Plans

• Set up NGO platforms/networks in five countries

Capacity building; Collaboration

TD

• Online toolkits and resources in local languages were developed

• Strengthened partner relationships

• Trained trainers

Capacity Building; Collaboration; Learning

TSC

• Organisational systems developed

• Open-access training, tools, and resources were created

• Funding secured for future use of the Philippines and Regional rosters

• Pakistan roster has been taken over by the National Humanitarian Network (NHN), with an MOU with government body to continue collaboration


UEWEA

• Developed and tested an urban surveillance system that has been adopted by the Nairobi county government

• Contributed to an amendment of the Disaster and Emergency Management Act (DEM Act), 2015 to include food security and a MOU with the Nairobi City County Government to ensure the government’s role in addressing urban food security issues

Early warning system development

Table7.2:contributionsofprojectstowardssustainableoutputs

Sustainable Outcomes

At the outcome level, sustainable changeswere more difficult to detect due to the shortprogramme time frame.As described in chapters 4 and 7, there have been no significantimprovements in knowledge on core humanitarian competencies of humanitarian staff, onself-reportedknowledgeondisasterpreparedness,responsetodisastersandemergenciesorongender,ageordisability-related issues indisasters.conversely, therehavebeennotableimprovements in organisational and community preparedness among DEPP beneficiaries,


especiallyamonglocalorganisations.QualitativedatahavealsodemonstratedthecontributionofDEPPbeneficiariestowardsimprovedresponseinseveraloftheintensivesetcountries(thePhilippines,KenyaandEthiopia).

The extent towhich these outcomes are sustainable remains uncertain at the time of theprogramme’sclose.Sustainingpreparednesslevelsandresponsecapacitiesaredependentonmaintainingskills,knowledge,andcompetenciesattheindividual,organisationalandcommunitylevels.Thesecapacitiesarealsoaffectedbyshiftsinhumanresourcesandstaffturnover,whichwere reported as consistent and recurring issues throughout the programme. furthermore,increasedcapacity also leads to thepotential for individuals tobe recruitedelsewhere, andlocalorganisationsthatparticipatedinDEPPcapacitybuildingactivitiesrisklosingkeystafftoInGOs.Aslongastheseresourcesremaininthedisasterpreparednesssystem,however,theywillbesustainabledespitethetransferandexchangeofresources.

Additionally, preparedness and response are influenced by many factors external to DEPPincludingorganisationalpolicy,leadership,andculture,andchallengesengaginggovernment.Inmostcases,thecontributionsthatDEPPhasmadeinimprovingoutcomesaroundpreparednessandresponsewilllargelybedependentonsecuringadditionalfundstocontinueprojectactivities,maintainsystems,infrastructures,andcapacitiesbuilt,andensurequickresponsestodisasters.

The Role of Community and Beneficiary Stakeholders in Design and the Impact on Sustainability

Asoutlinedinchapter3,beneficiariesatalllevelsaswellaskeylocalstakeholderswerenotadequately involved in thedesign stage formostprojects.While thishadsignificanteffectsontherelevanceandpotentialeffectivenessoftheselectedinterventions,andledtodelaysin implementation as projects had to be contextualised, it also is a threat to the long-termsustainability of the projects and in turn the programme. Overall, beneficiary engagementand feedbackmechanisms to thecommunity shouldbeprioritisedasa strategy to increasesustainability.Infuturesuchprogrammes,beneficiariesshouldbeinvolvedinthedesignphaseinordertoachievehigherlikelihoodofsustainability.Acrosstheboard,interviewsandsitevisitsdemonstrated that using a bottom-up approach of supporting organisations and individualstoleadtheirowncapacitybuildingprocessesinfluencesthelikelihoodofbuildingsustainablecapacities.

Programme Visibility and Sustainability

Programme visibility, often built through strategic communication about a programme,contributes to community, stakeholder and government level support and is an importantconsideration with respect to sustainability. Site visit observations by the evaluation teamandintervieweereportsindicatedthatbothinternal and externalprogrammevisibilityatthecountry levelwere low. Internalvisibility,orawarenessof theDEPPby in-country staffandconsortiamembers involved inDEPPprojects increased through theDEPP learning events,butremainedlimited.Externalvisibility,orawarenessoftheDEPPbynon-DEPPstakeholderswasevenloweratthecountrylevel.WhilebothtypesofstakeholderswereawareofspecificDEPPprojects,mostwereunawareofthelargerDEPPportfoliointheirowncountrysuggestingstrategiccommunicationabouttheportfoliomayhavebeensuboptimal.

AsmanylocalDEPPprojectstaffdidnotidentifythemselvesasbeingpartoftheDEPP,orworkjointlywiththeotherin-countryprojectstowardsthecommongoalsandobjectivesoftheDEPP,itisunsurprisingthattherewereminimaljointadvocacyorcommunicationefforts.Whilemostprojectstaffdidfeelastrongsenseofconnectiontotheirownprojectsandownorganisations,thelackofconnectiontotheDEPPbrandandthelimitedfocusonoverallprogrammeobjectivesaffected longer-termprogramme sustainability through reduced community and stakeholderdialogueandprogramme-levelbuy-in.Ultimatelytherewaslessintegrationoftheprogrammeinto relevant structures inDEPP countries.At the regional or global level, one stakeholderdescribedlimitedawarenessandunderstandingoftheDEPPatimportantexternalevents,suchastheWorldHumanitarianSummit.Anumberofrespondentsalsosuggestedthat increasedadvocacyinitiativesandpromotionoftheDEPPwereneededtoraisethelocalprofileoftheprogramme.


Innovation, Sustainability and Contingency Planning

TheDEPPportfoliowasenvisionedtoincludeamixofflagshipprojectsthatemployedmoretraditionalstrategiesandapproachesaswellasothermoreinnovativeandsometimesriskierprojectswithahigherpotentialtofail.Elementsoftheportfoliothatwereparticularlyinnovativeinclude:

• flexiblefundingmechanisms(i.e.,fE,PIP,cDAc-n,lPRR)• Urbanearlywarningsystemstoenabledetectionandresponseofemergenciesin

urbansettingsincludingrelatedtofoodsecurity(i.e.,UEWEA)• Regionalpooledsurgeplatforms(i.e.,TSc)

Incaseswhereinnovationmeanshigherriskoffailure,sustainabilityplanningdoesnotmakesenseandisnotagooduseofresourcesuntilresultscanbedemonstrated.Theseshouldbetreatedaspilotprojectsandthereforeshouldnotberequiredtoestablishsustainabilityplanningandconsiderationsduringthedesignphase.Insteadtheseprojectsshouldberequiredtohavestrongmonitoringandevaluationplansinordertoinformcoursecorrectionandpotentialscale-up.Inaddition,allprojects,regardlessoflevelofrisk,shouldhaveappropriatecontingencyplansinplacetonavigateexpectedorunexpectedcontextualfactorssuchasdisruptionofprojectactivitiesduetodisastersandemergencies.Therewasatleastoneexamplewheresuchplanswerenotinplace(cDAc-nprojectinSouthSudan),resultingincriticaleffectsontheprojectwhenadisasteroccurred.Therewereseveralotherexamples,albeitonasmallerscale,whereprojectactivitieswereaffectedbyemergencies(i.e.,reportingandlearningActivitiesrelatedtocDAc-ninbangladeshwerelimitedduetofocusonRohingyacrisis164,andbothdroughtandinsecurityhinderedEWEAprojectimplementationinEthiopia165).

Likelihood of Impact and Summary

Overall,moreattentiontosustainabilitywasneededearlierinthecourseoftheprogramme,withamoreholisticviewonsustainabilitygoingbeyondsimplysecuringfundingstreams.Alongerperiodforprogrammeimplementationmayhaveledtogreaterchangesinoutcomesandincreasedthelikelihoodofprogrammesustainability.Thedatasuggestthatlikelihoodofsustainabilitywasenhancedforprojectsorprojectcomponentsthatdemonstratedthefollowingcharacteristics:

• builtonexistingwork(i.e.,UEWEA,TD),andexistingpartnershipsandconsortia(i.e.,STPbangladeshandnationalAllianceofHumanitarianActorsbangladesh(nAHAb);cDAc-ninthePhilippineswherethecWcgroupsareembeddedwithinpre-existingstructures)

• Workedwithgovernmentand/orhadapolicyoradvocacyelement(i.e.,lPRRinKenya,PHEPGambellainEthiopia)

• Workedtowardssystems-levelchange(i.e.,UEWEA)• Paidgreaterattentiontoexitplanningearlierinprogrammecycle(i.e.,UEWEA)• Developedtools,guidelines,orsystemsthatfillanimportantgap(i.e.,ADcAP’s

minimumstandards,PIP’sprotectionmainstreamingtoolkit)• Involvedbeneficiariesand/orlocalstakeholders(i.e.,UEWEA)• ExhibitedgoodVfM(i.e.,cDAc-nfoundationaltrainingcalled“unsustainably

expensive”suggestinglowVfMandchallengesbringingthiscomponenttoscale)• builtlinkageswithotherentities(i.e.,STARTfund)andotherDEPPprojects(i.e.,the

Philippines)• Implementedincontextsthatweremorefertileforchange(i.e.,thePhilippines)

164 cDAc-nfinalEvaluationReport.165 EarlyWarningEarlyActionfinalEvaluationReport.

183 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES


HUMANITARIAN PREPAREDNESS AND RESPONSE CASE STUDIES

cHAPTER8


Evaluation Questions: What are the impacts of violence on project implementation and lessons learned on implementing emergency preparedness and response programming in a conflict-affected setting? How has disaster preparedness translated into actual response? Sub-questions:a) InwhatwayshaveDEPPcapacitybuildingprogrammesstrengthenedpreparednessandresponse

capacityamongstparticipants?b) To what extent is DEPP contributing to greater preparedness and response among local

organisationsandcommunities?c) HaslocalcapacitytorespondtodisasterschangedsincethestartofDEPP?Ifyes,how?Ifnot,whynot?d) HasDEPPledtoimprovedknowledgeandunderstandingofbestpracticesrelatingtodisasterand

emergencypreparednessandresponse?Ifyes,inwhatways?e) Whatwas the role of government and/or pre-existing systems?How did these facilitate the

responses?Orhowdidalackoftheseinhibitresponse?Method and data sources: Twocasestudieswereconducted.caseStudy#1examinedtheimpactofescalationofviolenceinSouthSudanonDEPPprojectimplementationandcaseStudy#2assessedthe relationship between emergency preparedness and humanitarian response in Ethiopia andKenya.Thefollowingdataformedthefindingsofthetwocasestudies:• Documentreview(Qualitative)• Semi-Structuredinterviews(Qualitative)• SiteVisits(Kenyaonly,sitevisitsinSouthSudanandEthiopiawerenotpossibleduetosecurityissues)• Datacollectedduringtheminimumsetdatacollection(Qualitative)Key Findings:Case Study #1onSouthSudandemonstratedthatoperatinginaconflict-affectedcontextrequiresprocedures that allow for flexible and adaptive management, rather than traditional and rigidcontractualrelationships.Thefindingsemphasisetheimportanceofhavingclearagreementsaboutdutyof carewhencreatingmulti-lateral agreements amongmultiple institutions in a consortiumand the responsibilities of care for the host agency in relation to consortiumpartners. findingsalso highlight the need to have a contingency plan including budget contingencies in case ofdisruptiontotheproject.finally, thecasestudycitedchallengesofoperating inconflictsettingswhereengagementwithgovernmentstakeholdersisnotalwayspossibleandthatcapacitybuildingapproachesandmaterialsmayneedmoreadaptationsforconflictsettings.Case study #2,onEthiopiaandKenya,demonstratedthattheconsortiummodelandcollaborationwithgovernmentwerecriticaltosuccessfulimplementationofearlywarningsystemsandpermittedcoordinated humanitarian responses that improvedwith each subsequent emergency in Kenya.Availabilityof immediate response funds andpreviously agreedupon response actionswere keytoensuringmorerapidresponses.InEthiopia,self-reporteddatasuggestthatexperientiallearningrelatedtofundraising,resourcemobilisationandimplementationofhumanitarianresponseactivities,alongwithaccess toflexibleresponse fundingmechanismscontributedto responsesbysevenl/nnGObeneficiariesoftheDEPPtoaddressthedrought.Keythemesthatemergedincluded:• communityandgovernmentengagementwereessentialtosuccessfulprojectdesignandimplementation• communityandorganisationalcapacitybuildingstrengthenedpreparednessandresponsethrough

improved awareness of humanitarian standards, coordination, and procedures, and increasedvisibilityoftheorganisationsandthecredibilityoftheiremergencypreparednessandresponseactivitiestodonorsandotherstakeholders

• collaborationimprovedcommunicationandefficiencyofresourcesused• Surveillanceandearlywarningsystemswerefundamentaltobothprojects(KenyaandEthiopia),

andweredescribedasthepillarsofemergencypreparednessandresponse.beyondcollectingdata, respondentsnoted that these also servedasmechanismsof collaborationandprovidedopportunitiesforbehaviouralchangeforcommunitiesandorganisations.Definitionsofdisastersdifferedbetweenurbanand rural contexts.Povertyand food insecurityare rarelyconsidereddisastersandgetvariedlevelsofattentionfromdonorsandstakeholders.

• EngagementwithDEPPprovidedopportunitiesfororganisationstosharelearningandbestpractices• Individualconfidencetorespondtoadisasterincreased;however,therewerefewexamplesof

behaviourchange• Organisationalpoliciesandprocedureswerestrengthened,andcollaborationwiththegovernment

inKenyaallowedforsystems-levelchanges,includingalignmentandintegrationofprojectactivitieswithinbroaderplatforms,suchasconstitutionalprioritiesandnationalpolicies

• Theconnectionbetweenemergencypreparednessandresponseactivitiesweredisjointedandneedtobecloselylinked


INTRODUCTION AND RATIONALE FOR THE CASE STUDIES: SOUTH SUDAN AND KENYA AND ETHIOPIA In-depth case studieswere added to the evaluationmethodologyduring the formative andinterim phases to provide further perspectives on emergency preparedness and responseactivities in the programme, and to assesswhether improved preparedness translated intomoreefficientandtimelyhumanitarianresponse.SimulationexerciseswereinitiallyproposedduringtheinceptionphaseasamethodtomeasureDEPPbeneficiaries’performanceduringhumanitarian response activities in a controlled manner. The intentionwas to collect suchdataduringsimulationsthatwereplannedbyDEPPprojectsthemselves.However,overthecourseofthethree-yearprogramme,projectschangedtheirplansregardingsimulations,anditwasalsonotedthatseveraloftheDEPPprojectsrespondedtoactualdisastersintheprojectcountries.Thisprovidedanopportunitytodirectlyassesscontributionstoactualhumanitarianresponses. After reflection and discussion with the learning Project, the evaluation teamdecided to conduct two in-depth case studies.The first case studywas focused on SouthSudantocaptureexperiencesandperspectivesofprojectstakeholdersrelatedtotheJuly2016escalationofviolenceandhowthisimpactedprojectimplementation.Thesecondcasestudyinvolved assessing humanitarian responses in Ethiopia and Kenya during the course of theDEPPtocapturetherelationshipbetweenpreparedness,inparticularearlywarningsystems,andresponseefforts.

SouthSudanwasselectedasoneofthefivecountrieswhereintensivedatacollectionwouldtakeplaceduringtheevaluation.Datacollection,however,waspostponedduetoincreasedviolenceinJuly2016,whichsignificantlyaffected theDEPPprojectsoperating in thatcountry.AfterdiscussionswiththetwoprojectsthatwereoperatinginSouthSudan(cDAc-nandProtectioninPractice),andmonitoringofthesecuritysituationinSouthSudan,adecisionwasmadeinOctober2016todiscontinuethein-countrydatacollectionplannedforSouthSudan.Instead,theevaluationteamplannedtodevelopacasestudy,toexplorehowthesetwoprojectswereaffectedbytheviolenceandthelessonsthatcanbeextractedfromthisexperience.Thiscasestudywillbepresentedfirstasitwillprovideinsighthowacrisiscanaffectprojectoperations.

Thesecondcasestudyfocusedonpreparednessandresponseeffortsintwoselectedcountries:Kenya(UrbanEarlyWarningEarlyActionproject)andEthiopia(ShiftingthePowerproject).Aspartoftheexternalevaluation’sdocumentreviewprocess,a listwasgeneratedofcountriesinwhichatleastoneresponsetoadisasteroremergencywasnotedinDEPPprojectreportsupuntilendof2017.forthecasestudy,thecountriesinwhichintensivedatacollectionwasconductedaspartoftheexternalevaluationwereconsidered(Kenya,Ethiopia,thePhilippines,andMyanmar).ThePhilippineswasexcludedasacountryoffocusgivenacasestudyhadbeenrecentlyconductedbythelearningProject.Myanmarwasexcludedgivenongoinginsecurity.Responseefforts inKenyaandEthiopiawerethereforeselectedtobethefocusofthecasestudy.

Thetwocasestudiesaimedtoexplorethefollowingevaluationquestions:

• What are the impacts of violence on project implementation and lessons learnedon implementing emergency preparedness and response programming in a conflict-affectedsetting?(SouthSudan)

• InwhatwayswereDEPP’sdesignandcorecomponents(capacitybuilding,collaboration,earlywarningsystems,andlearning)effectiveorineffectiveinstrengtheningemergencyresponseinDEPPtargetcountries?(EthiopiaandKenya)

• InwhatwayshaveDEPPcapacitybuildingprogrammesstrengthenedpreparednessandresponsecapacityamongstparticipants?(EthiopiaandKenya)


• TowhatextentisDEPPcontributingtogreaterpreparednessandresponseamonglocalorganisationsandcommunities?(EthiopiaandKenya)

• HaslocalcapacitytorespondtodisasterschangedsincethestartofDEPP?Ifyes,howhasitchanged?Ifnot,whynot?(EthiopiaandKenya)

• HasDEPPledtoimprovedknowledgeandunderstandingofbestpracticesrelatingtodisasterandemergencypreparednessandresponse?Ifyes,inwhatways?(EthiopiaandKenya)

• Whatwastheroleofgovernmentand/orpre-existingsystems?Howdidthesefacilitatetheresponses?Orhowdidalackoftheseinhibitresponse?(EthiopiaandKenya)


CASE STUDY #1: EMERGENCY PREPAREDNESS AND RESPONSE IN SOUTH SUDAN BACKGROUND

following South Sudan’s independence in 2011, political struggles led to renewed conflictin2013166.Within thecontextofongoingconflict,violenceescalated inJuly2016betweengovernment and opposition forces in Juba, affecting both civilians and aid workers167,168.Deepeninganalreadycomplexhumanitariancrisis,theviolenceresultedindisplacementandfamine,withanestimatedsixmillionSouthSudanese inneedofhumanitarianassistancebyAugust2016169.Thehumanitariansituationcontinuedtodeteriorate,andatthetimeofthiscasestudyin2017,therewasongoingviolenceinotherstatesofSouthSudantargetingbothciviliansandaidworkers170.TheviolenceandinsecurityaffectedtwoDEPPprojectsinSouthSudan:1) thecDAcnetworkcommunicatingwithcommunities (cWc)project and2) theSTARTnetworkProtectioninPracticeproject171,172.

The cWc project was delivered as part of a consortium amongWorld Vision, Internews,bbcMediaAction and national and local organisations, and aimed to improve emergencypreparednessby increasing informationexchangeandcommunityengagement173.ThecWcproject supportednovel, experimental ideas, andmodels to increase information sharing toimproveresponseandenablethosemostaffectedtoinformtheresponse.Theprojecthadfivedifferentcomponents:1)aflexiblefundingmechanismwherebylocalorganisationsappliedforfundsthroughthebroaderprojects,2)acWcinEmergencyWorkingGroupofagenciesco-chairedbytheUnitednationsOfficeforthecoordinationofHumanitarianAffairs(OcHA),3)cWctrainings,4)exchangevisitsandnetworking,and5)asimulation174.PreliminaryproposalsformajorworkstreamswerepresentedattheMarch2016meetingoftheSouthSudancWcworkinggroup.AsofJuly2016,theworkinggrouphadbeenestablished,afoundationtraininghadbeenconducted,andaspartoftheflexiblefundingmechanism,fivepotentialsub-projectswereidentifiedrightbeforetheviolenceescalatedinJuba.

TheProtectioninPracticeprojectwasdeliveredaspartofaconsortiumamongOxfam,WorldVision,andInternationalRescuecommittee,andaimedto increaseprotectionexpertiseandcapacity innational-levelorganisations175.Theprojecthadthree interlinkedaims:1) tobuildgreater capacity for protection mainstreaming among national humanitarian staff, nGOs,

166 GuardianT.AttackonaidworkersinSouthSudan:‘Therewasincrediblenaivety’2016[cited201714April].Availablefrom:https://www.theguardian.com/global-development-professionals-network/2016/oct/17/attack-on-aid-workers-in-south-sudan-there-was-incredible-naivety.

167 centerforciviliansinconflict.Underfire:TheJuly2016ViolenceinJubaandUnResponse.UnitedStates:centerforciviliansinconflict,2016.

168 GuardianT.‘Untolddevastation’inSouthSudantriggersgravehumanitariancrisis:TheGuardian;2016[cited201714April].Availablefrom:https://www.theguardian.com/global-development/2016/aug/09/untold-devastation-in-south-sudan-triggers-grave-humanitarian-crisis-un.

169 Unnewscentre.Afterattacks,UnseniorofficialinSouthSudanurgesprotectionofaidworkersandcivilians2016[updated8April2017;cited201715April].Availablefrom:http://www.un.org/apps/news/story.asp?newsID=56528-.WPIcD7uGPfy.

170 cDAcnetwork.communicatingwithcommunitiesinSouthSudan2016[updated11March2016;cited201714April].Availablefrom:http://www.cdacnetwork.org/i/20160311131929-0c4ru.

171 STARTnetwork.ProtectioninPractice[cited201714April2017].Availablefrom:https://startnetwork.org/start-engage/protection-in-practice.

172 RogueAmoeba.Piezo.2016.173 STARTnetwork.ProtectioninPractice[cited201714April2017].Availablefrom:https://startnetwork.

org/start-engage/protection-in-practice.174 Ibid.175 RogueAmoeba.Piezo.2016.


andcommunity-basedorganisations(cbOs);2)tohelpfivenationalnGOsorcbOsbecomeprotectionspecialistsbyassistingthemtoimplementprotectionactivitiesandcoordinatewithprotectionactors;and3)tocreateaneffectiveevidencebaseofwhatworksinpracticeandsharethiswithotherprojectsandinitiatives.AsofJuly2016,theProtectioninPracticeprojecthad implementedthefirstoutputonprotectionmainstreaming.Workshopsandtrainingsonmainstreamprotectionhadbeenconductedand localorganisationshadcreatedtheiractionplanstoidentifyandaddressprotectiongaps.Inaddition,localorganisationshadbeenidentifiedandselectedtoreceivefunding.

Theviolence inJubadirectly impacted thestaffandpartnersof thecWcproject176,177.Theviolence also impacted project implementation and activities for both the cWc projectandProtection inPracticeproject.Thiscasestudy,conductedaspartof theDEPPexternalevaluation,aimstoexplorehowviolenceimpactedprojectimplementationandlessonslearnedon implementingemergencypreparedness and responseprogramming in a conflict-affectedsetting.

FINDINGS FROM THE CASE STUDY

Intotal,fourindividualswereinvitedtoparticipateinaninterviewandthreeindividualsagreedtoparticipate.Demographicdatawerenotcollectedduringthecasestudyinterviewstorespecttheprivacyofindividualsinterviewed.

Keythemesemergedfromtheinterviewsrelatedtothefollowingbroadcategories:1)impactsofviolenceandinsecurityonprojectimplementation;2)impactsonprojectmanagementanddecision-makingprocesses;3)impactsonprojectmulti-stakeholderplatformsandcollaborativepartnershipswithnationalandlocalorganisations;4)emergencypreparednessandresponseinaconflict-affectedsetting;and5)lessonslearned.Withinthosecategories,multiplesub-themesemergedasdescribedbelow.Directquotesfromtheinterviewsarenot includedinordertoprotecttheprivacyofrespondents.

1.1. Impact of Violence and Insecurity on Project Implementation: Project Activities, Management, and Staffing

beyondthedirectimpactonprojectstaffbasedinJuba,theviolenceimpactedprojectactivities,projectmanagement, and the budget in differentways between the two projects in SouthSudan.

AsaresultoftheviolenceinJuba,thedecisionwasmadetotemporarilysuspendthecWcproject. In the immediate period following the violence, international staff members wereevacuated fromSouthSudan.Decisions related to staffprotectionweredeterminedby thein-countryhostorganisationasperdutyofcareprotocols.WhilethehostorganisationofthecWcprojectalsorelocateditsinternationalstaff,itwasabletocontinueitsdailyoperationsthroughouttheperiodofviolencethroughitsnetworkoflocalpartnersandstaff.Theprojectmanagerposition,previouslyaninternationalstaffposition,shiftedtoanationalstaffposition.Projectimplementationwasdelayedbyseveralmonthsbetweentheperiodofevacuationofinternationalstaffandthere-entryofstaffwithrespondentsnotinglengthynegotiationsamongconsortium partners about re-entry terms and conditions.With that said, respondents alsonotedthatimplementationdelayscouldariseinanyprojectoperatinginachallengingcontext.

SeveralmonthslaterdiscussionsresumedamongconsortiumpartnerstoconsiderthefutureofthecWcproject,re-entry,andcontractingoftheflexiblefundingmechanismprojects.Afterconductingariskassessment, the in-countryhostorganisationdeterminedthatthreeofthefivepreviouslyidentifiedsub-projectscouldproceedasplanned.Itwasultimatelydecidedtoimplementthethreeprojectsbutatareducedscaleandscope.Thecontractingprocessfor

176 centerforciviliansinconflict.Underfire:TheJuly2016ViolenceinJubaandUnResponse.UnitedStates:centerforciviliansinconflict,2016.

177 GuardianT.‘Untolddevastation’inSouthSudantriggersgravehumanitariancrisis:TheGuardian;2016[cited201714April].Availablefrom:https://www.theguardian.com/global-development/2016/aug/09/untold-devastation-in-south-sudan-triggers-grave-humanitarian-crisis-un.


thethreeprojectswasstartedinnovember2016,wascompletedinDecember2016,andtheprojectsbeganimplementationinJanuary2017.Anationalstaffprojectmanagerwashired.Infebruary2017,budgetrevisionsandreductionswerefinalised.

followingtheescalationofviolenceinJuba,theProtectioninPracticeprojectalsotemporarilyevacuateditsinternationalstaff,whileitslocalpartnerscontinuedwithprojectimplementation.OneofthefacilitatingfactorswasthatthelocalandnationalorganisationssupportedbytheprojectwerelocatedinareasofSouthSudanthatwererelativelyunaffectedbythatparticularviolenceinJuba;hence,theprojectexperiencedfewerdelaysinimplementation.Theprojecthadalsocompleteditsfirstoutputonprotectionmainstreamingbythetimetheviolenceoccurred,whichmeantfewerdelays inoverallproject implementationandprojectactivities.However,thesecondoutputoftheproject(focusedonsupportinglocalandnationalorganisations)wasaffectedbytheviolencebecausetheinternationalstaffmembersresponsibleforadministrationofthisoutputwereevacuated.

ArespondentfromtheProtectioninPracticeprojectalsocitedthattheadaptiveprojectdesigninherentlyallowedformoreflexibilityinimplementationduringtheescalationofviolence.Theprojectwas designed from the outset around the national conflict, such that themore theviolence increases, themoreflexibilityprojectmanagementhad in termsof implementation.Asnotedbyprojectmanagementmostotherprojectsinthiscontextwouldhavebeenhaltedcompletely;however,therewasflexibilitybuiltintotheprojectdesigntorespondtochangesinviolence,fromthedonortotheprojectmanagertotheincountrystaff.Theprojectwasdesignedsothatintheeventofanescalationofviolencetherewouldbelessrelianceoninternationalstaff,inthatanythingthatwouldnormallybedelegatedtoaninternationalorganisationwouldbehandledbythelocaland/ornationalorganisation.

1.2. Impact of Violence on Project Implementation: Flexible and Emergency Funding Mechanisms

bothprojectsweredesignedwith aflexible fundingmechanism;however, theviolence andsubsequentdecisionsonprojectimplementationhaddifferentimpactsontheflexiblefundingavailabletolocalandnationalorganisations.

followingtheviolenceinJuba,thecontractingprocessfortheflexiblefundingmechanismwithinthecWcprojectwasputonhold.Aspartofthere-entrynegotiations,itwasdeterminedtoproceedwiththreesub-projectsandarevisedprojectactivitymixasdescribedabove.However,giventhecWcprojectenddateofSeptember2017,theprojectactivitiesandtheoverallbudgetneededtoberealignedtoaccountforthedelays.Whiletheconsortiumpartnersconsideredapplyingforanocostextension,itwasdeterminedthatthiswouldnotbesufficienttocoverthecostsofsalarysupportforprojectmanagementthatwouldbeneedediftheextensionwasgranted.Therefore,theoverallbudgetwasreducedandfunds(approximatelyone-thirdofthebudget)werereallocatedtoacWcworkinggroupinthePhilippines.Additionally,therewassupposedtobeasecondroundofflexiblefundingtosupportthegrowthofthosepilotprojectsthatwere successful.Due to the delays and thewindowwithinwhich to apply for projectfunding,asecondroundofflexiblefundingwasnotpossible.

for the Protection in Practice project, within the budget available to local and nationalorganisations,therewasasumofmoney(approximately£10,000)setasideasemergencyfunds.Theseemergencyfundsdidnothavetobeallocatedatthebeginningoftheprojectandcouldbedrawnuponiftherewasanemergencyorescalationinviolence.Theprojectwasdesignedsuch that this emergency funding could be accessed and utilisedwith the least amount ofadministrationaspossible,suchasbysendingamessagetotheprojectmanagerthatthemoneywasneeded.ThefundingwasdisburseddirectlyfromtheInGOtothelocalorganisationsasacashaward, thus, increasing theflexibilityof the funding rather thanadministering it asagrantwith reporting.beyond the fundingprovided to the local andnationalpartners, therewas a contingency plan built into the funding so that in the event internationalnGO staffwereevacuated,the internationalnGOcouldsupportthe localpartnersremotely,providingmentoringonproject implementationandsupportwith logisticsandadvocacy, soasnot todelaythelocalimplementationoftheproject.


Asdescribedbyonerespondent,fundingtranslatesintodifferentvalueindifferentcontexts—suchthattheimpactoffundinginSouthSudan,giventhehighcostsofoperatinginthecountry,ispotentiallyreducedcomparedtootherhumanitariansettings.ThisrespondentdescribedtheexpensivecontextofoperatinginSouthSudanandthatthisshouldbeaccountedforinfutureprojects.

2. Decision-making Processes and Project Management

Respondentsfrombothprojectsdescribedthecomplexitiesofthedecision-makingprocessesinthemonthsfollowingtheviolence.Whiletheimmediateconsiderationwastowardstaffsafetyandsecurity,severalotherkeyfactorsemergedduringtheinterviews.RespondentsfrombothprojectsreportedthenecessitytoadapttochangingcircumstancesinSouthSudanandofferedreflectionsonprojectmanagementinaconflict-affectedsetting.

forthecWcproject,thedecisionoptionsweretoresumeprojectactivitiesasplanned,resumeactivitiesbutwithreducedscaleandscope,ormovethefundingfromSouthSudantoanothercountry.Itwasdecided,asdescribedabove,toimplementthreeoftheplannedfivesub-projects.Asplanswereoutlined to implement thoseprojects, differingopinionsemergedamong theconsortiumpartnersabout theplannedapproach.At thattime,given travel restrictions, theinternationalstaffmembersforthein-countryorganisationwerestillbasedoutsideofSouthSudan.Theprimerecipientorganisationofthegrantadvocatedforanofficewithpermanentofficestafftoallowfordirectsupervision.Thein-countryhostorganisationnotedthatithadstronglocalteamsontheground.Itwashereinthedecision-makingprocessthat importantquestionsemerged,exposingalackofclarityanduncertaintyabouttherolesofeachconsortiumpartner.Who makes the decisions? Is it the in-country host organisation or the organisation managing the grant, or is it the donor? Where does duty of care fall?Itwasnotedthatthedialoguearoundthedecisionwasparticularlychallengingduetothemultiplelevelsofpeopleinvolved,thatevenminordecisionsbecameoverlycomplicated.

Respondents from the cWc project described the complexities of bringing numerousorganisations,eachwiththeirownindividualmandatesandorganisationalprocedures,toreachadecision.Despiteattempts tooutlineorganisationaldecision-makingmatricesevenbeforetheviolence,therewasstillalackofclarityabouttherolesofthedonor,projectmanagerorglobalgrantholder,and thehostagency,whichalsoactedasadonor for its localpartners.ThiswasfurthercomplicatedbythefactthattheonebranchofthehostagencyalsohostedthecDAcnetwork.Eventually,adecisionwasmadethroughconsensusbetweentheglobalgrant holder and the in-country host organisation.Amanagement advisory group providedfurthersupportduringthenegotiations.Thein-countryhostorganisationnotedstrongworkingrelationshipswith its localpartnersandstaff,but itwasnotclearwhat levelofengagementnationalorganisationshadinthenegotiationsaroundre-entry.AnotherchallengenotedbyarespondentfromthecWcprojectwashowthetraumaexperiencedbyorganisationsandtheirstaffasaresultoftheviolenceaffecteddecision-making.Aspeoplewerestilladaptingtothetraumaticeventsthathadtakenplace,therewastheneedtocontinuetomakedecisionsfortheprojectandorganisation,whichwasnotedtobechallenging.

TheProtectioninPracticeproject,ontheotherhand,haddesignedtheprojectinSouthSudantobeadministrativelyflexibleandwithfewbureaucraticprocedures.Asarespondentfromtheprojectnoted,withoutthatflexibility,theprojectwouldhavefailedinSouthSudan,astheywouldnothavebeenabletoimplementoutput2.Thedecision-makingprocessesfortheprojectweredescribedasadaptive,direct,andstreamlined. Itwas thedirect lineofcommunication fromthefieldwithdecisionmakersthatwasdescribedasthemostimportanttosupportongoingproject implementation in the context of insecurity. facilitating factors cited included opencommunication,trust,anddelegatedresponsibilitytotheprojectmanagerswhoarethepeopleinclosestcommunicationtothefieldteams.Projectmanagersreportedanelementofcontrolandthattheyfeltedtrustedtomakedecisions.Asanexample,theprojectwasdesignedsuchthatthepersonmanagingthefundingisalsoabletomakeprojectdecisions.

fortheProtectioninPracticeproject,oneofthekeyfactorsnotedthatallowedforadaptivemanagement was trust. This was described on one level as trust between DEPP and theconsortium organisations and then on the next level as trust between the consortium


organisationsandlocalandnationalpartnerorganisations.Thetrustbetweenprojectmanagersandthetechnicalteaminthecountrywasattributedtothebeliefthatthelocalandnationalorganisationsunderstandtheprotectionneedsandhowtobestservethoseneeds.

3. Impact on the Multi-stakeholder Platforms and Collaborative Partnerships

When asked about the impact of the violence on the multi-stakeholder platforms and oncollaborativepartnershipswith local andnational stakeholders, bothprojects acknowledgedtheimpactoftheprojectsuspensiononthelocalandnationalorganisationalpartners.

InregardtothecWcproject,asnotedabove,the in-countryhostorganisationwasabletocontinue project implementation through its network of local partners. Respondents alsopositivelynotedthatthecWcworkinggroupdidmeetduringtheperiodofprojectsuspension.However, there was concern that the group lost momentum during the period of projectsuspensionandthatthemeetingsmaynothavebeenaseffective.Asonerespondentnoted,withinthecontextofviolenceandwiththeresultingfamineandhumanitariancrisis,thecWcworkinggroupmaynothavebeenprioritisedgiven thecompetingpriorities forOcHA, theworkinggroupco-chair.Anotherrespondentacknowledgedthatthegroupwasatanascentstagewhentheviolenceoccurred.

Oneof theProtection inPractice respondentsdescribed the impact that theevacuationofinternationalorganisationshadonthelocalandnationalorganisationsthatremainedinSouthSudanwithactiveprogramming.RecognitionoftheworkbeingdoneonthegroundbylocalandnationalorganisationschangedthewaytheInGOsviewedtheirpartnershipswithintheproject.Thesepartnershipsshiftedtobeviewedascomplementarypartnerships,ratherthana partnership built on unidirectional capacity building. In doing so, power dynamics shiftedbetween the InGO and the local and national partner organisations and the relationshipsevolvedtobemorereciprocal innature.Whiletheviolencebroughtto lightthe importanceof reciprocal relationships, this approach to building partnerships existedwithin the projectevenpriortotheviolence.Asanexample,fromthebeginningoftheprojectdesign,thelocalandnationalorganisationsweregivenanopportunitytodecidewhichinternationalprotectionpartnerorganisationcouldprovidethetrainingandcapacitybuildingtheywereseeking,ratherthantheInGOdecidingwhatandwhomtoteach.

When considering these data together with the data from the evaluation’s minimum setinterviews,itwasnotedthatthedepartureofInGOsfollowingtheviolenceleftavoidinthehumanitarian community and led to a complete lack of responsibility. The respondent feltthat this void encouraged further violence by armed groups and encouraged unnecessarycompetitionbetweennGOsremaininginSouthSudan.

4. Emergency Preparedness and Response in a Conflict-affected Setting

Respondents from both projects had opinions and broader reflections about focusing onemergencypreparednessinthemidstofaconflict-affectedsetting.

ThecWcprojectwaspositionedinSouthSudanasanemergencypreparednessprojectandnot an emergency response project.Meanwhile, the in-country host organisation’s broaderportfolio,beyond thecWcproject,waspartially focusedonemergency responseactivities.Onerespondentreflectedonwhetheritisappropriatetotrytofocusonpreparednessinsuchacontextwherethereisongoinginsecurityandhowtomosteffectivelyconsideremergencypreparednessinsuchsettings.Thisrespondentnotedthatitisdifficulttogetpeopletothinkaboutpreparingwhentheyneedtobethinkingaboutfunctioningandhowtorespond.

WithinthecWcproject,thehostorganisationoperatedasbothadevelopmentprojectandahumanitarianproject.becauseofitsinternalflexibility,theorganisationisabletopivotactivitiesandfundinginresponsetoviolence.butbecauseofthemandateoftheDEPPonpreparedness,thiskindofpivotingwasnotpossible.Themandateofemergencypreparednessappliestotheentireconsortiumandmaintainingtheattentiononpreparednessamongconsortiumpartners,someofwhomalsohaveresponseactivities,wasnotedtobeparticularlychallenging inthemidstofanactiveconflictandfamine.

A respondent from the Protection in Practice project described the importance of linking


preparedness and response—that these are not necessarily two separate concepts. Thisrespondentnotedthatoneofthebestwaystostrengthenpreparednesswastoalsorespond,sothatthelessonslearnedwhilerespondinginturnstrengthenpreparednessactivities.Theprocessofrespondinginandofitselfallowsforpreparedness,andallowsforlearningtobeappliedifthesamesituationhappensinadifferentlocationatadifferenttime.combiningpreparednessandresponseactivitiesalsoallowsforopportunitiesforcombinedtraining,mentoring,andlearning,inadditiontoaperceivedbettervaluefortheinvestmentandultimatelyabetterresponse.

Itwasnoted in interviews thatmuchof theevidenceonpreparedness todatecomes fromnaturaldisastersettings,inwhichthereisacertainpredictabilityofeventsthatcanbepreparedfor inanorganisedmanner.However,preparednessforconflictwasdescribedasadifferentprocessthathasnotbeenexploredinasmuchdepthasemergencypreparednessandresponsetonaturaldisasters.Mostoftheevidenceonpreparednessfocusesondisaster-affectedareasanddevelopmentsettingsthathaveatendencytobecomehumanitariancontextsatcertaintimesduringtheyear(duetoadroughtortyphoon,asexamples).Respondentsidentifiedagapinthelearningandevidenceonpreparednessinconflict-affectedsettings.

5. Lessons Learned about Implementing a DEPP Project in a Conflict-affected Setting

Respondentswereaskedtoreflectonlessons learnedaspartofprojectmanagement inthecontextofviolence.Prominentpointsraisedrelatedtodutyofcareandtothemanagementarrangementsthatwereinplace.

5.1 Protocols, Duty of Care and Operating in a Conflict-affected Setting

RespondentswereaskedaboutprojectplanningandimplementationinSouthSudanandwhatlearningcouldbeappliedtootherDEPPprojectsoperatinginsettingsatriskofviolenceandinsecurity.Inregardtosecurityprotocols,itwasnotedthattheviolenceinJubawasunexpectedandthatthesecurityplanninghadprioritisedotherareas inSouthSudanthatwerethoughttobehigher risk. Inaddition to securityprotocols, respondentsemphasised the importanceofhavingclearagreementsaboutdutyofcarewhencreatingmulti-lateralagreementsamongmultiple institutions in a consortium and the responsibilities of care for the host agency inrelationtoconsortiumpartners.

5.2 Clarifying Consortium Agreements and Streamlining Contractual Arrangements

Inadditiontohavingclearagreementsofdutyofcare,respondentshighlightedtheimportanceofstreamliningthecontractualarrangementsamongorganisations.Keytooperatinginaconflict-affectedcontextwastohaveproceduresthatallowedforflexibleandadaptivemanagement,rather than traditionalandrigidcontractual relationships.Asanexample,cWcrespondentssuggested that futureprojectsensure that theglobalgrantholdingorganisationand the in-countryorganisationarethesamewhenpossible,suchthat ifWorldVisionUKistheglobalgrant holder that the in-countryorganisationwouldbeWorldVision in SouthSudan, as anexample.Thiswassuggestedtostreamlineadministrativeprocedures,flowoffunds,anddutyofcare.

5.3. Lessons Emerging from Minimum Set Interviews on the DEPP Approach in South Sudan

Theinterviewsconductedforthecasestudywereanalysedinconjunctionwithdatapreviouslycollected during theMinimum Set evaluation activities.These interviews offered additionalperspectives on project implementation in a conflict-affected setting and reinforced theimportanceofconsideringcontext inprojectdesignandimplementation.Thedatawerealsoanalysedwiththeexternalevaluationtheoryofchangeinmind,consideringthecorecomponentsofDEPP(capacitybuilding,collaboration,earlywarningsystems,andlearning).

OneoftheobservationsonDEPP’sconsortiumapproachandemphasisoncollaborationwithstakeholdersisthatengagementofgovernmentstakeholdersinprojectsisnotalwayspossibleinconflict-affectedsettings,suchas inSouthSudan.Giventhegovernment’s involvement intheconflict,theDEPPprojectsinSouthSudanwereunabletocollaboratewiththegovernmentasapartner.

Inregardtocapacitybuildingonemergencypreparednessandresponse,mostoftheexisting


trainingtoolsandapproachesaredesignedforotherdisasterandemergencycontexts,andnotforconflict-affectedcontexts.ArespondentfromthecWcprojectutilisedcasestudiesduringtrainingstoincreasecapacity;however,thosecasestudiesweretoodifferentfromtheSouthSudancontextandweren’taseffectiveasintended.

Respondentsreflectedonchallengespredictingviolenceinconflict-affectedsettings,citingalackofstandardisedriskassessmentsor indicators,suchasearlywarningsystems,to informprogramming.Asanexample,arespondentreflectedhowothersitesoutsideofJubahadbeenidentifiedasathigherriskofviolenceandthusprogrammingwascentredonthisassumption.

Inregardtolearning,onerespondentnotedthatexpectationsneedtoberealistictothecontext.ThisrespondentobservedinSouthSudanthatorganisationalcultureandpoliciesandownershipofcommunicationmessagesdifferedamongorganisations,suchthatorganisations,includingInGOs,nationalandlocalnGOsinSouthSudan,werenotedtobelesswillingtoshareinformationcomparedtootherDEPPprojectcountries.

KEY FINDINGS

• Thekeytooperatinginaconflict-affectedcontextwastohaveproceduresthatallowedfor flexible and adaptive management, rather than traditional and rigid contractualrelationships.Aspects of one of the projectswere able to continue because of thisflexiblemechanism.

• In addition to security protocols, respondents emphasised the importanceof havingclear agreements about duty of carewhen creatingmulti-lateral agreements amongmultipleinstitutionsinaconsortiumandtheresponsibilitiesofcareforthehostagencyinrelationtoconsortiumpartners.

• Engagementofgovernmentstakeholdersinprojectsisnotalwayspossibleinconflict-affected settings, such as in South Sudan. Given the government’s involvement inthe conflict, theDEPPprojects in SouthSudanwereunable to collaboratewith thegovernmentasapartner.


CASE STUDY #2: STRENGTHENING PREPAREDNESS AND RESPONSE IN KENYA AND ETHIOPIA Thesecondcasestudyfocusedonpreparednessandresponseeffortsintwoselectedcountries:Kenya(UrbanEarlyWarningEarlyActionproject)andEthiopia(ShiftingthePowerproject).

URBAN EARLY WARNING EARLY ACTION PROJECT

TheUrbanEarlyWarningEarlyAction(UEWEA)project178innairobi,Kenya,wasimplementedbyaconsortiumcomprisedoffournGOs(concernWorldwide,KenyaRedcross,Oxfam,andWorldVision)andnairobicountyGovernment.Theprojectaimedtodevelopearlywarningsystems and early action responses to food insecurity and health-related crises in selectedurbanslumsofnairobi(Kibera,Mukuru,Korogocho).TheprojectbuiltonconcernWorldwide’s5-year IndicatorDevelopment for theSurveillanceofUrbanEmergencies (IDSUE)project179 toestablishthresholdsforhumanitarianresponserelatedtofoodinsecurityandrelatedhealthcrises.Theprojectaimedtoachievefourkeyoutcomes:180:

• facilitatetheset-upofandcapacity-buildacoordinatedurbanearlyactionmechanismwithinthenairobicountyGovernmentwithagreedactions

• Strengthen the capacity of six nairobi sub-counties and one informal settlementcommunitytomitigateandrespondquicklytotheimpactsofslow-onsetemergencies

• Ensure routine surveillance in urban informal settlements in Kenya by the countyDisasterManagementcommittee(cDMc)bytheendof2017

• AdvocateforUEWEAasabestpracticeandmostcost-effectivemodelforrespondingtourbanemergenciesintheworld

SHIFTING THE POWER PROJECT

The Shifting the Power project was implemented in five countries (bangladesh, Pakistan,DemocraticRepublicofthecongo(DRc),EthiopiaandKenya)byaconsortiumcomprisedoffivenGOs:ActionAidandcAfOD(co-leads),Oxfam,concern,Tearfund,andchristianAid.

Theseorganisationsworkedtogethersupporting55oftheirlocalandnationalnGOpartners(10inEthiopia)whosharethevisionandambitionofplayingaleadingroleindecision-makingandrespondingtocrisesintheircountriesandregions.Theaimoftheprojectwastosupportlocal actors to take their place alongside international ones in order to create a balancedhumanitariansystem.Theprojectwastoproducefiveoutputstoachieveoneoveralloutcome:

Outcome1:nationalorganisationsinfivecountrieshaveincreasedtheircapacitytodetermineanddeliveremergencypreparednessandresponse

• Output1:nationalpartnerorganisations infivecountrieshavetheknowledge,skills,processes,andpoliciestoprepareforandrespondeffectivelytoemergencies

• Output2:nationalorganisationsarebetterrepresentedandhaveastrongervoicein

178 STARTnetwork.TheUrbanEarlyWarningEarlyActionProject:foodSecurityandnutrition,2017.179 IndicatorDevelopmentforSurveillanceofUrbanEmergencies(IDSUE).UrbanEarlyWarningEarlyAction

finalEvaluationReport,p1.180 STARTnetwork.TheUrbanEarlyWarningEarlyActionProject:foodSecurityandnutrition,2017.


relevanthumanitarianplatformsandnetworksintheircountries• Output 3: Shifting thePower consortiummember InGOs recognise and respond to

local/nationalorganisations’capacity,leadershipandvoice• Output4:ShiftingthePowercollaborateswiththeotherDEPPprojectstomaximise

collaborativeadvantage• Output 5:The project provides evidence of good practice in strengthening national

organisations’ humanitarian preparedness and response work and their role andinfluenceinhumanitarianaction

FINDINGS

Atotalof22in-depthinterviews(n=16inKenyaandn=6inEthiopia)wereconductedforthecasestudy(Table8.1)witharesponserateof100%(16/16inKenyaand6/6inEthiopia).Duetosecurityreasons,theteamcouldnotreachallofpotentialrespondentsinEthiopiaandhenceasmallernumberofinterviewswerepursuedandconductedinthatsetting.

Kenya

InKenya,ofthe16respondents,11(68.8%)weremaleand5(31.2%)werefemalewithameanageof36.7years[25-56years](seeTable8.1).Eachoftheorganisationsandkeygovernmentpartners(ministriesandcountygovernment)oftheprojectwasrepresentedintheinterviews.Respondentsreportedworkingonaverage7.6yearsinthehumanitariansectorand6.2yearsfortherespectiveorganisation.

Ethiopia

InEthiopia,ofthe6respondents,5(83.3%)weremaleand1(16.7%)wasfemalewithameanageof42.8years [36-62years].Thesixorganisations representedbeneficiaryorganisationsinEthiopia affiliatedwith the Shifting thePowerproject. Respondents reportedworkingonaverage7.8yearsinthehumanitariansectorand9.2yearsfortherespectiveorganisation.

KENYA ETHIOPIAVariable N (%) N (%)Sex of RespondentMale 11 (68.8%) 5 (83.3%)Female 5 (31.2%) 1 (16.7%)Total 16 6

Mean Age of Respondent (Years) 36.7 years [25 – 56 years] 42.8 years [36-62 years]

Highest Educational Degree ObtainedBachelor Degree 7 (43.8%) 2 (33.3%)Master’s Degree 6 (37.5%) 4 (66.7%)High School Diploma 3 (18.7%)

OrganisationsInternational NGOs 7 0National and/or Local NGOs 6 6Government (i.e., Ministry, County Offices)

3 0

Table8.1:DemographiccharacteristicsofrespondentsinKenyaandEthiopia


Respondentswere asked to summarise the types of emergencies that occurred during theprojectperiodandcorrespondingpreparednessandresponseactivities.Datafromsitevisitsandprojectreportswerecompiledtodescribetheresponseactivitiesandarepresentedbelow(seefigures8.1and8.2).

Humanitarian Response: Focus on urban emergencies in Kenya

What:

Anurbanearlywarningsurveillancesystemsetupininformalsettlementsinnairobicountytomonitorfoodsecurity,householdshocksandpublichealthindicatorsonaroutinebasis(everytwomonths)detectedthreeemergenciesandtriggeredresponseactivities.

1. Diarrhoealoutbreak:february2017toJune2017

2. choleraoutbreak:April2017tofebruary2018

3. foodinsecurity:OctobertoMarch2018

Who:

TheDEPP’sUrbanEarlyWarningEarlyAction(UEWEA)projectdevelopedandimplementedthe urban earlywarning system through its consortiummembers concernWorldwide,Oxfam,WorldVisionandKenyaRedcrossSociety,insevensub-counties(Kibera,Kasarani,Ruaraka,Makadara,Starehe,KamukunjiandEmbakasiEast).Thesystemaimedtoidentifyslowonsetorchronicemergenciesinurbancontextsthroughroutinemonitoringofasetofsensitiveindicatorsthatweredevelopedalongwithspecificthresholdspriortothestartof theprojectand tosetupacoordinatedresponsemechanismwithin thenairobicitycountryGovernment.

Response Activities:

TheDEPP’sUrbanEarlyWarningEarlyAction(UEWEA)projectdevelopedandimplementedthe urban earlywarning system through its consortiummembers concernWorldwide,Oxfam,WorldVisionandKenyaRedcrossSociety,insevensub-counties(Kibera,Kasarani,Ruaraka,Makadara,Starehe,KamukunjiandEmbakasiEast).Thesystemaimedtoidentifyslowonsetorchronicemergenciesinurbancontextsthroughroutinemonitoringofasetofsensitiveindicatorsthatweredevelopedalongwithspecificthresholdspriortothestartof theprojectand tosetupacoordinatedresponsemechanismwithin thenairobicitycountryGovernment.

• Diarrhealoutbreak:healtheducation,massscreeningofchildrenunder5years (8,338screened in Korogocho, 13,284 in Embakasi), and distribution of water purificationproducts

• choleraoutbreak:healthpromotion(171,139households,2630studentssensitised),solidandliquidwastemanagement,andwaterqualityandfoodsafetymonitoring

• foodsecurity:Massscreeningofchildrenunder5years(44,825)andcashtransfersof2000KSH(roughly20USD)providedto3,034households(2,085women,949men)whowereseverelyfoodinsecureweremadeviamobilemoneyinKorogochoandKibera

figure8.1:HumanitarianResponse:focusonUrbanEmergenciesinKenya


Humanitarian Response: Focus on drought in Ethiopia

What:

Ethiopiahasbeenexperiencing largescaledroughtssince2015. Initially inducedbythe2015/2016Elnino,andfurtherexacerbatedbylow2016autumnrains,andbelowaverage2017Spring rains, at least8.5millionpeoplewere in needof relief food assistancebymid-2017inSouthernandSouth-easternEthiopia.Theeffectsofthedroughtwerefurtherexacerbatedbydiseaseoutbreaks,largescalelossoflivelihoodassets,anddisplacement.conflict intheOromiaandSomali regionsbeginning inSeptember2017resulted inthedisplacementofhundredsofthousandsofpeopleoftenindroughtaffectedareas.

Who:

• fourDEPPprojectswereimplementedinEthiopia:ShiftingthePower(STP),PublicHealthPreparednessinGambella,TalentDevelopmentandImprovedEarlyWarningEarlyAction–Ethiopia.

• Self-reporteddatasuggest7outof10localpartnersoftheShiftingthePowerproject,whichhadaimedtobuildlocalandnationalnGOcapacityandpreparedness,respondedtothedroughtintheareaswheretheymanagedevelopmentprogrammes.Theseincluded:community InitiativefacilitationAssistance (cIfA),communityDevelopment Initiative(cDI),RiftValleychildrenandWomenDevelopmentOrganisation(RVcWO),ActionforDevelopment,SOSSahel,HUnDEEandAVHO.

Response Activities:

Specificresponseactivitiesledbythesel/nnGOsin2017-2018included:

• cIfA vaccinated 263,187 cattle from 35,060 target households inMoyale,Miyo andGuchiworedas.Improvedresponsetimeduetostrengthenedorganisationalprocurementpolicieswasreported

• cDIsecureda£10,000responsegrantfromOxfamandSTPwhichenableddistributionofseedsto200householdsincommunitiesaffectedbythedroughtinWestArsiZone,Oromia

• RcVWOobtaineda£60,000grantfromActionAidtoprovidelivestockfeedsupportto1,030households

• Action for Development provided £60,000 in cash transfers to drought-affectedcommunitiesinborena

• SOSSahelsecured$530,000directlyfromUnOcHA’sEmergencyResponsefundtoprovideanimalfeedinthreeworedas(Wachille,DubuluqandDire),indirectlybenefitting28,256individuals.ThisisoneofthefewinstancesalocalEthiopiannGOhasbeenabletosecurefundsthroughthismechanismwithoutanintermediarypartner

• HUnDEE obtained $410,000 from Kindernothilfe (KnH) to provide emergency foodassistanceto10,000peoplein4KebelesoftwoworedasintheborenaZone

figure8.2:HumanitarianResponse:focusonDroughtinEthiopia

belowarekeythemesidentifiedduringthedataanalysis,withexcerptsfromtheinterviewsorsummariesfrominterviewnotes.

I. Key project design and implementation factors that strengthened emergency preparedness and response included community and local organisational engagement, government engagement, and flexible funding mechanisms.

Ia. Community Engagement and Emergency Preparedness and Response


Among key project design and implementation factors, respondents highlighted the importance of community engagement, especially in the design of the surveillance system, the selection of beneficiaries, and the response activities.

Inbothcountries, keycommunitymembersand stakeholderswere identifiedaspartof theprojectdesignandengagedinvariousrolesthroughouttheproject.communityengagementnotonlymeantprovidinginput,butalsobeingengagedinselectingbeneficiariesanddesigningtheresponsepackageofactivities.AsdescribedbyarespondentinEthiopia:“youhavetobetransparentandinvolveallthepeople,notonlymajorityordominantpeopleinthecommunity,aswellaswomen.Duringbeneficiaryselection,everyoneshouldbe involvedandhelpwiththeverification.”Asaresultofengagingthecommunity intheselectionofbeneficiariesandthe response activities: “…therewere no complaints frombeneficiaries about the selection.Thereusually[are]complaintsaboutselectionerrorbutthistimetherewasnot…communitymemberswereabletogiveopinionswhichmadeitsuccessful.”

Respondents inboth countries commented that communityengagementhelped tomanagecommunityexpectations,especiallyinregardtosurveillanceorneedsassessments.Oneofthechallengingaspects,asdescribedbytherespondent,was“pushback”fromthecommunityduringdatacollectionbecausecommunitymembersareaskedtoprovidedata,butmaynotreceiveanydirectaid.Toaddressthisproblem,theprojectinEthiopiacreatedseveralcommunity-basedcommittees,includingabeneficiaryselectioncommitteeandanemergencyitemsmanagementcommittee.Thisallowedtheprocessestobemoretransparenttothecommunityandprovidedamechanismbywhichcommunitymemberscouldbringforthconcernsorcomplaints.Similarly,theprojectinKenyaidentifiedcommunityfocalpeopleandcreatedopportunitiesfordialogueamongstakeholdersfromcommunityrepresentativestofoodandwatervendorstogarbagecollectors.

Ib. Government Engagement and Emergency Preparedness and Response

Respondents in both countries noted that partnership with government was crucial to project design and implementation from facilitating administrative procedures to fostering trust.

Governmentengagementwasnotedoveralltofacilitateprojectimplementation.Ononelevel,inclusionof government in theproject design and implementation facilitated administrativeprocedures.As noted by a respondent in Ethiopia: “This time itwas different because thegovernmenthashelpeduswithfacilitatingthesigningoftheagreement.Eventhoughalreadytheeffectofthedroughthasstartedtobeobserved,afterthefund[was]released…wehavebeenabletointerveneafteramonthortwo.Thisisdifferentfromotherinterventionsbecausethepreviousbottlenecksandbureaucraticstepshavebeenalleviatedandwewereallowedtoproceedwithsomeoftheimportantproceduresafterwehavestartedtheproject.”

Onanotherlevel,engagementwithgovernmentfosteredtrustbetweenprojectorganisationsandthegovernment.forexample, inKenya,theengagementwiththegovernment,earlyoninprojectdesignandimplementation,allowedfortrustinthesurveillancesystemandinthesurveillancedatacollectedbyprojectorganisations.Thistrustwasespeciallyimportantwhensurveillancedataindicatedanoutbreakofcholeraintheurbanslums,andhelpedtofacilitateamoretimelyresponse.

Ic. Funding Mechanisms and Emergency Response

Respondents from both projects reported that timely access to funding and flexible funding are essential to the speed and timeliness of humanitarian response.

InKenya,theUEWEAprojecthadaflexiblefundingmechanism,knownasa“crisismodifier”,which is funding set asidewithin the project to allow organisations to act and respond tosurveillanceindicatorsthatdetectanemergency.Aspartoftheproject,asystemwasdesignedsuchthataresponsewouldberequiredinrelationtofivesurveillanceindicatorsandthatthiswouldinturntriggertheflexiblefundingmechanism.

A respondent inKenyaexplained that thecrisismodifierwasessential to their response tothediarrhoeaoutbreakinfebruary2018.However,itwasnotedthateventhoughthecrisismodifierexisted,therewasn’tinitiallyasysteminplacebywhichtotriggerthecrisismodifier.Thissystemwassubsequentlyrefinedduringthecourseoftheproject.


Whilethecrisismodifierwasalignedwithsurveillanceindicators,therewasconcernexpressedthatitmayhavenotbeenlinkedtothemosturgentneed,suchascholera.“Ithinkthey’dalreadyplannedorhadintheirmindwhatthey’duseit[crisismodifier]for,theyweremonitoringfoodpricesthosehavegoneup,andthey’vebeenupinnairobiforacoupleofyearsso,butyouknowtheyhadtriggersthatwerethereandthattriggeredafoodsecuritycashtransferresponse.buttomeitwasfrustratingthatindicatorsmayhavepointedtowardsacholeraoutbreakandthat’sclearlyformeoneofthebiggest,oneofthebiggestthreats, immediatethreats isacholeraoutbreak.”ThedelaysinrecognisingandplanningfortheemergingcholeraoutbreakledtoaslightlydelayedresponsecompoundedbytheneedtoseekfundsfromoutsidetheDEPPtorespondtothecholeraoutbreak.

WhiletheprojectinKenyahadaflexiblefundingmechanismbuiltintotheprojectbudgetfromtheoutset, noneof the respondents inEthiopia reportedhaving access to aDEPPflexiblefundingmechanismtosupportemergency responseactivities.Document reviews,however,foundthatShiftingthePowerdidprovideasmallfundforresponseactivities.Organisationsthataimedtoimplementresponseactivitiesreportedsubmittingproposalstoexternaldonorstoseekrapidfundingwhenanemergencywasdetected.notingthatthiswasachallengeformanylocalorganisationsduetolimitedcapacity,theShiftingthePowerprojectshouldcontinuetostrengthenorganisationalcapacitytoprepareandsubmitthesefundingproposals.

Despite the increased capacity around fundraising for a response, interviewees in Ethiopianotedthatresponseeffortsweresignificantlydelayedduetothetimerequiredtoseekresponsefundingfromoutsidesources.forexample,oneintervieweeinEthiopiaexplainedthatasearchfor funds led to anearly six-monthdelay in implementationof response activities.AnotherintervieweeinEthiopiadescribedhowtheprocessofseekingfundingforresponseactivitiesalsoaffectedtheirabilitytorespondtoshiftingneeds:“Developaproposalandwithin72hoursthereshouldbearesponse,butit’sactuallynotlikethat.Insteadyouactuallyhavetowaitonemonthortwomonths,prepareaproposal,getcomments,thengetpaidfor1-3months.Thecallsforproposalsaren’tincontextofthecurrentposition...thus,someimmediateneedsdon’tgetanimmediateresponse.”

OneofthepositiveoutputsoftheShiftingthePowertrainingsinEthiopiawasthatorganisationshad proposals ready that can be adjusted immediately to the context. However, despitestrengthenedcapacitytoseekfundingforresponseactivities,manynationalnGOsinEthiopiadonotqualifytoreceivefundingfromcertaindonorsbecausetheydonothavetherequiredforeigncurrencybankaccounts.Anotherrespondentnotedthatlocalorganisationsoftenaren’teligibleforamultitudeofreasonstoreceivefundingdirectlyandthatthefundingstreamsarebasedoutsideof thecountry, suchas in theUnitedStates,UnitedKingdom,andEuropeancountries, leading to additional delays in response activities. for this reason, more flexiblefundingoptionsthatcouldpotentiallyovercomethesebarrierswouldbeaneffectivesolutiontoincreasingtimelinessofresponse.

II. How key components of DEPP – capacity building, collaboration, early warning systems, and learning – influenced emergency preparedness and response activities in Kenya and Ethiopia.

IIa. Capacity Building and Emergency Preparedness and Response

Capacity building through trainings at community and organisational levels strengthened emergency preparedness and response through improved coordination, surveillance, proposal development, and project implementation.

ThemajorityofrespondentsprovidedexamplesoforganisationalcapacitybuildingaspartoftheDEPP.However,therewerealsoseveralexamplesofcapacitybuildingatthecommunitylevel.ArespondentinKenyareportedthatafteracommunity-levelcapacitybuildingtraining,communitymemberswereabletorespondeffectivelywithoutfurtherassistance.TheprojectinKenyaalsotargetedcommunityhealthvolunteersandcommunityhealthworkerstosharemessagingandbuildcommunitycapacityinregardtoemergencypreparednessandresponse.

As described by another respondent in Kenya, “We have been doing capacity building of[nGOs],capacitybuildingofthecountygovernmentinthesurveillance,andcapacitybuildingofcoordination.Thebiggerpartofthisprojectistobuildthecapacity.Andalso,capacitybuildinginpreparedness,capacitybuildinginresponse,capacitybuildingintheoverallthinking.”During


thecholeraoutbreakinthenairobislums,Oxfamprovidedtechnicalcapacitybuildingoveraperiodofseveralmonthstoorganisationsinvolvedintheresponseandtocommunitynetworksaffiliated with the Kenya Red cross, including health workers and volunteers. However, arespondentdescribed the challenges inbuilding capacity andmeasuring change in capacityinsuchashortperiod:“fromacholeraresponseperspectiveIthinkitisverydifficulttobuildcapacity inthreemonthsperiodotherthanworkingwitheachother, trusting, learningfromthat.Howyouthenmeasurethatpastthreemonthsisadifferentquestion.”

The most common capacity building activities mentioned among respondents in Ethiopiaincluded training provided by the Shifting the Power project on conducting rapid needsassessments,preparingproposals,andimplementingcashtransferprogrammes.Thecapacitybuildingextendedbeyondtheorganisationitself,asonerespondentinEthiopianoted“…theShiftingthePowerprojecthashelpedusa lot.Weareworkingwith thezonal task force inrelationtocapacitybuilding.Wehavenotkeptsilentafterourcapacityisbuilt;wehavegonedowntotheworedalevelandwehavegiventrainingsinordertobuildthecapacityofthestaff.”

IIb. Collaboration and Emergency Preparedness and Response

Collaboration strengthened emergency preparedness and response in the project target areas through improved communication, coordination, and resource efficiency.

collaborationsupportedimprovementsincommunication,coordination,andsharingofideas.As an example from theUEWEAproject inKenya, the formalDEPP consortium facilitatedmeetings, bringing together the organisational representatives and government officers, todecideandagreeoninputsandtheinterventionstrategy.“basically,thereisalotofsharing[of] technical ideasonwhatwillworkbest, onwhatwill notworkbasedonbothpartners’experiencesontheground…Soyouarenotworkinginisolation.”

TherolesofcollaboratingorganisationsandinstitutionsinKenyaaspartoftheUEWEAprojectwereclearlyoutlinedanddescribedbyrespondents.Organisationalmandatesandrolesweredescribedtobewellalignedwiththetechnicalexpertiseofeachorganisation.OnethemethatemergedduringtheanalysisoftheKenyainterviewsistheimportanceofincludingadvocacyrolesandresponsibilitieswithinthemandateoftheconsortium(i.e.,anorganisationalpartnerwhoisfocusedonadvocacy—whetherthisbeaformalroleoramediapartner).

local nGOs interviewed in Ethiopia developed a similar model of collaboration betweenthe organisations, government, and other nGOs; however, thiswasmore often through ahumanitariancluster-basedsystemratherthanaformalconsortium.collaborationallowedfordelegationofresponsibilitiesanddutiesamongorganisationsandorganisationalstaff.Italsoallowedforimprovedresourceefficiencyandavoidedduplicationofpreparednessandresponseactivities.AsdescribedbyarespondentinEthiopia:“Whenwesayitiscluster-baseditmeanstheapproachisnotsingle-based.InborenawewereworkingwithUnOcHA,cARE,Acf…wehavebeenworkingtogetherinordertoidentifytheareaswecollaborateandwewereworkingtogetherinordertoavoidduplicationofeffortsbecausethereisconstraintofresources…wehavetouseourresourcesinanefficientwayandwehavebeendoingthat.So,bycluster-basedwemeaneverystakeholderwasinvolvedandcollaboratedinthatoperationalarea.”

OneofthemostnotablesuccessesdescribedbyUEWEAprojectrespondentsinKenyawasthat the consortium structure facilitated the response to the cholera outbreak.While thefundingforthecholeraresponsewasseparatefromtheDEPPfunding,thestructureoftheconsortiumthatwasalreadyinplacefacilitatedatimelyresponseastheywereableto“injectintotheconsortiumtodoacholeraresponsethroughtheKenyaRedcross—throughthesameconsortiumsystem.”beyondfinancing,theconsortiumstructure,includingitsproceduresandmanagementstructure,facilitatedaneffectiveresponsetothecholeraoutbreak.

Whenaskedaboutchallengesrelatedtotheconsortiumapproach,respondentsinKenyanotedthattherewereinevitablechallengestobringmultiplepartnerstogether,includinggovernmentandcountyoffices.TheyalsodescribedthelengthoftimethatittooktogetagreementsandMOUs signed among partner organisations and government officials, especially consideringtheDEPPtimelineexpectations.Despitethesechallenges,therewasoverallconsensusthata consortium approach strengthened preparedness and response activities. As noted by arespondentinKenya:“It’s[theconsortium]alwaysgoingtocreatechallenges,anditwouldn’tbeinterestingifitdidn’t.butit’sclearlygoingtohavemoreimpactifyoulocateitwithincountystructuresbothintermsofrollout,butalsointermsofinfluencinginthefuture.”


IIc. Early Warning Systems and Emergency Preparedness and Response

Surveillance and early warning systems were fundamental to effective preparedness and response in the project target areas by providing data, systems, and procedures.

AsarespondentinEthiopiaconveyed:“Thefundamentalpillarofanyemergency—itcouldbeanemergencyoritcouldbedevelopment—youhavetohaveanearlywarninginformation[system].you have to have adequate information.There needs to be an information sharing systematacommunitylevel.”ArespondentinKenyaechoedtheimportanceofdata:“Preparednessrequiresdata.Itrequiresdataaboutasetofindicators…noonewouldhaveknownthatfoodinsecurityisarealdisasterinourinformalsettlementsinnairobi.It[data]alsotellsyouaboutwhosuffersor…whereisthepotentialofthisdisasterhittingmost.”

Surveillance Systems and Procedures

The surveillance system of theUEWEA project in Kenya comprised data collection on keyindicators,dataentry,andusedadashboardtohighlightalertsandtrends.Respondentsreportedthattheprocessofdeterminingindicatorswascollaborativeamongconsortiumpartnersandtooktime, nearly fouryears (the surveillance systemhadbeen indevelopmentprior to theDEPP.)Therespondents inKenyaalsodescribedsomeof the intricaciesneededtoconductsurveillanceonfoodinsecurityinanurbancontext.Asnotedbyrespondents,foodinsecuritytendstohaveasloweronset,asopposedtoasudden,naturaldisaster;thus,requiringadifferentsurveillancesystemandsetofindicators.

SimilarlyinEthiopia,therewassurveillanceintheformofneedsassessments.OnerespondentreportedthataftercollaboratingwiththeShiftingthePowerproject,needsassessmentswereincorporated intotheorganisations’responseactivities.Theseassessmentsaimedto includerelevant stakeholders, community, and government offices, such as the disaster risk andpastoral office.Theneeds assessments subsequently identified food insecurity,malnutritionamongwomenandchildren,contaminationofwatersources,andcompromisedhealthstatus.needsassessmentsandsurveillancesystemswerealsodescribedasprovidingmechanismsforcollaborationwithgovernmentpartnersandstrengtheningthesepartnerships.

Early warning systems: people, coordination, community engagement, and trust

Dataarenot theonly importantcomponentsofanearlywarningsystem.Respondentsalsoemphasised that these early warning systems require people, coordination, communityengagement, and trust. In the case of the UEWEA project, community ambassadors werecriticaltothesurveillancesystem.Asanexample,arespondentinKenyareported:“basedonthosevariablestheyareabletotellandtheambassadorsraisetheredflag.So,theredflagcanberaisedtothecountyofficialsthatwethinkthereisaproblem.So,itdoesn’tneedtowaitforthenationalgovernmentministertoannouncethatwehaveaproblem.”

Early warning systems require coordination. Having an established surveillance system isimportant,butitdoesnotguaranteeatimelyresponseasnotedbythisrespondentinKenya.“Theresponseisdependentonhowstrongisthecoordination…becausethetimeshouldbeshort…Aswemoveon,weneedtoimprovebecausethesuccesswillbehowtimelyitis.Andhowtimelyisdependentonhowthepreparednessisdoneverywell.becausethemoreyouprepare the lesstimeyouneed.So, themechanism [earlywarningsystem,surveillance]andotherthingshelpedalotbutstillnowthereisaroomto[improve].”


furthermore,communityinvolvementinthedesignandimplementationoftheearlywarningsystemiscrucial.ArespondentinEthiopiaprovidedanexampleofcommunityengagementinthesurveillanceactivities:“Everymonthmembersofthecommunitysitanddiscusslivelihoodissues of the community (availability ofwater, livestockmarket info, conflicts, anything thatrelates to livelihoods.” Several respondents acknowledged that engagement of communitymembersandstakeholdershelpedtofostertrustintheearlywarningsystemandsurveillancedata.

Emergingfromtheinterviewdata,itwasevidentthatrespondentsbelievedthatearlywarningandsurveillancesystemsallownotonly for improvedpreparedness,suchas inpredictionofeventsandprevention,butthattheyalsoprovideanopportunityandmechanismforbehaviourchange.Itisthisbehaviourchangethatwasnotedtoleadtomoreeffectiveresponseefforts,suchasnotedbythisrespondentinKenya:“Havingbeen[involved]inthesystems,mytakehasbeennowthatifwemovetoriskmanagementwhichyougettheearlywarningsignsandactonthem,earlyactionyouactonthemitslesscostly,yousavealotofresourcesandavoiddeaths.”

IId. Learning and Emergency Preparedness and Response

Respondents identified various platforms, including committees within projects and learning events within the DEPP that facilitated sharing of best practices on emergency preparedness and response.

Respondents in both countries shared examples of learning and sharing of best practices.However,itwasnotablethatrespondentsinEthiopiaattributedimprovedaccessandvisibilitytoabroaderplatformasaresultofDEPP.RespondentsinEthiopiareportedaccessingplatformswithintheShiftingthePowerprojectthatfacilitatedsharingandlearningamongorganisations.AsdescribedbythisrespondentinEthiopia:“Thereisahumanitarianforumwhichisestablishedby the Shifting the Power project.We aremember of the steering committee there, so allmemberswereaskedtosharetheirexperiences.”

Respondents inEthiopia alsodescribed the impactofDEPP learningevents asmechanismsfor learning: “These arevery important, preparedness and response is not enoughby itself,collaborationwithotherorganisations is important.Ourparticipation in learningeventsandengagingindifferentplatformshashelpedusalot.ThankstotheDEPPprogrammewehavebeenabletoparticipateindifferentplatformswhichweappreciate.becausewecanlearnfromeachotherintheseplatforms.Ithasgivenusanopportunitytoreflectourideasandalsoheardifferentstudiesfromotherparticipants.”

Whilenotdirectlyassessedduringtheinterviews,respondentsdidnotmentionanylearningandsharingbetweenDEPPprojectsinthesamecountry.

IIe. Enhanced Visibility of Organisations as a Result of DEPP

Working as an organisation within the DEPP programme also helped to enhance visibility and credibility of local organisations to donors, government, and other stakeholders.

Thetrainingandcapacitybuildingallowedforl/nnGOstobecomemorevisibletodonorsandotherstakeholders.Asanexample,theShiftingthePowercapacitybuildingactivitiesinEthiopiahelped localorganisationsevaluatetheircommunicationanddisseminationplatformsandtodevelopwebsitestocommunicatetheirprojectactivities.

One of the respondents in Ethiopia commented how the DEPP components, especiallycapacitybuildingandcollaboration,helpedtheorganisationgainvisibilityandrecognitionasanorganisation that could respond todisasters.Prior to the capacitybuildingactivities andcollaborationwiththeShiftingthePowerproject,theorganisationwouldbeinvitedtoattenddistrictmeetings;however,now,after thecapacitybuildingactivitiesandcollaborationwithlocalgovernment,theyarecallednotonlytodistrictmeetingsbutalsotoUnOcHAmeetings.

III. Levels of action – Examples of individual, organisational, and systems-level changes in emergency preparedness and response

IIIa. Individual-level Changes Related to Emergency Preparedness and Response

Emergency preparedness and response capacity building activities resulted in increased confidence among individuals to respond to a disaster.


WhilemostofthecapacitybuildingactivitiesaspartoftheUEWEAprojectandShiftingthePowerprojectfocusedonimprovingpreparednessandresponseattheorganisationallevel,manyrespondentsnotedimpactattheindividuallevel.forthoserespondentsreportingindividual-levelchanges,thecapacitybuildingactivitiesledtoimprovedskills,knowledgeofproceduresandresources,andconfidence.

Asanexample,arespondentinEthiopiarecountedthatwithadequateresourcesandtraining,confidence in emergency response increased such that “you now knowwhere to start andwheretoend”.ArespondentfrominKenyawhoreceivedtraininginsearchandrescueandfirstaid,reportedimprovedcapabilitytorespondandto“respondwithoutfear”.

IIIb. Organisational-level Changes Related to Emergency Preparedness and Response

Emergency preparedness and response capacity building activities resulted in strengthened procedures and systems at the organisational level

Allrespondentsidentifiedorganisational-levelchangesasaresultofcapacitybuildingactivities,withthemostdetailedexamplescomingfromtherespondentsinEthiopiawhoreceivedcapacitybuilding training from the Shifting the Power project.Themost common changes reportedincludedshiftsinmindsets, inter-departmentalcoordination,andstrengthenedorganisationalprocedures—in other words, “institutionalising preparedness”. Some of the organisationalchangesincludedcreatinganemergencyresponseteamwithintheorganisationwithdefineddelegationofresponsibilitiesandimprovedprocurementpolicies.Developmentoforganisationalemergencypreparednessplanswasreportedtobecriticaltoimprovedorganisationalcapacityto respond. The capacity building activities also helped beneficiary organisations to linkpreparednessandresponseactivitiestohumanitarianstandardsandguidelines.

AsdescribedbythisrespondentinEthiopia:“Afterwehaveimplementedtheprojectwehavealsostartedtothinkasateam.Thebigthingisthereisashiftinourmindsets...AfterShiftingthePowerwestartedtolook[at]thingsindetail.Weidentifiedourgapsandwewereabletoidentifytheareaswemustengage.Eventhoughthereisnostand-alonedepartment,wehaveorganisedandformedanemergencyresponseteam.”

AsanotherrespondentinEthiopiareflected:“Previouslytherewasnoemergencypreparedness,everythingwasdoneoncetheemergencyoccurredbutnowwehaveemergencypreparednessplansandjobdescriptionsofeachmemberofresponseteamsready-made.Hazardsareidentified.Whodoeswhatisidentified…Inthiscasewheneveremergencyoccursweareready.Itisnotdifficulttotransferfrompreparednesstoresponse.Wealreadyhavepreparedproposalsontheshelf.ThankstoShiftingthePowerwearepartoftheteamnowandwegettheinformationfromthenational,regionalandlocal levelsandourvisibilityhasincreasedandinformationisreadilyavailable.”

IIIc. Systems-level Changes Related to Emergency Preparedness and Response

Collaboration with the government resulted in numerous systems-level changes to improve emergency preparedness and response.

While the respondents in Ethiopia commonly reported organisational-level changes, therespondents inKenya reportednumerous systems-level changes as a result of theUEWEAproject.Asa respondent inKenyareported: “…[when] theconsortiumwastryingtoget intogovernment,totransferthistechnologytothegovernment,initially,theywenttotheassembly.AndtheymanagedtoconvincetheassemblytopassamotiontoestablishEarlyWarningEarlyActioncentresineveryward.

Arespondent inKenyaalsohighlightedhowtheprojectwasalignedwithbroaderplatforms,includingtheconstitutioninKenya:“Theotheractivities—thechanceofbeingintegratedinthecountyishigh.Why?‘causeitis[oneofthe]fundamentalactivitiesthatneedstobeaddressedbytheKenyanconstitution.KenyanconstitutionsaysanyKenyanswholivewhetherininformalsettlementsor ina townhasa right togetbasic services,whichwasnot thecasebefore…beforesixyears,theoldKenyanconstitution,informalsettlementsareinformalsoyouarenotsupposedtoprovidethoseservices.Sonow,theconstitutionisnowgivingthatasplatform…Soitisapath…itisnotanisolatedactivity.”


IIId. Emergent Themes on Project Sustainability

Government engagement was key to sustainability of emergency preparedness and response projects.

TheinterviewswereconductedastheprojectswerereachingtheendoftheDEPPtimelineinKenyaandEthiopia. Assuch,manyrespondentsreflectedonthesustainabilityofprojectactivities.

TherespondentsfromtheUEWEAprojectinKenyahighlightedtheimportanceofgovernmentengagement to project sustainability. As a respondent in Kenya reported: “This project isfinishinginMarch[2018]now.Thecountygovernmentisgoingtotakeitforward.Thecounty[representative]said,“Weareprivilegedtohavethisprogramme.Itshouldbeustogoandaskyou,canyoujoinus?butyoudidallthejobsthatwearesupposedtodo.Anditisourdutytotake it forward.formethat isbigsuccess.”Therespondentsfromthe localorganisationsin Ethiopia expressed concern about local staff being able to continue and provided fewerexamplesofprojectsustainabilityduringtheinterviews.

IV. Emergent themes on disaster preparedness and response

IVa. Innovation is Needed in the Definitions of Disaster, Preparedness, and Response

Definitions of disasters and approaches to emergency preparedness and response differ between urban and rural contexts.

Many respondents reported the need for innovation in how the humanitarian communitydefines disasters.As an example, poverty and food insecurity, especially in urban contexts,are rarelyconsideredasdisasters;but ratherasdevelopment issues,andassuchgetvariedlevelsofattention fromdonorsandotherstakeholdersduringemergencypreparednessandresponseprogramming.Onerespondentnotedthatpovertyandfoodinsecuritygetpackagedintodroughtandfamineresponseinaruralcontext,butnotinurbancontexts.AsarespondentinKenyanoted:“It’salsoverydifficulttoraisemoneyquicklyforshorttermresponseandurbanareasinparticular…they[thedonors]lovefundingpeopleinruralareas.”Thisalsohighlightsthatdisasterandinthiscase,drought,affectsruralandurbanpopulationsdifferentlywhereruralpopulationsdirectlyloseaccesstofoodwhereasurbanpopulationsmaystillhaveaccessbutathighercostswhichtheycannotafford.

AdifferentrespondentinKenyadescribedthechallengesofaddressingfoodsecurityaspartofpreparednessandresponseprogramming: “The foodsecurityaspectof it isabitnew…Indevelopingtheresponsepackage,wearestartingfromscratchbecauseeverysystemthatisthereinresponsetofoodsecurityisrural-based,soyoucannotdothesamethings-livestockthingsandthingstheydothereforlivelihood-thatdoesnotworkhere.”

OneofthenotedsuccessesoftheUEWEAprojectinKenyaandasystems-levelchangeisthattheprojectwasabletobringattentiontofoodinsecurityaspartofdisastermanagement:“…Rightnowtheyjustputitintothelaw.Disaster,countydisastermanagement,andemergencymanagement[act].They’vejustinsertedfoodsecurityaspartofthedisastersothatitattractstheattentionof,ofbudgeting,becausenowtheyhavetoimplementthatlaw.And[through]thatlaw,afund,thedisasteremergencyfundiscreatedthatwillfundpartoftheresponsesthatwillemerge.”

Respondents inEthiopiaalsodescribedchallenges inseekingfundingtoaddressdisasters inaruralcontext.comprehensiveresponseprogrammesinaruralcontextneedtoincludelife-savingmeasuresnotonly forhumanbeings,butalso for livestock.Arespondent inEthiopiareported that due to budget restrictions and donor preferences, it ismore difficult to seekfundingforlivestockprogrammesasopposedtothoseprogrammeswithhumanbeneficiaries.

IVb. Standards on Emergency Preparedness and Response

Knowledge of and adherence to humanitarian standards strengthened emergency preparedness and response.

Respondentsinbothcountriesdescribedtheimportanceofhumanitarianstandards,particularlythosefocusedonemergencypreparednessandresponse,toallowforacommonunderstanding,


framework, and measure. The Shifting the Power project was successfully noted to havestrengthenedcapacityamonglocalorganisationsondevelopmentofemergencypreparednessplansaccordingtodefinedstandards.AsarespondentinEthiopiadescribed:“Whenweworkonsavingthelivestockweareworkingusingthelivestockguidelinestandard.nowwehaveasateambecomeawareofthespherestandardandcorehumanitarianstandardandsoasateamhavingacommonunderstandingisbigachievement.”

IVc. Linking Emergency Preparedness and Response Programming

Emergency preparedness and response were described as disjointed activities that need to be linked for effective response.

In general, respondents highlighted the importance of linking preparedness, in this case,surveillance,withresponseactivities.Manyacknowledgedthatthetwoweredisjointedintheproject,withone respondent inKenyapointingout that theirworkandpartneragreementswerefocusedonlyonsurveillance,andnotonresponse:“Aquicksurveillance,aquickresponse.butmoreimportantly,inmyview,you’dsavelives.Thereisonesayingwehavewiththisproject,“Actearly,savealife”.Thatisourmotto.Actearly,savealife.So,whatwewantissurveillancetofeedintoresponseandpreferably,immediatelysothatnolifeislost.yeah,butthetwoasofnowaredisjointed.”

Respondentswereaskedtoprovidetheirperspectivesonwhetherpreparednessleadstomoreeffectiveresponse. Ingeneral, respondents identifiedthatthecohesionbetweenemergencypreparedness and emergency response needed to be strengthened.A respondent inKenyaacknowledgedthatitshouldbutthat itwasn’tyetdoingso,andtherearestepsbeingtakenin thatdirection.As this respondentnoted: “It should. Itshouldbut it isnothappening.forreasons,wearelearning.”Oneofthereasonsmentionedbyrespondentsisthatfundingsourcesareoftenseparateforemergencypreparednessandemergencyresponse,whichimpedesthetransitionfromemergencypreparednesstoresponse.

IVd. The Transition from Preparedness to Response

Key components to transition from preparedness to response included community engagement, response training, and systems.

ArespondentinEthiopiahighlightedtheimportanceofcommunityengagementinfacilitatingthetransitionfrompreparednesstoresponse.Asanexample,theircommunityRiskReductionApproachprovidescommunitymemberswithpre-disasterinformation,trainingonpreparedness,andtrainingonwhattodoifadisasteroccurs.

AnotherrespondentinEthiopiaalsonotedtheimportanceofnotonlypreparednessactivities,but also response training and reported that the Shifting the Power project has filled thegap between preparedness and response. before the project, the organisation only knewpreparednessand thatnow,after thecapacitybuilding training fromtheproject, theyknowwhattheyneedtodotorespond.

Respondents also identified the importance of having a system in place to transition frompreparednesstoresponsetofacilitatereactivityanddecision-making.However,preparednessplansareonecomponentofasystemthatisneededtofacilitateanemergencyresponseasarespondentinKenyanoted:“Thattransitionfrompreparednesstoactualresponseneedsaveryrobustsystem.Andyouhavetobereadyalsotochangesomekeythingsveryfastiftheydon’tworkintherealcasescenario.”

IVe. The Transition from Development Activities to Emergency Response

Capacity building activities on emergency preparedness and response strengthened organisations’ capacity to transition from development activities to emergency response activities.

InEthiopia,severaloftheDEPPprojectbeneficiaryorganisationswerefocusedondevelopmentandnotondirecthumanitarianaidprovision.AsaresultoftheDEPP,theorganisationsgainedcapacityonhowtorespondtoahumanitariancrisis.Eventhoughtheynotedthatdevelopmentactivities,suchasaddressingfoodinsecurity,werenotthatdifferentfromhumanitarianactivities,theyneeded tochange their systems.Asa respondent inEthiopianoted, they revised their


organisationalstrategy.Theychangedprocurementandpolicies,andalertedstakeholdersandgovernmentthattheyarenowimplementinghumanitarianactivities.ThetrainingfromShiftingthePowerhelpedtosupportthisprocessofexpandingwhatwaspreviouslyconsideredtobedevelopment activities and shifting the organisational strategy to now focus on emergencypreparednessandresponseactivities.

IVf. Measuring Success of Emergency Response

Respondents lacked clear measures by which to evaluate the success of emergency preparedness and response activities.

Respondents were asked to identify examples of successful emergency preparedness andresponseactivitiesandalsoshareexamplesoflesssuccessfulprogramming.Themostcommoncriteriaidentifiedwereatimelyresponseandtheabilitytominimisesuffering.Onerespondentin Ethiopia described: “One of the objectives of the responsewas to minimise death andhumansuffering…becausewerespondedontime,wewereabletominimisethesuffering.”Therespondentcontinued:“[We]dotheseactivitiestosavelivesandimprovewellbeingofhumanbeings.Whenweseetheeffectofdrought,itistheworst.Itcauseshugeloss.Visionistoseeaworldwherebyallhumanbeingsliveindignityandrespecteachotherinprosperity.”

Whilerespondentsinbothcountrieshighlightedtheimportanceofminimisinghumansuffering,theylackedaframeworkbywhichtoevaluatethesuccessofprogramming.fewrespondentswereabletoprovidepost-disastermonitoringframeworksordata.Onereasonprovidedwasthattherewerebudgetrestrictionsonemergencyresponseactivitiesthatdidnotincludepost-disastermonitoring.

KEY FINDINGS

SummaryThis qualitative case study provides an in-depth analysis of two DEPP-funded projectsoperatingintwodifferentemergencycontexts—theUrbanEarlyWarningEarlyActionintheslumsofurbanKenyaandtheShiftingthePowerproject inruralEthiopia. Inbothcontexts,thepopulationswereaffectedbypoverty,foodinsecurity,andpoorhealthoutcomesincludingmalnutrition and diarrheal disease.While therewere similar components to the emergencypreparednessandresponseactivitiesbetweenthetwoprojects,respondentsfromeachprojectoffereduniqueperspectivesonstrengtheningemergencypreparednessandresponse.Thecasestudy integratesperspectivesfrominternationalandnationalnGOsandcountygovernmentrepresentationpartofaformalDEPPconsortiuminKenyaandfromlocalnGOsbenefittingfromaDEPPfundedprojectinEthiopia.

Major Result AreasResultswereanalysedandgroupedintofourbroadcategories:

1. Identification of key project design and implementation factors that strengthenedemergencypreparednessandresponse

2. characterisation of theways inwhich key components of DEPP (capacity building,collaboration, early warning systems, and learning) strengthened emergencypreparednessandresponse

3. Description of the levels of action (individual, organisational, and systems level) ofemergencypreparednessandresponseactivities

4. Recognition of gaps between emergency preparedness and response activities andopportunitiestostrengthenthelinkageandtransition

Key Findings• communityandgovernmentengagementwasnotedtobeessentialtoprojectdesignand

implementation,especiallyinthedesignofthesurveillancesystem,responseactivities,andbeneficiaries.Importantly,engagingcommunitymembersandgovernmentofficialsfostered trust and strengthenedpartnerships that facilitatedproject implementation.


Governmentengagement,especiallyfortheprojectinKenya,wasnotedtobekeyforsustainabilityandintegrationintofutureprogrammingbeyondtheDEPPtimeline.

• capacitybuilding,intheformofcommunityandorganisationaltrainings,strengthenedemergencypreparednessandresponsethroughimprovedawareness,coordination,andprocedures.Thesetrainingsalsostrengthenedawarenessofhumanitarianstandards.forlocalorganisationsinEthiopia,thesecapacitybuildingactivitieshelpedtostrengthenthevisibilityoftheorganisationsandalsothecredibilityoftheiremergencypreparednessandresponseactivitiestodonorsandotherstakeholders.

• collaboration, both formally through project consortia and informally throughhumanitariannetworks,improvedcommunicationandefficiencyofresourcestoavoidduplicationofactivitiesinthetargetedareas.TheconsortiumapproachfortheUEWEAprojectinKenyaprovidedaneffectivemechanismforrespondingtoemergentdisastersthataroseduringtheprojectduration,suchasacholeraoutbreak.

• Surveillanceandearlywarningsystemswerefundamentaltobothprojects,describedas the pillars of emergency preparedness and response. beyond collecting data,respondentsnotedthatthesealsoservedasmechanismsofcollaborationandprovidedopportunitiesforbehaviouralchangeforcommunitiesandorganisations.

• EngagementwithDEPPprovidedopportunitiesfororganisationstosharelearningandbestpracticesthroughwidernetworksandplatforms.DEPPlearningeventswerenotedtobeparticularlyhelpfulplatformsfor learningforthosewhoattended.Engagementwith DEPP improved visibility of organisations among key stakeholders, includinggovernmentanddonors.

• Attheindividuallevel,emergencypreparednessandresponsecapacitybuildingactivitiesresultedinincreasedconfidencetorespondtoadisaster;however,therewerefewerexamples of individual-level changes. At the organisational level, capacity building,especiallyamongorganisationsinEthiopia,strengthenedproceduresandpoliciessuchasthedevelopmentofemergencypreparednessteamsandplans.collaborationwiththegovernmentfortheprojectinKenyaallowedfornumeroussystems-levelchanges,includingalignmentandintegrationofprojectemergencypreparednessandresponseactivitieswithbroaderplatforms,suchasconstitutionalprioritiesandlocalandnationalpolicies.

• Definitionsofdisastersdifferedbetweenurbanandruralcontexts.Povertyandfoodinsecurity,especiallyinurbancontexts,arerarelyconsideredasdisastersandgetvariedlevelsofattentionfromdonorsandotherstakeholders.

• Emergency preparedness and response activitieswere described by respondents asdisjointed. They described a need for a system to link preparedness and response.facilitators of this transition from preparedness to response include communityengagement, response training, and organisational procedures and standards. Theneed to seek separate funding for response activitieswas described as a barrier totransitioningfromemergencypreparednesstoresponse.

• When asked to evaluate the success of emergency preparedness and responseprogramming,respondentslackedclearmeasuresorframeworksbywhichtoevaluateeffectiveresponseprogramming.

Overall Conclusion of the Two Case Studies

TheSouthSudancasestudydemonstratedthatispossibleforprojectstooperateinhigh-risk,conflict-affectedcontexts;however,securityprotocols,flexibleandadaptivemanagementandbudgets,contingencyplans,aswellasengagementandintegrationoflocalpartnersandstaffarevital. While the case study inSouthSudan found that engagementof thegovernmentstakeholderswas not always possible, the case study in Ethiopia and Kenya demonstratedthat theconsortiummanagementmodelandcollaborationwithgovernmentwerecritical tosuccessfulimplementationofearlywarningsystemsandpermittedcoordinatedhumanitarianresponses that improvedwith each subsequent emergency. both case studies stressed theimportanceofhavingflexiblebudgetsandrapidfundingmechanismseithertomakechangesinoriginalprojectplannedactivitiesortosupportresponseactivitieswhenadisasteroccurs.


CONCLUSIONS &RECOMMENDATIONS

cHAPTER9

209 SUMMATIVEPHASEREPORT-cOnclUSIOnSAnDREcOMMEnDATIOnS

OVERALL CONCLUSION TheDEPPwasa3-year,£40millionhumanitariancapacitydevelopmentprogrammefundedbytheDepartmentforInternationalDevelopment(DfID).Itincluded14distinctprojectsaimingtobuildhumanitariancapacityoflocalactorsusingacollaborativeapproachin10countriesatriskforbothnaturalandmanmadedisasters.Theprogrammewasdesignedin2014,andwasaheadofitstimeinmanyways,especiallywithitsstrongfocusonlocalisationandmulti-stakeholderconsortia. It includedamixofbothflagshipprojectsusingmore traditionalapproaches (i.e.,in-personindividualcapacitybuilding)withsomemoreinnovativeelements(i.e.,a£10millionInnovationWindow;flexiblefundingmechanismstofinanceboth localcapacitybuildingbutalsoemergencyresponse;pooledsurgeplatforms;urbanearlywarningsystemsandinclusionoffoodsecurityasanemergency).

Asitsufferedfromanumberofdesignchallengesincludingathree-yeartimeframethatwasunrealistictomeettheobjectives,anduniversalimplementationdelays,thedecisionbyDfIDtono longerfundtheprogrammelimitspotential long-termimpactoftheDEPP.Regardlessof the fundingdecision,DEPP’s focusonemergencypreparednessand localisation is still arelevantone.Italignswellwiththegrowingdialogueamongtheglobalhumanitariancommunityandwithmore recent policy commitments such as theGrand bargain and thecharter forchangewhichadvocateforincreasedvoiceforlocalactors,andforrebalancingofpowerandrelationshipswithintheexistinghumanitarianarchitecture.

Despite its initialdesignflaws,positivefindingsrelatedtothecollectiveactionoftheDEPPand to individual DEPP projects have been documented. These include the following keyachievements:

• based on project reports, 33,388 individuals were exposed at varying degrees tocapacitybuilding

• Stakeholders across the programme noted that themost significant change due totheDEPPwasachangeinattitudestowardandincreasedsupportoflocalisation(i.e.,contributingtotheGrandbargain)

• changesincapacityoflocalorganisations(intermsofsystemsandprocesses,fundraising,conductingneeds assessments) and their preparedness levels, that led to outcomessuchasbecomingeligibleforUnpooledfunding,successinsecuringexternalresponsefunds,andbeinggivenleadershiproles

• Significant changes in organisational policy with respect to inclusion of vulnerablegroups(Ethiopia,localorganisationsacrossallintensivesetevaluationcountries),andlocalisation(Ethiopia)

• Someevidenceofstrengthenednetworksandincreasedcollaboration(Ethiopia,Kenya,thePhilippines)

• contributiontoatleast42humanitarianresponsesin11countriesandsomequalitativeevidenceinseveralsettings(ThePhilippines,Kenya)ofimprovedspeed,efficiencyandinclusivenessofresponse

• cross-countrylearningintermsoftheamountoflearningdocumentsproduced,andsharingofideasacrosscountriesandprojects

• Strongcross-projectcollaborationinthePhilippinesdemonstratingbenefitsofcollectiveaction

nevertheless,agreater impact,asoriginallyenvisioned,could likelyhavebeenachievedhadtherebeenamorecohesivedesign,alongertimeframe,amorerobustprogrammeM&Esystemandamorestrategicapproachwithregardstothetypeofcapacitybuildingimplementedateachlevelofaction(individual,organisational,community,systems)andthebalancebetweentheselevels.Overall,theexternalevaluationfoundthefollowingfindingsthatneedtobeexaminedfurtherinfutureevaluationsandprogrammingrelatedtoemergencypreparedness:


• noquantitativeevidence todate that the three-yearprogramme led tomeasurableimpactwithregardstomoreefficient,andtimelyhumanitarianresponse

• noquantitativeevidencetosupportthehypothesisthatstrengthenednetworkscanlead to improved emergency preparedness and response (mainly due to the shortobservationperiodavailabletoassessthis)

• noquantitativeevidencethatDEPPimprovedindividualknowledgeoncorehumanitariancompetencies,preparedness,inclusionofvulnerablegroups,orprotectionissuesinthefourintensivesetcountries

• fewchangesinquantitativepolicy-relatedindicators,thoughthereweresomeexamplesofimprovedpoliciesdocumentedbyprojects

• Toowideofavarietyofindividualcapacitybuildinginitiativesandlackofstrategyontypeofcapacitybuildingapproachesateach level (individual,organisational,system,community)andeffectdesiredateachlevel

Asnotedabove,thethreeyearsallottedforanemergencypreparednessprogrammetoachievefivemacroresultsin10countrieswasanambitioustimelineandagenda.ThelevelofexposuretotheDEPPprogrammeactivitieswasunevenlydistributedacrossthe10DEPPfocuscountries;eachcountry receivedonlyasubsetof the14projects.forexample, threeDEPPcountriesbenefittedfromsixprojectseach,whiletwocountrieshadonlyoneprojecteach(notincludingthelearningProject;seeTable4.7forthefulllistofprojectnumberspercountry).ThisunevenlevelofexposuretoDEPPcompoundedwithvaryingcountry-specificcontextualfactors ledtovaryingresultsinspecificcountries.HadtheDEPPbeenabletocontinueforanotherthreeyearsormoreandbeenabletointegrateitslearningsintoarevisedprogrammestrategy(i.e.,permittingtimetorestructureitsmanagementstructure,scaleupprojectsthatshowpromiseand reviseprojects that are less promising), therewouldbe amore feasibletimeperiod toobtainmoreconclusivefindingsaboutDEPP’sachievementstowardsitsfiveresultareas.

FINDINGS AND THE PROGRAMME THEORY OF CHANGE

TheevaluationwasnotbuiltaroundtheoriginalprogrammetheoryofchangeaspresentedinthebusinesscaseduetothefactthatitdidnotalignwellwiththetypesandscopeofDEPPactivitiesthatwereapprovedforimplementation.However,itmaystillbeworthwhiletoexaminetheDEPPfindingsagainstthattheoryofchangeandtoassesswhethertheassumptionsmadeduring thebusinesscasedevelopmenthavebeenproventrue.Table9.1presents the threekeyDEPPactivitiesintheoriginaltheoryofchange(capacitybuilding,improvingpreparednesssystemsforcommunities,establishingmulti-stakeholderplatforms),andtheexpectedoutputs,shorter-termoutcomesandlonger-termoutcomesagainsttheresultsthatwereachievedbytheDEPP.Table9.2presentstheassumptionsthatwerearticulatedintheoriginaltheoryofchangeandtheavailableevidencecorrespondingtoeachassumption.

Ascanbeseen,theevaluationfindingsdemonstratethattheoriginalprogramme-leveltheoryofchange(aspresentedinthebusinesscase)couldnotbeprovenordisproven.Mostoftheassumptionswerenotmetandanumberofcomponentsoftheoriginaltheoryofchangewerenotimplementedasoriginallydescribed.forexample,“multi-stakeholderplatforms”comprisea largepartof the theoryofchangeandweredescribed in thebusinesscaseas “platformswithabroadmembershipe.g.,InGOs,nationalnGOs,localgovernment,nationalgovernment,academia,privatesectorcompanies,theRedcrossRedcrescentMovement,andUnagencies.Successful platforms should improvepolicy atnational and international level.Weenvisageasingleplatformpercountryor regionwhere theprogramme isactiveandan internationalplatform. Possible products of the platforms are: synthesised lessons identified, jointevaluationsandcommonpositionpapersforadvocacy.”Theseplatformswerehypothesisedtobethemechanismthroughwhichpolicychangeandevidencewouldoccur.TheDEPPfocusedmuchmoreonconsortiaratherthanonlargermulti-stakeholderplatformsasdescribedinthebusinesscase(thoughsomeprojectsasTransformingSurgecapacity,cDAc’scommunicatingwithcommunitiesprojectdidincludethis).Similarly,advocacywasnotaslargeofaprogrammeemphasisasoriginallyintended.Ultimately,becauseofthedifferencesinwhatwasimplementedandthefactthatmostoftheassumptionsdidnothold,itisimpossibletoproveordisprovethetheoryofchange.Itshouldberevisited,revisedandre-testedinanyfutureprogrammedesignprocess.


DETAILS RESULTS LESSONSAC

TIV

ITIE

S

1. Capacity Building

capacitydevelopmentinterventionsreachnationalactors(nationalstaffofInGOsandnationalpartners,governmentstaff)andimprovetheorganisationalandsocialenvironmentnecessaryforeffectiveresponse

33,388individualswereexposedtocapacitybuilding,butequalgendertargetnotmet

•levelofexposurewasvariable(manyshorttrainingsorwebinars) •focusonquantityoverquality •Didnotmeetequalgendertarget •Accessibilityissuesrelatedtolackofavailabilityoftrainingsandmaterialsinlocallanguages •comparisongroupalsoexposedtocapacitybuildingindicatingthattherearemanyindividual-levelcapacitybuildinginitiativesbeingimplemented

Organisationalandsocialenvironmentnotconsistentlyorsystematicallytargetedthroughactivities

•Structuralbarriersrelatedtoorganisationalandsocialenvironmentnotadequatelytargeted;difficulttoachieveorganically

Raisingawarenessofgoodpracticeinpreparednessandresponse

learningevents,conferences,disseminationoflearningproductshelpedraiseawareness

•Amoresystematic,strategicawarenessraisingapproachmighthavebroadenedreachandbeenmoreimpactful

Enhancingskills(personalandtechnical)

Manycapacitybuildingprojectsfocusedonskillbuilding(learningbydoing,simulationsetc.)

•Structuralbarriersatorganisationswhichimpedestafffromapplyingnewskillsnotadequatelyaddressed

Improvingteamwork

Activitiesemphasisedstrengtheningcollaborativewaysofworking

•Perhapstoomuchemphasisoncollaborationwithoutenoughconsiderationformorenuancedapproachesandcontextualdifferencessuchaskeyinfluencersinthecountry-specifichumanitariannetworks

•Therewaslimitedcapacitybuildingonhowtoeffectivelycollaborate

fosteringcoalitionsandnetworks

Activitiesfocusedonstrengtheningcoalitionsandnetworks

•Greateremphasisonconsortiastrengtheningthanthebroaderhumanitariannetworks

Advocacystrengthenspolicy

Advocacyeffortswerelimitedandtherewasnoprogrammewidestrategy

•Mainstreamingofadvocacyandpolicyeffortsacrossprogrammemighthaveenhancedresults


ACTI

VIT

IES

2. Projects to improve preparedness systems for response for communities at risk of disaster

Projectstoimprovepreparednesssystemsimplemented

4projectsfocusedondevelopingorimprovingpreparednesssystemsforcommunities

•Diversegroupofprojects(includesurbansettings,publichealthsurveillance,earlywarninginconflict-affectedareas)

3. Multi-stakeholder platforms established to enable collective action for capacity development

Enabledcollectiveactionforcapacitydevelopment

collaborativestructuresweresupportedacrosstheprogramme,includingmulti-stakeholderplatformsinseveralsettings

•Multi-stakeholderplatformswerenotemphasisedasmuchasconsortia-basedapproachesandmoregeneralcollaboration

•Multi-stakeholderplatformswerenotimplementedineverycountryorregionasarticulatedinthebusinesscase

capturedlessonsandevidence

Over600learningproductswereproducedacrossDEPP,butqualitywasvariable

•focuswasonquantityoverqualityandtherewasminimalempiricalevidence

Advocatedforchangebasedonemergingevidence

Someadvocacyeffortswereimplementedbutnosystematicadvocacycomponentattheprogrammelevel,andnotnecessarilybasedonevidence

•Advocacystrategyshouldbeincorporatedatprogrammelevel

OU

TPU

TS /

SH

ORT

-TER

M O

UTC

OM

ES

1. Improved knowledge and understanding of people in the system regarding best practices for humanitarian preparedness and response

Improvedknowledgeandunderstanding

noquantitativeevidenceofimprovedknowledgeoncorehumanitariancompetencies,preparedness,inclusionofprioritisedgroups,VAWG

Qualitativedatasuggestssomeimprovementinknowledge

•Toomanydifferenttypesofcapacitybuildingstrategiesused.focusshouldinsteadbeonknowledge,skillbuildingandretentionandshouldconsidermulti-prongedapproaches

2. Increased number of coalitions and partnerships developed

Increasednumberofcoalitionsandpartnerships

IncreaseinnumbersofcoalitionsandpartnershipsacrossDEPP

EvidenceofstrengthenednetworksinEthiopia,Kenya,thePhilippines

•Insufficienttimetotestthehypothesisthatstrengthenednetworksleadtoimprovedpreparedness


OU

TPU

TS /

SH

ORT

-TER

M

OU

TCO

MES 3. Emerging evidence

base for what works in building humanitarian capacity

Evidencebaseestablished

Evidencebasethathasbeencreatedismainlylearning-canteredbutnotforprojectmonitoringandimproving

notenoughempiricalevidencegeneratedthroughouttheprogrammecyclebyindividualprojects

•limiteduseofevidencebaseandlesssoatcountrylevel

•AccessibilityofevidencegeneratedwasalsoproblematicasmostproductswereonlyavailableinEnglish

4. Improved institutional and policy environment for building humanitarian capacity

Improvedinstitutionalandpolicyenvironment

Somequalitativeevidenceofstrengthenedpolicybutonlyinsomesettings

•Insomecases,multi-stakeholderplatformscontributedtothese,inothercasesitwasgeneratedbytheprojects

5. Improved preparedness systems for communities at risk of disaster

Improvedpreparednesssystemsforcommunitiesatriskofdisaster

5of6preparednesssystemsarefullyoperational

•Earlywarningsystemsrequirealongertimeframetodevelopandimplement •Thetwosystems(one-thirdofallthesystems)thathavebeentestedand/orusedinresponsehavedemonstratedimprovedpreparednesssystems

LON

GER

- TER

M

OU

TCO

MES

Improvement in effective delivery of humanitarian assistance to disaster affected communities by national individuals and organisations

Improvedeffectivedeliveryofhumanitarianassistance

Somequalitativeevidenceofimproveddeliveryofhumanitarianresponseinafewsettings

•Timeframetooshorttoachievethis •limitedaccesstoresponsefundshindered/preventedpotentialresponses

DiD NOT AcHIEVE

PARTIAllyAcHIEVED

fUllyAcHIEVED

cAPAcITybUIlDInGAnDDOWnSTREAMOUTPUTS/OUTcOMES

PREPAREDNESS SySTEMSAnDDOWnSTREAMOUTPUTS/OUTcOMES

MUlTI-STAKEHOLDER

PLATFORMS AND DOWnSTREAMOUTPUTS/OUTcOMES

Table9.1:Assessmentoftheoriginalprogrammetheoryofchangeagainstfindingsfromtheevaluation


DETAILS RESULTSA

SSU

MPT

ION

S

Capacity sufficiently improved to meet humanitarian standards relevant to target group

•Datasuggestthatthecapacitybuildinginterventionsdidnotleadtosignificantquantitativeimprovementsinindividualcapacity

Trained staff and local leaders with enhanced skills participate and remain in sector

•Staffandlocalleaderswereexposedtotrainingandthereisnoevidencetosuggestthattrainedstaffandlocalleadersleftthehumanitariansector

Platforms are the appropriate mechanism to strengthen vertical and horizontal practice, learning and accountability networks

•Somedatafoundthatcollaborativestructures(i.e.,consortia)cancontributetoimprovedpreparedness,responseandlearning

•notenoughevidencearoundplatformsandaccountabilitynetworks

Platforms and emphasis on M&E allows project learning to be taken up across both organisations and the system

•ProgrammeM&Ewasweak

•Someevidenceoftakeupofprojectlearningwithinorganisations

•limitedevidenceoftakeupacrossthesystem

•UnabletotesttheassumptionthatplatformsplusstrongM&EcontributetotakeupofprojectlearningduetotheweakM&E

Platforms improve collective advocacy or higher-level change (e.g., organisation and policy)

•challengesinsettingupandsustainingmulti-stakeholderplatforms

•Multi-stakeholderplatformswerenotaswidelyimplementedasoriginallyintended

•Platformscontributedtoorganisationalchange,butlimitedevidencethattheycontributedtoimprovedadvocacy

Sound analysis of hazard and differential capacity / vulnerability risks form basis of projects

•needsassessmentsandinvolvementoflocalstakeholdersandbeneficiariesdidnotoccurduringdesignphase

Projects inclusive of all relevant population groups under good local leadership

•Projectsfacedchallengeswithinclusionofbothwomenandofprioritisedgroups(bothintermsofdesign,implementationandinclusivenessofactivitiesandcapacitybuildingefforts)

DiD NOT AcHIEVE

PARTIAllyAcHIEVED

fUllyAcHIEVED

Table9.2:Assessmentoftheassumptionsoftheoriginalprogrammetheoryofchange

RECOMMENDED ACTIONS FOR PROGRAMME COMPONENTS

Given the key evaluation findings, there is scope to replicate and further test, or scale upsomecomponentsoftheprogramme.ThiswillensurethatthepreviousinvestmentinDEPPissustained,andthatanyadditionalresourceswillbeusedeffectively,efficientlyandstrategically.Ascanbeseen,noneoftheDEPPprogrammecomponentshavebeenrecommendedtobeceasedandnotbepursuedinthefuture.ThisinitselfisastrongindicationthattheDEPPcoretheoryofchangeandinitialassumptionsareworthyofbeingre-examinedforfutureinvestment.Specially,ninecomponentswererecommendedtobemodifiedandre-tested,whileeightwererecommendedtobereplicatedindifferentcontextsandpotentiallyonalargerscale,enablingmore rigorousevaluations.note that inaprogrammeeachof thesecomponentswouldnotbeexpectedtofunctioninisolation,andtheinteractionbetweencomponentsshouldalsobeconsidered.


DEPP PROGRAMME COMPONENT RECOMMENDED ACTION KEY FINDING / LESSONS LEARNED

Individual Capacity Building Modifyandre-test

• Prioritisemulti-prongedapproaches(in-person+coaching,mentoring,simulations)

• Qualityoftrainingsshouldbeprioritisedoverquantityofindividualstrained(webinarscanreachalotofpeoplebutmaybelesseffective)

• considerstructuralbarrierswithinorganisationsthatmayimpedeapplicationoflearning

Organisational Capacity Building Modifyandre-test

• Prioritiseapproachesthatpermitorganisationalself-assessmentsandcapacitybuildingapproachestailoredtoorganisations’needs

• focusonstrengtheningofadministrativepolicies,proceduresandsystems(i.e.,humanresources,procurement,financesystems)hasbeeneffectiveforl/nnGOs

• Approachesshouldbebetteradaptedtodifferentcontexts

• Accesstofundingiskeyfororganisationstoputlearningintopractice

• conductcomparativeassessmentsoftheeffectivenessofdifferentapproaches

Community Capacity Building Modifyandre-test

• fewprojectsaddressedthis,butthosethatdidindicatedsomepotential;furtherevidenceisneededonhowbesttobuildcommunitycapacity

Systems Capacity Building Modifyandre-test

• fewprojectsdirectlyaddressedthis,buttherehasbeensomeevidenceofqualitativeattitudinalchangethatcouldbeaprecursortoeventualsystems-levelchange;moretimeandfurtherevidenceareneededonhowbesttobuildcapacityatthesystemslevel

Early Warning Systems Replicateandtestonlargerscaleindifferingcontexts

• UEWEA’surbansurveillancesystemshouldbetestedinotherurbansettings

• Otherearlywarningsystemprojectscouldbeadaptedforandimplementedinothercontextswithsufficientprojecttimelines

Pooled Surge Platforms Replicateandtestonlargerscaleindifferingcontexts

• Pooledsurgeplatforms,especiallynational-levelplatformshaveshownsomeevidenceofeffectiveness.Pooledsurgeplatformsshouldensurealocallydrivendesignprocessandimplementationtoensurecontextualisationandlocalownership

• Replicateandtesttheseindifferentcontextstogeneratemorerigorousevidenceontheireffectiveness


Flexible Funding Mechanisms

Replicateandtestonlargerscaleindifferingcontexts

• flexiblefundingmechanismswerefoundtobeapromisingapproachtosupportl/nnGOsself-directedcapacitybuildingefforts.Inaddition,flexiblefundswereessentialtotranslatingemergencypreparednessintotimelyandefficientresponseactivities,especiallyforl/nnGOs

• flexiblefundingmechanisms(bothforcapacitybuildingandresponseefforts)shouldbereplicatedinothersettings,andmorerigorouslyevaluated

Multi-stakeholder Platforms Modifyandre-test

• Theseplatformsweremoreeffectiveincontextswhereexistinghumanitarianstructuresweremorematureandwithsupportinggovernments.

• furthermodificationandtestingoftheapproachisneededtounderstandhowbesttoutilisethisstrategyandinwhichspecificsettings

Consortia Modifyandre-test

• consortiathatbuildonexistingrelationships,commonvaluesandwaysofworking,andthataresmallerwithinclusionofl/nnGOsshouldbefavoured

Strengthening Networks Modifyapproachandre-test

• Useknowledgeanddataoncountrynetworkstocreatemoretargetedapproachestostrengthennetworks;allowenoughtimetotestwhetherstrongernetworkscanleadtoincreasedemergencypreparedness

Advocacy and Policy Mainstreamacrossprogramme

• Someadvocacyeffortswereimplementedbuttherewasnosystematicadvocacycomponentattheprogrammelevel.Itwasrecognizedthatadvocacyandpolicychangeareneededtoovercomebarrierstochangeandtoincreaseprogrammesustainability

• Ensureacohesiveprogramme-widestrategyrelatedtoadvocacyandpolicy

Learning Project Modifyandre-test

• Theapproachofhavinganindividualprojectdedicatedtolearningandgeneratingevidenceisuniquebutfindingsweremixedduetothelackofprogrammemonitoringandthelackofadirectlinktothemanagementteam.Alternativemodelsforanindependentlearningprojectcouldbeenvisionedandpotentiallyassessedinafutureprogramme

Evidence Generation Modifyapproach

• Prioritisequalityoverquantity

• Includeempiricalevidencegeneration

• Adaptevidenceuseandknowledgetranslationstrategies

Protection Mainstreamacrossprogramme

• Ensureacohesiveprogramme-widestrategyrelatedtoprotection

Inclusion of Prioritised Groups


• Ensureacohesiveprogramme-widestrategyoninclusionofprioritisedgroups(suchaswomen,theelderlyandpeoplewithdisabilities)


Localisation contextualiseandreplicateandtestonlargerscalein

differingcontexts

• Definelocalisation(whatitis,whichentitiesareconsideredlocal);andtestapproachestoenableInGOstotakeamoresupportiveroletowardsl/nnGOs,andbetteroperationaliselocalisationindifferentcontexts

DO NOT PURSUE

FURTHER

MODIfyAnDRE-TEST

cOnTExTUAlISEAnDREPlIcATEInAnOTHER

SETTiNG OR MAiNSTREAM AcROSSPROGRAMME

Table9.3:RecommendedactionsforeachDEPPprogrammecomponent

Inaddition,anumberofbestpracticesrelatedtotheDEPPprogrammecomponentshavebeencompiledtoguidefutureprogrammedesignandimplementation.ThesebestpracticescanbefoundinAnnex11.

SPECIFIC CONCLUSIONS AND RECOMMENDATIONS(SUMMATIVE PHASE) This section presents specific conclusions and recommendations related to each evaluationquestion.

1. RELEVANCE AND VALIDITY OF DESIGN CONCLUSION 1: WhiletheobjectivesandtargetsoftheDEPPwereappropriate,relevantandalignedwithDfIDpriorities,theDEPPdesignprocesssufferedfromnumerousweaknessesthathindered itspotentialfor impact.AnyfutureprogrammewouldrequireasignificantredesignbasedonthelessonslearnedandtheevidencegeneratedfromtheDEPP.forfuturedesign,programmeresultsthatarerealisticwithinathree-yeartimeframemustbeset,ortheprojecttimeframeshouldbeextendedtoallowformoretimetomeetlonger-termoutcomes.Amorelogical,coherentdesignprocess,which is locally led,drawsonneedsassessments,andusesparticipatoryapproachesshouldbeprioritisedtoensurethatprojectsarecontextualisedandbasedonexistingneedsatthecountrylevel.Thedesignprocessshouldadequatelyconsiderprojectalignmentandcomplementarityaswellasrisksattheprogrammelevel.Itshouldalsoensurethedevelopmentandimplementationofprogramme-levelsystemsandprocesses,suchasarobustM&Esystem,toensurecoursecorrectionandthatprojectsfunctionasaportfoliorather than as standalone projects. This includes elaborating a programme-level theory ofchange,definitionofkeyterms,cross-projectlinkages,andstreamlinedinclusionofgenderandprioritised groups alongwith appropriate considerationof cross-cutting themes. Emergencypreparednessandresponseprojectsshouldalsoconsiderurbancontextsand issuessuchaspovertyandfoodinsecuritywhichlieatthehumanitarian-developmentnexus.budgetsshouldincludehigherallocation fornPAcs,programmemanagement functionsandM&Eacrossallprojects.

RECOMMENDATION 1.1: [DEPP PROGRAMME BOARD AND DFID] The design process should be re-envisioned to consider alternative programme models and governance structures. To increase the local relevance and effectiveness of theprogramme, the design process should be locally driven from the outset (i.e., led by


localstakeholders),andincludesufficienttimeandfundsforprojectdesign,localisation,consortiadevelopmentandcollaboration.

Potential Model 1: The process could involve several stages: After the overallprogrammatic business case and theory of change are developed, global goalscould be established and local andnational ownership prioritised.A programme-level focal point could be appointed in each country to conduct contextualisedneedsassessmentswhichwould feed into thedevelopmentof theportfolio-levelstrategy.Theproject designprocesswouldoccur at the country level, under theoversightofthefocalpointsandtheProgrammeboard.Thisapproachwoulddrivethe localisation agenda and allow for country-based organisations to determinethemostneededinitiativeswithinthegoalsoftheoveralltheoryofchangeaswellasriskassessment.Itwouldhelpalignprojectstowardcommonprogrammegoals,identifycomplementaryelementsofdifferentprojects,mainactorsandthosethatneedtobebolsteredaswellasmoreriskyprojectsworthyofsupport.Itwouldalsoensuresufficient investment ineachcountry inordertoaccomplishthetheoryofchange,andwouldenablemorecomprehensiveassessmentsofrisks.Also,areturnoninvestmentassessmentcouldbeconductedtoinformthefinalinvestment.focalpointscouldworkwitharegionalorcountryM&EandlearningAdvisortodevelopandimplementprogramme-andproject-levelM&Esystems.Asaprogramme-levelactor, the focalpointwouldhelpbuildcohesivenessandenhance thevisibilityofDEPPasaunifiedprogrammeineachcountry.

Potential Model 2: Alternatively, consider a multi-phase programme in which aseriesofpilotinterventionsaredevelopedandimplementedonasmallscalewithstrong but targeted M&E systems integrated to evidence changes and provideaccountabilitymechanisms.Successfulpilotinterventionscouldbescaledupduringthesubsequentphase,buildingonthelessonscapturedduringthepilot.Astrongprogramme- and project-levelM&E system during the scale up phasewould beintegraltomeasuringchangeandensureregulartrackingofoutputstomodifytheprogrammewhenneeded. Inaddition, this typeofmulti-phaseprogrammecouldincludeapre-pilotphasewhereseedfundingisprovidedtodeveloppromisingideas,toconceptualiseprojectsandtobuildconsortiaaroundtheprojects.

RECOMMENDATION 1.2: [DEPP PROGRAMME BOARD AND DFID] The programme portfolio should be balanced in terms of risk, types of activities, size of projects, geographic scope and cost. To maximise impact, the link between emergency preparedness and response should be better articulated; preparedness programmes should strongly consider embedding flexible funding mechanisms in order to facilitate contribution to humanitarian response by local actors.

Theportfolioshouldincludeamixofmoreestablishedideasandprojects,thosethatareconsideredriskierandwithahigherlikelihoodoffailure,aswellasinnovativeelements.Prioritisinghumanitarianneed,examiningcountrydisasterandemergencyprofiles,aswellasaligningwithoverallprogrammestrategyduringcountryselectionprocesswouldensureamoreoptimal,balancedsetoffocuscountries.forafundofthissize,concentratingonasmallergroupofprojects(atleastinanyscaleupphaseofamulti-phaseprogramme),andpotentiallyfeweroverallcountriesmayhelptoensurethatinvestmentsarenotspreadtoothinandthatthereisbothsufficientinvestmentandalargeenoughnumberofprojectsineachcountrytoreachacriticalmassandachievesystems-levelchanges.Acomplementaryandstrategicmixofactivitiesattheprogrammelevelandespeciallyatthecountryleveliswarrantedtomaximise impacts;but this tooneeds tobefocused.forexample, theDEPP’sopenapproachledtotoomanydifferenttypesofcapacitybuilding,toogreatanemphasisonindividualcapacitybuilding,andlackofconsensusondesiredeffectsateach


level.Instead,amoredeliberateandevidence-basedglobalcapacitybuildingstrategyisneeded,withmorebalancebetweenlevelsofaction(individual,organisational,systems),aswellasconsiderationofstructuralbarrierstochangethatmighthinderbehaviourchangeor operationalising organisation-level changes. Inmany cases advocacy, policy changeorother system-based approachesmaybe critical to easing someof theseobstacles;integratingaprogramme-wideadvocacyorpolicycomponentshouldbefurtherexplored.Emergencypreparednessmustnotbethoughtofinisolationandmechanismstoembedlinkagestohumanitarianresponse(includingresponsefunding)areneeded.

RECOMMENDATION 1.3: [PROGRAMME BOARD AND DFID] Portfolio-level harmonisation should be prioritised, including mainstreaming of gender and inclusion of prioritised groups.Toensurealignmentoftheprojectsandacohesiveprogramme-wideapproach,portfolioobjectives,aswellaskeyterms(e.g.,collaboration,preparednessandlocalisation),andapproaches(e.g.,typesofcapacitybuildingefforts)shouldbefullydefined,andthedefinitionsstandardisedacross theprogramme.cross-cuttingthemesthatareprioritisedbythedonorand/orprogrammeandprojectstakeholders, suchasgenderandaddressingVAWG,mustbegivenadequateattentionandembeddedacrossthe programme. Inclusion of prioritised groups and gender considerations are equallyimportantandshouldbemainstreamed.

RECOMMENDATION 1.4: [PROGRAMME BOARD AND DFID] Ensure objectives are realistic and feasible within the programme’s time frame.Developprogrammeswithrealistic objectives that are feasiblewithin a three-year time frame.Thesemost likelywould focus on outputs and shorter-term outcomes. Alternatively, consider creatinglonger (i.e.,five-yearor longer)programmetimelines if longer-termoutcomesmustbeincluded.WhilelongerprogrammetimelinesmaynotcorrespondtocurrentDfIDfundingcycles,amulti-phaseprogrammecouldbeaneffectivesolutionthatiscompatiblewithfundingtermsaswellasactivitiesaimedatlonger-termimpact.

RECOMMENDATION 1.5: [PROGRAMME BOARD AND DFID] Ensure M&E processes are integrated within the programme design from the outset and that there is an appropriate balance between M&E and learning activities. Itisimperativethataprogramme-levellogicalframeworkbeinplacepriortoimplementationofanyprogrammeorproject activities.Developmentof this framework shouldoccur inparallelwith theprogrammeandwithinputfromlocalstakeholders.Ideally,indicatorsshouldbestreamlinedacrosstheprogramme,andshouldcontainacoresetofprogramme indicatorsagainstwhichallprojectsreport,withpotentiallyaseriesofadditionalsite-specificorproject-specific indicatorswhere relevant. Incorporate quantitativeVfM indicatorswithin thecoresetofindicatorstoensureVfMandefficiencycanbetrackedacrosstheprogramme.TheM&Esystemshouldenablethecollectionandreportingofdatadisaggregatedbysexandotherprioritisedgroups.Inatleastoneprogrammefocuscountry,considerprioritisinga randomised controlled trial (RcT) or other rigorous evaluation design to evaluateprogrammeeffectiveness.Thiswouldrequireinvolvingtheevaluationteamintheinitialstagesoftheprogrammedesignprocesstoensurealignmentoftheevaluationquestions,randomisationoftheinterventionsites,andadequatebaselinedatacollection.Thiswouldallowfortestingoftheeffectivenessoftargetedcomponentsoftheprogrammeaswellastheprogrammeoverall,isolatingtheroleofdifferentcontextualfactors.Technologicalsolutionssuchasdashboardswith real-timedatavisualisationsofkey indicatorscouldhelpenhanceroutinemonitoring,andallowformorerapidcoursecorrection.SufficienttrainingforprogrammeandprojectfocalpointswithrespecttotheM&Esystemmustbeprovidedandappropriatemechanismstoensurehigherqualitydatamustbeinplace.learningisimportantbutrequiresawell-designedM&Esysteminordertobeofmostvalue.


RECOMMENDATION 1.6: [DFID] Re-examine the development of the business case, as well as its timing and its content, and broaden definitions of disasters, emergency preparedness and response. for any future programme at the scale of theDEPP, tofacilitateprojectdesign, thebusiness case shouldprovidetimelyand sufficientdetailssuch as focus countries, the criteria for country selection, a set of definitions of keyconcepts(suchascollaboration,institutionalarrangements,emergencypreparednessandlocalisation),andaconcisesummaryoftheDfIDpolicyoninclusionofprioritisedtargetgroupsinprogrammeandprojectdesign.furtherconsiderationsforhowtothinkaboutemergency preparedness in urban contexts and conflict-affected settings are needed.considerationsforissuessuchaspovertyandfoodinsecuritywhichlieattheintersectionof development and humanitarian sectors should be considered when designingemergencypreparednessandresponseprojectsdependingonthecontext.

2. RELEVANCE AND EFFECTIVENESS OF THE INTERVENTIONS CONCLUSION 2:DEPPinterventionswereoverwhelminglyfoundtoberelevantandappropriateandweredemonstrated tofill importantgaps thathavenotbeenpreviouslywelladdressed(surveillance of emergencies in urban settings, conflict-sensitive emergency preparedness,inclusionofaginganddisability,etc).However,start-updelayslimitedprojectimplementationperiodsandreducedpotentialimpact.Optimisingthedesignprocessasdescribedaboveandstreamliningadministrativeandcontractualprocesswouldensuresmoothandtimelyprojectstart-upandminimiseimplementationdelays.Attheprogrammelevel,therewasnoevidenceofimprovedknowledgeoncorehumanitariancompetencies,inclusion,oremergencypreparednessduetotheDEPPbutsignificantchanges inorganisationalcapacityandpreparednessamongDEPP beneficiaries occurred, in particular among local organisations. furthermore, DEPPorganisationscontributedtoimprovedresponseinseveralcontexts(ThePhilippines,KenyaandEthiopia).Agreaterimpactcouldlikelyhavebeenachievedwithalongerimplementationperiod,andwitheasieraccesstoemergencyresponsefunding.Thereisscopeforcertainprogrammecomponentstobereplicatedorscaledupormodifiedandfurthertested.

Conclusion 2a: Collaboration and Consortiafunctioning of consortiawas suboptimal and contributed to implementationdelays and slow decision-making. A smaller, more strategic set of consortiamembers, with more consistency between UK and in-country members foreach projectwould streamline project delivery and should be considered infutureprogrammes. Inclusionofmorel/nnGOswithin consortiamighthelpshiftpowerbalancesandcontributetolocalisation.collaborationswithexternalstakeholdersincludinggovernmentandUn,thoughessential,weredifficultandlessonslearnedshouldinformfutureprogramming.IntwosequentialdiarrhoealoutbreaksinKenya,theconsortiamodelandcollaborationwiththegovernmentenabled coordinated effective response activities that improved with eachsubsequentemergency.

Conclusion 2b: Capacity BuildingOverall individual training targets at the programme level were exceededsuggesting successful implementation of DEPP capacity building activities.However,thesedataincludedindividualswhoparticipatedincapacitybuildingeffortswithminimal exposure such aswebinars and forums.Accessibility totrainingremaineduneven.Thetargetofincluding50%womentraineesacrosstheprogrammewasnotmet,andsometrainingswereonlyavailableinEnglish.Structuralbarriers,suchasorganisationalandmanagementarrangements,were


akeybarriertoapplyingandintegratingknowledgegainedfromDEPPcapacitybuildingeffortsat theorganisational levelandshouldbeaddressed in futureprogrammes. Multi-pronged capacity building efforts that include practicalapproachestosustainandapplyknowledgewereperceivedtobemosteffective,butquantitativedatashownoevidenceofknowledgechange.Organisational-level capacity building efforts where several strategies were combined andtailoredtothespecificgapsofeachorganisationwereperceivedtobeeffective,and evidence of increased organisational capacity of local organisationswasdemonstratedinanumberofsettings.

Conclusion 2c: Evidence Generation and LearningGenerationoflearningandevidencewashighintermsofquantity,butevidencewasmainlyanecdotalandlackinginscientificrigor.Arguablytoomuchemphasiswasplacedongenerationoflearningproductscomparedtoroutinemonitoring.Differentialreportingbyprojectsagainsttheprogramme-levellogicalframework(once itwas finalised and fully implemented inAugust 2017) occurred, andtherewasevidenceofweakqualityassurance.Inaddition,therewasalackofdisaggregateddatacollectedwithrespect to importantsubgroups.Sharingoflearningdidoccurbut there is noevidence that this translated tobehaviourchange.learningeventswereausefulapproachtosharelearning,howeveranincreasedfocusonthedisseminationofevidenceattheseeventswouldhavebeenvaluable.ThelearningPlatformwasanimportantresource(thatnowservesaspartof theDEPP’s institutionalmemory)but itwasnotusedconsistentlyacrossprojectsandfeaturedvaryingamountsofevidenceandlearningsharedbyeachproject.Suchplatformsshouldbemoreheavilypromotedtoincreaseuse by in-country partners, represent resources from all projects and ensureavailabilityofresourcesinrelevantlanguages.TheDEPPlearningProjecthadbeenoriginallyconceptualised to takeon theM&Efunctionsof theDEPP inadditiontolearning.Whileauniqueidea,thismodeldidnotworkwell–M&Efunctionsandresponsibilitywouldhavebeenbetterplacedattheprogrammemanagementlevel.Placingthesefunctionsasoneprojectalongsidetheotherswithlimitedauthorityledtodiffusionofaccountabilityandresponsibility.Asastandalonelearningproject,ithashadmixedresults,butcouldbere-envisionedforfutureprogrammes.

Conclusion 2d: Early Warning SystemsEarly warning systems are important as they provide (in theory) accurate,predictive and timely data to support emergency preparedness. However,findingswithrespecttoimplementationofearlywarningsystemsaremixed.Sixdiverseearlywarningsystemsweredevelopedandfivewerefullyoperationalatthetimeofthisreport.However,becauseofimplementationdelays,thelengthoftimethesesystemswerefunctionalwasverylimited.Severalofthesystemshavebeenwellintegratedintoappropriatestructurescontributingtoincreasedlikelihood of sustainability and impact. Only two systems (UEWEA, PHEPGambella)havebeenusedinanemergencyresponse.UEWEAinparticularwasshowntobeaneffectivesystemthatwasabletodetectseveralemergenciesthatwouldhaveotherwisegoneunnoticedininformalsettlementsaroundnairobiandthiscontributedtoimprovedtimelyandeffectiveresponse.Thissystemhadbeenoperationalthelongest,andalsoestablishedaconcreteexitstrategy,withthelocalcountygovernmenttakingovermanagementofthesystem.Thisprojectbenefittedfromalonger“effective”timelinecomparedtotheotherprojectsas


ittookforwardworkfromapreviousproject181thatconceptualisedthesystem,developedandtestedtheindicatorsandestablishedgovernmentcollaborations.Withoutthatexistingworktobuildon,UEWEAwouldlikelyhavefacedsimilarchallengesastherestofthecohortintermsofestablishingafunctionalsystemwiththeprogrammeperiod.

Conclusion 2e: Emergency Preparedness and ResponseThere are some emerging examples in several countries of perceivedimprovementsinemergencypreparednessattheorganisationalandcommunitylevel.DocumentreviewhasdemonstratedthattheDEPPhascontributedtoatleast42responsesin11countriesandthereareseveralself-reportedexamplesofstrengthened,morelocalisedresponse.

RECOMMENDATION 2.1 [DEPP PROGRAMME BOARD] The Learning Project and its placement should be re-examined, with its role, responsibilities, objectives and accountability mechanisms clearly defined at the outset of any future programme. Programme-level M&E should sit within programme-level management, but the casecould bemade to test a differently structured standalone learning project in a futureprogramme.furtherconsiderationstoensureabalancebetweenquantityandqualityoflearningandevidencegenerated,includingtheburdenandtimerequirementsplacedonprojects,andthetimingofM&Eactivitiestobeginbeforeprojectsstartimplementingareneeded.

RECOMMENDATION 2.2 [DEPP PROGRAMME BOARD] Project-level M&E systems should directly feed into programme-level systems (via a common, core set of indicators), and should be in place prior to implementation with timely reporting to permit agile course correction.StrongerM&Esystems includingcollectionofempiricaldatawouldsupport projects in accessing additional funding andyield better quality projectswitha greater likelihood of impact. Accountability mechanisms, including accountability tobeneficiaries,shouldbestrengthened.

RECOMMENDATION 2.3 [DEPP PROGRAMME BOARD] To increase the effectiveness of capacity building interventions, emphasis should be placed on quality not on quantity.Attheindividuallevel,approachesthatbuildskillsandreinforcelearning,andcombineseveralstrategies,shouldbeprioritised.Attheorganisationallevel,approachesthataretailoredtoidentifiedgapsandaimtostrengthensystemsandprocessesshouldbe emphasised. flexible funding mechanisms providing small or in some cases micrograntsforcapacitybuildingwereonesuccessfulapproachtoempowerlocalorganisationstobuild their capacity and shouldbe considered in futureprogrammes. Systems-levelcapacitystrengtheningshouldalsobefurtherpursued.furtherattentiontothebalanceofactivitiesacrossthesedifferentlevelsandthedesiredeffectateachlevelwillbeneededto maximise impacts. Structural barriers need to be addressed, and capacity buildinginterventionsshouldbebettertailoredtospecificcontexts.

RECOMMENDATION 2.4 [DEPP PROGRAMME BOARD] Development of early warning systems should be taken forward within future programmes but only when sufficient implementation periods are available.Athree-yeartimeframewasinsufficienttodesignandoperationaliseanearlywarningsystem;morerealistictimelinesareneededtoensurecompletionofthesystemandallowingtimetotestthesystemwithinemergencyresponses (or simulations). community members and government officials should be

181 IndicatorDevelopmentforSurveillanceofUrbanEmergencies(IDSUE).UrbanEarlyWarningEarlyActionfinalEvaluationReport,p1.


activelyinvolvedindesigningthesurveillancesystems,beneficiarycriteria,andresponsepackagedetails.consensusonindicatorsandearlywarningsystemsshouldbereachedin collaboration with community and government stakeholders. Early and sustainedengagementshouldbeemphasisedtofostertrustandstrengthenpartnerships.

RECOMMENDATION 2.5 [DEPP PROGRAMME BOARD] Future programmes must consider emergency preparedness and response on a continuum with more attention paid to the response component. The programme should strategise on how best toenableimprovedresponsethroughemergencypreparednessactivitiesandensuresuchmechanisms are embedded within the programme, rather than expecting responsecontributiontooccurorganicallyasanaturalby-productofpreparednessactivities.Thiscouldtakevariousforms.flexibleresponsefundscouldbebuiltintotheprogramme(andpotentially the individual projects) to provide a rapidmechanism for local programmebeneficiariestoimplementresponseactivities.Asthisapproachattheprojectlevelhasworkedwell inseveralcontexts itshouldbeviewedasapromisingstrategyworthyofreplicationinmoresettingsandtestedasaprogramme-levelcomponent.Alternatively,orinaddition,moreformallinkswithexistingemergencyresponsefunds(suchastheSTARTfund)couldbepursued.Inaddition,furtherthinkingaroundemergencypreparednessinconflict-affectedsettingsandinurbancontextsisneeded.

RECOMMENDATION 2.6 [DEPP PROGRAMME BOARD] To leverage the global momentum around localisation of humanitarian response and build on the contribution of DEPP in advancing this agenda, more nuanced considerations on localisation are needed. This includes articulating a clear definitionofwhat localisationmeans,whichentitiesareconsidered localandwhat the implicationsof thesedefinitionsare fromapractical standpoint. Strategically, laying out and potentially testing approacheswhichcouldenableInGOstotakeamoresupportiverole,andbetteroperationaliselocalisation(movefromtheoreticaltoactual).Organisationalpoliciesaroundlocalisationcouldbeonetarget,butunderstandinghowtooperationalisethepoliciesintoactualprocesseswouldbean importantelement.Additionally, thehumanitarian landscape indifferentsettingsand its relationship to localisation need to be considered to ensure that appropriate,contextuallyrelevantstrategiestosupportlocalisationareimplemented.finally,localisationfromthedonorperspectiveshouldbefurtherexploredtounderstandstrategieswhichmightenabledonorstomoreeasilysupportlocalcounterpartswithinthecontextofriskaversionandduediligenceprocesses.

3. EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS CONCLUSION 3: collaborationishighlyvaluedandDEPP’sconsortiaapproachwasuniversallydeemedtobethemostappropriate,preferreddeliverymechanismforhumanitariancapacitybuildingandemergencypreparednessandresponseactivities.Therewassomeevidencethattheconsortiamodelenabledcoordinated,timelierresponseinsomesettings.Globally,effectivenessof consortia couldbe increased in futureprogrammes througha smaller,more strategic setof consortia members, improved communication, opportunities for better coordination ofactivities, and provision of sufficient resources for strategic and organised collaborations.Inclusionofmorel/nnGOswithintheconsortiacouldcontributetothelocalisationagenda.network analysesdemonstratedevidenceof strengthenedhumanitarian responsenetworksin thePhilippines,KenyaandEthiopia,while thenetwork inMyanmarremainedunchanged.Arangeofdifferenttypesofemergencypreparednessandresponsenetworksexist inthesecountries, from highly isolated (Myanmar) to highly distributed, locally led (The Philippines).Inanyfutureprogrammeaimingtostrengthennetworks,thesedifferencesinthesize,scopeand priorities of networks across countries should be appropriately considered during the


bothdesignandimplementation.Humanitarianlandscapesandexistingnetworksacrossfocalcountriesshouldbeconsidered,particularlyinthedesignoftheprojectsandattheprogrammeleveltoleverageexistingcollaborationsandlocalpartnerships.Targetingofnetworkmembersforcapacitybuildingandevidencesharingshouldbemorestrategicandincluderelevantkeyinfluencers.

RECOMMENDATION 3.1 [DEPP PROGRAMME BOARD] In future programmes, refine and optimise the consortium model building on lessons from the DEPP, ensuring context-specific considerations. Workingeffectivelywithcollaborativestructuresinthefuturewillrequirebetterunderstandingofwhattypesofconsortiaareappropriateandeffectivewithin different contexts.Overarching factorswhich should be incorporatedinprogrammesof thisnature includesupportingcollaborativestructures thatbuildonexistingsuccessfulrelationshipsandthatworkwithasmallernumberofmembers,andthatworktowardsstrategiccollaborations.Toincreasethespeedofdecision-makingandeffectiveness of consortia, streamlined contracting and reporting processes should beestablished.Ensuringl/nnGOsareincludedinconsortia,andassessmentofstrategiestoaddressriskaversionamongdonorstosupportingsuchconsortiaareneeded.

RECOMMENDATION 3.2 [DEPP PROGRAMME BOARD] Network assessments could better inform targeted strategies to enhance collaboration and to effectively strengthen emergency preparedness and response networks. country-specificnetworkmappingandassessmentsshouldbeundertaken,whenappropriate,tofacilitatetheuseofmoretargetedapproachestobuildingnetworksandpartnerships,suchasthroughtheidentificationandtargetingofkeyinfluencers.Deeperexplorationonthelevelofinfluenceandroleofdifferentorganisations(i.e.,asresourcehubs,knowledgebrokersetc.)withinthenetworkinfocalcountrieswouldbeextremelyvaluabletoinformstrategiesaroundwhichorganisationstotargetandhow.Theseassessmentscouldalsoenableoptimisationofcollaborationwithlocalandnationalactors.forexample,localisationeffortscouldberefinedinthePhilippineswherenationalnGOsarealreadyverystronglyinvolvedwithinemergencypreparednessandresponsenetworks.

4. EFFICIENCY AND VALUE FOR MONEY (VFM) CONCLUSION 4: Goodindicatorsofcosteconomywereevident,butinsomecasesprojectbudgetsweretooleanwithinsufficientresourcesforconsortiamanagementandcollaborativeactivities.Attheportfoliolevel,resourcesdedicatedtoportfoliomanagementandcollaboration,M&EandtonPAcswereinsufficientwithrespecttodesiredprogrammeobjectives.Infutureprogrammes,morestrategicallocationoffundsshouldbeundertaken.Inaddition,systemsandresources for governance, strategy, portfoliomanagement, decision-making and consortiumarrangementsshouldberestructuredtoimproveefficiency.Adaptivemanagementprocessespermittedcoursecorrectionand revisionsofprojectplans to reflectchangingcontexts, andVfMreportingimprovedoverthecourseoftheprogramme.futureprogrammesshouldbetteradoptandstandardiseVfM indicators inorder tocollectdata for internalprojectpurposes,cross-projectcomparisonsandgenerallearning.TherewerenostrongVfMfindingstosuggestthat emergency preparedness improved the efficiency of humanitarian responsewithin theprogrammeduration.

RECOMMENDATION 4.1 [DEPP PROGRAMME BOARD] Streamline contractual processes, management decisions and flow of funds. Whilerecognisingtheimportanceofsubcontractingprocessesandorganisations’internalprocedures,futureprogrammingshouldtakestepstominimiseandstreamlinecontractualprocesses.forexample,pre-agreementlettersofcommitmentcouldhelpclarifysomecontractualissuespriortothestartofprojects.Theprogrammemanagement couldalsoprovidea template for sub-


contracts and consider settinga standardtimeline for contract turnaround. If projectsare structuredwith a global grant holder and in-country host or partner, streamliningmanagementdecisionsandflowoffundssothatthesameorganisationservesinbothroleswheneverpossiblecouldbeconsidered.

RECOMMENDATION 4.2 [DEPP PROGRAMME BOARD AND PROJECTS] Adopt and standardise VFM indicators at programme and project levels and implement routine reporting.DfIDguidanceonVfMinhumanitarianprogrammesorotherrelevantVfMguidanceshouldbeusedtoenhanceVfMmonitoringandreportinginfutureprogrammes.AdoptingVfMindicatorsattheprogrammelevel,aswellastheprojectlevelwillensurethatVfMcanbecloselytrackedandanalysedatbothlevels.

RECOMMENDATION 4.3 [DEPP PROGRAMME BOARD] In a future programme, stra-tegically allocate funds to achieve programme objectives, with greater funds desig-nated for programme management costs, M&E and NPACs. foralargeportfolio,pro-grammemanagementcostsshouldbeintherangeofaround10%,toenablesufficientresourcestoundertakeitskeyfunctionsofmanagement,M&Eandstrategictasks.nPAcscanbeupto20%andashighas25%,butshouldbeconsideredrealisticallytoensurethatin-countryandbackofficecostsarecovered.WithrespecttoM&E,ensuresufficientresourcesareavailableattheboththeprogrammeandprojectlevelstoefficientlyimple-mentbothroutinemonitoringandmorerigorousevaluations.

5. SUSTAINABILITY OF THE INTERVENTION AND LIKELIHOODOF IMPACT OF THE PROGRAMME CONCLUSION 5: Insomesettings,DEPPhascontributedtostrengtheningnationalemergen-cypreparednesssystemsbut typicallyonasmallgeographicscale,andby individualprojectconsortia.TherehavebeenseveralconcreteexampleswhereDEPPhasimpactedgovernmentpoliciesorsystemsor increasedpoliticalcommitment.Stakeholdersnotedashift inthewayorganisationsconsider,addressandinvolvelocalactorsandcommunities,andattributeditasoneofDEPP’smostsignificantcontributions.However,therewasalackofadetailedstrategyfortheinclusionofgenderandprioritisedtargetgroupsattheprogrammelevelresultinginafragmentedprogrammewideapproachandunevenresults. Inaddition,cross-cuttingthemessuchasaddressingVAWGwerenotadequately integrated intotheprogramme.Ratherthanonlyincludingoneortwoprojectswithaninclusionandprotectionfocuswithinaportfolio,amoreintegratedapproachinvolvingmainstreamingacrosstheprogrammeshouldbeprioritised.Thisshouldentaildisseminationofdetailedexpectationsandguidelinesoninclusionofgenderandother prioritised groups and reporting requirements during thedesignphase to ensurethatprojectsaredevelopedaccordingly.Aswell,consistencyinimplementationofgenderandinclusion considerationswith regularmonitoring shouldbeestablished. In addition to inclu-sion,sustainabilityplanningwasweakandwasnotclearlyplannedordocumentedatboththeprogrammeandprojectlevels.Thisfailureseemstohavebeenlinkedtoexpectationsaroundthepossibilityof receiving additional funds fromDfID for a secondphaseof theDEPP, asarticulatedinthebusinesscase,combinedwithlackofguidancefromtheprogrammeonsus-tainability.Inanyfutureprogramme,moredeliberate,strategicsustainabilityplanningisneededatall levels,beginningduringthedesignphase,andwithtransparencyaroundthepossibilityoffurtherfunding.Attheprogrammelevel,thereissomeevidencethatbenefitshavebecomeembedded–withexamplesofsystemandpolicychangeaswellasstrengthenednationalpre-parednesssystems,albeitonasmallgeographicscale.However,thepotentialforlonger-termeffectsisdependentontheextenttowhichdifferentcomponentsareabletocontinuebeyondthecloseoftheprogramme.futureprogrammesshouldprioritiselongerimplementationperi-


odstoincreaselikelihoodofimpact,andincorporategovernmentcollaboration,systems-levelchange,policyoradvocacycomponents,involvementofbeneficiariesandgoodVfMinordertopromotesustainability.

RECOMMENDATION 5.1 [DEPP PROGRAMME BOARD AND PROJECTS] Develop a strategy for inclusion of gender and prioritised target groups at programme level, ensure projects adopt an approach which aligns with this strategy and monitor im-plementation at all levels. In thegovernancecriteria, considerbroadening thegenderstatementtoastatementoninclusionofprioritisedtargetgroups.Developguidelinesforprojectsonrequirementsforthisstatement.Thiscouldincludelinkstokeydocumentsonbestpracticesandthelawandpoliciesgoverninginclusionofprioritisedtargetgroupsindevelopmentprogrammingandprojectdesign.Ensuregenderconsiderationsareconsis-tentlyimplementedandreported.

RECOMMENDATION 5.2: [DEPP PROGRAMME AND PROJECTS] Within any future programme, integrate deliberate, strategic sustainability planning that takes a more holistic view on sustainability going beyond simply securing funding streams. Pro-grammeandprojectsustainabilityplansshouldbedevelopedasearlyaspossible,prefer-ablyinthedesignphaseandupdatedregularly.However,forriskierprojectswithahighpossibilityoffailure,sustainabilityplanningdoesnotmakesenseandisnotagooduseofresourcesuntilresultscanbedemonstrated.futureprogrammesshouldincreasethelikelihoodofsustainabilityandlonger-termimpactsbybuildingonpreviouseffortsandexistingpartnerships/consortia,workingwithgovernments(whenappropriate),incorpo-ratingapolicyoradvocacyelement,developingsustainableoutputssuchastools,guide-linesorsystems,increasingcross-projectlinkages,fosteringsustainablerelationshipsthatcancontinuebeyond theDEPP, improvingbeneficiaryengagement,andstrengtheningfeedbackmechanismsforthecommunity.GoodVfMshouldalsobeprioritisedtomaxi-misesustainability.Programmevisibilityatthecountrylevelshouldbemaximisedthroughincreasedadvocacyinitiativesandthedevelopmentofstreamlinedcommunicationwithexternalstakeholders.

227 SUMMATIVEPHASEREPORT-cOnclUSIOnSAnDREcOMMEnDATIOnS FORMATIVE PHASE REPORT 227