THINKINGEVALUATIVELYIN

PEACEBUILDINGDESIGN,

IMPLEMENTATIONANDMONITORING:

ThreeReflectingonPeacePractice(RPP)andDoNoHarm(DNH)-infusedoptionstostrengthentheeffectivenessofpeacebuildingstrategiesandprograms

DevelopedbyAnitaErnstorfer,IsabellaJean,andPeterWoodrow

ofCDACollaborativeLearningProjects,withinputsfromDianaChigasofTuftUniversity

November2016

Thisreportwasmadepossibleby

theCarnegieCorporationofNewYork

ResourceGuide

ThisresourcehasbeendevelopedbyCDACollaborativeLearningProjectsinpartnershipwiththePeacebuildingEvaluationConsortium(PEC)andwithfundingfromtheCarnegieCorporationofNewYork.

Particular thanks go to Cynthia ClappWincek (Alliance for Peacebuilding), LeslieWingender(Mercy Corps), Mark M. Rogers (independent consultant), and Kiely Barnard-Webster (CDACollaborativeLearningProjects)fortheirreview,andhelpfulsuggestions.

ThePeacebuildingEvaluationConsortium(PEC)

The Peacebuilding Evaluation Consortium (PEC) is a project of Alliance for Peacebuilding (AfP) inpartnership with CDA Collaborative Learning Projects, Mercy Corps and Search for Common Ground(SFCG).TheprojectisfundedbytheCarnegieCorporationofNewYork(CCNY)andisfield-wideefforttoaddresstheuniquechallengestomeasuringandlearningfrompeacebuildingprograms.ThePECconvenesdonors, scholars, policymakers, local and international practitioners, and evaluation experts in anunprecedentedopendialogue,exchange,andjointlearning.Itseekstoaddresstherootcausesofweakevaluationpracticesanddisincentivesforbetter learningbyfosteringfield-widechangethroughthreestrategic and reinforcing initiatives: 1) DevelopingMethodological Rigor; 2) Improving the Culture ofEvaluationandSharedLearning;and3)FosteringtheUseofEvidencetoInformPeacebuildingPolicy.

CDACollaborativeLearningProjects

CDA (www.cdacollaborative.org) is a US based non-profit organization committed to improving theeffectiveness of thosewhowork internationally to provide humanitarian assistance, engage in peacepractice,supportsustainabledevelopment,andconductcorporateoperations inasocially responsiblemanner.Ourdonorsandpartners supportCDAbecausewecombine rigorousanalysiswithpragmaticfield-levelworkanddeliverpracticaltoolstofieldstaffandpolicymakersalike.

CDA is known as a leader in peacebuilding effectiveness work. We help peace practitioners, andorganizationsimprovetherelevanceandaccountabilityofprogrammingthroughbettertoolsforconflictanalysis,programstrategy,design,andmonitoringandevaluation.CDAhasalsocontributedtoinfluentialpolicy guidance, suchas theOECD/DACguidanceonevaluating conflict preventionandpeacebuildingactivities.

TABLEOFCONTENTS

1.Introduction................................................................................................................................................11.1WhatisthisGuide—andwhatisitnot?.............................................................................................................11.2WhoshouldusetheGuide?................................................................................................................................11.3OriginsoftheGuide............................................................................................................................................21.4WhyistheOptionsGuideneeded–whyevaluativethinkinginpeacebuilding.................................................31.5WhenisthisGuideuseful?.................................................................................................................................6

2.EmergingCriteriaandProgrammingConceptsinPeacebuilding............................................................72.1BackgroundRPPandDNH...................................................................................................................................72.2KeylessonsfromCDA’sReflectingonPeacePracticeProgram(RPP)................................................................82.3KeylessonsfromCDA’sDoNoHarmProgram(DNH).......................................................................................14

3.Whatistherightoptionformyteam?Whatarethebasicsweneedtohaveinplacetobenefitfrom

anyoftheseoptions?....................................................................................................................................16

4.ProgramQualityAssessment(PQA)–criteriaforassessingpeacebuildingprogramquality............194.1Overview:PurposeofaProgramQualityAssessment......................................................................................194.2KeyelementsofaprogramqualityassessmentbasedonRPPcriteria............................................................19

5.EvaluabilityAssessment(EA)–areyoureadyforanevaluation?.........................................................275.1Overview–PurposeofanEvaluabilityAssessment..........................................................................................275.2KeyelementsofanevaluabilityassessmentbasedonRPPcriteria:.................................................................28

6.StrategyandProgramReflectionExercise...............................................................................................356.1Overview–PurposeofaStrategy/ProgramReflectionExercise......................................................................356.2KeystepsoftheStrategyandProgramReflectionExercisebasedonRPPcriteria..........................................35

7.AdaptivePeacebuildingProgrammingthroughEffectiveFeedbackLoops..........................................41

8.SummaryOverviewofOptionscoveredinthisGuide............................................................................43

9.Recommendationsforpractice................................................................................................................46

10.Bibliography.............................................................................................................................................50

11.Annexes:ToolsandHand-outs...............................................................................................................53Annex1:FiveBuildingBlockstowardsPeace.........................................................................................................53Annex2.1:RPPMatrix............................................................................................................................................54Annex2.2:RPPMatrix‘Plus’...................................................................................................................................55Annex3:RPPProgramPlanningChart...................................................................................................................56Annex4:DoNoHarmActionPatterns(ResourceTransferPatterns).....................................................................57Annex5:ToolforPeacebuildingQualityAssessment............................................................................................23Annex6:ChecklistforEvaluabilityAssessmentofPeacebuildingPrograms.........................................................29Annex7:CDACollaborativeLearningProcessMethodologyandSteps.................................................................33

GLOSSARY

AfP - AllianceforPeacebuilding

CDA - CDACollaborativeLearningProjects

DNH - DoNoHarm

DM&E - Design,Monitoring,andEvaluation

EA - EvaluabilityAssessment

FCAS - FragileandConflictAffectedStates

M&E - MonitoringandEvaluation

OECD - OrganizationforEconomicCooperationandDevelopment

PEC - PeacebuildingEvaluationConsortium

PQA - ProgramQualityAssessment

pwl - peacewritlittle

PWL - PeaceWritLarge

RPP - ReflectingonPeacePracticeProgram(CDA)

TOC - TheoryofChange

UN - UnitedNations

USIP - UnitedStatesInstituteofPeace

1

1.Introduction

1.1WhatisthisGuide—andwhatisitnot?

TheGuideputsforwardthreeoptions:(i)ProgramQualityAssessments, (ii)EvaluabilityAssessments,and(iii)StrategyandProgramReflectionExercisesusingfindingsandlessonsfromCDA’sReflectingonPeacePractice(RPP)andDoNoHarm(DNH)Programsascriteriaforeffectiveandrelevantpeacebuildingengagement.

ThisisnotaGuideonhowtoevaluatepeacebuildingefforts…itisabouthowtoapplyEvaluativeThinkingtopeacebuildingstrategiesandprogramming.

This resource responds to theneed forclearerevaluative thinkingandpractice duringpeacebuildingstrategyandprogramdesign,andimplementation,andstrongermonitoringmechanismstoimprovetheevaluability of peacebuilding initiatives. It providesconcrete guidance for practitioners on how toimplement different ‘evaluative’ options – short offormalevaluations.

ThisGuide does not provide guidance on planning orconducting an evaluation. Nor does it provide acomprehensive overview of other evaluationapproachesavailabletoaddressthequestionsraisedintheprocessesoutlinedbytheGuide.

TheGuide can be used during peacebuilding strategyandprogramdesign,duringprogramplanningandset-up,duringimplementation,andduringthepreparationfor a formal evaluation. It is also relevant forengagements that are ending, instead of orcomplementary to other assessment processes or aformalevaluation.

1.2WhoshouldusetheGuide?

ThisGuideisintendedforusebydifferentaudiences:

§ Peacebuildingpractitionersdesigningandimplementingprograms;

ThisGuideprovidestoolsandguidanceforintegratinggreaterEVALUATIVETHINKINGintodesign,implementationandmonitoringofPEACEBUILDINGinitiatives,toenhanceprogramquality,ensurethattheymaximizetheirpotentialforimpactandarepreparedforaformalevaluationwhenitisnecessary.

Evaluative Thinking | ongoing process ofquestioning,reflecting,learning,andmodifying.

“Critical thinking applied in the context ofevaluation, motivated by an attitude ofinquisitiveness and a belief in the value ofevidence, that involves identifying assumptions,posing thoughtful questions, pursuing deeperunderstandingthroughreflectionandperspectivetaking,andinformingdecisionsinpreparationforaction.”aea365.org/blog/tag/evaluative-thinking

Peacebuilding|thisGuideusesthefollowingtwointerpretationsof‘Peacebuilding’:

(1) “Direct work that intentionally focuses onaddressing the factors driving and mitigatingconflict”;and

(2)“Effortstocoordinateacomprehensive,multi-leveled, multi-sectoral strategy, includingdevelopment, humanitarian assistance,governance, security, justice and other sectorsthat may not use the term peacebuilding todescribethemselves.”

(Schirch2013,Glossary)

2

§ Peacebuildingmanagerswhosuperviseothersonprogramdesign,implementationandMonitoringandEvaluation(M&E),andwhoareinvolvedinstrategicprogrammingandfundingdecisions;

§ M&Eprofessionalsoperatinginconflict-affectedcontexts,andworkingonpeacebuildinginitiatives;§ Commissionersormanagersofevaluations;§ Evaluatorswillingtoexperimentwithalternativestoformalevaluationprocesses.

1.3OriginsoftheGuide

ProgramQuality Assessments (PQAs), Evaluability Assessments (EAs) and strategy evaluation1 alreadyexistinevaluationpractice,andhavebeenusedwidelyinnon-peacebuildingfields.Aspartofitsbroaderefforts to enhance program relevance and effectiveness within the peacebuilding field, CDA hasdevelopedaspecificapproachtoStrategyandProgramReflectionforpeacebuildingengagements,andadaptedPQAsandEAstothespecificneedsofpeacepractitioners.Forthispurpose,lessonsfromCDA’sReflectingonPeacePracticeandDoNoHarmProgramshavebeen integratedandusedascriteria foreffectivepeacebuildingwork.2 Sections4,5and6willdescribePQAs,EAs,andStrategyandProgramReflectionExercisesinmoredetail.

TheseprocessesweredevelopedfollowingrecommendationsofastudybyCheyanneScharbatke-Churchin2011,inwhichsheexploredhow“ReflectingonPeacePractice”(RPP)conceptsandtoolswerebeingusedinevaluation.3ShefoundthatRPPconceptswerebeingusedinmanypeacebuildingprograms,oftenimplicitly, as de-facto ‘standards’ for understanding peacebuilding results, as a frame of inquiry forevaluations,asameanstoassessrelevance,aswellastostrengthenprogramdesignduringthedesignandimplementationphase.4ThepartsofRPPthatseemtobeusedthemostweretheRPPmatrix,keydrivingfactorsofconflict,thebuildingblocksforpeace/criteriaofeffectiveness,linkages,determiningadd-uptoPeaceWritLarge,andsystemsapproachestoconflictanalysis.TheseapproacheswillbeexplainedinmoredetailinSections3-6.

The Guide will use lessons and findings from RPP as criteria for

assessing peacebuilding program quality, with the goal to worktowardsgreaterimpact.ThisalsoincludesinsightsfromCDA’sworkon systems approaches to conflict analysis, peacebuilding programdesign,andM&E.Inaddition,toalesscomprehensiveextentbutinacomplementaryfashion,insightsandlessonsfromCDA’sDoNoHarm(DNH)programandongoingworkonconflict-sensitivitywillalsobeusedasprinciplesforeffectiveprogramming.5

OnemightlegitimatelyaskwhywechosetouseRPPandDNHcriteriaandtools,ratherthanothernormsforpeacebuildingprogrammingandconflictsensitivity.Onthepeacebuildingside,wefindthattherearefew,ifany,widelyacceptednormsforprogramdesignandeffectivenessbeyondbroadgeneralities.AsScharbatke-Churchfound,manypolicymakersandpractitionerswerealreadysuccessfullyusingRPPasa

1See(PatriziandPatton2011),forexample,onstrategyevaluation2See(Reimann,ChigasandWoodrow2012)3Scharbatke-Church,TheUseofReflectingonPeacePractice(RPP)inPeacebuildingEvaluation.Review&Recommendations4TherecommendationfromScharbatke-Church’sreporttoCDAwasto(i)buildonexistingworkinRPPtosupportandstrengthenthequalityofpeacebuildinginterventiondesigninawaythatsetsthestageformoreeffectiveevaluation,and(ii)forCDAtouseitslessonsfromRPPtocontributetotheunderstandingofhowtoapplysomeoftheOECD-DACevaluationcriteriaforconflictpreventionandpeacebuildingwork.5Itisoften–wrongly–assumedthatpeacebuildingworkisautomaticallyconflict-sensitive.Lessonsfrompracticeshowthatthisisnotnecessarilythecaseandthatpeacebuildingprogrammingneedstoapplythesamerigorofconflict-sensitivityassessmentas,forexample,developmentandhumanitarianprogramming.Atthesametime,thereisalsoasignificantlevelofconfusionbetween‘conflict-sensitiveprogramming’and‘peacebuildingprogramming’.Ausefulresourceinthisregardis(WoodrowandChigas2009)

Moreinformationaboutpeacebuildingevaluationandwhichevaluationoptions:

® dmeforpeace.org/introduction-to-the-field-Guide

® dmeforpeace.org/evaluation-planning

*ThePECOnlineFieldGuidetoPeacebuildingEvaluationonDM&E

3

setofreasonablenorms—sowecouldbuildonandenhancethatapplication.And,whilethereareothermodels for conflict sensitivity (such as PCIA – Peace and Conflict Impact Assessment), they do notcontradictDNH.

Section2ofthisGuidewillintroducethemainapproachesandtoolsofbothRPPandDNHasafoundationtounderstandhowRPPandDNHapproachesandlessonshavebeenintegratedintothethreeoptionspresentedinthisresource.

TheGuidebringstogetherCDA’sexperienceworkingwithvariousprogramteamsandprogrampartnersontheapplicationofRPP,andDNHtostrengthenpeacebuildingstrategyandprogrammingwithaviewto enhance relevance, effectiveness, and conflict-sensitive programming. TheGuide alsobuilds on findings from two RPP and DNH infused Program Quality Assessments and Evaluability Assessmentsconductedin2013and2014withfourpartnerorganizations.6Inaddition,thecollectiveexperienceofthePeacebuildingEvaluationConsortiumhasbeenleveragedtoproducethisGuide.

1.4WhyistheOptionsGuideneeded–whyevaluativethinkinginpeacebuilding

PeacebuildingandEvaluation–wheredowestand?

Thepeacebuildingfieldhasbeenstrugglingtoagreeonandadheretouniversallyrecognizedprinciplesand standards for quality and accountability (e.g., for conflict analysis, theories of change, clearlyarticulated goals that are relevant from a peacebuilding perspective). In other development orhumanitariansectorstherearestandardssuchastheSPHEREstandardsinthehumanitariansectorortheINEEminimumstandards for education in emergencies. Likewise, theevaluation communityoperateswith clear standards and principles.7On the other hand, definitions ofwhat constitutes relevant andeffectivepeacebuildingvariesgreatlyacrossorganizations.

Atthesametime,overthepasttenyears,thefieldofpeacebuildingevaluationhassignificantlymatured,anddonorshaveincreasedtheirpressuretoshowconcreteresults.ArangeofGuidelines,frameworksandtoolkitshavebeendevelopedbypeacebuildingandevaluationorganizationsandpractitioners.TheOECDDevelopmentAssistanceCommittee(DAC)Guidelinesontheevaluationofconflictpreventionandpeacebuildingactivities(OECD/DAC20128)arenowonesetofstandardsinthefield.CDAprovidedkeycontributionstotheOECD/DACGuidelines,basedonfindingsfrombothitsReflectingonPeacePractice(RPP)aswellasDoNoHarm(DNH)collaborativelearningprograms.

Increasingly, development and peacebuilding organizations are making a conscious effort toinstitutionalizepeacebuildingprogramdesignandrelatedmonitoringandevaluation(M&E)approachesintheirwork.Thisincludesprominentinternationalnon-governmentalorganizationssuchasSearchforCommonGround,9MercyCorps,CARE,Saferworld,andCatholicReliefServices.Multilateralorganizations

6ThosewereaPQAfacilitatedwithInternationalAlertintheSouthCaucasus(facilitator:IsabellaJean/CDA),aPQAfacilitatedwithInterpeaceinMali(facilitator:AnitaErnstorfer/CDA),anEAfacilitatedwithNorwegianChurchAidinAfghanistan(facilitator:MarkRogers/independentconsultant),andanEAfacilitatedwithWorldVisioninSriLanka(facilitator:CordulaReimann/CoreConsulting&Training).7See,forexample,“AmericanEvaluationAssociationGuidingPrinciplesForEvaluators,”AmericanEvaluationAssociation,http://www.eval.org/p/cm/ld/fid=518InadditiontotheOECD/DACevaluationcriteriaforallfields(relevance,effectiveness,efficiency,impact,sustainability,andcoherence),(OECD/DAC2012)recommendsthreeoptionalcriteriatobeusedforpeacebuilding,specifically:linkages,coverageandconsistencywithvalues/conflictsensitivity.9See“DM&EforPeacePortal,”SearchForCommonGround(SFCG),http://www.dmeforpeace.org/

4

havealsomadeeffortstobemorerigorousaboutstrategydesignandM&E,suchastheUNPeacebuildingFund.

Despiteallthepositivedevelopmentsmentionedabove,thepeacebuildingfieldhasbeenstrugglingtoapplyevaluationsasasystematicprofessionalpractice.In2011,areportbytheAllianceforPeacebuilding(AfP)notedthat“[...]thepeacebuildingfieldseemstohavereachedafrustratinglylongplateauintheuse,understanding,andapplicationofevaluation.Asaresult,mostpeacebuildingfundersandimplementersexpressdissatisfactionat thecurrent stateofevaluation.”10Progressandpractical learninghavebeenslow.Thereisstillnowidelyacceptedmethodologicalagreementabouthowbesttoconductevaluationsincomplexandconflict-affectedcontexts(Paffenholz2011).11

Four challenges are worth highlighting in this regard:

1.Thefirstchallengeishowtomeasuretheeffectsofmicro-levelinterventionsonmacro-levelconflict

dynamics.Thisrefersnotonlytothechallengesofattribution,butalsotothehighlycomplexnon-linearprocessesofsocialchangewhichcannotbecapturedbylinearcause-effectlogic(e.g.,asmanifestedinsomeofthestandardlogicalframeworksusedinM&Esystems).Forthatreason,systemsapproachestopeacebuilding and peacebuilding evaluation are capturing increased attention, but how to use thempracticallyandsystematicallyfromamonitoringandevaluationperspectiveisyetunderexplored.

2.Thesecondchallengeisthatmanypeacebuildinginitiativesarenotbasedonclearlydefinedstrategy

and design principles or criteria, with either limited or no conflict analysis, unarticulated theories ofchange,or ‘fuzzy’peacebuildinggoals.Thismakesevaluationextremelychallengingandhighlights theneedtoapplyanevaluativeapproachfromtheinitialstagesofprogramdesign,throughimplementation,andM&E–notsimplyattheendofaproject.

3.Thethirdchallengeisthatthereisnotyetawell-established‘cultureof evaluation’within the peacebuilding field,which often results invery limited M&E systems and capacities. Many peacebuildingpractitionersdon’thaveexpertisewithM&E,andmanyM&Eexpertsdon’t have practical experience with peacebuilding. Manypeacebuildershaveresistedasystematicapplicationofrigorousandprofessionalresultsmeasurementtoolsandframeworkstotheirwork.Many peacebuilding practitioners claim that it is too difficult tomeasureimpact,giventhecomplexityoftheconflictfactorsatstake,highlydynamicandquicklychangingenvironments,andthelong-termnature of conflict transformation and peace efforts.12 Severalorganizationshavemadeprogress in strengtheningmore integratedDesign,M&E and Learning capacities; however, often capacities areconcentratedinheadquartersandarenotreplicatedatthesamelevelincountryandfieldoffices.Atthesametime,thecombinedpressuresof scarce funding, the requirement to demonstrate results and the10Kawano-Chiu,StartingontheSamePage:ALessonsReportfromthePeacebuildingEvaluationProject,811Oneofthemostrecentcontributionsinthisregardis(Andersen,Bull,andKennedy-Chouane2014).Seealso(CorlazzoliandWhite2011)12Thisisoneoffive‘mythsandmisconceptions’identifiedbytheAfPreport(Kawano-Chiu2011,9andfollowing.)Theotherfourare:“Staffincountryofficesmustbetrainedsocialscientists”;“Theprimarypurposeofevaluationsistohighlightflawsandfaultsandassesswhenaprogramisa‘success’or‘failure’”;“Theexpectationisthatnearlyallprojectswillbe‘successful”;and“Countervailingforcesagainstgoodevaluationpracticesaretooentrenchedtochange”.

Moreinformationaboutstandardsinevaluation:

® dmeforpeace.org/introduction-to-evaluationunder“ArethereQualityStandardsforEvaluations?”

Resourcesonevaluatingpeacebuildingprogramminginaparticularsector:

® dmeforpeace.org/evaluation,under“IamInterestedinEvaluatingPeacebuildingProgramminginaParticularSector.ArethereSpecialConsiderationsandTools?”

5

need to establish peacebuilding as a legitimate field have led to a tendency among peacebuildingprogramsto‘over-claim’results(achallengethatisnotuniquetothepeacebuildingfield).13

4.Commitmenttoactuponassessmentandevaluationresults,andestablishingacultureoflearningisafourthchallenge.Thisisclearlynotuniquetothepeacebuildingfield.Iftheobjectives,unit(s)ofanalysis,andpurposesofevaluationsarenotclearlyarticulated,itdecreasesthelikelihoodthatevaluationswillleadtoachangeinpracticeandcontributetotheuptakeoflearninginanorganization.Often,localstaffarelefttotheirowndeviceswithlittlecontinuingexternalsupporttoimplementtherecommendationsof evaluation reports.14 Most literature on (peacebuilding) evaluation is clear about the fact thatevaluation should never be “[...] an end in itself. It should be a mechanism that contributes toaccountability and learning at a variety of levels: project team, office, organization or peacebuildingfield.”15 Inpractice,however, thedonoraccountabilityaspectofevaluationshas receivedmuchmoreattentionthanthelearningandprogramqualityimprovementaspects.Frequently,evaluationprocessesarenotdesignedinawaythatsupportsaconsciousprocessoflearningandadaptation.

Whydoweneeddifferentevaluativeoptionstostrengthenpeacebuildingstrategy,designandimplementation?

In timeswhenthere is increasedrecognitionof theneedformoreadaptiveandflexibleprogrammingapproaches in highly complex and conflict-affected contexts (by donors and policymakers aswell asprogram partners and local stakeholders), the application of different evaluative options becomesincreasinglyrelevant.Theycanprovideafoundationfornimblerandmoreadaptabledecision-making,coursecorrectiononprogrammingdirections,anddifferentoptionsforengagement.

CDAhasfoundseveralissuesinthepeacebuildingfieldthattheuseoftheseapproachesaddresses:

§ Manypeacebuildingprogramsarenotreadyforformalevaluations—eitherbecausetheywerenotdesignedwith evaluation criteria inmind or because they are not prepared in otherways.Whiledonorsoftenimposearequirementthatprogramsperforman“evaluation”,itisnotalwaysclearwhatstandardtheyareapplyingandwhatthepurposeofsuchanevaluationwouldbe—norareadequatefunds provided for a full evaluation.Many donors (and implementing organizations)will bewell-served by other evaluative processes that meet more modest objectives for program qualityimprovement.

§ Conduciveenvironment:Insomesituations,theconditionsofconflict,violenceandinsecurityarenotconducive to robust evaluation per the highest accepted standards of the American EvaluationAssociationorotherinternationalbodies,suchastheOECD/DAC.Datamaynotbeavailable.Accesstocertainareasmaybelimited.Posingcertaintypesofkeyquestionsmayexacerbateconflictorputprogramstaffindanger.

§ Inmost cases, program teams also have significantM&E capacity development needswhich areusuallynotmetandcannotbemetbyevaluationsalone.

Basedontheabove,thepeacebuildingandevaluationcommunityhavebeenexploringarangeofnewapproaches to evaluation.16 This guidance builds on these developments to focus on guidance for

13Scharbatke-Church,PeacebuildingEvaluation:NotYetAllItCouldBe,BerghofHandbookforConflictTransformation.SectionII,476

14ActionAsia,ExaminingRPPasAToolforEvaluation-TheActionAsiaExperience,16

15Scharbatke-Church,PeacebuildingEvaluation:NotYetAllItCouldBe,BerghofHandbookforConflictTransformation.SectionII,471

16Theseinclude“mostsignificantchange”technique(DaviesandDart2005),developmentalevaluation(QuinnPatton2011),andoutcomemapping.

6

programsthatseektoreviewandstrengthenprogramqualityandstrategy, thatarenotyetreadyforevaluation,orwhereformalevaluationmaynotbeappropriateordesired.

1.5WhenisthisGuideuseful?

TheGuideisusefulforbothstrategyandprogramdesign:itcanbeusedtoreviewandassessmacro-levelpeacebuilding strategies, involving various projects and programs within one organization or acrossagencies,aswellasattheindividualprojectandprogramlevel.

Therefore,thisGuideisintendedtoberelevantforafairlybroadrangeofinitiativespursuingeitheroftheabove,includingshorter-term,medium-term,aswellaslonger-terminitiatives.

TheStagesofConflictFigure(CDA2016)impliesafairlysmoothsetofstagesthatmoveinexorablytowards“stablepeace.”Therealityisquitedifferent,asprocessesstartandstop,returntoearlierphases(suchasviolence),makesomeprogressandstall,andsoforth.

Peacebuildingprogramsmaybeorientedtowardsanyofthesestagesormayaccompanyapeaceprocessthroughseveralphases.

The threeoptionsoffered in thisGuideare intended tohelppractitionersanddecisionmakers locatethemselvesinarangeofpeaceeffortsandachievecontinuousimprovementineffectivenesstowardsthelargergoalofstablepeace—whatRPPcalls“PeaceWritLarge.”Therefore,theGuideisexpectedtobehelpfulinarangeofdifferentconflict-affectedcontextsandnotlimitedtoaspecific‘conflictphase’.

…theGuideisexpectedtobehelpful

inarangeofdifferentconflict-

affectedcontextsandnotlimitedtoa

specific‘conflictphase’.

Figure:StagesofConflict(CDACollaborativeLearningProjects2016)

ManifestConflict

Pre-ConflictArguments,disagreements,culturalandstructuralviolence

Post-ConflictPoliticalsettlement,peaceagreements

LatentConflictAggressivepositioning,targetedattacks,

culturalandstructuralviolence

PoliticalInstability,frequentphysicalviolence,'civilwar'

StablePeace

7

2.EmergingCriteriaandProgrammingConceptsinPeacebuilding

AsnoteveryreaderofthisresourcewillbefamiliarwiththecoreconceptsofRPPandDNH,thissectionprovidesabrief introductiontothemainconceptsandtools–thefoundationofwhyandhowweareusingRPPandDNHascriteriaforPQAs,EAs,andstrategyandprogramreflections.Foramorein-depthoverviewofeachoftheseapproaches,pleaserefertothebibliography.

2.1BackgroundRPPandDNH

ReflectingonPeacePractice

Launched in1999,CDA’sReflectingonPeacePracticeProgram (RPP)offeredpractical answers to thecorequestionsaboutrelevanceandeffectivenessinthepeacebuildingfield.RelevanceandEffectivenessarealsotwoofthekeyOECD/DACcriteriaforpeacebuildingevaluation,anddefinedasfollows:

CDAbuildsonthisdefinitionofeffectivenessbyintroducingadistinctionbetweenprogrameffectivenessandpeaceeffectiveness:

CDA’sworkonpeacebuilding effectivenessbegan in 1999with the launchof theReflectingonPeacePracticeProgram(RPP).RPPposedasimple,albeitcomplexquestion:Whatworks-andwhatdoesn’t

work – in peacebuilding? RPPworkedwith hundreds of agencies and individuals, and conducted 26

Relevance|assessestheextenttowhichtheobjectivesandactivitiesoftheintervention(s)respondtotheneedsofthepeacebuildingprocess,i.e.whethertheyaddressthekeydrivingfactorsofconflictrevealedthroughaconflictanalysis.Relevancelinkstheoutcomesoftheconflictanalysiswiththeintervention’sobjectives,althoughtherelevanceoftheinterventionmightchangeovertimeascircumstanceschange.Understandingrelevancemayalsoinvolveanassessmentoftheextenttowhichaninterventiontiesinwithoverallstrategiesandpolicyframeworksofthecountryorexternalpartners.Differentconflictgroupsoractorsmayhavedifferentperspectivesontherelevanceofaninterventionanditsresults.(OECD/DAC,2012,56)

Effectiveness|isusedtoevaluatewhetheraninterventionhasmetitsintendedobjectiveswithrespecttoitsimmediatepeacebuildingenvironment,orislikelytodoso.Thekeytoevaluatingeffectiveness–andthusthelinkagebetweenoutputs,outcomesandimpacts–isfindingouttowhatdegreetheenvisagedresultshavebeenachievedandnotingchangesthattheinterventionhasinitiatedortowhichithascontributed.[…]Itisimportanttodrawadistinctionbetweentwokindsofresults.Oneis“programmeeffectiveness”,i.e.towhatextenttheprogrammeachieveditsstatedobjective.Theotheris–iftheprogrammemetitsobjectivesorgoal–theimmediateorsecondaryoutcomesastheyrelatetopeacebuildingandconflictdynamicsidentifiedintheanalysis.(OECD/DAC2012,57)

ProgramEffectiveness|focusesonassessingwhetheraspecificprogramisachievingitsintendedgoalsinaneffectivemanner.Thiskindofevaluationaskswhethertheprogramisfulfillingitsgoalsandissuccessfulonitsownterms.

PeaceEffectiveness|askswhether,inmeetingspecificgoals,theprogrammakesacontributiontoPeaceWritLargeandhasapositiveeffectbyreducingkeydrivingfactorsofconflict.Thisrequiresassessingchangesintheoverallenvironmentthatmayormaynotresultdirectlyfromtheprogram.InmostinstancesthisrequiresidentifyingthecontributionofthespecificprogramtoPWL,ratherthanseekingclearattributionofimpactsfromdiscretepeaceinitiatives.ImpactsatthelevelofPWLtypicallycannotbeachievedbysingleactivitiesandprojects,butratherarecumulative,resultingfrommanydifferenteffortshappeningsimultaneously,especiallywhentheseeffortsaredeliberatelydesignedtocomplementoneanother.Strategiclinkagesamongeffortsinasinglecontextarethereforecritical.(CDACollaborativeLearningProjects2013,28)

8

peacebuildingcasestudiesthroughouttheworldtogleanlessonsapplicableacrossconflictcontextsanddevelopuser-friendlytoolkits.TheresultinglessonsarepresentedinConfrontingWar:CriticalLessonsforPeacePractitioners(2003).Between2007and2009,CDAundertook16casestudiesthatinvestigatedthecumulativeimpactsofpeacebuildingprograms inEurope,Africa,MiddleEast,AsiaandLatinAmerica.Findingsfromthesehavebeenpublishedinissuepapers,selectpublications.17

Buildingonthiscumulativeimpactwork,CDAhasdevelopedspecificapproachestosystemsthinkingandpeacebuilding,includingsystemicconflictanalysis,systemsmapping,andtheidentificationofleverage

pointsforchangeasanothermeansofexpandingthepeacebuildingeffectivenessfield.Theexperienceand lessons gained through the years of RPP’s operation are the foundation of CDA’s currentPeacebuildingEffectivenesspracticearea,whichcontinuestopromotelearninginthisfield,boththroughadvisoryservicesandthroughongoingcollaborativelearningefforts.18

DoNoHarm

CDA’sworkonconflictsensitivitybeganin1993withthelaunchoftheLocalCapacitiesforPeaceProject,whichcametobeknownastheDoNoHarmProgram.Overtheyears,DoNoHarmhasinvolvedhundredsofaidagencies,andmorethan1000aidpractitionersfromallovertheworldinitscollaborativelearningprocesses.TheresultinglessonsarepresentedinDoNoHarm:HowAidCanSupportPeace–OrWar,aswell asmany other publications andGuides available on the CDAwebsite. Today, “DoNo Harm” isrelevantinpracticeasaprincipleandasatool(theDNHFramework),andisusedbymanypractitionerstodescribetheirworkonconflict-sensitivity.

ThepurposeofthefollowingsectionistointroducebasicRPPandDNHapproachesandtools,whichareused as criteria for program quality assessments, evaluability assessments, as well as strategy andprogramreflectionexercisesashighlightedinthisGuide.

2.2KeylessonsfromCDA’sReflectingonPeacePracticeProgram(RPP)

CDAhasdistilledthefollowingkeylessonsthroughthevariousphasesofpracticallearningfromRPP:

I. PeacebuildingprogramsshouldbeaccountabletoPeaceWritLarge;II. Conflictanalysisiscrucial.Goodconflictanalysisshould:

a. identifyKeyFactorsandKeyActorsvis-à-vispeace/conflict,b. identifytheRelationshipsandDynamicsamongthem,andc. clarifypointsofpossibleintervention.

III. AnalysismustbelinkedtoprogrammingandtoPeaceWritLargethroughastrongtheoryofchange;

IV. ProgramsmustreachtheSocio-PoliticalLevelinordertoaffectPeaceWritLarge;V. “MorePeople”workmustengage“KeyPeople”andviceversa;VI. Itisimportanttoengagethehard-to-reach;VII. ItispossibletoassesstheimpactofprogramsonPeaceWritLargeiftheyarebasedonconflict

analysis,strongtheoriesofchange,androbustprogramgoals.

17Formoreinformation,pleasevisit:http://cdacollaborative.org/18FormoreinformationaboutCDA’sworkand/orRPP,pleasevisit:http://cdacollaborative.org/

9

KeyLessonI:PeacebuildingprogramsshouldbeaccountabletoPeaceWritLarge

PeaceWritLarge(PWL)isconcernedwiththe“biggerpicture”ofaconflict.This“biggerpicture”referstotheoverallsocio-politicalconditionsinagivencontext.Itcaninvolvenationallevelconflictdynamics(orin some contexts, sub-national or regional dynamics). Being accountable to PeaceWrit Largemeansensuringthatinitiativesaddresskeydriversofconflictandmakeacontributiontothe'biggerpicture'.Thisrequiresanexplicitstrategyforinfluencingthosedrivers,andawaytomonitorandevaluateeffectsbeyondthelifeoftheproject.Itdoesnotmeanthatallprogramsshouldbeexpectedtoproduceconcretechangesatthelargersocietallevel.Infact,manyprogramsaresuccessfulatsmallerscaleinterventions,suchasoperatingatthecommunitylevel,orwithsmallgroupsofpeople,thuscontributingto‘peacewritlittle’. The impact of these interventionswill not be directly observable at a societal level. However,CDA/RPPhasfoundthatmanypractitionersassumethattheirprograms,becausetheyhavesolidgoals,willsomehowleadtoorsupportPeaceWritLarge.Thisisnotalwaysthecase.

AssessingcontributiontoPeaceWritLargeisdifficult,asmostpeacebuildingprogramsarediscreteeffortsaimedataffectingone(oftensmall)pieceofthepuzzle,andnooneprojectcandoeverything.RPPhasfound that certain elements of program strategy and logicmake it more likely that peacebuilding

projects/programswillhaveaninfluenceonPeaceWritLarge.Thosearefurtheroutlinedbelow–andarealsoatthecoreofthePQA,EA,andStrategyandProgramReflectionExercisedescribedinthisGuide.

KeyLessonII:ConflictAnalysisisCrucial

Basedonareviewofdozensofpeacebuildinginitiatives,thereisstrongevidenceabouttherelationshipbetweenconflictanalysisandstrategicandrelevantprogramminginconflict-affectedcontexts.

Goodconflictanalysisshould:

§ identifyKeyFactorsandKeyActorsvis-à-vispeace/conflict,§ identifytheRelationshipsandDynamicsamongthem,and§ clarifypointsofpossibleintervention.

Key driving factors of conflict are factors without which the conflict would not exist or would besignificantly different. Key actors are people or groups that can significantly influence the conflictdynamics.

Foramoredetailed introductiononhowtoconductconflictanalysis,pleaserefertotheCDATrainingManualsandtheConflictAnalysisFrameworkdevelopedbytheGlobalPartnershipforthePreventionofArmed Conflict in collaboration with CDA Collaborative Learning Projects and Norwegian Church Aid(2016)(seeSection8,Resources).

“Analysisisnotoptional;itisessentialandobligatoryforpeacework”

RPPconsultationparticipant

10

KeyLessonIII:AnalysismustbelinkedtoprogrammingandtoPeaceWritLargethroughastrongtheoryofchange

ExamplesofTheoriesofChange19

Projectlevel(individuallevelchange)20 Portfolio/Sectorlevel(socio-politicallevelchange)

If[activity]childreninthisschoolaregivenindividualtreatmentfortraumarecovery,

then[change]theywilldevelopincreasedabilitytocontroltheiremotionsandnotactoutagainstothers,especiallythosewhoaredifferentfromthem;

because[rationale]theactivitieswillhavehelpedthembegintohealfromthepsychologicalwoundsofwarandreducetheiroverallfearandsenseofvulnerabilityatschool.

Ifwewantedtomovethisengagementtopotentiallyshowresultstowardssocio-politicalchange:

[Note:undertheseconditions,ifweintroduceinter-groupskills(negotiation,mediation,problem-solving)tochildrenofdifferentreligiousgroupstogether,thentheywillbeabletolearnthemandusethemtoresolvedisputesatschool,includingthosethatmayarisebetweenreligiousgroups.21]

If[activity]westrengthenthecapacitiesofselectlocalandnationallevelgovernmentinstitutionsinviolencepreventionandcoexistence;

then[change]interactionswithinthegovernmentandbetweenstateandcivilsocietywillbemoreconstructiveandinclusive,

because[rationale]localandnationalgovernmentinstitutionswillbebetterequippedtodealwithtensionsmoreconstructivelyandengageinforwardlooking,preventiveapproacheswithingovernmentandinstate-societyrelations.

[Note:thissectortheoryofchangeisverymacro-levelandneedstobeaccompaniedbymoreconcreteandmeasurabletheoriesofchangeattheprogramandprojectlevel.Thiswouldhelpfurtherembedthetheoryinspecificoperationalcontexts.]

19Formorebackgroundonthedifferentlevelsatwhichtheoriesofchangecanbeuseful(activity,project,program,portfolio/sector,countryleveletc.)pleasesee(WoodrowandOatley2013)20Babbittetal,TheoriesandIndicatorsofChange:ConceptsandPrimersforConflictManagementandMitigation,921Ibid.

ATheoryofChange|isanexplanationofhowandwhyanactionisbelievedtobringaboutitsplannedobjectives,i.e.thechangesithopestocreatethroughitsactivities,therebyrevealingunderlyingassumptions.Acleartheoryofchangehelpstoarticulatethelogicalflowfromthestartingpoint(analysis)tothegoaloftheinitiativetothebroaderchangetheorganizationplanstoachieve.

Apracticalformulaforarticulatingatheoryofchangeisthefollowing:

Ifx[activity],

theny[expectedchange],

becausez[rationale-whydoyouthinkthischangewillhappen?]

11

Theoriesofchangeoperateatdifferentlevels.Theycanrelateto micro-level changes (e.g. project level), usually associatedwith specific activities. They can describe how an overallprogramapproachandgoalwillbeachieved(e.g.howvariousjustice and human rights initiatives achieve progress in thatsector), as well as how achieving the goal will contribute toPeace Writ Large (e.g. how different sector activities acrosshuman rights, rule of law, and dialogue promotionwill worktowardsreconciliation– if that iswhathadbeen identifiedasthevisionforPeaceWritLarge).Theoriesofchangeneedtobegrounded in the particular context, and should be specific

enoughtobetestable.

KeyLessonIV:ProgramsmustreachtheSocio-PoliticalLevelinordertoaffectPeaceWritLarge

For peacebuilding programs to be effective, they must linkchangeattheindividual/personalleveltochangeatthesocio-

politicallevel.Theindividual/personallevelincludesattitudes,perceptions, behaviors, skills and interpersonal relations. Thesocio-political level includes relations among social groups,public opinion, social norms, societal institutions, and deeperelements embedded in social and economic structures andculture.

Individual/PersonalChange

Healing/recoveryPerceptionsAttitudesSkillsKnowledge

BehaviorIndividualrelationships

Socio-PoliticalChange

Groupbehavior/relationshipsPublicopinionSocialnorms

Institutionalchange

Structural+culturalchange

RPP found that programming which focuses on change at the individual/personal level but nevertranslatesthisintoactionorresultsatthesocio-politicallevelhasnodiscernibleeffectonpeace.Inmanycases, it is also important to link change at the socio-political level back to individual/personal levelchange—especiallyifthechangesaretobemeaningfulandsustainable.

While the desired changes do not necessarily need to be observable at the national level,programs/projects should, at the level atwhich they are operating, affect the creationof institutions(formalorinformal),resultinlocally-drivenpeaceagendasandaction,orleadtocollectiveattitudesand

Morein-depthreadingontheoriesofchange:

® dmeforpeace.org/evaluation-planning® WoodrowandOatley2013® Babbit,ChigasandWilkinson2013

Cartoonist:SidneyHarris

12

behaviors that reflect improvements in the key driving factors of conflict or peace at that level (e.g.,increased security or perceptions of security, improved group attitudes or relations, resistance toviolence,etc.).22

KeyLessonV:“MorePeople”workmustengage“KeyPeople”andviceversa

Effectiveprogramsalsolinkworkwith“morepeople”—i.e.,peopleatmanylevelsofsocietyandinmanysectors—to“keypeople,”peopleorgroups thathave thepoweror influence todecide fororagainstprogresstowardspeace.

MorePeople KeyPeople

Peaceneedssupportandparticipationofthegeneralpopulation.

Peacecannotbeachievedwithoutinvolvementofcertainpeoplewithmajorinfluenceonthesituation.

Work that influences “more people” or“keypeople”butdoesnotconnector linkto efforts to affect the other has limitedimpact.

InsightsIVandVcombinedpromptedCDAto develop a very practical tool used toassessprogramstrategies:theRPPMatrix(SeeAnnexes2.1and2.2)

KeyLessonVI:Itisimportanttoengagethe‘hardtoreach’

Manyprogramsoperateoncertainbiases.They…

§ engagewiththe“easytoreach”(thosewhowanttoworkwithus),§ workwiththoseseenasnon-political,willingtocooperate,lesscommittedtoviolence,and§ focusondoing“good”vs.stopping“bad”(e.g.,participatorycommunitydevelopment,inter-ethnic

dialogue,etc.)anddonotdealwithdynamicsandpeoplethatpromoteorperpetuateviolentconflictand/orfragility.

Therefore,itiscriticalforpeacebuilderstoaskthemselvesthefollowingquestions:

§ Whatgroupsaresystematicallyleftoutofpeacebuildingefforts?§ Whoisthepeacebuildingcommunityavoiding?§ Whichgroupsmighthaveanegativeeffectonpeaceefforts,andcouldunderminefragilegains?§ Who(ifanyone!)hasaccesstothosegroups?

22Forfurtherreadingreferto(Ernstorfer,ChigasandVaughan-Lee2015)

Healing/recovery

Perceptions

Attitudes

Skills

Group behavior/

relationships

Public opinion

Social norms

MORE PEOPLE KEY PEOPLE

VISION: A desired futureSocietal change/Peace Writ Large

CURRENT SITUATION:

Conflict AnalysisKey Driving Factors of Conflict and “Key People” or

Actor Analysis

INDIVIDUAL/

PERSONAL

CHANGE

SOCIO-

POLITICAL

CHANGE

Institutional

change

Structural

change

Behavior

Individual

relationships

Programme Activities

Program Theory:

How do the activities

lead to the goal?

Socio-political

goal

Theory of Change:

How does the goal contribute

to Peace Writ Large?

What is the

gap between

the current

situation and

the desired

future? à

“peace

needs” and/or

strategic

space.

What needs to change

and how?

Programme Activities

13

The ‘hard to reach question’ will not beexamined to a great level of detail in thisGuide. However, it is often one criticalquestion in program relevance (next toothers).23

KeyLessonVII:ItispossibletoassesstheimpactofprogramsonPeaceWritLarge

For many years, the peacebuildingcommunity has been avoiding amore rigidapproachtomonitoringandevaluation(seeSection2).However, it ispossibletoassesstheimpactofprogramsonPeaceWritLargeIFprograms:

§ arebasedongoodconflictanalysis–toensurerelevance,§ haveawell-articulatedtheoryofchange,and§ haverobustandwell-definedgoals.

Wehavealreadytoucheduponconflictanalysisandtheoriesofchange.Robustprogramgoalsneedtobeset in termsofdesiredchanges.Manyprogramssetgoals forpersonalchange, includingattitudes,perceptions,personalbehaviors,skills,andrelationsamongindividuals—andchangeatthislevelisoftennecessary,thoughrarelysufficient.

However, programs that formulate goals as desired changes at the socio-political level, and/or havestrategiesinplacethatworkwithotherprogramsandorganizationsoperatingatthatlevel(iftheydon’tdoitthemselves),aremorelikelytohaveimpactsonPeaceWritLarge.

BuildingBlocksforPeace(CriteriaofEffectiveness)

Basedonextensivecaseanalysisandpractitionerreflection,theRPPprocessidentifiedfiveintermediateBuildingBlocksforPeaceorCriteriaofEffectivenesstosupportprogresstowardsPeaceWritLarge.

Thesecanbeusedtoassess,acrossabroadrangeofcontextsandprogrammingapproaches,whetheraprogramismakingameaningfulcontributiontoPeaceWritLarge.

The effort results in the creation or reform of political institutions to handle grievances insituationswheresuchgrievancesdo,genuinely,drivetheconflict.

The effort contributes to amomentum for peaceby causingparticipants and communities todeveloptheirownpeaceinitiativesinrelationtocriticalelementsofcontextanalysis.

Theeffortpromptspeopletoresistviolenceandprovocationstoviolencemorefrequently.

Theeffortresultsinanincreaseinpeople’ssecurityandintheirsenseofsecurity.

Theeffortresultsinmeaningfulimprovementininter-grouprelations.

TheseBuildingBlockscanbestbethoughtofasintermediate-levelbenchmarksorindicatorsofsuccessapplicable to thebroad rangeofpeaceworkbeingdone.TheBuildingBlockscanbeused inprogram23Forfurtherreadingpleasesee(CDACollaborativeLearningProjects2012)

1

2

3

4

5

It’salmostenoughtomakeanevaluatornostalgicforfuzzygoals.Cartoonist:MarkM.Rogers

14

planningtoensurethatspecificprogramgoalsarelinkedtothelargerandlong-termgoalof“PeaceWritLarge.” Theycanbeusedduringprogram implementation to reflectoneffectivenessandGuidemid-coursechanges,andasabasisforevaluationaftertheprogramhasbeencompleted.

Foramorein-depthintroductiontoRPP’slessonsandinsights,pleaserefertotheresourceslistedinthebibliography(Section10).

2.3KeylessonsfromCDA’sDoNoHarmProgram(DNH)

A conflict-sensitive approach minimizes the negative and maximizes the positive impacts of anyinterventionsonpeaceandconflictdynamics.Manyorganizationsworkonconflict-sensitivity,andusethe DNH approach developed by CDA – both as a tool, a framework, and a ‘standard’ for conflict-sensitivity.

ThecollaborativelearningprocessthatCDA’sDoNoHarmprogramwentthroughledtothefollowingsixmainconcludinglessons:

1. Whenaninterventionofanykindentersacontext,itbecomespartofthatcontext;2. AllcontextsarecharacterizedbybothDividersandConnectors;3. AllinterventionswillinteractwithbothDividersandConnectors,makingthembetterorworse;4. InterventionsinteractwithDividersandConnectorsthroughtheirorganizationalActionsandthe

Behaviorofstaff;5. TheDetailsofaninterventionarethesourceofitsimpacts;6. TherearealwaysOptions(e.g.forprogramre-designordoingthingsdifferently.)

TheselessonsresultedinthecreationoftheDNHframework:

CONTEXTOFCONFLICT

Options Dividers Interventions Connectors Options

Redesign

Who?What?Where?How?When?Why?

Redesign

ForfurtherintroductiontotheDNHframeworkpleaserefertotheDNHresourceslistedinSection10.

Actionsand

Behaviors

HeadquartersMandateFunding

15

ThefollowingSections4,5,and6willprovidedetailedguidanceandastep-by-stepapproachtoProgramQualityAssessments,EvaluabilityAssessmentsandStrategyandProgramReflectionExercisesusingRPPandDNHascriteria.

Principles of participation, partner and local stakeholder engagement throughout thisdocument

Program Quality Assessments, Evaluability Assessments, and Strategy and ProgramReflectionExercisesprovideexcellentopportunitiesforprogramteams toworkwith theirprogrampartners,localstakeholdersanddonorstojointlydevelopavisionforstrengthenedstrategiesandprogramsinaparticipatoryfashion.

Eachteamwillneedtodeterminewhattypeandwhatlevelofpartnerengagementseemsmostappropriateanduseful.Ingeneral,alltheseprocessesencourageamaximumlevelofprogram partner and local stakeholder engagement. No program quality assessment orEvaluabilityAssessmentcanbecrediblydonewithoutawell-thought-outprocessofprogrampartnerengagement.Ingeneral,StrategyorProgramReflectionExercisesshouldalsoinvolvepartnerstoacertainextent.However,StrategyandProgramReflectionExercisesmightbethe option most frequently used to support internal team alignment around programstrategy–andnotalwaysinvolveprogramandotherlocalpartners.

ThequestionofhowprogrampartnersandlocalstakeholdersareengagedinEAsandPQAswill need to be determined case by case. In some instances, the facilitation of a jointworkshopwithallrelevantteams,partners,anddonors isfeasibleandadvisable. Inotherinstances, separate processes of consultations and bi-lateral interviews are moreappropriate,alsoregardingoverallpoliticalandculturalsensitivities.Inanycase,especiallyforEAs,acertainamountofindependenceindatacollectionwithlocalpartnersisessentialforthecredibilityandrobustnessoftheprocess.

16

3.Whatistherightoptionformyteam?Whatarethebasicsweneedtohaveinplacetobenefitfromanyoftheseoptions?

AllthreeoptionsputforwardinthisGuidearelearningtoolswithdifferentfoci,differentpurposes,anddifferentdegreesofrigor(seeSection7forasummaryoverviewofkeyfeaturesofPQAs,EAsandStrategyandProgramReflectionExercisesvis-à-visformalevaluations).Allthreeoptionsreviewprogramdesignandstrategy,inordertomaximizethepotentialimpactofpeacebuildinginitiativesonconflictsystems.However,theseoptionsarenotdesignedasevaluationstoassesstheactualchangesin-country.

Beforedecidingwhichevaluativeoptiontochoose,afewkeyquestionswillneedtobeanswered:

� Whereareyouinthelifeoftheprogram?

� Whatpromptedtheneedtoundertakeoneoftheseexercises?

� Whatisthepurposeandexpectedresultoftheprocess?Howwillyouusewhatyoulearnthroughthisprocess?

� Whatisthenatureoftheexercise(whichoptionsandforwhatpurpose)andrelateddemandsonteam,staffandprogrampartners:areexistingcapacitiesandresourcessufficient?

� Whatarethemainlinesofinquiry?Isdataavailabletoanswerthekeyquestions?Howmuchdatacollectionisneeded?Dataavailabilitymightalsoinfluencethechoiceofanevaluativeprocess.

� TeamCohesion:IsSeniorManagementonboard?Dothedonorsneedtobeinvolved?Isthereacommon vision about the purpose/timing/requirement amongst staff and/or partners andeveryonewhoneedstobeinvolved?Howwillpartnersbeincluded?

� Whowillleadandfacilitatetheexercise?

� Whatismoreappropriate–aninternal,externalorblendedprocess?Sometimesalessformalprocess ismoreconducivetostrengtheninganinitiative. Itcanbefacilitatedinternallyorwithexternalsupport.

Moreresourcesonwhetheryourprogramisreadyforanevaluation:

® dmeforpeace.org/evaluation-planningunder"Istheprogramorinitiativereadyforevaluation?”

17

ChoosingtheRightEvaluativeOption

Thebelowtablemighthelpwithdecisionmakingbasedondifferenttypesofpurposesandintentions:

Ifyourgoalisto… …youshould…

…conductanindependentassessmentofyourprogramstrategyreviewingconflictanalysis,goals,theoriesofchange,andprogramstrategy…

→ …chooseaprogramqualityassessment.

…assesshowreadyyourprogramisforanevaluation… → …chooseanevaluabilityassessment.• …assesstheperformanceoftheinterventions,review

outputs,outcomes,andresults,andassessimplementationpractices…

→ …conductanevaluation.

• ...facilitateaninternal,fairlyinformalprocessofreflectionandimproveyourteam’s(andpossiblyyourpartner’s)understandingofwhatmakesapeacebuildinginitiativerelevantandeffective…

→ …conductastrategy/programreflectionexercise–orpossiblyaprogramqualityassessment(thelatterappliesthepeacebuildingcriteriamoresystematically).

…obtainanindependentassessmentofyourprogramtoshowaccountability24toyourdonorand/orprogrampartners…

→ …conductanevaluation.

…obtainaninternalassessmentofyourworktoshowaccountabilitytoadonorand/orprogrampartners

…conductaninternalevaluationorself-evaluation.

…analyzedataavailability,andunderstandtheconducivenessofthecontextforyourprogram’seffectiveness…

→ …conductanevaluabilityassessment.

…developacommonunderstandingwithintheprojectteam(andpossiblyamongstpartners)aboutthecontext,overallgoals,theoriesofchange,andprogramstrategy,aswellasstrengthenskills…

→ …initiateastrategy/programreflectionexercise.

…strengthenthecapacityofyourteamandpartnersinprogramstrategydevelopmentandimplementationbyassessingyourprogrambasedonRPPcriteria…

→ …planforaprogramqualityassessment.

…totrainyourstaffand/orpartnersinRPPand/orDNHtoolsandapproaches…

→ …notconductanyoftheseprocesses,butdevelopa/RPP/DNHtrainingprogramforstaffandpartners.

…tointroducebasicM&Eframeworksandapproaches… → …notconductanyoftheseprocesses,butdevelopanM&Etrainingprogramforstaffandpartners.

…analyzewhetheryourinitiativemightneedtobeadaptediftherearemajorchangestothecontext…(e.g.anelection,orsigningofapeaceagreement)

→ …consideranyoftheseoptions.Butastrategyandprogramreflectionexercisewill,inmostcases,bethemostappropriate.

24Notallformalevaluationsfordonorsonlyservethepurposeofaccountability.Somedonorsalsoconductlearningevaluations.

18

ConditionsforEngagementinRPPinfusedEvaluabilityAssessments,ProgramQualityAssessmentsandStrategy/ProgramReflectionExercise

Regardlessofwhichmodalityischosen,thefollowingconditionsshouldbeinplacebeforeconductinganyoftheseprocesses:

Condition Explanation

Willingnesstochallengeassumptions

RPPtoolsandlessons-learnedoccasionallyprovidechallengingmessagesaboutwhatmakesforeffectivepeacebuilding.Inparticular,theprocessinvolvescarefulconsiderationofthetheoriesofchangeunderpinningtheprogram–i.e.theveryassumptionswehaveabouthowtheprogramwillcontributetochange.Theprogramteamandprogrampartnersneedtobeopenandreflective,reflexiveandwillingtochallengetheassumptionsuponwhichprogrammingisbased.

Abilitytoadjustprogram

Adjustmentstoprogramdesignand/orimplementationarelikelytoresultfromeitherprocess.Thereneedstobeawillingnesstoadaptprogrammingonthepartofallrelevantstakeholders,includingdonors.Thismaybeparticularlychallengingiftheprogramisbeingimplementedbyaconsortiumwheredifferentagenciesareimplementingdifferentpartsoftheprogram.

OpennesstoRPPtoolsandmethods

SometeamsandprogramwillhavehadpriorexposuretoRPPtoolsandapproaches,somewon’t.Whileitisnotapre-requisitethatprogramteamsand/orprogrampartnershavebeenexposedtooralreadyknowRPP,moretimewillneedtobeallocatedifteamsarenewtothemethodologies.Inanycase,theprogramteamwillneedtobeopentousingthesetoolsandmethods.

Realisticsenseofcapacitydevelopment

TeamsandSeniorManagersneedtohavearealisticsenseofhowmuchcapacitydevelopmentispossiblethroughastrategy/programreflectionexerciseoraprogramqualityassessment.ForteamsandprogrampartnerswithnopriorexposuretoRPPtools,thebasicconceptscanbeconveyedthrougheitherofthose.However,thisdoesnotreplaceamoreformaltrainingand/orcapacitydevelopmentprocessforstaff.EvaluabilityAssessmentsusuallydonotincludeafocusoncapacitydevelopment.

HavingtheseconditionsinplacewillmaximizethebenefitofanyofthethreeoptionsinthisGuideandfacilitateuptakeofthefindingsandfollow-up.

19

4.ProgramQualityAssessment(PQA)–criteriaforassessingpeacebuildingprogramquality

4.1Overview:PurposeofaProgramQualityAssessment

ThepurposesofaPQAareto:

§ reviewandlearnaboutthequalityofaprogram’sdesign,§ strengthen/adaptprogramdesign throughapplicationofa clear setofprofessional standardsand

criteria,§ maximizepotentialforprogramtocontributetoPeaceWritLarge,§ laythefoundationforadaptiveprogrammingandactionplanningwithprogramteamsandprogram

partners,and§ supportstrategicandlong-termlearning.

A PQA examines how the program is functioning in practice, but only to the extent necessary tounderstandifandhowaprogramismeetingthegivencriteria.Particularly,thisincludesanassessmentof the quality of the program strategy, logic and theory of change, and how conducive these are toachievingtheenvisionedoutputs,results,andoutcomesoftheprogram.Thus,itexaminesifitislikelytoachieveitsgoals,aswellasachieveacontributiontoPeaceWritLarge.Itdoesnotassessresults,outputs,andoutcomesindetail.

It is similar toa formativeormid-termevaluation,butdiffers in that itdoesnot involvea systematicevaluationprocessnorapplyevaluationstandards,andhasastrongfocusoncapacitybuilding.PQAsdonot substituteanevaluation.However, theremightbe circumstances (e.g. anextremelydynamicandquickly changing context) under which a PQA might be the appropriate level of rigor, and a formalevaluationmightnotbepossible.

4.2KeyelementsofaprogramqualityassessmentbasedonRPPcriteria

ThePQAinvolvesareviewofkeyprogramdimensionsandassessmentagainstRPP-basedcriteria:

1. Performanceofaconflictanalysiscontributingtoprogramrelevance;2. Clearandappropriateprogramgoals;3. Well-formulatedandplausibletheory(ies)ofchangeatdifferentlevels(macro–meso-micro);4. Programstrategyandlogic,includingincorporationofanM&Esystemaspartoftheprogram

design;255. Inclusionofconflict-sensitivityindesignandimplementation.

Thefollowingoverviewtablesprovidekeyassessmentcriteria,andexamplesofcommongapsandweaknessesfoundinmanyprograms.

Annex5willprovideapracticalPQAtoolthatteamscanusetodocumenttheoutcomesofprogramqualityassessmentofeachoftheseprogramdimensions.

25WhereasinanEvaluabilityAssessmenttheM&Esystemismuchmoreinthefocusintermsofdatacollectionmechanisms.

20

PQATable1:ConflictAnalysisbasedonRPPcriteria

RPPCriteria Commongaps/weaknessesorcautionsconcerninganalysis

1 Theanalysisidentifieskeydrivingfactorsandkeyactorsforconflict/peace.26

§ Analysisistoocomprehensive:toomanyfactorswithnoprioritiesidentified.

§ Analyzesentirecontext,butdoesnotfocusonconflictdeterminants;everythingisseenasrelevanttopeacebuilding.

§ Factors/issuesareidentified,butnotdynamicsamongthemorwhicharemore/lessimportant(priorities).

§ Analysisisimplicit,andthusnotsharedamongteamandprogrampartners.

2 Theanalysisconsiderswhatneedstobestopped(andwhowillresist)andwhatforcespromotepeaceinthiscontext.

§ Analysisfocusesonpositivefactorsthatmightbestrengthened,butdoesnotconsidercountervailingnegativeforces.

§ Analysisdoesnotanalyzewhatfactorsconnectpeopleorpromotepeaceinthiscontext.

3 Theanalysisisupdatedandtestedregularly/periodically.

§ Analysisisperformedonceatbeginningofprogram,butnotupdated.§ Conclusionsaboutdriversanddynamicsofconflictarenotutilizedto

strengthenprogram.

4 Analysisofsimilarprogrameffortsincludinganylessonsfromtheirresults(programeffortsincludethosefromthepastorongoing).

§ Analysisdoesnotidentifyresults/lessons§ Theprogramteamhasnoknowledgeofwhathasbeentriedbefore(or

resultingeffects).§ Programsrepeatfailedapproaches.§ Programs/projectsduplicateeffortsofotherswithoutaddedvalue.

5 Theprogramstrategybuildsontheanalysis:identifiespeacebuildingneedsorpointsofleverageforchange.

§ Programgoalsanddesigndonotaddressfactorsidentified.§ Analysisdoesnotenableprogramdesignerstoidentifywhattodoto

changeconflictdynamic.

6 Thescopeoftheanalysisisappropriate(nottoobroadornarrow);andmitigatesbiastowardsagency’sexpertiseorgeneralbeliefsaboutconflict.

§ Analysisisperformedtojustifyfavoredprogramapproach(methodology,focus,constituency).

§ Analysisisbasedonbeliefsabouthowtobringaboutpeacegenerally(andnotcontextualized).

§ Analysishasomittedorexcludedsignificantperspectives.§ Analysisexistsatonelevel,butdoesnotconsiderotherlevels(e.g.,at

locallevel,missingwiderdynamics;national/regionalanalysiswithoutlocalparticularities;nationalanalysiswithoutinternational/regionaldimensions).

7 Theanalysisprocesshasbeenconflictsensitive—consideredpotentialharmitmightcause

§ Teamcompositionexposesteammembers,partners,orintervieweestodanger.

§ Teamcomposition,behaviorperceivedasbiased.§ Analysisprocessdeepenspolarization.§ Causesofconflictarecontestedamongkeyparties—analysisprocessis

fraught.

26Keydrivingfactorsareelements/dynamicswithoutwhichtheconflictwouldnotexistorwouldbesignificantlydifferent.Keyactorsarepeopleorgroupsthatcansignificantlyinfluencetheconflictdynamics.

21

PQATable2:RPPcriteriaforprogramgoals

RPPCriteria Commongaps/weaknessesorcautionsingoals

1 Thegoaladdresses,directlyorindirectly,keydriversofconflictorpeace.

§ Thegoalisnotappropriateforconflictcontext;othergoalsmaybemoreappropriate.

§ Thegoal(s)addressessymptomsorconsequencesofconflictbutnotdrivers,oritaddressesfactorsofsecondaryimportance.

2 Thegoalisstatedasadesiredchange. § Intendedorexpectedchangesfromtheeffortarenotclear.

§ Goalsarestatedasactivities,outputsortasks.

3 Thegoalisspecificandrealisticforthetimeframe—neithertoobroad(along-termvision)nortoonarrow(attheactivitylevel).

§ Goalsarevague,grandioseand“overclaim”.§ Goalsareexpressedatavisionlevel.§ Goalsstatedareprocesses(ratherthanthe

outcomeofprocesses)oraseriesofactivities.§ Programteamisunabletoarticulateclear

benchmarks,indicatorsorothersignsthatwouldhelpthemknowifprogressisbeingmade.

4 Thegoalisstatedasadesiredchangeinthesocio-politicalrealm.Ifnot,thereisanexplicitlonger-termstrategyforeffectingsocio-politicallevelchange,ortheprogrammakeslinkagestotheactivitiesofotheragenciesinthesocio-politicalrealm.

§ Theprogram/projectseekschangeattheindividual-personallevelonly(attitudes,skills,etc.),andunrealistically“hopes”orassumesthatchangesatthesocio-politicalwillcomeabout.

§ Theprogramgoalattheindividual-personallevelisappropriate,butlinkagestootherprogramsorstrategiesforfollow-onworktomovetothesocio-politicalleveldonotexist.

§ Theprogramassumes(withoutcontext-basedevidenceorconflictanalysis)thatalotofworkatthemicro(community)levelwillsomehow“addup”tosignificantchangesathigherlevels(PeaceWritLarge).

§ Changesdesiredarenotsustainable,bigenoughinscaleorfastenoughinthiscontext.

5 Thechangescontributetooneofthefollowingbuildingblocksforpeace:

1. Politicalinstitutionsthataddresskeydriversofconflictarecreatedorreformed.

2. Locallydrivenpeaceinitiative/saddress(indirectlyordirectly)keydriversofconflict/peace.

3. Peopleincreasinglyresistviolenceand/orprovocationstoviolence.

4. Peoplegainincreasedsecurityand/orasenseofsecurity.

5. Inter-grouprelationsimprovesignificantly(e.g.,groupattitudes,publicopinion,socialnorms,publicbehavior).

§ Goalsaimatindividualattitude,skillorbehavioralchangeonly.

§ Teamsand/orpartnersdonotagreeonthetypeofchangetheyarepursuing.

§ Programgoalsrepresentmeaningfulchange,butitisnotclearhowthechangemightbesustained.

22

PQATable3:RPPcriteriaregardingoveralltheoryofchange

RPPCriteria Commongaps/weaknessesorcautionsconcerningoveralltheoryofchange

1 TheOverallTheoryofChangeisexplicit,withclearandunderstandableconceptualizationofpathwaytochange.

§ Theoveralltheory(ies)ofchangeisimplicit,unstated.§ Itisunclearhowtheprogram,ifsuccessful,willaffect

keydriversofconflictorpeace.§ Theoryofchangeisbasedonfalseassumptionsabout

howchangecomesaboutinthiscontext.

2 TheOverallTheoryofChangeprovidesaplausibleexplanationofhowachievingthegoalwillaffectkeydriversofconflictorpeace.

§ Programsseekchangesthatarereasonableinthemselves,butwillultimatelyfailtoachievesustainablepeace(e.g.,passingalawwillnotaffectconflictdriversbecauseagreementonprinciplesandenforcementmechanismsarenotbeingworkedon).

3 TheoverallTheoryofChangeisgroundedinanunderstandingofhowchangehappensinthiscontext

§ Waysinwhichchangeprocessesaredifferentfromcontexttocontextarenotexaminedwhenprogramideasorapproachesfromonecontextareappliedinanother.

Insightfrompractice.NoneoftheprogramsexaminedundertheCDAfacilitatedEAsandPQAshadbeenplannedaccordingtoRPPcriteria.OneofthePQAshappenedshortlyafterprogramdesign,sosomeoftheRPPtoolsandapproacheswereintroducedaspartofaprogramre-designthathappenedduringthePQA(questionremainedwhetheraProgramReflectionExercisewouldhavebeenmoreappropriateforthatpurposeinsteadofaPQA).Asfortheothers,someoftheRPPelementswerealignedwithotherapproachesused,or freshly introducedaspartof theAssessmentFramework. Incases inwhichRPPprinciplesarenew,additionaltimeandarelatedbudgetneedstobebuiltintofamiliarizeparticipantswiththeconceptsand/orplantimeforrelatedcapacitydevelopment.

Insightfrompractice.InoneoftheCDAfacilitatedPQAs,theCDAfacilitatorworkedwiththeprogramteamandpartnerstoturntheprogramgoal,whichwasstatedatthemacro-level/long-termvisionlevel,intoamorerealistic,andachievablegoalfora2-yeartimeframe.Thishelpedclarifyprogramactivitiesandbenchmarksfor theprogram,whileadoptingthemacro-levelarticulationof theobjectiveastheoverall and long-termvisionof the initiative (so the team felt theywere going in the rightdirectionbroadlyspeaking,butnowwithaclearerunderstandingofactualprogramachievements).

23

PQATable4:RPP-basedcriteriaforprogramstrategyandlogic

RPP-BasedCriteria Commongaps/weaknessesinprogramstrategyandlogic1 Theactivities“addup”to

thegoal—thereisanexplicit,rationalandplausiblelinkbetweencomponentsoftheprogram(input,output,outcome,impact)thatisvalidinthiscontext.

§ Programtheory/logicisunclearornotexplicit.§ Programlogicisweak/illogicalorbasedonfalseassumptionabout

howchangecomesabout.§ Therearegapsorleapsoflogicinthepathwaytothegoal.§ Assumptionsabouthowonechangewillleadtoanother(howthe

programactivitieswill“addup”)havenotbeenexploredorarticulated.

§ Teammembersandprogrampartnersproceedonverydifferentassumptionsaboutprogramgoals,objectivesandchangeprocesses.

§ Programsfailtoaccountforkeyrequirements(e.g.,willingness&availabilityofparticipants).

§ Assumptionshavenotbeenchallengedorthoughtthrough.

2 Theprogrammakeslinkagesbetweenactivities/changesattheindividual/personallevelandthesocio-politicallevel.

§ Programactivitiesandchangesareexclusivelyattheindividual/personallevel(attitudes,skills,relationships),withnostrategytotranslatethesechangestosocio-politicalchange(eitherthroughfollow-upactivitiesorprograms,orlinkageswithotherefforts).

§ Linkagesoreffectsfromtheindividual/personaltothesocio-politicallevelbasedonlyon“hopes”orassumptions.

3 Theprogrammakeslinkagesbetween“morepeople”and“keypeople”.

§ Programfocusesonthe“easytoreach”withnostrategyforreachingbeyondtoaffectthe“hardtoreach”orconstituenciesignored.

§ Programhasnotincorporatedstrategiesforaffecting“keypeople”(ifworkingwith“morepeople”)or“morepeople”(ifworkingwith“keypeople”).

§ Programworksbothwith“morepeople”and“keypeople”andassumeslinkagethatmaynotoccur,i.e.areunrealisticinthiscontext.

§ Programbelievesitisworkingwithkeypeoplewhenitisnot(e.g.,assumesgovernmentofficialsare“key”whentheyhavelittleinfluenceontheconflict;assumespeoplekeytoimplementationoftheprogramortothemissionoftheagencyare“key”toconflict;assumesvictimsofconflictarekey).

§ Analysisdefinesentiregroupsofpeople(e.g.,youth15-25)askeybutdoesnotexaminewhetheritwillreachthoselikelytoperpetrateviolence.

4 Thescaleandleveloftheoutputsarereasonableinrelationtotheintendedimpactsandthesizeoftheissueinthiscontext.

§ Thereismismatchbetweenscaleofgoal(e.g.,toleranceorreconciliation)andscaleandlevelofoutput(e.g.,numberofparticipants,communities,etc.)

§ Theprogramisnot“big”enough—doesnothaveenoughscaletohavemeaningfulinfluence—andthereisnostrategy(eitherwithinorbeyondtheprogramorinconjunctionwithotherefforts)forachievingmeaningfulscale.

24

RPP-BasedCriteria Commongaps/weaknessesinprogramstrategyandlogic5 Theprogramdesignhas

accountedforfactorsthatcouldimpedesuccess,includingwayssocial&politicalsystemsmightresistchangesthattheprogramistryingtoworktowards.

§ Programhasnotaccountedforhowthesocialandpoliticalsystemswillpushbackagainstchangeefforts.

§ Programcan/doesachievemeaningfulinfluence,butitisnotclearhowchangescan/willbesustained—how,whyandbywhom?

6 Theeffortrelatestootherinitiativesinthesameorrelatedareasofwork,intermsofnecessarycomplementarities,linkagesorduplicationofeffort.

§ Programduplicatesothereffortsunnecessarilyorfailstoidentifyandlinktocomplementaryeffortsinthesamedomain.

7 ThereisanM&Eplanorfeedbackmechanismthatwillprovidetimely,accurateandusefulinformationaboutprogresstowarddesiredchangesandaboutassumptionsunderlyingthetheoryofchange.

§ Thereisnoprocessinplaceformonitoringandtestingtheprogramlogicandensuringtheprogramisnotcreatingunintendednegativeeffects.

§ Indicatorsareunrealistic.

RPPcriteriaonprogramstrategycombinedwithotherspecializedandsector-specificstandards

InadditiontotheapplicationofRPPcriteriaassessthequalityofprogramstrategy,specializedand/orsector-specificstandardsforthespecifictypeofpeacebuildinginterventionsalsoneedtobeconsideredaspartofaPQA-wheretheseexistorcanbeinferred.

Forinstance,ifthecentralprogrammethodologyinvolvesmulti-stakeholderdialogue,therearenumerous studies and scholarly articles that posit best practices and norms that can bereferenced when assessing a dialogue program. Likewise, programs that focus on electoralviolencepreventionnowcanbuildonvariousstandardsandtoolboxes.RPPcriteriaaregeneraltoalltypesofpeacebuildingprogrammingandfocusontheeffectivenessandrelevanceoftheprogrammingmorebroadly.

25

PQATable5:Inclusionofconflict-sensitivityintoprogramdesignandimplementation

DoNoHarmCriteria Commongaps/weaknessesinprogramstrategyandlogic1 Theprogramisconflict

sensitive—itconsiderspotentialunintendednegativeimpactsitmightcause.

§ Programhasunintendednegativeeffects.§ Programdesignand/orimplementationhasnotexamined

commoncausesofunintendednegativeeffects(thepotentialnegativeimpactsofchoicesaboutprogrampartners,contractors,suppliers,location,distributionofbenefits,timingofprogrammingetc.).

2 Theinitiativeisconsciousoftheimpactofitsactionpatterns/resourcetransferpatternsinfiveareas:1. Theft2. MarketEffects3. DistributionEffects4. SubstitutionEffects5. LegitimizationEffects[SeeAnnex4,DNHActionPatterns]

§ Goodsormoneyintendedfordistributionofpaymentisstolenorusedbyactorsintheconflictand/ortosupportongoingviolence.

§ Localmarketsaredestroyed,localpeoplearepricedoutoftheirownmarkets.

§ Unevendistributionofgoodsandservicesalongconflictlines.§ Unintentionalweakeningofthestate’sabilitytorespondand

manageconflictsanddisaster,anditsowndevelopment.§ Inadvertentlylegitimizingagovernment,institution,orleader.

3 TheinitiativeconsidersunintendednegativeimpactsoftheprograminitsM&Esystem

§ M&Esystemsarenotdesignedtocapturethedetailsofhowaninterventioninteractswiththeconflictcontext.

§ Therearefeworganizationalincentivestocaptureunintendedimpactsandtoactuponthosefindings.

Insightfrompractice.Manyprogramsarefamiliarwithriskassessmentaspartofprogramming–andmanypeacebuildingprogramsassumethattheyareautomaticallyconflictsensitive.TheCDA-ledPQAshelpedtounpackthedistinctionsbetweenconflictsensitivityandpeacebuildingprogramming,astheyprovidedaspaceforconversationsaboutriskmanagement(anentrypointfordiscussionsaboutconflictsensitivity).Theyalsohelpedclarify therequirements forconflict-sensitiveprogrammingandrelatedM&E.

Insight frompractice.TheCDA-ledPQAsandEAsrevealedagreaterneedfor capacitydevelopmentthanwasinitiallyanticipated.OnlyafewindividualswithinthoseteamsandpartnershadpriorexposuretotheRPPandDNHapproachesandtools,andonlyafewhadin-depthM&Eand/orpeacebuildingskills.Allprocessesendedup includingsignificantportionsof capacitydevelopment,which, insomecases,compromisedthedepthoftheactualassessment,asparticipantswerelearningthetoolsatthesametime as engaging on the assessments. Some prior training would have been useful, as would havefactoringinmoretimefortheassessmentsgiventheextensivecapacity-developmentneeds.

26

KeypreparationstepsforaProgramQualityAssessmentbasedonRPPcriteria

Step1:ClarifyoverallpurposeofthePQAwiththeteam.Stressthelearningaspect,thefactthatPQAsarenotevaluations.Clarifywhatwillhappenwiththeresultsoftheprocessandhowthefindingswillbeusedinternally,withpartners,andpossiblywithdonors.IdentifycapacitydevelopmentneedsonRPPandDNHtoolsandapproaches,aswellasM&Eofpeacebuildingskillsaspartoftheprocess.Determinehowmuch is realistic tobedone in termsofcapacity-developmentaspartof thePQA-andwhatneeds tohappenseparately.

Step2:GetSeniorManagementonboardwiththeprocessand/ortoparticipate(atleastinpartsofit).Re-confirmavailabilityofbudgetfortheprocess(stafftime,venue,facilitator).

Step3:DevelopTORforexternal facilitatorandrecruitsomeonewiththerightskillset–e.g.,amixofM&E,peacebuilding,facilitation,andcapacitydevelopmentskills.

Step4: Identifyparticipants in thePQA:programteam,programpartners,donors.Partneranddonorparticipation in a PQA is critical, the type of engagement can vary based on context and needs (e.g.workshopwitheveryone,orworkshopwithsomeandbi-lateralinterviewsandfocusgroupswithothers).Clarify expectationswith the team: an open mind set, active participation, willing to constructivelychallengethemselvesandcolleagues,abilitytoengagewiththeresultsandengageinfollow-up.

Step 5: PQA Facilitator: Review relevant documents, program monitoring data, and conduct selectinterviewswithprogrampartnersandotherlocalstakeholders.

Step6:JointlydeterminewiththefacilitatorwhetherthereareanyparticularelementsofthePQAthatshouldreceiveaparticularfocus.ClarifycapacitydevelopmentexpectationsonRPPandDNHandhowtheywill be addressed through PQA – or a follow-up process. Consult SeniorManagement on thesequestions.

Step 7: Logistics:Make sure sufficient time is being set aside for the PQA and that Sr.Managementreleasesstafffromotherobligationsduringtheexercise.Arrangeforaconducivevenue–ifintheoffice,ensurepeoplecommittoparticipating(asopposedtobeingpulledbacktotheirofficestodootherworkorrespondtoemails)

Step8:ConductPQAthroughafacilitatedworkshopwithallrelevantparticipants,includingadditionalbi-lateralinterviewsasneeded.

Step9:SummarizefindingsfromthePQAincludingafilled-inassessmenttemplate(seeAnnex5),withclear recommendationsonhowto improveprogramquality.Developstrategyonhowto include intorevisedprogramdocumentationandplan,andhowtouseasareferenceforfutureexercisesofthiskind(e.g.updatedanalysis,reformulatedgoalsortheoriesofchange,RPPMatrixetc.).

Step10:Developafollow-upactionplanonhowtousethefindingsfromthePQAsinanongoingprocessof learning, reflection, and program improvement. Determine engagement of partners and donors.Determinefuturecapacitydevelopmentneedsandplans.

27

5.EvaluabilityAssessment(EA)–areyoureadyforanevaluation?

5.1Overview–PurposeofanEvaluabilityAssessment

Thepurposesofanevaluabilityassessmentareto:

§ determinewhetheraprogramcanbemeaningfullyevaluated(isit‘evaluable’?),§ determinetheobstaclestoaneffectiveandusefulevaluation,andhowtostrengthentheprogramto

increaseitsevaluability,§ reviewthecoherenceandlogicoftheprogram,§ clarifydataavailability(qualityandquantityofdataavailable),§ analyzetheextent towhichprogramteamsandprogrampartnersare likely tousetheevaluation

findings,§ determinecapacityandorganizationalsystems’needsinrelationtodatacollectionandanalysis;§ laythefoundationforadaptiveprogrammingandactionplanningwithprogramteamsandpartners,

withaviewtowardstheimpendingevaluation,§ informthedesignoftheimpendingevaluation,andto§ supportstrategicandlong-termlearning.

EAsmaybeparticularlysuitedforlargeandcomplexprograms,whereitwillbehelpfultoclarifywhatwillbeevaluatedandhow.ItallowsanexternalorinternalEAfacilitatortoobservetheprograminaction,engagekey stakeholdersand todemonstrate thevalueofevaluativeprocessesandevaluation to keydecision-makersandprogramstakeholders.

Just like PQAs, EAs do not take the place of an evaluation. However, they are useful precursors toevaluations,especially fordeterminingwhetheranevaluation isworthwhileand feasible,andatwhatstageofprogramimplementation.Evaluabilityassessmentsaskabouttheplausibilityofresultsinorderto assess the plausibility of the theories and assumptions underpinning a program, but donot try toevaluatetheactualresultsachieved.AnEAalsoexamineswhethertheprogramisbeingimplementedasdesigned,i.e.ifthetheoryandprogramimplementationalignandwhetherthedatamonitoringsystemsareappropriateandfunctioning.

Therefore,anEAshouldcoverthefollowingthreeelements,asillustratedinthefollowingdiagram:27

Thethreeelementsarebestunderstoodas interwovenandinterrelated.

Theresultsofanevaluabilityassessmentfallalongaspectrumsuchasthefollowing:28§ Fullyevaluable;§ Mostlyevaluable:canimprove;§ Limitedevaluability:needssubstantialimprovement;and§ Notevaluable.

27Reimann,Chigas,andWoodrow,AnAlternativetoFormalEvaluationofPeacebuilding:ProgramQualityAssessment.Thethree-elementapproachissimilartothatfoundin(UNIFEM2009)28See(InternationalLaborOrganization2011)

Evaluability Assessment

Data Availability

Conduciveness of Context

Program Design

28

Clearly,ifaprogramorprojectwasfoundtobe“notevaluable,”ornotworthyofanevaluationfromapeacebuildingperspective(seeSection5.2),afullevaluationwouldnotbewarranted.Ifaprogramhasbeenfoundtobemostlyorfullyevaluable,theproductofanEAmightbeaprogramevaluationproposalorevaluationplan.Ontheotherhand,iftheresultsindicatedthattheefforthadlimitedevaluabilityorwas mostly evaluable, it might make sense to postpone an evaluation and concentrate instead onstrengthening theprogram inways the EAprocess has suggested are needed in order to increase itsreadinessforevaluation.29

Annex 6 provides a check-list for Evaluability Assessments of Peacebuilding Programs based on RPPcriteria.

5.2KeyelementsofanevaluabilityassessmentbasedonRPPcriteria:

TheEAinvolvesareviewofkeyprogramdimensionsandassessmentagainstthefollowingcriteria:

1. Strengthofprogramdesign;2. Availabilityofdataandinformation;3. Conducivenessofthecontext.

Thebelowtablepointsoutkeyprogrammingdimensionsthatdeterminewhetheraninitiativeis‘readyforanevaluation’and‘worthyofanevaluation’–fromapeacebuildingperspective.

ThiscanbeusedinconjunctionwithAnnex6,whichprovidesa‘trafficlight’checklistinordertodeterminethelevelofevaluabilityofanintervention.

29Reimann,ChigasandWoodrow,EvaluabilityAssessmentsinPeacebuildingProgramming,5

29

EATable1:ReadyforevaluationANDworthyofevaluation(fromapeacebuildingperspective)?

EvaluabilityRubrics

ProgramDimensionsProgramDesign DataAvailability ContextualConduciveness

Readyforevaluation

Cleardefinitionofintendedchangebasedonconflictanalysis Capacitytoprovidedata Adequatesecurityand

access

Thoroughstakeholderidentification Repeatableanddurablebaselinemeasures Availabilityofstakeholders

Plausibletheoriesofchange SMARToutcomeindicators Sufficientresources

Goals&objectivesfitneeds(relevance/appropriateness)

Monitoringsystemlinkedtodecision-makingoncoursecorrectionsorprogramsteering(adaptiveprogramming)

Conflictsensitivity

Participantbuy-in

Worthyofevaluation(fromapeacebuildingperspective)

Important&significantinitiative(i.e.effectonPWL–basedonconflictanalysis-orimportancetothefieldofpeacebuilding)

Necessarydataisobtainableatreasonablecosts

Anticipatedutilityoffindingsisworththeassociatedcosts

BasedonworkMarkRogersdidforCDAin2014,and(Reimann,ChigasandWoodrow2012b)

30

Thefollowingtablesprovidekeycriteria,andexamplesofcommongapsandweaknessesfoundinmanyprograms.

EATable2:ProgramDesign–EvaluabilityAssessment(Criteria1)

Evaluabilitycriteriaforpeacebuilding

programming

Commongaps/

Weaknessesinprogramstrategyandlogic

1 ConflictAnalysis

§ Isconductedatthebeginningofdesignandthroughoutprogramimplementationthroughtheintegrationoftheperspectivesofmultiplestakeholders

§ Identifieskeydriversofconflict,andtherelationshipsanddynamicsbetweenstakeholdersandkeyconflictdrivers.

§ Iswelldocumented,updatedthroughoutprogramimplementation,andaccessibletotheteam

§ Isusedasafoundationforprogrammingdecisions(analysis–strategy/programlink).

§ Noconflictanalysisisconductedasthebasisforprogramming.

§ Onlypartialanalysisisconducted.§ Analysisisnotupdated.§ Othertypesofanalysismistakenas‘conflictanalysis’.§ Analysisistoogeneral(contextanalysis),narrow(e.g.

onlysectorspecific),ortoocomprehensive(e.g.notprovidinginformationonlocallevelconflictdrivers.)

§ Analysisisbiasedanddoesnotincludeamultitudeofperspectivesfromdifferentpartsofsociety.

§ Analysisisnotdocumentedand/oraccessibletoallrelevantteammembersandprogrampartners.

§ Analysisisnotlinkedtostrategy.

2 ProgramGoals

§ Clearlydescribethesocio-politicalchangethatisexpected/desired.

§ Areformulatedinaclearandmeasurableway,arespecificandachievablewithinthedescribedtimeframe

§ Goalsaretoobroad/general,statedatthemacro/visionandnotprogramlevel.

§ Goalsareunclear.§ Goalsaretooambitious.§ Goalsaredefinedasactivities,notasdesiredchange.§ Goalsarenotappropriateforthe(conflict)context.

3 Linkagesbetweenactivities,programgoals,andtheoverallobjective/vision

§ Noexplicitlinksexistoronlyimplicitlinksexistconnectingactivities,programgoalsandtheoverallvision/PeaceWritLarge.

§ Only‘hopelines’exist(linksbetweenactivitiesandoutcomesatvariouslevelsthatarebasedonassumptionsonly).

4 Theoriesofchange(TOCs)

§ Areexplicitlyarticulated§ Areclearandrealistic

§ TOCsareimplicitandnotarticulated.§ TOCsarenotsharedwithintheteamand/orwith

partners.§ TOCSarenotrealisticforthescopeoftheprogram.§ TOCsareunclear/fuzzy/don’texplainhowthebroader

changewillbeachieved.

Insightfrompractice.Manyprogramsfocusonthe‘lowhangingfruit’–theyworkwithpartnerswhowanttoengage(notthe‘hardtoreach’),andonprogrammaticareasfamiliartotheorganization.Itisequallyimportanttoworkonwhatneedstobe‘stopped’–keyconflictdriversanddynamicsidentifiedintheconflictanalysis thatnurturetheconflictonanongoingbasis–ongoingsocialexclusion,inter-ethnictensions,mistrustetc.Hence,itiscriticaltobaseprogramdesignandtheoriesofchangeontheoutcomesoftheconflictanalysistoensurerelevance.

31

EATable3:DataAvailability–EvaluabilityAssessment(Criteria2)

Evaluabilitycriteriaforpeacebuildingprograms Commonevaluabilitygapsandweaknesses

1 Theoriesofchange(TOCs)

§ Areexplicitlyarticulated§ Areclearandrealistic

§ TOCsareimplicitandnotarticulated.§ TOCsarenotsharedwithintheteamand/or

partners.§ TOCsarenotrealisticforthescopeoftheprogram.§ TOCsareunclear/fuzzy/don’texplainhowthe

broaderchangewillbeachieved.§ “Theoriesofchange”indesigndonotmatch

“theoriesinuse”–anEAservesthepurposeofrevealingwhethertherearegapsbetweenwhatwasdesignedandwhatisbeingimplemented.

2 Baselines

§ Arecompleted,basedonconflictanalysis§ Areadaptedtothecontext,e.g.recognizing

volatilityinconflictcontexts

§ Baselinesarenotbasedonfindingsfromconflictanalysis.

§ Baselinesareincomplete.§ Baseline’snotdefinedinawaythatwouldallowfor

monitoringofchangesinthebroaderconflictcontext.

3 ProgramMonitoringSystem

§ Isestablishedwithclearresponsibilitieswithinandacrossprogramteams

§ Issetuptoinformchangesinprogrammingandsupportflexibleprogramming

§ Measuresbroaderchangesinconflictcontext§ Monitorsunintendedimpactsoftheprogram.

§ MonitoringSystemsareconsideredonlyafunctionofprogrammanagementanddonoraccountability–notasakeyelementofadaptiveandflexibleprogramminginconflictcontexts.

§ Insightsfrommonitoringdonotinformdecision-makingandprogramchanges.

§ Aggregatedanalysisofmonitoringdataflawedornon-existent.Trendsanalysisnotcommunicatedtokeydecision-makersintimelyandactionableway.

§ Monitoringsystemonlymonitorsdirectprogramresults,andnotlargerimpactsonconflictcontext.

§ Monitoringsystemdoesnotmeasureunintended(negative)impacts(conflictsensitivity).

4 Indicators§ Areclear,realisticandmeasurable§ Arequalitativeandquantitative§ MeetsSMART30criteria,especiallyatthe

outcomelevel§ Expressthebroaderchangeintheconflict

contextthattheprogramintendstoachieve§ Aresetuptomeasureconflictsensitivity

considerations

§ IndicatorsarenotdesignedbasedonSMARTcriteria(especiallyattheoutcomelevel).

§ Indicatorsareunrealistic,un-measurableorunclear.§ Noindicatorsexist.§ Onlyquantitativeindicators.§ Arenotdesignedtocapturethebroaderchangeon

theconflictcontext.§ Donottakeintoaccountmeasurementof

unintended(negative)impacts(conflictsensitivity).

5 Accesstostakeholders§ Forprogramdataverification§ Forongoingfeed-backfromprogram

participantsandpartners

§ Noaccesstoprogrampartnersandotherlocalstakeholders.

§ Onlyaccesstothe‘easytoreach’.§ Nomechanismsinplacetocreativelymanage

remoteimplementationscenarios.

30SMART=specific,measurable,achievable/attainable,result-oriented,andtime-bound.FormorebackgroundonSMARTindicatorsseehttp://dmeforpeace.org/sites/default/files/3.9%20Indicators.pdf

32

Evaluabilitycriteriaforpeacebuildingprograms Commonevaluabilitygapsandweaknesses

§ Toassessbroaderchangesintheconflictcontextandpossibleunintended(negative)impacts

§ Tojointlyagreeonprogramchangesandadaption

§ Noongoingfeed-backmechanismsinplacewithpartnersandstakeholders.

EATable4:Conducivenessofthecontext–EvaluabilityAssessment(Criteria3)

Evaluabilitycriteriaforpeacebuildingprograms Commonevaluabilitygapsandweaknesses

1 Generalconditionsarefavorable(weather,security,availabilityofstakeholders,currentpoliticaleventsetc.)

§ Limitedaccessb/cofweatherorsecurityrisks.§ Currentpoliticalclimateand/oreventsimpede

access.§ Restrictedaccesstoprogrampartnersand/or

localstakeholders

2 Financialresourcesavailabletoconductevaluation,includinglogistics

§ Evaluationplannedasan‘afterthought’andnotasakeyelementoftheprogram,leadingtolimitedresourcesbeingavailable.

§ Limitedfinancialresourcesimpacttherangeofstakeholdersbeingconsulted(e.g.inremotelocations).

3 Availablestaffcapacitiestoparticipatein,shape,andfollow-uponanevaluation

§ Limitedstaffawarenessaboutkeyelementsneededtomakepeacebuildingprogramsevaluable(conflictanalysis,DNHprinciples,basiscriteriaofstrategydesignandprogramlogic,theoriesofchangeetc.).

§ Nocapacitytofollow-uponimplementationofevaluationfindings.

4 Commitment/Internallydrivenprocess § BothEAsandevaluationsoftendonor-driven,ordrivenbySr.Managementwithoutbuy-infromstaff.

§ Partnersnotinvolved.§ UnderstandingofwhatanEAorevaluationis,

andhowthefindingswillbeusedvaryandimpactthesuccessoftheEAorevaluation.

5 Evaluationprocessdesignedinaconflict-sensitivemanner

§ Conflict-sensitive/DNHprinciplesarenotappliedtothedesignoftheEAortheevaluationprocess,e.g.inrelationtotiming,whoisinvolved/selectionofinterviewees,locationofEAand/orevaluationetc.

33

KeypreparationstepsforanEvaluabilityAssessmentforpeacebuildingprograms

Step1:ClarifyoverallpurposeoftheEAwiththeprogramteamandmanagement.Clarifywhatwillhappenwiththeresultsoftheprocess andhowthe findingswillbeused internally,withpartners, and possibly with donors. Identify capacitydevelopment needs on RPP and DNH tools, on M&E ofpeacebuilding, and approaches as part of the process.DeterminehowmuchcapacitydevelopmentcanbedoneaspartoftheEA–andwhatneedstobeachievedinaseparateprocess.

Step2:ClarifythetypeofevaluationthattheEAprecedes.

Step3:GetSeniorManagementonboardwiththeprocessand/ortoparticipate(atleastinpartsofit).Re-confirmavailabilityofbudgetfortheprocess(stafftime,venue,facilitator).

Step3:DevelopTORforexternalfacilitatorandrecruitsomeonewiththerightskillset–i.e.,mixofM&E,peacebuilding,facilitation,andcapacitydevelopmentskills.

Step4:IdentifyparticipantsintheEA:team,partners,donors.Programpartnersandlocalpartnersneedtobeengagedwithacertainlevelofindependence.EngageprogramstaffandpartnersinarticulatingEAquestionstobuildownershipandcapacity.Clarifyexpectationswiththeteam:anopenmindset,activeparticipation,willing toconstructivelychallenge themselvesandcolleagues,ability toengagewith theresultsandengageinfollow-up.

Step5:EAFacilitator:Reviewrelevantdocuments,programM&Esystem,availableprogramdata.

Step6:JointlydeterminewiththefacilitatorwhetherthereareanyelementsoftheEAthatshouldreceivea particular focus. Clarify capacity development expectations on RPP and DNH and how theywill beaddressedthroughEA–orafollow-upprocess.ConsultSeniorManagementonthesequestions.

Step7:Logistics:MakesuresufficienttimeisbeingsetasidefortheEAandthatSr.Managementreleasesstafffromotherobligationsduringtheexercise.Arrangeforaconducivevenue–ifintheoffice,ensurepeople commit toparticipating (asopposed tobeingpulledback to their offices todootherworkorrespondtoemails).

Step8:ConductEAthroughatwo-stepprocess:i)individualinterviewsconductedindependently(e.g.,aswouldbedoneforanevaluation),and(ii) facilitatedworkshopwithall relevantparticipants, includingteam,partnersanddonorstoreviewthethreekeyareasofanEA(programdesign,dataavailability,andconducivenessofcontext).

Step9:SummarizefindingsfromEAincludingafilled-inEAcheck-list(seeAnnex6)

Step10:DevelopstrategyandimplementationplanonhowtousefindingsfromEA:Istheprogramreadyforaformalevaluation?Forwhattypeofevaluationandwhen?Ifnot,whichprogramelementsneedtobe strengthened for the program to become ready? Determine engagement of partners and donors.Determinefuturecapacitydevelopmentneedsandplans.

Moreresourcesonethicalandconflict-sensitivityquestionsinrelationtoformalevaluations:

® dmeforpeace.org/evaluationunder"Whatareethicalandconflict-sensitivityissuesinevaluation,andhowdoImanagethem?”

34

Insightfrompractice.Clarifyingthepurposeoftheassessmentprocessandhowtheresultswillbeusediscritical.Often,programteamsandprogrampartners arehesitant todiscussdifficulties inprogramstrategyandimplementation–andthereissignificantfearwithinteamsaboutmakingmistakes.Thisisa challenge especially in organizations that do not have an explicit culture of learning or adaptivepractice–orothertypesofstaffincentivesthatarecontrarytoalearningculture.Theremightbefearof negative repercussions, or negative reactions from donors if challenges are discussed openly.Ensuring that teams (including local partners) understand that the focusof theseassessments is onlearningandnotonaccountabilityiscritical.SeniorManagementbuy-intocommunicatetheprocessofhowfindingswillbeused,includingrelatedfollow-upprocesseswithbothprogramstaffandprogrampartners, iscritical inthisregard.Alloftheseoptionsrequirestrongsupportfromtheteamandlocalpartners,andshouldnotbeconductedifthereismajoruncertaintyaboutthepurposeand/ornatureoftheexercise.

35

6.StrategyandProgramReflectionExercise

6.1Overview–PurposeofaStrategy/ProgramReflectionExercise

ComparedtoaProgramQualityAssessmentorEvaluabilityAssessment,aStrategyorProgramReflectionExerciseisoftenamoreinformalandinternalprocessthatcanbeconductedatprettymuchanymomentofprogramdesignandimplementation.

ThepurposesofaStrategyandProgramReflectionExerciseareto:

§ improvespecificelementsofprogramstrategy,design,andimplementation,

§ maximizepotentialforprogramtocontributetoPeaceWritLarge,andto

§ contributetoacommonunderstandingwithintheteam,andpossiblyprogrampartners,aboutkeyelementsofeffectiveandrelevantpeacebuildingprogramming.

6.2KeystepsoftheStrategyandProgramReflectionExercisebasedonRPPcriteria

ReflectionExerciseStepsPossibleimplicationsofthisstepduring

Strategyand/orProgramDesign

Possibleimplicationsofthisstep

duringImplementation

Step1:Reviewconflictanalysis Domoreanalysisifconclusionisthatconflictanalysisisnotgoodenough.Possiblyadaptprogramdesignifcontexthaschangedrecently.

Updateconflictanalysisandpossiblymodifystrategyandactivitiesifcontexthaschanged.

Step2:Reviewprogramgoal Possiblymodifyprogramgoal(andrelatedactivities).

Reviewandpossiblymodifygoal-activityalignment(istheprogrammakingprogresstowardsthegoal?).Possiblegoalre-designed(e.g.inchangingcontext)?

Step3:Identifyprogramactivities,intendedchanges,andtheoriesofchange

Jointarticulationofkeycomponentsoftheprogram(activities,theoriesofchange).Validateassumptionsandjointapproach.

Takestockofpossibledifferencesbetweendesignandimplementation:aretheredifferencesandwhy,doestheteamhavethesameunderstanding?

Step4:Plottheprogramgoal,activities,andchangesontotheRPPmatrix

DesignprogramstrategyusingtheRPPmatrix.

ReviewprogramstrategyandpossiblyupdateitusingtheRPPmatrix.

Step5:Assesstheprogram’stheoriesofchange

Developrealistictheoriesofchange;revealunderlyingassumptionswithintheteamandwithpartners.

Review&possiblymodifytheoriesofchange

Step6:Exploreprogramlogic Defineandtestdesignlogic. Reviewandpossiblyadaptprogramlogic.

Step7:Assessconflict-sensitivedesignand/orimplementationoftheengagement

Designconflictsensitivitymechanismsfortheengagement.

Reviewwhetherexistingconflictsensitivitymechanismsareworkingandappropriate,orwhetheradaptionisrequired.

Step8:Reflect&recommendchangesinprogramdesignand/orimplementation

Agreeondesignchangesandrelatedresponsibilities,involvingteam,partners,anddonors.

Agreeonchangesrequiredandrelatedresponsibilities,involvingteam,partners,anddonors.

36

Noteveryprogramneedstoexamineallthesesteps.Determinewhetherallstepsneedtobeperformed,orwhichonesshouldbeselected.Thebelowsectionillustrateskeyquestionsthatprogramteamsandpartnerscanusetoreflectoneachoftheabovesteps.

ReflectionExercise–Step1:Reviewtheconflictanalysis

Programdesignandperiodicreviewshouldbebasedonanup-to-dateconflictanalysis.Thisstepisaimedatensuringthe“relevance”oftheprogram—thatis,whetheritisworkingontherightissuewiththerightpeopleattherighttimeusinganappropriatemethodology.

§ Istheavailableanalysiscurrent(withinafewmonths)? Hasthesituationchangedsignificantlysincepreviousanalyses? Is theavailableanalysis in factageneral ‘context’analysis rather thanaconflictanalysis?

§ Doestheanalysisidentifythekeydrivingfactorsofconflict(bothissuesandpeople)?[Asareminder:Keydrivingfactorsarefactorswithoutwhichtheconflictwouldnotexistorwouldbesignificantlydifferent.Keyactorsarepeopleorgroupsthatcansignificantlyinfluencetheconflictdynamics.]

§ Doestheanalysisexploreordepicttherelationshipsamongfactorsand/oramongfactorsandactors?

§ Doestheanalysisidentifyactualorpotentialfactorsforpeace?Whataretheforcesinthesituationthatcanbebuiltupontopromotemovementtowardspeace?Whatconnectspeopleacrossconflictlines?Whoexercisesleadershipforpeaceandhow?

§ Whatneedstochange?Whoorwhatneedstochangetotransformanegativeanddestructivedynamicintoamoreconstructiveone?Doestheanalysisidentifywhatmustabsolutelybestoppedorreducedbeforepeaceeffortshaveachancetomakeanimpact?

§ Doestheanalysisconsiderpastandongoingeffortsatpeace,andwhatcanormustbelearnedfromtheirperceivedeffectiveness—orlackthereof?

§ Doestheanalysissuggestpossiblepointsofleveragetocreatechangeinconflictdynamics?Giventhenatureofyourorganization,whatdoyouseeaspointsofleverage,andwhy?Toeffectchange,willyoureffortsalonebesufficient,ordotheyneedtobesupportedandcomplementedbyotherefforts?Ifso,bywhom?

§ If theprogram isalreadybeing implemented:Hasanupdatedconflictanalysisbeenperformed?Doesthebasicapproachorprogramfocusneedtochangeasaresult?

WhoseAnalysis?Ajointprocessofconflictanalysisandrelatedreflectionwiththeteamandprogrampartnershelpstoestablishacommonunderstandingaboutthekeydriversofconflict–andwheretheremightbedifferenceswithintheteamandwithpartnersonwhatthespecificconflictdynamicsare,andhow that difference in understanding might impact program design and implementation. Often,differencesinvisionaboutthelargerchangeaprogramintendstoachieve(workingtowardsPeaceWritLarge)isbasedonalackofsystematic,andsharedanalysis–eventhoughpeopleoftenassume‘theyareonthesamepage’,whentheyactuallyhaveadifferentunderstandingof‘theproblem’.Agreeingontheunitofanalysisisequallyimportant:doteamandpartnersneedabroadmacro-levelviewofnationalconflictdynamicsoristheanalysisofasub-regionmoreimportant?Wouldasystemicconflictanalysishelptore-assessandre-confirmleveragepointsforchangeandprogrammaticentrypoints?

.

37

ReflectionExercise–Step2:Reviewtheprogramgoal

Programgoalsshouldbearticulatedasintendedchangesatanappropriatelevelofambition!

§ Istheprogramgoal,asstated,‘robust’?Isitchange-oriented,realistic,timeconscious?

§ Istheprogramgoaltoogeneral(atthelong-termvisionlevel)ortooambitious(over-claiming)?Oristheprogramgoaltoospecific—thatis,isitmorelikeanactivity?

§ Howdoestheprogramgoalrelatetothedynamicsofconflictandpeaceasanalyzed? If there isnoapparentlinktothekeydriversofconflictandpeace,theprogrammaybeworthwhile,butwillnothaveapeacebuildingimpact.

§ Iftheprogramgoalenvisionschangeatthelocallevelorofanon-criticalcomponentoftheconflictandpeacedynamics(peacewritlittle),howmightitcreatelinkagestowiderpeaceatthesocietallevel(PeaceWritLarge)intheparticularcontext?

§ Iftheprogramgoalisstatedas‘acontributionto[anelementofpeace]’,isitclearhowitwillmakesuchacontributionandhowtoobserveormeasureit?

§ Doestheprogramgoalseekchangesatthesocio-politicallevel?Peaceeffortsthatfocusonlyonchangeattheindividual-personallevelanddonotlinkthoseeffortstochangeatthesocio-politicallevelwillhavenodiscernibleimpactonpeace.

§ How does the program goal relate to any of the RPP Building Blocks for Peace (‘Criteria ofEffectiveness’)?

§ Ifaprogram isalreadybeing implemented: Is theprogrammakingreasonableprogress towardsthegoal?Whatkindoffeedback(monitoringdata)indicatessuchprogress?Isthegoalstillappropriate—orhasthesituationchangedsignificantly,requiringredesign?

ReflectionExercise-Step3:Identifytheprogramactivities,intendedchangesandtheoriesofchange

ThisstepusestheRPPprogramplanningchartshowingactivities,expectedchanges,theoryofchange,andotherassumptions(seeAnnex3).

§ Inthetoprowofthechart,enterthegoal/objectiveoftheprogram/projectandtheassociatedoveralltheoryofchangeatthislevel.

§ Identifyfiveorsixkeyactivitiesinthisproject/program?Entertheminthefirstcolumnoftheprogramplanningchart.

§ Intheothercolumns,identifytheactualorexpectedchangefromeachindividualactivity,aswellasthetheoryofchangeandotherassumptionsassociatedwiththeactivity.

ReflectionExercise-Step4:UsetheRPPMatrixtoassessprogramstrategy:Plottheprogramgoal,activities,andchangesontotheRPPmatrix(seeAnnexes2.1and2.2)

§ Startwith locatingthegoalontheRPPmatrix. Is itattheIndividual/Personal levelofchange,ortheSocio-Politicallevel?Isitmoreintherealmof‘MorePeople’or‘KeyPeople’?

§ Plot the program activities and their intended changes onto the matrix. Use different visualidentifications for ‘activities’ and ‘changes’. (Note: for a program already being implemented,

TheMovingGoalPost.Oftentherearedifferentviewswithinaprogramteam(includinginternationalstaff,localstaff,partners,localstakeholders)onwhattheprogramgoalis–andhowbesttoachieveit.Adeliberateprocessof jointreflectionandgoalarticulationhelpstoputdifferentunderstandingsonthetableandtodevelopconsensusonarealisticandachievablegoalwithintheavailablescopeofwork.Inputandguidancefromthosewithanin-depthunderstandingofthelocalcontextiscritical.

38

completed activities and actual results can be plotted, as well as further planned activities andexpectedresults.)

ReflectionExercise-Step5:Analyzetheprogram’stheoriesofchange

Analyzing theprogramtheoryofchangewillhelp toseehowtheconflictanalysis isconnected to theprogramgoal,andultimately,toPeaceWritLarge.Itwillalsohelptheteam,partners,anddonorsseewhethertheinitiativeisonitswaytocontributingtoPeaceWritLargebeyondthelifeofprogram,andwhetherthereareanyassumptionsmadethatwouldneedtobeaddressedintheprogramstrategy.

§ Arethetheoriesofchangeappropriateandrealisticinthecontext?Willchangecomeaboutinthewaysenvisionedasaresultoftheplannedactivities?

§ How would successful achievement of the program goal make a significant contribution to therealizationofPeaceWritLarge?Whatisthetheoryofchangeatthislevel?

§ If the program is already being implemented: Have the activities completed so far resulted in theexpected changes? Are there any unexpected positive or negative outcomes? Are the theories ofchangeprovingviableinthecontext—orisrethinkingindicated?

ReflectionExercise-Step6:Exploretheprogramlogic

This sectionuses theRPPprogramplanningchart,and theRPPMatrix together (Annexes3and4). Inadditiontoreviewingthetheoryofchangeassociatedwithindividualactivitiesinthepreviousstep,itisimportanttomakesurethattheoverallprogramwilladduptotheintendedgoal.

§ Examinethelogicbetweentheactivitiesandthegoal.Wouldachievementoftheactivitiesleadtothegoal?Isanythingmissing?

§ Are there unexamined assumptions underlying the links between the different activities, such aswillingness,availability,externaleventsetc.?

§ Whatkindsofobstaclesmighttheprojectencounterinitsimplementation?Who/whatmightgetintheway?

§ Are“hopelines”revealedontheMatrixorinthelogicpresentedintheFour-ColumnChart?(“Hopelines”are leaps in logicorgapsbetweenactivitiesanddesiredresults—depictedontheMatrixbydottedlines.)Howmighthopelinesbeconvertedtodesiredchanges?

§ If the program is already being implemented: Is the program on track to achieve itsgoal/objective? Have new gaps in program logic or other obstacles appeared during

ActivityDisharmony.Staffandprogrampartnersoftenworktowardanagreed-upongoalinsurprisinglydifferent ways. This compromises program quality and causes friction within a team, and possiblyconfusionwithpartnersandwithdonors.Often,theunderlyingreasonisadifferentunderstandingoftheories of change. Articulating the different assumptions team members, partners, other localstakeholders,anddonorsmakeonhowtoachievetheenvisagedchangehelpsteamsandallpartnersinvolvedmoveinasimilardirection.

TheBusinessasUsualTrap.Manyprogramsareexcellentatimplementingcertainstrategies.Butoftentheworkhastakenona“cookiecutter”pattern,wherebydifferentproblemsarealltackledviasimilarstrategiesandactivities(e.g.dialogueeffortsasa‘recipe’foreverything),ratherthanbeingaddressedbytailoredinterventionsspecifictothe‘problem’orthecontext-specificconflictdriver.Reviewingtheunderlying analysis and related key drivers of conflict, as well as using the RPP Matrix to revealassumptions,linkages(andthelackthereof)inprogramlogichelpstorevealhowtheprogramcanbemademorerelevanttotheconflictcontext.Inclusionoflocalpartnersinthisprocessiscritical.

39

implementation, requiringadjustments in the futureplannedactivitiesoranewapproach?

ReflectionExercise-Step7:Assessingconflict-sensitivedesignand/orimplementationoftheinitiative.Istheinitiativeawareoftheactualorpotentialunintendednegativeimpactsitmightcause?

§ Hastheprogramdesignexaminedcommoncausesofunintendednegativeeffects,suchase.g.thechoiceofprogrampartners,contractors,suppliers,locationoftheengagement,distributionofbenefits,timingoftheprogramming?

§ Does the initiative consider conflict-sensitivity considerations (mainly possible unintended negativeimpacts)initsM&Esystem?

§ Dostaffandpartnershaveskillsinconflict-sensitiveprogramimplementationand/orbeentrainedinDoNoHarmapproaches?

ReflectionExercise-Step8:Reflect and recommend changes in programdesign and/orimplementation

The fundamental purposeof this reflection exercise is to strengthenprogramdesignor to encouragechanges indirectionor implementation.

§ Whatinsightshaveyougainedregardingthisprogram?Whatchallengeshavebeenraised?

§ Basedonthepreviousstepsandassociatedreflections,howmightthisprogramoritscontinuationbestrengthenedoritsconceptsfurtherelaborated?

§ Doesthisprogramneedtolinkmoreactivelywithotheragencies,withotherefforts?Whichonesandwhy?

§ Reflectonotherdimensionsofthisprogram,consideringthecontextandwhatyouknowaboutthepeaceeffortsofotheractors/agencies:

§ Is thisprogram ‘big’ enough,does it haveenough ‘scale’ tobeable tohave somemeaningfulinfluence?Why?

§ Isthisprogrammovingattherightpace,nottoofastandnottooslow?Why?

§ If thisprogramachievesmeaningful influenceand impact, can thisbe sustained?Whyandbywhom?

§ Ifyouareproposingchangestotheprogramdesignorimplementationprocess,areyouconfidentthattheredesignedprogramwillindeedbestrongerormoreeffective?Why?

§ Willitbenecessarytoseekapprovalforprogramdesignchangesfromothersintheorganizationorfromaprimarydonor?Whatwillbethebeststrategyforgainingdonorapprovalforchanges?

§ Finally,assessthisReflectionExerciseitselffromaprocessperspective.Whatwashelpfulornotsohelpful?Whatsuggestionswouldyoumakeforimprovingtheprocess?

The peacebuilding and conflict-sensitivity confusion. Often, there is an assumption that allpeacebuildingprogrammingthatfocusesonaddressingkeydriversofconflictdirectlyisautomaticallyconflictsensitive.Thisisnotthecase.Inanytypeofprogramming,beitdevelopment,humanitarian,orpeacebuilding,thedetailsofaninterventionmatterfromaconflictsensitivityperspective.Theactionsand behaviors within programs, as well as programs themselves can have significant unintendednegativeconsequencesandimpactsontheconflictcontext.It’simportanttoconsidercriticalprogramdetailssuchaschoiceofpartners,location,timing,orchoiceswithinprocurementandhumanresources.

40

KeypreparationstepsforaStrategy/ProgramReflectionExercisebasedonRPPcriteria

Step1:Clarifyoverallpurposeofthereflectionexercisewiththeteam.Stressthelearningaspectofastrategy/programreflectionexercise–theyarenotassessmentsorevaluations.Clarifywhatwillhappenwiththeresultsoftheprocessandhowthefindingswillbeusedinternally,withpartners,andpossiblywithdonors. IdentifycapacitydevelopmentneedsonRPPandDNHtoolsandapproaches,aswellasM&Einpeacebuildingtools,aspartoftheprocess.DeterminehowmuchcapacitydevelopmentcanbedoneaspartoftheStrategy/ProgramReflectionexercise.

Step2:GetSeniorManagementonboardwiththeprocessand/ortoparticipate(atleastinpartsofit).Re-confirmavailablebudget–costsforstafftime,venue(possibly).

Step3:Nominatefacilitatorandclarifyhis/herrole&TOR(inmostcasesaninternalfacilitator,butcanbeexternal).ClarifyinformationneedsofthefacilitatorbeforetheReflectionExercise(documents,bi-lateralconversationsetc.)

Step 4: Identify participants in the Strategy/Program Reflection exercise: team, partners, donors.Programreflectionexercisesmightbebestconductedwiththeteaminternally iftherearesignificantgapsinunderstandingoftheoverallprogramwithintheteam,differenttheoriesofchangeetc.Inclusionof localpartnersand stakeholders,on theotherhand, is critical for anuancedunderstandingof thecontext,overalladviceonfeasibilityoftheprogram,anddevelopmentofajointvisionbetweenprogramstaffandprogrampartners. Itneeds tobedetermined ina context-specificwaywhat levelofdonorengagementisdesired–e.g.,participationofdonorsinpartsoftheprocessaftertheteamhashadanopportunitytodiscussinternallyandwiththeirlocalpartners.Clarifyexpectationswiththeteam:anopenmindset,activeparticipation,willingtochallengethemselvesandcolleaguesconstructively,abletoengagewiththeresultsandengageinfollow-up.

Step5:ReflectionExerciseFacilitatorandprogramteam:Reviewrelevantprogramdocumentation.

Step6:Determine(jointlybetweenfacilitator,teamandpartners)whichelementsoftheprogrammightrequirespecificattentionduringtheStrategy/ProgramReflectionExercise.ClarifycapacitydevelopmentexpectationsonRPPandDNHandhowtheywillbeaddressedthroughthestrategy/programreflectionprocess–orafollow-upprocess.ConsultSr.Managementonthisquestion.

Step7:Arrangeforconducivelogistics:MakesuresufficienttimeisbeingsetasidefortheexerciseandthatSr.Managementreleasesstafffromotherobligationsduringtheexercise.Arrangeforaconducivevenue–ifintheoffice,ensurepeoplecommittoparticipating(asopposedtobeingpulledbacktotheirofficestodootherworkorrespondtoemails)

Step 8: Conduct Reflection Exercise through a facilitated workshop with all relevant participants,includingadditionalbi-lateralinterviewsasneeded.

Step9:SummarizefindingsfromtheReflectionExerciseinashort,action-orientedformat.Capturekeyoutcomesfromtheworkshoptofeedintorevisedprogramdocumentationandasareferenceforfutureexercisesofthiskind(e.g.updatedanalysis,reformulatedgoalsortheoriesofchange,RPPMatrixetc.).

Step 10: Develop a follow-up action plan on how to use thefindingsfromtheReflectionExercise.Inanongoingprocessoflearning, reflection, and program improvement. Determineengagementofpartnersanddonors.Determinefuturecapacitydevelopmentneedsandplans.

Moreinformationonpreparationstepsforformalevaluation,bymeansofcomparison:

® dmeforpeace.org/evaluation-planning

41

7.AdaptivePeacebuildingProgrammingthroughEffectiveFeedback

Loops

Acrosstheinternationalaidsector,thereisagrowinginterestinevaluativethinkingandfeedbackloops.Thereisalsoanincreasingcommitmenttoengagefrontlinestaff,localpartners,communitymembersandprogramparticipantsas“users”ofdata,informationandlessons.Asreflectivepractitioners,ifwetrulystriveforrapidfeedbackandlearningcycles,weneedskillstobeabletothinkincriticalwaysaboutwhatwe do, and observe and learn in the course of designing and implementing programs and initiatives.Listeningbroadlyandintentionally,andsolicitinglocalanalysisandfeedbackareallcriticalrequirementsfor organizations supporting local peacebuilding efforts. An effective feedback process goes beyondcollecting participant satisfaction data using pre-determined questions. Well-designed and context-appropriatefeedbackprocessesprovideanopportunitytoregularlyengagekeyprogramconstituentsinevaluativeconversationsabouttheassumptionsthatunderpininterventions,toquestionthevalidityoftheseassumptionsandprogrammaticchoicesandtooffersuggestionsforprogramqualityimprovementduringimplementationandevaluationphases.

CDA’sresearchoneffectivefeedbackloopsandfactorsthatenablefeedbackutilization31hasproducedcase studies and evidence-based practical guidance to assist donors and operational agencies in thehumanitarian anddevelopment fields. CDAhas identified several problems that apply equally to thepeacebuildingfield:

§ Therearefrequentlymissedopportunitiesforengaginglocalpeople,notonlythroughlisteningto their perspectives, but also by engaging them in analysis of feedback and identification ofprogramchanges.

§ Staff and local partners often need enhanced individual skills in active listening, appreciativeinquiry,anddataanalysis.

§ Skillsandprocessesarenotenough;institutionalstructures,decisionmakingthatincorporatesfeedback,managementsupportandincentivesarenecessary.

§ Increaseduseoftechnologycansupportsustainedandreal-timefeedback loops,oftenasoneelementofanM&Esystem.Buttechnologydoesnotaddressliteracylevels,culturallyappropriatemonitoring/feedbackprocesses,language,andwhoisengagedandwhoisleftout(bygender,age,elitesvs.marginal/vulnerablestatus,etc.)

In2016-2017,CDAiscollaboratingwithpeacebuildingorganizationstodocumenteffectivepracticeswithfeedback and how it supports adaptive programming through improved monitoring and evaluationpractice.

Feedback mechanisms can serve as a powerful means to support evaluative processes such as

Evaluability and ProgramQuality Assessments. For peacebuilding programs, feedback loops broadlyservetwomainpurposes:programqualityimprovementandparticipatorycontextmonitoring.

Programqualityimprovement:Solicitedandunsolicitedfeedbackcanbegatheredtoinformreal-timeprogram improvements and adaptation. To this end, feedback is sought on the quality andappropriatenessofprograminterventions,staffperformanceandprogramresults.Feedbackisgatheredaspartofroutinemonitoringprocessesorbyestablishingadditional,accessibleandconfidentialfeedback

31“EffectivelyUtilizingFeedback,”CDACollaborativeLearningProjects,http://cdacollaborative.org/cdaproject/effectively-utilizing-feedback/.

42

channels.Suchpracticeshavebecomecommonplaceinmanyhumanitarianoperations,particularlytheuseofcomplaintsresponseandgrievancemechanisms.32Challengesremaininactiveconflictareaswhererestrictedaccessandsecurityconcernsprohibitregularcommunicationchannels.

Inadditiontofeedbackonprogramquality,localviewscanbesoughtonbroader,strategicareastoinformstrategyreviewforprograms,sectorsorcountryspecificpolicies.Bothprogram-levelandstrategy-levelfeedbackloopsrequireanintentionalandpurposefulapproachtofeedbackcollection,acknowledgement,analysisandmakingsenseofdata,opinionsandperceptions.ProgramQualityAssessmentscanbenefitfromthisadditionaldatacollectionmethod.PQAfacilitatorscanexamineaccumulatedfeedbackgatheredonprogramqualityanduseformalandinformalfeedbackchannelstogatheradditionalperceptionsfromawiderrangeofstakeholders,toincludeinformedobservers.

Participatorycontextmonitoring.Organizationsseekingtoidentifyunintendedandunanticipatedeffectsof their programs need to establish feedback channels and practices that reach beyond their targetprogramparticipants.Thisisparticularlytruewhenover-relianceonindicator-basedmethodologiescanresultinoversightofdynamicchangesinthecontextandaprogram’seffectsonthelocalcontext.Well-functioningfeedbackmechanismscancaptureandrespondtoreal-timeinformationaboutunintendedimpactsofprogramsoninter-groupandintra-grouprelationsandavoiddoingharm.

Afunctioningfeedbackloopgoesbeyondfeedbackcollectionandanalysis.Itrequiresaresponseand/orcorrectiveaction,inotherwords--theclosingoftheloop.33Ultimately,forfeedbackloopstobeeffective,thefeedbackneedstobeutilizedindecision-makingandevaluativeprocesses.

32See(CDACollaborativeLearningProjects2011)33See(Bonino,JeanandKnox-Clarke2014)and(Bonino,JeanandKnox-Clarke2014b)

43

8.SummaryOverviewofOptionscoveredinthisGuide

TheGuide provides a selectmenuof three options for reviewing and strengthening of peacebuildinginterventions,namely:

1. ProgramQualityAssessments;2. EvaluabilityAssessments;3. StrategyandProgramReflectionExercises.

Thetableonthenexttwopagesprovidesanoverviewoftheobjectives,characteristics,purposesandbenefitsofeachoption.

44

ComparisonofcharacteristicsoftheStrategyandProgramReflectionExercise,PQA,andEAvis-à-visEvaluation

StrategyandProgramReflectionExercise ProgramQualityAssessment(PQA) EvaluabilityAssessment(EA) Evaluation(notcoveredinthisGuide)

Purpose(s)

§ Improvespecificelementsofprogram

strategyordesign

§ Maximizepotentialforprogramto

contributetoPeaceWritLarge

§ Contributetoacommonunderstanding

withintheteam(andpossiblywith

partners)aboutkeyelementsof

effectiveandrelevantpeacebuilding

programming.

§ Learnaboutqualityofprogram

design

§ Adapt/improveprogramdesign

andimplementationthrough

applicationofaclearsetof

professionalstandardsand

criteria

§ Maximizepotentialfor

programtocontributetoPeace

WritLarge.

§ Assesswhetheraprogramisreadyforaformal

evaluation

§ Identifyareasforimprovementin(specifically)

datacollection,programlogic,andsupport

evaluationplanning.

§ Identifyandassessworthof

results/outcomesofprogram

§ Learnaboutqualityandvalueof

program,includingareasandoptions

forimprovement(ifformative

evaluation)

§ Fulfillobligationsofaccountability(to

donors,toparticipants,communities,

ororganizations,etc.)

Timing

§ Duringprogram/designphase

§ Atkeymomentsduring

implementation.

§ Mid-term

§ Limiteduseatthebeginningof

program,butcanbeusedto

validatetheoriesofchangeand

programstrategywithteams

andprogrampartners

§ Possiblyusefulatendofa

projecttodrawlessonsfor

futureengagement.

§ Beforeaformalevaluation-ideallyonceitis

knownwhattypeofevaluationwillbe

conducted

§ Mid-termreviewstagetoidentifyareasfor

improvementbeforeconductinganevaluation.

§ Atendofprojectorprogram

(summative)or

§ Mid-term(formative,developmental)

or

§ Throughout(developmental)

AssessmentCriteria

§ Notan‘assessment’

§ RPPandDoNoHarmconceptsand

toolsrelatedtoimpactsonPeaceWrit

Largeandconflict-sensitivedesign&

implementation.

§ Qualityanduseofconflict

analysisinprogramming

§ Articulationofprogramgoals

§ Theory/iesofchange

§ Programstrategy&logic

§ StrengthofM&Esystem–

relevancefromapeacebuilding

perspective

§ Applicationofconflict

sensitivity.

§ Qualityofprogramdesign

§ Conducivenessofcontext

§ Dataavailability.

§ OECDDACevaluationcriteria(impact,

relevance,sustainability,efficiency,

andeffectiveness)

§ Contextually-relevantstandardsof

achievementsetbyprogram/project

DataNeeds

§ Reliesonknowledgeandexperienceof

participants

§ Generalunderstandingoftheoverall

(conflict)contextisimportant.

§ Reliesonprogram’smonitoring

data,documentreviewand

someinterviewswithprogram

teamandpartners.

§ Publiclyavailablerelevantdata

(e.g.,violencestatistics,or

externalconflictanalyses).

§ Reviewstheavailabilityofdata,including

baselines(basedonconflictanalysis)

§ ReviewsstrengthofM&Esystems,relevanceof

collecteddata,accesstostakeholderstocollect

data.

§ Checklistsandsimplepresence/absencetests.

§ Testing,smallscalesamplingwithoutregardto

generalizability

§ Standardsfordata

collection/methodologyarefollowed

§ Triangulationofevidencesought.

§ Mixedmethodswherefeasible.

45

StrategyandProgramReflectionExercise ProgramQualityAssessment(PQA) EvaluabilityAssessment(EA) Evaluation(notcoveredinthisGuide)

Whoconducts

§ Inmostcasesinternalfacilitator

§ Externalfacilitatorrecommendedifno

internalfacilitationskillsavailable.

§ Inmostcases,external

facilitatorwithsubstantive

knowledgeofprogramarea,

programassessment

experience,andfacilitation

skills.

§ Ifinternalfacilitator(e.g.team

memberofprogrampartner)is

chosen(self-assessment),

personneedsacertainlevelof

independenceinadditionto

therightskillset.

§ IdeallyexternalfacilitatorinclosecoordinationwithprogramteamsandSeniormanagement.

§ Internalfacilitationpossible.

§ Externalevaluatorwithevaluation

expertiseandcredentialsand

understandingoftheprogramarea.

§ Self-evaluationusingsameskills,

standardsandtechniquesasother

evaluationsalsopossibledependingon

purpose.

§ Blendedexternal/internalevaluators;

Programteam;Seniormanagement;

Programparticipants;Subjectmatter

experts;Programpartners;Donors;

Hostgovernmentstakeholders;

Spoilers;Otherstakeholders.

WhoParticipates

intheprocess

Programteam;SeniorManagement;Possibly

programpartners;Possiblyanexternal

facilitator;Donor(s).

Facilitator(externalorinternal);

Programteam;SeniorManagement;

ProgramPartners;Donor(s).

Programteam;Externalorinternalfacilitator;Senior

Management;ProgramPartners,Donor(s).Programteam;SeniorManagement;

ProgramPartners;Donor(s);Blended

external/internalevaluators;Senior

management;Subjectmatterexperts;Host

governmentstakeholders;Other

stakeholders.

LevelofCapacitybuilding

KnowledgeaboutbasicRPPandDNH

conceptsrequired,andoftenbuiltinto

processtosomeextent

Capacitybuildingisanintegral

element—thoseimplementing

findingsofqualityassessmentsare

supportedonhowtoapplythe

assessment’sfindings.Criticalto

determinehowmuchcapacitydev.is

neededbytheteamandprogram

partnersaspartofPQA.Some

capacitybuildingonapproachesand

toolsforM&Eofpeacebuilding

mightalsoberequired.

CapacitybuildingandknowledgetransferonRPPand

DNHapproachesnotarequiredelement,butare

oftennecessarytoapplytheassessmentcriteriaina

processwiththeteamandprogrampartners.Some

capacitybuildingonapproachesandtoolsforM&Eof

peacebuildingmightalsoberequired.

§ Capacitybuildingisnotanelementin

mostcases.

§ Learningisanimportantcomponentof

evaluations,ideallycontributingtothe

capacitiesofstaff.Inmanycases,there

isnotsufficienttimebuiltinto

evaluationstofulfilthispurpose.§ Capacitybuildingcomponentmore

pronouncedininternalevaluations.

Resourcesrequired

Limited:resourcestocoverstafftime,venue

andpartnerparticipation,and,wherethere

isexternalfacilitator,feesforhis/her

services.

Medium:resourcestocoverstaff

time+interviews,facilitator,venueMediumastheseareusuallyshortduration:

resourcestocoverstafftime,andfacilitator,

resourcesforfacilitation.

§ MediumtoHigh:resourcesforevaluator/evaluationteamtodesign

andimplementrigorousdatacollection

andanalysisprocess.

§ Highforlargeexternalevaluation.

§ Mediumforblendedteam

§ Limited-mediumforself-evaluation

dependingontheevaluationquestions

anddatacollectionmethodology.

Benefitofthismodality

versusothers

Moreinformalprocessthatcanbe

conductedmorefrequently,islessexpensive

whenexternalfacilitationisomitted.

Strengthensthecapacityofteams

andprogrampartnerstoimprove

programqualityonanongoingbasis.

Iffindingsareusedandaddressed,EAhasthe

potentialtosignificantlyimproveafutureevaluation.Independentevaluationandexternal

assessment;credibilityvis-a-visdonors

9.Recommendationsforpractice

EventhoughdecisionstoconductRPP infusedStrategy/ProgramReflectionExercises,ProgramQualityand/orEvaluabilityAssessmentcanbetakenad-hoc,itisgenerallyadvisabletoplanforsuchprocessesaspartofanagreementofprogrammanagementstepsaspartoftheoverallplanningandprogrammingcycle.Thefollowingrecommendationsarefocusedondesigningprocessesfortheoptionsdealtwithinthisresource:

RECOMMENDATION1:Beclearabout thepurposeand timingof theRPP-infusedStrategy/ProgramReflection Exercise, EA or PQA as well as how the process fits into a larger plan for programimprovement,includingstaffandpartnercapacitydevelopment.

Beforeanyengagementisplanned,therightprocessneedstobedeterminedandexpectationswithintheteamandwithprogrampartnersneedstobeclear(seeTable3).

Someteamsmightbereadyorhaveaclearneedforassessingtheirprogramsagainstclearpeacebuildingprogramqualitycriteria(PQA),othersmightbenefitfromamoreinformalStrategy/ProgramReflectionExercise,whileothersneedtogetreadyforanevaluationandbenefitmostfromanEA.Whateverthefinalchoiceis,teamsneedtobecomfortablewiththerequirementsandconditionsforeachoptionasoutlinedinthisresource.

OnekeyconsiderationishowreadyteamsandprogrampartnersaretojumpintoanyoftheseoptionsrightawaywithoutpriorexposuretoRPPandDNHtools.SometeamsmightbeabletodothisandpickuponsomeofthetoolsastheygothroughaReflectionExercise,PQAorEA.Otherteamsmightbebestadvisedtoconductsometraining,alsojointlywithlocalpartners,oratleastexposurebeforeanyofthemodalitiesarechosen.Itisalsocriticaltoreflectonhoweachofthemodalitieswillwork,basedonthetoolsusedduringprogramdesign–e.g.ifnoconflictanalysiswasdone,whataretheoptionsfore.g.aPQA? If different typesofprogram strategy toolswereused,howdo theyalignwith theRPP-infusedprocesses?

RECOMMENDATION 2: Evaluative processes need to be integrated into organizational policies andprocesses,suchastheplanningandprogrammingcycles,andadequatefinancialresourcesneedtobeallocated.

Evaluative processes need to become part of regular organizational routine. For this to happen,organizationsneedseniormanagementdecisionsandadequatepriority setting.Theyneed to identifyexisting institutional capacities for the facilitation of evaluative processes and reach out for externalsupportwhennecessary.Documentingthelessonsanditerativeprogramadaptationstepsareusefulfromaninstitutionallearningandmemoryperspectiveandcanbearichsourceofdataforexternalevaluationswhenthesetakeplace.

Integratingevaluativeprocessesfromthebeginningisalsocriticalfromafinancialperspective:ifanyoftheoutlinedoptionsinthisresourcecameinonlyasan‘afterthought’whenbudgetshadalreadybeenallocatedandspent,possibilitiesforanyoftheseoptionsbecomemuchmorelimited.Clearallocationofadedicatedbudget (stafftime,costsforexternalfacilitator,meetingvenue, logisticsandtravelwithincountryetc.)isarequirement.

47

Thebelowfigure(CDA,2016)demonstratesonescenarioforsequencingthedifferentevaluativeoptions:this isonewayaprogramcouldintegrateeachoftheoptionsintoitsprogramreflection,learningandimprovementcycle.

RECOMMENDATION3:Seniormanagementbuy-in,partneranddonorengagementarecritical.

ItiscriticaltogetSeniorManagementanddonorbuy-intosupporttheintegrationofevaluativetoolsintopeacebuildingprograms.Anyevaluativeprocessisof limitedvalueifperceivedonlyasatechnicalexercisedrivenbytheimplementationteam.Follow-uptoanyoftheevaluativeprocessesneedstobeensured and must be supported by Senior Management, including difficult decisions about coursecorrection and strategy review. Some donors are already leading on the promotion of evaluativeapproaches,butthereisaneedformoreworkwithothers.Thiscouldincludecloseengagementbetweendonorsregardingrelevantevaluativeapproaches,aswellascloserengagementofdonorsregardingtheintegrationofOECD/DACcriteriaintopoliciesandprogramguidance.

AshighlightedthroughouttheGuide,Partnerinvolvementiscriticalinallthreeprocesses,mostpronouncedforPQAsandEAs.Thelevelofdetail(howmany,whattypesofpartners)needtobedeterminedindividually.

Insight from practice. A strong program management team committed to the reflection and/orassessmentprocess,aswellasbuy-infromSr.Management,anddonorsprovedcriticalduringtheCDA-led,RPP/DNHinfusedPQAsandEAs.Partnerswereinvolvedthroughouttheprocess,andthebenefitsoftheprocessaswellasfollow-upengagementwereclarifiedfromthebeginning.InbothPQAsandoneoftheEAs,theassessmentwasonlythebeginningofalongerprocessofprogramqualityenhancement,andthefindingsandresultsfromthePQAsandEAswereusedasinitialbenchmarksforongoingprogramqualityimprovement.

InthreeoftheEAsandPQAs,donorseitherparticipatedintheassessments,orwereinformedaboutthe outcomes. It was clearly an advantage to get the donor on board and increase mutualunderstanding,jointlearning,andsupportforfutureprogrammingdecisions.

ConflictAnalysis

(inthebeginingand throughouttheprogrammingcycle)

Phase:ProgramDesign

Includeevaluativethinkingfromthebegining; Possiblestrategy/programreflectionexerciseearlyon

Phase:ProgramImplementation

Strategy/ProgramReflectionExercise(PQA)

Phase:Mid-term Review

Strategy/ProgramReflectionExercise(PQA)

Evaluability Assessment

Phase: Evaluation

Isformalevaluationrequired&appropriate?

48

RECOMMENDATION4:DeterminetherightamountofdatacollectionandanalysisforbothEAandPQA.

Akeyquestion inboth EAandPQAprocesses is howmuchdata (in addition toprogram reports andinformation, baselines and other data collected by the program itself) needs to be collected for theEA/PQAassessments.

Fornewerprograms,datacollectionislessapplicable;instead,thefocusisonreviewingprogramdesignandM&Esystems.ForprogramsthatundertakeanEAorPQAlaterintheprogramcycle,thequestionarisesregardingwhattypeofdatathefacilitator/evaluatorshouldcollect–orfocusmoreonthequestionofwhattypeofdataisavailablewithoutcollectinganyofitdirectly.Datacollectionshouldbetargetedandlimited,asnoneoftheseprocessesareactualevaluationsand,especiallyinthecaseoftheEA,shouldnotbecomeone.

Datacollectioncouldbeusefulinseveralareas:

§ Conflictanalysis:Thiscanbecometricky,especiallyincasesinwhichaconflictanalysishadnotbeendoneorupdated.Itmightbenecessarytoconductatleastavalidationexercisewithlocalpartnersonthekeyconflictdriversinordertodeterminethepeacebuildingrelevanceoftheprogram.

§ Understanding theoryof change: Inmany cases, theories of change arenot explicitly articulated.SomeadditionaldatacollectionmightbeneededtoarticulatethetheoryofchangeduringtheEAand/orPQA.

§ Conflicting reports or beliefs about facts related to the program:Where there are conflictingaccountsoffactsrelatedtotheprogram(e.g.,programactivities,outputs,outcomes,theories,etc.),datacollectionmaybeneededforvalidationandclarification.

Sharing of information and data: A big shortfall for peacebuilding evaluation and gathering data forfeedbackmechanismsinfragileandconflictaffectedstatesisthatdataandevaluationfindingsarenotsharedwithinandacrossdifferentorganizationsworkingtowardssimilargoals.Sharinginformationanddata, particularly regarding conflict analysis, has the potential to decrease costs for all organizationsinvolved, increase quality of analyses due to thewide range of data included, aswell as provide thefoundationforjointplanningandimplementation.

Insight from practice. The CDA-facilitated EAs and PQAs included a mix between facilitated multi-stakeholderworkshops (withprogram teams,partners, anddonors)and individual interviews.Whilegroupreflectionbringsoutadditionalelements,andiskeytothere-designofprogramelements,aswellas for capacity-development purposes, individual interviews are critical from a data collectionperspective.Decisionswillhavetobemadeforeachcasespecifically.IndependenceofthePQAorEAfacilitatoriscriticalwhencollectingdata,asistriangulationofinformationindependentoftheamountofdatacollectionchosen.

49

RECOMMENDATION5: Facilitators need tobehighly competent in bothpeacebuildingpractice andevaluativemethods.Itisimportanttochoosetherightfacilitator/facilitationteam.

Thefacilitatorsofsuchevaluativeprocesseswearmany“hats."FacilitatingaStrategy/ProgramReflectionExercise,PQAorEArequiresagreatamountofadaptabilityonthepartofthefacilitators,whoneedtorespondflexiblytoarisingneedsthroughouttherespectiveprocesses,whileatthesametimestayingtruetothePQA/EAbenchmarks.

Facilitators of such processes needmultiple skills to be effective (independent of whether they areexternal or internal): skills in facilitation, coaching, training, peacebuilding and conflict sensitivityexpertise, and a strongM&Ebackground. In addition, for the specific CDAprocesses, knowledge andfluency in the application of RPP and DNH tools was critical. Given the sensitive contexts in whichpeacebuildinghappens,facilitatorsalsoneedtobewellinformedaboutthecontextandself-aware.

Arethereparticularbenefitstointernalversusexternalfacilitation?‘Outsiders’areoftenappreciatedasthey come with a fresh and unbiased perspective, and often bring additional and more in-depthknowledgeonRPPandDNH,evaluativethinking,M&Eskills,and/orfacilitationskills.Atthesametime,engagingexternalfacilitatorsobviouslyhasbudgetimplications.Consideringlocalfacilitators,e.g.fromwithin thenetworkof local stakeholdersofanorganization is anothergoodpossibility.Consequently,buildingregularself-evaluativecomponentsintoprogramsispotentiallyagoodalternative–facilitatedbyoutsidersonlywhenneeded.

Moreresourcesondatacollectionanddataanalysisinpeacebuildingevaluation:

® dmeforpeace.org/evaluation,under,"HowdoIcollectandanalyzedata?”

Insight from practice. In the two EAs facilitated by CDA, an international facilitator worked in a teamapproachwithalocalfacilitator.Whilethisarrangementisinvaluablefromtheperspectiveofbuildingonandworkingwithahigh levelof localcontextknowledgeandsharing facilitation, theset-upneeds tobecarefully planned, including the following questions: profile of the local facilitator, language skills,perceptionsofthelocalfacilitatorwithdifferentpartnergroupsbasedonhis/herbackground,rolewithinthe team, division of labor between international and local facilitator. In any case, both local andinternationalfacilitatorsneedtocollectivelycombinetheskillsetsoutlinedinthischapter.

50

10.Bibliography

ActionAsia.ExaminingRPPasAToolforEvaluation-TheActionAsiaExperience.ResultsfromtheActionAsiaPractitionersForumFebruary8-9,2011.Colombo:ActionAsia.2011.

AmericanEvaluationAssociation.“AmericanEvaluationAssociationGuidingPrinciplesforEvaluators.”AccessedJanuary5,2017.http://www.eval.org/p/cm/ld/fid=51.

Andersen,OleWinckler,BeateBull,andMeganKennedy-Chouane,eds.EvaluationMethodologiesforAidin

Conflict.NewYork,NY:Routledge,2014.

Anderson,MaryB.DoNoHarm:HowAidCanSupportPeace–OrWar.Boulder,CO:LynneRiennerPublishers,1999.

——— OptionsforAidinConflict:LessonsfromFieldExperience.Cambridge,MA:CDACollaborativeLearningProjects,2000.

Anderson,MaryB.,andLaraOlson.ConfrontingWar;CriticalLessonsforPeacePractitioners.Cambridge,MA:CDACollaborativeLearningProjects,2003.

Anderson,MaryB.,andMarshallWallace.OptingOutofWar:Strategiestopreventviolentconflict.Boulder,CO:LynneRiennerPublishers,2013.

Babbitt,Eileen,DianaChigas,andRobertWilkinson,withAMEXInternational.TheoriesandIndicatorsofChange:ConceptsandPrimersforConflictManagementandMitigation.UnitedStatesAgencyforInternationalDevelopment(USAID),2013.

Barnard-Webster,Kiely,NicoleGoddard,andIsabellaJean.“MonitoringandEvaluationofConflictSensitivity:PracticalConsiderations.”ConflictSensitivity:TakingittotheNextLevel,swisspeaceWorkingPaperSeriesno.2(2016):88-95.

BetterEvaluation.AccessedJanuary5,2017.http://betterevaluation.org/.

Blum,Andrew.ImprovingPeacebuildingEvaluation:AWhole-of-FieldApproach.UnitedStatesInstituteofPeace(USIP)SpecialReportno.280(2011).

Bonino,Francesca,withIsabellaJean,andPaulKnoxClarke.ClosingtheLoop-EffectiveFeedbackMechanismsin

HumanitarianContexts.ALNAP-CDAPractitionerGuidance.London:ALNAP/ODI,2014.

——— HumanitarianFeedbackMechanisms:Research,EvidenceandGuidance.ALNAPStudy.London:ALNAP/ODI,2014b.

CDACollaborativeLearningProjects.ReflectingonPeacePractice:ProgramReflectionExercise.Cambridge,MA:CDA,2014.

——— ReflectingonPeacePracticeParticipantTrainingManual.Cambridge,MA:CDA,2013.

——— StrategiesforDealingwiththeHardtoReach.Cambridge,MA:CDA,2012.

51

——— FeedbackMechanismsinInternationalAssistanceOrganizations.Cambridge,MA:CDA,2011.

——— EffectivelyUtilizingFeedback.AccessedJanuary5,2017.http://cdacollaborative.org/cdaproject/effectively-utilizing-feedback/.

Chigas,Diana,andNicoleGoddard.“MonitoringandEvaluationofConflictSensitivity–MethodologicalConsiderations.”EvaluationConnections(August2016):10-11.

Chigas,Diana,andPeterWoodrow.“SystemsThinkinginPeacebuildingEvaluations:ApplicationsinGhana,Guinea-BissauandKosovo,”inEvaluationMethodologiesforAidinConflict.EditedbyOleWincklerAndersen,BeateBull,andMeganKennedy-Chouane.NewYork,NY:Routledge,2013.

Corlazzoli,Vanessa,andJonathanWhite.“BacktoBasics.Acompilationofbestpracticesindesign,monitoringandevaluationinfragileandconflict-affectedenvironments.”AConflict,Crime,andViolenceResults

Initiative(CCVRI)product.London:UKDepartmentforInternationalDevelopment(DFID),2013.

Davies,Rick.PlanningEvaluabilityAssessments.ASynthesisoftheLiteraturewithRecommendations.WorkingPaperno.40.London:UKDepartmentforInternationalDevelopment(DFID),2013.

Davies,Rick,andJessDart.The‘MostSignificantChange’(MSC)Technique:AGuidetoItsUse.2005.

Duggan,Colleen,andKennethBush."TheEthicalTippingPointsofEvaluatorsinConflictZones,"inAmerican

JournalofEvaluation35(2014):1-22.

Ernstorfer,Anita,DianaChigas,andHannahVaughan-Lee.“FromlittletoLarge:WhendoesPeacebuildingAddUp?”inJournalofPeacebuildingandDevelopment10,no.1(2015):72-77.

Ernstorfer,Anita,andIsabellaJean.“AlternativeApproachestoEvaluatingPeacebuildingInitiatives.”Paperpresentedatthe11thEuropeanEvaluationSocietyBiennialConference,Dublin,October1-3,2014.

GlobalPartnershipforthePreventionofArmedConflict(GPPAC)incollaborationwithCDACollaborativeLearningProjectsandtheNorwegianChurchAid.ConflictAnalysisFramework:FieldGuidelinesand

Procedure.TheHague:GPPAC,2015.

Goldwyn,Rachel,andDianaChigas.Monitoringandevaluatingconflictsensitivity.Methodologicalchallengesand

practicalsolutions.AConflict,Crime,andViolenceResultsInitiative(CCVRI)product.London:UKDepartmentforInternationalDevelopment(DFID),2013.

Harris,Sidney.What'sSoFunnyAboutScience?fromAmericanScientist.LosAltos,CA:W.Kaufmann,1997.

InternationalLaborOrganization.“GuidanceNote11:UsingtheEvaluabilityAssessmentTool,”partoftheI-evalResourceKit.Geneva:InternationalLaborOrganization—EvaluationUnit,2011.

Kawano-Chiu,Melanie.StartingontheSamePage:ALessonsReportfromthePeacebuildingEvaluationProject.Washington,D.C.:AllianceforPeacebuilding,2011.

OECD/DAC.EvaluatingPeacebuildingActivitiesinSettingsofConflictandFragility:ImprovingLearningforResults.Paris:OECD,2012.

——— GuidanceonEvaluatingConflictPreventionandPeacebuildingActivities.WorkingDraftforApplication.Paris:OECD,2008

Paffenholz,Thania.“PeacebuildingEvaluation:AssessingtheRelevanceandEffectivenessofPeacebuildingInitiatives:LessonsLearnedfromTestingNewApproachesandMethodologies.”PaperpresentedattheAnnualConventionoftheInternationalStudiesAssociation,Montreal,March16-19,2011.

Patrizi,PatriciaA.,andMichaelQuinnPatton,eds.“EvaluatingStrategy:NewDirectionsforEvaluation.”Jossey-BassQuarterlyReportSeries128(2011).

52

PeacebuildingEvaluationConsortium.“PeacebuildingEvaluationOnlineFieldGuide.”AccessedJanuary5,2017.http://dmeforpeace.org/online-field-Guide.

QuinnPatton,Michael.DevelopmentalEvaluation:ApplyingComplexityConceptstoEnhanceInnovationandUse.NewYork,NY:GuilfordPress,2011.

Reimann,Cordula,DianaChigas,andPeterWoodrow.AnAlternativetoFormalEvaluationofPeacebuilding:

ProgramQualityAssessment.Cambridge,MA:CDACollaborativeLearningProjects,2012.

——— EvaluabilityAssessmentsinPeacebuildingProgramming.Cambridge,MA:CDACollaborativeLearningProjects,2012b.

Rogers,MarkM.EvaluatingImpactinConflictPreventionandPeacebuildingPrograms.Cambridge,MA:CDACollaborativeLearningProjects,2012.

——— EvaluatingRelevanceinConflictPreventionandPeacebuildingPrograms.Cambridge,MA:CDACollaborativeLearningProjects,2012b.

——— cartooninNewRoutes13,no.3(2008):8.

Scharbatke-Church,Cheyanne.PeacebuildingEvaluation:NotYetAllItCouldBe,BerghofHandbookforConflictTransformation.SectionII.Berlin:BerghofFoundation,2011.

——— TheUseofReflectingonPeacePractice(RPP)inPeacebuildingEvaluation.Review&Recommendations.FinalReport.Cambridge,MA:CDACollaborativeLearningProjects,2011b.

——— “PeacebuildingEvaluation:FromInfancytoTeenager.”NewRoutes13,no.3(2008):3-6.

Scharbatke-Church,Cheyanne,andMarkRogers.DesigningforResults:IntegratingMonitoringandEvaluationin

ConflictTransformationPrograms.Washington,D.C.:SearchforCommonGround,2006.

Schirch,Lisa.ConflictAssessmentandPeacebuildingPlanning:TowardAParticipatoryApproachtoHuman

Security.Boulder,CO:KumarianPress,2013.

SearchForCommonGround(SFCG).“Design,MonitoringandEvaluation(DM&E)forPeacePortal.”AccessedJanuary5,2017.http://www.dmeforpeace.org/.

UnitedNationsDevelopmentFundforWomen(UNIFEM)EvaluationUnit.“GuidanceNoteonCarryingOutanEvaluabilityAssessment.”NewYork,NY:UNIFEM,2009.

Wallace,Marshall.FromPrincipletoPractice:AUser’sGuidetoDoNoHarm.Cambridge,MA:CDACollaborativeLearningProjects,2015.

Woodrow,PeterandDianaChigas.“ConnectingtheDots:EvaluatingWhetherandHowProgramsAddressConflictSystems,”inTheNon-LinearityofPeaceProcesses:TheoryandPracticeofSystemicConflict

Transformation.EditedbyHansJ.Giessmann,DanielaKörppen,andNorbertRopers.Opladen&FarmingtonHills:BarbaraBudrichPublishers,2011.

——— ADistinctionwithaDifference:ConflictSensitivityandPeacebuilding.Cambridge,MA:CDACollaborativeLearningProjects,2009.

Woodrow,Peter,withNickOatley.“PracticalApproachestoTheoriesofChangeinConflict,SecurityandJusticePrograms.PartI:Whattheyare,differenttypes,howtodevelopandusethem.”AConflict,Crime,and

ViolenceResultsInitiative(CCVRI)product.London:UKDepartmentforInternationalDevelopment(DFID),2013.

53

11.Annexes:ToolsandHand-outs

Annex1:FiveBuildingBlockstowardsPeace

Fromanalysisofthecasesandpractitionerreflectionontheirownexperiences,theRPPprocessidentifiedfiveintermediateBuildingBlocksthatcansupportprogresstowardsPeaceWritLarge.Thesecanbeusedtoassess,acrossabroadrangeofcontextsandprogrammingapproaches,whetheraprogramismakingameaningfulcontributiontoPeaceWritLarge.TheBuildingBlockscanbeusedinprogramplanningtoensurethatspecificprogramgoalsare linkedtothe largerand long-termgoalof“PeaceWritLarge.” Theycanbeusedduringprogram implementation to reflect on effectiveness and Guide mid-course changes, and as a basis forevaluationaftertheprogramhasbeencompleted.

Theeffortresultsinthecreationorreformofpoliticalinstitutionstohandlegrievancesinsituationswheresuchgrievancesdo,genuinely,drivetheconflict. Asignificantcontributiontopeace is thedevelopmentorsupportforinstitutionsormechanismsthataddressthespecificinequalities,injusticesand other grievances that cause and fuel a conflict. This approach underlines the importance ofmovingbeyondimpactsattheindividualorpersonal(attitudinal,materialoremotional)leveltothesocio-politicallevel.Thisideamustbeappliedinconjunctionwithacontextanalysisidentifyingwhatthe conflict isNOTabout andwhatneeds tobe stopped. To reformorbuild institutions that areunrelatedtotheactualdriversofaspecificconflictwouldbelesseffective.

Theeffortcontributestoamomentumforpeacebycausingparticipantsandcommunitiestodeveloptheirownpeace initiatives in relation tocriticalelementsof contextanalysis. Suchanalysis,andresultingprograms,shouldaddresswhatneedstobestopped,howtoreinforceareaswherepeopleinteractinpositiveways,andtheregionalandinternationaldimensionsoftheconflict.Thisapproachstressestheimportanceof“ownership”andsustainabilityofactionandeffortstobringaboutpeace,aswellascreatingmomentumforpeace,involvingmorepeople.

Theeffortpromptspeopleincreasinglytoresistviolenceandprovocationstoviolence.OnewayofaddressingandincludingKeyPeoplewhopromoteandcontinuetensions(e.g.,warlords,spoilers)istohelpMorePeopledeveloptheabilitytoresistthemanipulationandprovocationsofthesenegativekeypeople. Inmost circumstances,one important aspectofPeaceWrit Large is a significant andsustainedreductioninviolence.ThisBuildingBlockisasteppingstonetothatlong-termgoal.

Theeffortresults inan increase inpeople’ssecurityand intheirsenseofsecurity. Thisapproachreflects positive changes, both at the socio-political level (in people’s public lives) and at theindividual/personallevel,aspeoplegainasenseofsecurity,animportantelementofPWL.Securityandpeople’sperceptionsofitcontainmanydifferentaspects,whichmustbeidentifiedandattainedbasedonthelocalcontext.

The effort results inmeaningful improvement in inter-group relations, reflected in, for example,changesingroupattitudes,publicopinion,socialnorms,orpublicbehaviors.ImprovedrelationshipsbetweenconflictinggroupsconstituteanimportantBuildingBlockforpeace—oftenapreliminarysteptowardsotherinitiatives. Itentailstransformingpolarized(andpolarizing)attitudes,behaviorsandinteractionstomoretolerantandcooperativeones,aspartofaddressingunderlyinggrievancesandbuildingthewillingnessandabilitytoresolveconflictsandsustainpeace.

TheseBuildingBlockscanbestbethoughtofasintermediate-levelbenchmarksofsuccessapplicabletothebroadrangeofpeaceworkbeingdone.

1

2

3

4

5

54

Annex2.1:RPPMatrix

MorePeople KeyPeople

Individual/PersonalLevel

Socio-PoliticalLevel

55

Annex2.2:RPPMatrix‘Plus’

MorePeople KeyPeople

Individual/PersonalLevel

Healing/recovery

Perceptions

Attitudes

Skills

Behavior

IndividualRelationships

Socio-PoliticalLevel

Groupbehavior/relationships

Publicopinion

Socialnorms

Institutionalchange

Structuralchange

Programactivities

Programactivities

Socio-politicalgoal

CurrentSituation:ConflictAnalysis

KeyDrivingFactorsofConflictand“KeyPeople”orActorAnalysis

Whatisthegapbetweenthe

currentsituationandthedesiredfuture?->“peaceneeds”and/orstrategicspace.

Whatneedstochange

andhow?

ProgramTheory:Howdotheactivitiesleadto

thegoal?

TheoryofChange:HowdoesthegoalcontributetoPeaceWritLarge?

Vision:AdesiredfutureSocietalchange/PeaceWritLarge

56

Annex3:RPPProgramPlanningChart

IdentificationofActivities,ChangesandTheoriesofChangeandAssumptions

Program Goal(determinetimeframe!): ProgramGoal–TheoryofChange:Ifwedoxxx[activities]Thenweachieveyyy[thetypeofchangeyouexpectfromtheactivity/ies]Becausezzz[Rationaleforwhythischangewillbeachieved:whydoes‘then’happen?]

Proposed ProgramActivity Expected Changes, due totheActivity/ies Whatassumptionsdoyoumake?1.

2.

3.

4.

57

Annex4:DoNoHarmActionPatterns(ResourceTransferPatterns)

TheImpactsofanorganization’sACTIONS

Howanorganizationtransfersresourcesintoacontextmatters.Thewaysthesetransfershaveimpactsinfivespheres,andtheseimpactscanbepositiveornegative.Ifanorganizationisstrategic,understandsthecontextandmakescontext-appropriateprogrammingchoices,theycanhavepositiveimpactsinthesefivespheres.But,organizationscanalso,throughlackofattention,orprogramplanningnotlinkedtocontextanalysis,havenegativeimpactsinthesefivespheres.

Incompleteanalysisand/orinappropriateprogramming Strategicandcontext-appropriateprogramming

1. Theft/PreventionGoodsormoneyintendedfordistributionorpaymentmaybestolen,andusedbyfightersorusedtopayforongoingfighting.

Theftcanbeprevented,money,time,andresourcesaresavedandusedtobenefitcommunities.

2. MarketEffects

Adverseimpactsonpricesofgoodsandservices.Pricinglocalpeopleoutoftheirownmarkets,Goodsthatareavailablelocallybroughtinfromoutsideandgivenawayforfreecandroppricesiflocalgoodsandforcefarmersandsellersoutoftheirjobs.

Balancingandstabilizingmarkets.Ensuringpeoplecancontinuetoaffordlocalgoodsandservices.

3. DistributionEffects

Unevendistributionalongconflictlinescanexacerbatetensions/divisions,unfairlybenefitonesideofaconflictoveranother.

Understandinglocaldefinitionsoffairdistributioncanhelptodeterminebeneficiaryselectionwithoutexacerbatingtensions.Fairdoesnotalwaysequal“even”distribution.

4. SubstitutionEffects

Freeingupgovernmentresourcestocontinuefighting.Weakeningthestate’sabilitytorespondandmanagedisasters,,conflictsanditsowndevelopment.

Strategic,short-term,negotiatedsubstitution.Involvegovernmentinprogramdesignsotheyunderstandandareheldaccountablefortheirroleintheprogram(incl.transferofresponsibilitiesandtimeframes)

5. LegitimizationEffects

Inadvertentlylegitimizingagovernment,institutionorleaderbyinvolvingthemintheaidprocess.

Strategicallylegitimizingagovernment,leaderorinstitutionwithaneyetochangingorimprovinglocalperceptionsoftheirabilitytomanagedevelopmentordisasterresponse.MustunderstandWHYandHOWtheywillbelegitimized.

Annex5:ToolforPeacebuildingQualityAssessment34

The following tool is an illustrative framework for organizing and documenting a peacebuilding program qualityassessment.

Intheleftcolumnarequalitycriteriaforaprogramdesign,basedonthekeyfindings,conceptsandtoolsofRPP.Theframeworkprovidesaseriesofquestionsdividedintoseveralbroadcategories:conflictanalysis,programgoals/outcomes,programstrategyandlogic,andunintendedimpact.

Thesecondcolumnasksforaratingoftheprogramdesign,rangingfrom1to3:

1. Programdesignmeetsnoneofcriteria;2. Program design meets some of criteria, but there are significant gaps and weaknesses in several

categories;3. Programdesignmeetscriteria,withnosignificantgaps,althoughtheremayberoomforimprovement.

Thethirdcolumnasksforevidenceonthebasisofwhichtheassessmentteamorassessorhasarrivedattherating.Itshouldreflectstrengthsandopportunities—thatis,whereaprogrammayfulfillqualitycriteriabutmightbestrengthenedfurther,aswellasgapsandchallenges—thoseaspectsofaprogramthatdonotmeetthecriteriaorcriteriaofeffectivepeacebuildingprograms.

Thefourthcolumnprovides forrecommendations for improvingthequalityof theprogramdesign. Thiscanincludewhatanagencycandotosustainorimprovetheprogramdesign,whatitshouldkeepinmindormonitorwhile implementing theprogram, and suggestions on things that canbedone to improve the quality of thepeacebuildingprogram.Herethequestionofcapacitybuildingforresponsiblestaffmightalsoarise.

ThisframeworktoolmaybestbeunderstoodasaframeworkforanRPP-inspiredlearningprocessforpeacebuildingorganizations.Itisnottobeunderstoodascomprehensiveandexhaustive,butaspreliminaryreflectionandfoodforthought.

34BasedonReimann,AnAlternativetoFormalEvaluationofPeacebuilding:ProgramQualityAssessment,withadaption

1. ConflictAnalysis

Criteria/QuestionsbasedonRPP Rating Evidence Recommendations/WaystoImprove

Doestheanalysisidentifykeydrivingfactorsandkeyactors?

Doestheanalysisconsiderwhatneedstobestoppedandwhatforcespromotepeace?

Istheanalysisupdatedandtested?

Hastheanalysisidentifiedandexaminedpastorongoingsimilareffortsandanylessonsfromtheirresults?

Doestheanalysisidentifypeacebuildingneedsorpointsofleverage?

Isthescopeoftheanalysisisappropriate(nottoobroadornarrow,mitigatesbiastowardsagency’sexpertiseorgeneralbeliefsaboutconflict)?

Wastheanalysisprocessconflict-sensitive?

2. ProgramGoals

Criteria/QuestionsbasedonRPP Rating Evidence Recommendations/WaystoImprove

Doesthegoaladdress,directlyorindirectly,keydriversofconflictorpeace?

Isthegoalstatedasadesiredchange?

Isthegoalspecificandrealisticforthetimeframe—neithertoobroad(long-termvision)nortoonarrow(activitiesoroutputs)?

Isthegoalstatedasadesiredchangeatthesocio-politicallevel,orisachievingitpartofalonger-termstrategyforeffectingchangeatthesocio-politicallevel?Doestheprogrammakelinkagestoeffortsofotheragenciesatthesocio-politicallevel?

Dothechangescontributetooneormoreofthefollowingintermediatebuildingblocksofpeace:

1. CreationorreformofpoliticalinstitutionsthataddressKDF?

2. Locally-drivenpeaceinitiativesaddressingKDF?3. Increasingresistancetoviolenceor

provocationstoviolence?4. Increasedsecurityorsenseofsecurity?5. Meaningfulimprovementininter-group

relations?

3. OverallTheoryofChange

Criteria/QuestionsbasedonRPP Rating Evidence Recommendations/WaystoImprove

IstheOverallTheoryofChangeexplicit,withclearandunderstandableconceptualizationofpathwaytochange?

DoestheOverallTheoryofChangeprovideaplausibleexplanationofhowachievingtheprogramgoalwillaffectkeydriversofconflictorpeace?

IstheOverallTheoryofChangegroundedinanunderstandingofhowchangehappensintheparticularconflictandcontextinwhichtheprogramisbeingimplemented?

4. ProgramStrategyandLogic

Criteria/QuestionsbasedonRPP Rating Evidence Recommendations/WaystoImprove

Doactivities“addup”tothegoal?Arethereexplicit,rationalandplausiblelinksbetweencomponentsoftheprogramthatarevalidinthecontext?

Doestheprogrammakelinkagesbetweenactivitiesandchangesattheindividual-personallevelandatthesocio-politicallevel?

Doestheprogrammakelinkagesbetween“morepeople”and“keypeople”?

Arethescaleandleveloftheoutputsreasonableinrelationtotheintendedimpactsandthesizeoftheissueinthiscontext?

Hasthedesignaccountedforfactorsthatcouldimpedesuccess(includingwayssocialandpoliticalsystemsmightresistchanges)?

Doestheeffortrelatetoorlinkwithotherinitiativesinthesameorrelatedareas?

WilltheM&Eplanorfeedbackmechanismprovidetimely,accurateandusefulinformationaboutprogresstowarddesiredchangesandaboutassumptionsunderlyingtheoryofchange?

5. Conflict-Sensitivity/DoNoHarm

Criteria/QuestionsbasedonDNH Rating Evidence Recommendations/WaystoImprove

Istheprogramconflict-sensitive?Doesitconsiderpotentialunintendednegativeimpactsitmightcause?

Istheinitiativeconsciousoftheimpactofitsactionpatterns/resourcetransfersinthefollowingareas:1. Theft;2. MarketEffects;3. DistributionEffects;4. SubstitutionEffects;5. LegitimizationEffects.

Doestheinitiativeconsiderunintended(negative)impactsoftheprograminitsM&Esystem?

Annex6:ChecklistforEvaluabilityAssessmentofPeacebuildingPrograms35

Thefollowingcontinuumsprovideaframeworkforassessingtheevaluabilityofapeacebuildingprogramand illustratewhatmakes a peacebuilding program or project less ormore evaluable. RPP tools andfindingshavebeenintegratedintothequestionsregardingprogramdesign.

Theredcolumnontheleftindicateslowevaluabilitywhilethegreencolumnontherighthighlightshighevaluability.Thedottedlinesbetweenthered,yellowandgreencolumnsindicatethatthedivisionisnotclear-cutorfixed.

Theteamortheevaluabilityassessorcanticktherespectiveboxestoindicatewheretheprogramstandsintermsofevaluability.Ifmostofthetickedboxesareintheleft,redcolumn,aprogramisnotreadyforevaluation,andthechecklistindicateswherechangesintheprogramdesignwouldneedtobemade.Theevaluabilityassessorcouldalsorecommendwheretochangeandimprovetheprogramdesignandhencemaketheprogrammoreevaluable.Thefieldstaffandprogramdesignersinthefieldcouldlikewiseusethechecklistasamonitoringtoolthathelpsthemtoworkontheaspectsoflowevaluability.

35Thischeck-listisbasedon(Reimann,Chigas,andWoodrow2012b)andhasbeenadaptedbyCDA

ProgramDesign(includesRPPtoolsandfindings)

Lowevaluabilityinpeacebuilding

Highevaluabilityinpeacebuilding

Noconflictanalysisatall

Partialanalysis

“Goodconflictanalysis”

prioritizingKDF,stakeholders

Implicitanalysis/Informalanalysis

Notupdated

Nodocumentedanalysis Contextanalysis

Analysisnotupdated

Notlinkedtostrategy

Biasedandnarrow

Toocomprehensive

Uncleargoalsandobjective

Goalsandobjectivetoogeneralandbroad

Well-definedandrobustgoalsand

objectives

Goalsdefinedasactivities

Goalsandobjectivetooambitious Goalsareformulatedas

desiredchangesatthesocio-political

level

Nolinksbetweenactivities,goalsandoverallobjective

Indirectundunclearlinksbetweenactivities,goalsandobjective

Explicitlinksbetweenactivities,goalsandobjective

ofPWL

“Hopelines”(linksbetweenactivitiesandoutcomesatvariouslevelsthatarebasedonweakassumptions)

ImplicitToC ImplicitandrealisticToC ExplicitToC

UnclearToC ExplicitbutinappropriateToC(i.e.ToCcontradict/sknowledgeofpeace-buildingpracticeorisnotsuitedto

thecontext.)

ClearandrealisticToC

DataAvailability

Lowevaluabilityinpeacebuilding

Highevaluabilityinpeacebuilding

ImplicitToC ImplicitandrealisticToC ExplicitToC

UnclearToC

ExplicitbutinappropriateToC(i.e.ToCcontradict/sknowledgeofpeace-

buildingpracticeorisnotsuitedtothecontext.)

ClearandrealisticToC

Nobaseline Condensedbaselinewithdata-gatheringisfocusedonafewkey

indicatorsforselectedgoals

Completebaseline,basedonconflictanalysis

Nobaselinebutamore

comprehensivemonitoringatthebeginning

Nomonitoringsystem

Monitoringsysteminplacebutnotused

Monitoringsysteminplacetogatherandsystematizeallnecessaryinformation

Insightsfrommonitoringarenottranslatedintoprogramchanges

Monitoringsystemusedtoinformprogramchanges

Monitoringsystemmonitorsimpactsonconflictcontextaswellasunintended(negative)

impacts

Noindicators Indicatorsexist,butunrealistic,un-

measurableorunclear IndicatorsareSMART

Indicatorsaredesignedtocapturechangesinconflict

contextandpossibleunintended(negative)impacts

Noaccesstostakeholders

Difficultandlimitedaccesstostakeholders Accesstostakeholders

Ongoingfeed-backprocesswithpartnersinplace

ConducivenessoftheContext

Lowevaluabilityinpeacebuilding

Highevaluabilityinpeacebuilding

Conditions(weather,security,availability…)

notfavorable

Someconditionsquestionable,butgenerally

workable

Noapparentimpedimentstoaccessorsecurity

Nofinancialresourcesavailabletoconduct

evaluation

Financialresourcesavailablebutlimited

Fullfinancialresourcesavailable

Nointernalcommitment/Internally

drivenprocess

Internallydrivenbutonlybytheheadsofagencies

CommitmentAvailable-

Internallydrivenandlocallyownedprocess

Noownershipofprocess

Notdonor-drivenprocessbutnolocalownershipof

process

Internalcapacitiesavailabletoengageandfollow-up

Donor-drivenprocess

DoNoHarmprinciplesnotappliedtotheevaluationprocess

DNHprinciplesonlypartiallyapplied

DNHapplied:Timing,locationofevaluationandselectionofinterviewpartnersareconflict-

sensitive

DNHprinciplesappliedoncebutnotupdated

Annex7:CDACollaborativeLearningProcessMethodologyandSteps