joint humanitarian evaluations:opportunities and challenges

8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges

1/15

Joint Humanitarian Evaluations:

Opportunities and Challenges

Scott Green

United Nations Office for the Co-ordination of Humanitarian Affairs (OCHA)


2/15

Drivers for Joint Humanitarian

Evaluations ALNAP research finds joint evaluations to be of higher quality than singleagency.Delivering As One Report:

Stresses importance of closer partnerships between the United Nations andNGOsIdentifies need for more accountability to both affected people and donorsEvaluation defined as a major driver for building system-wide coherence.Panel calls for humanitarian system getting more adept at using jointevaluations; or regular independent assessments of the performance of thehumanitarian system in responding to humanitarian emergencies

Humanitarian system becoming more adept at commissioning and using joint evaluations; OCHA playing a key role in moving the process forward.


3/15

IASC Real-Time Evaluations

Approach initially piloted (2007 2009) in Pakistan, Mozambique andMyanmar.

2009 IASC Review endorses approach for scale up and regularimplementation across the system

Approach now routinely implemented across all major humanitarian crisesthrough use of an automatic trigger mechanism: applies in cases whereaffected population exceeds 1 million and appeal exceeds US$50 million.

OCHA serves as lead managing agent; IASC RTEs are jointly funded bymember agencies.


4/15

Key Characteristics

IASC RTEs are rapid assessments which make use of participatorymethodologies; identify operational bottlenecks around co-ordination issuesand seek out the views of affected people.

Management StructureGlobal-level policy and Steering Group

Ad Hoc Management CommitteesIn Country Advisory Groups


5/15

IASC RTEs: Key Characteristics

Teams deployed no later than 3 months after disaster

Content based on an agreed global IASC RTE assessment framework and

processes are implemented based on an agreed set of standard operatingprocedures

Real-time feedback used to support remedial action planning for immediateimplementation; action plans developed with assigned roles andaccountabilities for follow up.

Major IASC RTEs: Haiti (2010 & 2011) Pakistan (2010 & 2011) Kenya(2011)


6/15

Important lessons from IASC RTE

experienceCreate the necessary political space for joint evaluation work: it isimportant to have clear sense of the purpose and use for joint evaluationwork and to make it a routine part of the culture

Have an agreed assessment framework in place. There needs to be astrong concept of what is being used as a baseline for the assessment.

Maintain a narrow scope to be most useful. Focus on issues of broad crosscutting concern rather than in depth assessments by sectors

Ensure good communication throughout the process.


7/15

Ongoing Challenges

Ensuring RTE results get used at both strategic and operationallevels

Ensuring real-time deployments: significant demand for IASC RTEsto be deployed earlier for greater impact.

How to assess joint humanitarian impact: system still unable todeliver multiple-agency impact assessments. What value-added if any?


8/15

JHIE ConsultationsTo assess the potential for future inter-agency impact evaluations,

OCHA led a series of consultations:Objective was to define feasible approaches to undertaking jointimpact assessments with possibility of having some pilot evaluations(group interviews held around issues of scope, focus, purpose, useand methodologies)Consultations were held with:The affected population in 15 communities in Sudan, Bangladeshand Haiti (including women, men, children, disabled, ethnic groups),mainly in focus groupsLocal government and local NGOs in the same countriesNational governments and international humanitarian actors in Haitiand Bangladesh67 international humanitarian actors, donors and evaluators in sevenmeetings in New York, Rome, Geneva, London and Washington


9/15

MethodologySystematic attempt to consultwith national governments andwith disaster-affected peopleduring the design phase of amajor evaluative exercise

A new approach to evaluationdesign is warranted because of the scale of and resourcesrequired for JHIE, but also tomodel a participatory approachto designing impact evaluations

Evidence that it is necessaryand useful to consult with theaffected population duringevaluation design.


10/15

Results and Implications: Possibility

of Pilot JHIEs95 percent of respondents were supportive and 75 percent were stronglysupportive.Differences in what JHIEs should look like not about their relevance

Main advantages seem to be the potential to offer a more comprehensivepicture of impact and look at areas that cannot be covered by single agencyevaluations

ImplicationsThere is a strong basis to proceed with some pilot JHIEs in the future


11/15

Results & Implications: Questions

of Purpose, Focus and UseStrongest support was for generalizable knowledge and accountability toaffected populationsMain focus should be on changes in the quality of life of affectedpopulationsFocus may need to differ between sudden onset natural disasters andcomplex emergenciesConcern expressed over limited use of current system-wide humanitarianevaluations at country level (e.g. IA RTEs and other system-wideevaluations such as TEC)

Strong support for national governments to become primary users


12/15


of Purpose, Focus and UseImplications

Focus on making JHIEs useful first at the national level by buildingknowledge which is useful for national and local governments innatural disaster situations; get better at engaging with

governments on evaluation issuesPilot JHIEs should not be used in support of lesson learning for on-going programme modification or for meeting upward accountabilityneeds;Consider a pilot JHIE with primary purpose of supportingaccountability to affected populations

Pilots should avoid mixing purposes as most evaluations currentlytry to do: one option would be to have different JHIE pilots bedesigned around different purposes.


13/15

Results & Implications: Meaning of

impact in humanitarian contextLittle interest in developing acommon definition of humanitarianimpactBoth short and longer-term changes

are relevantCommon indicators would be usefulbased on classification of phases:relief, transition and recovery

Implication:JHIE steering committees will need todecide on a case-by-case basis whathumanitarian impact means. Thereshould be an effort to focus on bothshort and longer-term changes, using

common indicators.


14/15


of evaluation methodologyStrong emphasis on the need for consultations with communities ahead of time; do not always assume communities will want to be involvedEvaluation design should build in learning for affected populations. Strongemphasis on follow up and feedback mechanisms at the community levelNo support for experimental design and only minority opinion supporteduse of formal quantitative population surveysStrong majority supported use of qualitativemethodologies. Participatory impact assessmenttools likely to be preferable.

ImplicationsCurrent fly -in fly-out model of Humanitarian

Evaluation not suitableLonger time frames needed for research andmultiple visits to communitiesGoal free evaluation might offer a better modelthan theory-based combined with QED


15/15

Results and Implications: linkagesto broader M&E efforts andmanagement arrangements

Consultations emphasized need fordevelopment of a common set of indicatorswhich could be tracked from baselineConsultations supported a two-tiermanagement structure with a steeringcommittee made up of key stakeholders and

a management groupCommunities expressed doubts in Haiti thatagencies could effectively work together toassess joint impact

Implications:JHIE should build a small set of quantitativeand qualitative indicators in key interventionareas that can be tracked throughout theJHIE processDevelop JHIEs as part of a broader effort toenhance M&E efforts and a commonprogramme cycle

PreliminaryScenarioDefinition Flash

Appeal

Revision ofFlash Appeal

Coordinatedsectoralassessments

Contingency Planning Preparedness

Multi-clusterRapid Assessment

RTE / Monitoring

Recovery Assessment

ImpactMonitoring &Learning

HumanitarianDashboardUpdates

Performance monitoring &RTEs

Evaluation

Cluster-level monitoring andreporting

joint humanitarian evaluations:opportunities and challenges

Documents