harnessing the power of evaluation in humanitarian action: an initiative to improve understanding...

7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva

1/26

ALNAP Working Paper

Harnessing the Power ofEvaluation in Humanitarian Action:An initiative to improve understanding and use of evaluation

Alistair Hallam

June 2011


2/26

1. Inroducion 3

Background 3Limiaions 4

Ouline o his paper 4

A ramework o improve undersanding and use o EHA 4

Acknowledgemens 6

1. Leadership, culure and srucure 7

Ensure leadership is supporive o evaluaion and monioring 7

Promoe an evaluaion culure 7

Increase he inernal demand or evaluaion inormaion 8

Creae organisaional srucures ha promoe evaluaion 9

2. Evaluaion purpose and policy 10

Clariy he purpose o evaluaion (accounabiliy, audi, learning) 10

Clearly ariculae evaluaion policy 11

Ensure evaluaion processes are imely and orm an

inegral par o he decision-making cycle 12

Emphasise qualiy no quaniy 12

3. Evaluaion processes and sysems 14 Develop a sraegic approach o selecing wha should be evaluaed 14

Involve key sakeholders hroughou he process 14

Use boh inernal and exernal personnel o encourage a culure o evaluaion 16

Improve he echnical qualiy o he evaluaion process 16

Assign high prioriy o eecive disseminaion o ndings, including

hrough new media (video, web) 17

Ensure here is a managemen response o evaluaions 18

Carry ou periodic mea-evaluaions and evaluaion synheses,

and review recommendaions 19

4. Supporing processes and mechanisms 20

Improve monioring hroughou he programme cycle 20

Provide he necessary human resources and incenive srucures 20

Secure adequae nancial resources 21

Undersand and ake advanage o he exernal environmen 22

Nex seps 24

Reerences24

A B L E O F C O N E N S

2


3/26

Background

ALNAP has done more work on evaluaions o humaniarian acion (EHAs) during he pas ew

years han a any oher ime in is hisory. Tis has included research on evaluaion uilisaion, join

evaluaions, a pilo guide on real-ime evaluaions, a meeing and sudy on impac assessmens, a

new look Evaluaion epors Daabase, and he new Sae o he Humaniarian Sysem repor, which

included an exensive evaluaion synhesis.

However, despie hese eors, and some real improvemens wihin agencies in how hey carry

ou evaluaions, here remains a eeling among many in he secor ha he ull poenial bene

o humaniarian evaluaions is no being realised. oo oen, humaniarian evaluaions exis as

a disconneced process, raher han becoming embedded as par o he culure and mindse o

humaniarian organisaions. Te secor has made real advances in he qualiy o evaluaions; i would

seem ha he challenge now is o ensure ha evaluaions are acually used o improve operaional

perormance.

ALNAP is no alone in coming o such conclusions. Te UN Evaluaion Group (UNEG)1 and he

wider OECD-DAC group2 have been carrying ou imporan work on improving approaches o

evaluaion or some years across he UN sysem, and also carrying ou peer-reviews o members

evaluaion sysems and srucures. Following a UNEG peer-review, he Oce o Evaluaion ohe World Food Programme (WFP) commissioned work o enhance he learning purpose o

he organisaions evaluaions (WFP 2009). Similarly, he UKs Deparmen or Inernaional

Developmen (DFID) has recenly commissioned a sudy rom he Overseas Developmen Insiue

on how o improve is learning rom boh evaluaion and research (Jones and Mendizabal 2010).

UNICEF has carried ou similar work (Soddard 2005).

Tere have been similar debaes on evaluaion eeciveness wihin he developmen communiy

or decades. Ta he humaniarian communiy is coming o hese issues relaively lae may refec

he simple ac ha i also came laer o evaluaion isel. However, here is perhaps also somehingdieren abou humaniarian aid in he scale and iner-connecedness o is ambiion, and he

subsequenly greaer diculies o undersanding and atribuing impac.

Agains his background, ALNAP has commissioned a process o acion research, o which his paper

is a par, ocusing on srenghening insiuional undersanding as well as capaciies and processes in

order o beter harness he power o EHA. Te iniial research used lieraure review and inerviews

across he ALNAP member organisaions and idenied areas or urher examinaion. A cenral

emerging lesson is ha improving humaniarian evaluaion capaciies and processes is a complex

underaking ha involves a wide range o acors and acors including allocaion o resources;

1 . I N O D U C I O N

1. htp://www.uneval.org/2. htp://www.oecd.org/deparmen/0,3355,en_2649_34435_1_1_1_1_1,00.hml

3
http://www.uneval.org/http://www.oecd.org/department/0,3355,en_2649_34435_1_1_1_1_1,00.htmlhttp://www.oecd.org/department/0,3355,en_2649_34435_1_1_1_1_1,00.htmlhttp://www.uneval.org/


4/26

engaging senior leaders; and realigning operaional procedures.

Tis paper sars he process o developing he ideas generaed hrough he lieraure review and

inerviews. I is hoped ha by illuminaing he range o signican acors involved, his paper will helpALNAP members ineresed in srenghening heir evaluaion eors by helping hem o ideniy

prioriy areas o concern, share ideas across he membership abou wha has and has no worked, and

develop new sraegies or ackling longsanding issues.

A dra o his paper was presened a a workshop in Sepember 2010 ha brough ogeher evaluaors

rom UN, NGO, ed Cross and donor organisaions. Te paper was subsequenly redraed o

ake ino accoun commens made a ha workshop. Te paper was also peer-reviewed by ALNAP

members and changes were made o he dra as a resul o he helpul and insighul commens

received. Publicaion o his paper represens he end o Phase I o his projec.

Limiaions

Te work discussed in his paper is preliminary, and remains a work in progress. Te inerviews

inorming his paper generally involved only one or wo people per organisaion usually evaluaion

specialiss. In uure, we would need more inpu rom programme, eld and managemen personnel.

Tereore, his paper is a sepping-sone o a broader undersanding o rends and issues in

srenghening organisaional capaciies, processes or, and undersanding o he evaluaion o

humaniarian aciviies. I is inended as a saging pos on a roue o opening discussion and asking hekinds o quesions which will help o advance he debae in pragmaic and pracically useul ways.

Ouline o his paper

Te remainder o his Inroducion presens a simple dra ramework or increasing he impac and

use o evaluaions o humaniarian acion. Tis ramework akes inspiraion rom general lieraure

on evaluaion uilisaion and capaciy building, bu is modied according o he specics o he

humaniarian conex.

Te ramework is hen populaed in Chapers 1 o 4 wih discussion on he specic issues ha

seem mos likely o increase impac and uilisaion as drawn rom he lieraure review and rom

inerviews wih a small sample o ALNAP members. Te concluding Nex seps secion oulines

plans or coninuing and developing he work presened in he preceding chapers.

A ramework or improving undersanding and use o EHA

A dra analyical ramework is presened here as an aid o urher analysis o wha infuences

undersanding and use o EHA. Te purpose o he ramework is o help shape discussions among

4


5/26

ALNAP members and ohers involved in evaluaing humaniarian acion on how o address he

problems o poor evaluaion impac. Te ramework draws on lieraure on capaciy developmen,

and was rened ollowing discussions wih humaniarian evaluaors in order o make i moreappropriae or use by hose involved in EHA.

Te ramework is a work in progress. Framework caegories ha do no add much will be removed,

while ohers will be added as needed. Te aim is o keep he ramework as ligh and as easy o

use as possible, wih a clear ocus on he key issues. Te aim is also no o be overly prescripive.

A imes, where here appears o be consensus rom inerviews and he lieraure, he discussions

wihin he ramework poin owards a soluion. However, elsewhere, he answer o a problem may

be very agency-specic, when an exploraion o he pros and cons o dieren approaches is more

appropriae.

Te bare bones o he ramework are presened here, wih discussion o each capaciy area in he

our chapers ha ollow. Overarching issues o leadership, culure and srucure are presened rs,

ollowed by issues o purpose and policy. Nex are more specic evaluaion processes and sysems.

Te ourh and nal capaciy area is supporing processes and mechanisms, some o which are o

wider relevance, and do no relae solely o evaluaions. Eecive capaciy building requires atenion

a all levels o he ramework, wih much iner-dependence beween hem.

Capaciy Area 1: Leadership, culure and srucure

Ensure leadership is supporive o evaluaion and monioring

Promoe an evaluaion culure

Increase he inernal demand or evaluaion inormaion

Creae organisaional srucures ha promoe evaluaion

Capaciy Area 2: Evaluaion purpose and policy

Clariy he purpose o evaluaion (accounabiliy, audi, learning)

Clearly ariculae evaluaion policy Ensure evaluaion processes are imely and orm an inegral par o he decision-making

cycle

Emphasise qualiy no quaniy

Capaciy Area 3: Evaluaion processes and sysems

Develop a sraegic approach o selecing wha should be evaluaed

Involve key sakeholders hroughou he process

Use boh inernal and exernal personnel o encourage a culure o evaluaion

5


6/26

Improve he echnical qualiy o he evaluaion process

Assign high prioriy o eecive disseminaion o ndings, including hrough new media

(video, web) Ensure here is a managemen response o evaluaions

Carry ou periodic mea-evaluaions and evaluaion synheses, and review

recommendaions

Capaciy Area 4: Supporing processes and mechanisms

Improve monioring hroughou he programme cycle

Provide he necessary human resources and incenive srucures

Secure adequae nancial resources

Undersand and ake advanage o he exernal environmen

Use peer neworks o encourage change

Engage wih media demands or inormaion

Engage wih donors on heir evaluaion needs

Acknowledgemens

Many hanks o everyone a he ALNAP Secrearia or heir suppor. John Michell, Ben

amalingam and Yuka Hasegawa all gave considerable ime o orienaing he work and commeningon dras. Josh Harris helped o se up he pre-workshop quesionnaire, and Colin Hadkiss helped

wih he organisaion o he workshop. Tanks o he many ALNAP and non-ALNAP members

who agreed o be inerviewed. Tanks, in paricular, o achel Bedouin, Aine Hearns and Karen

Proudlock, who gave deailed commens as ormal or inormal peer-reviewers.

6


7/26

Ensure leadership is supporive o evaluaion and monioring

A high proporion o he inerviews carried ou or his sudy idenied leadership as key in

improving he impac o EHA. Tis is refeced in he academic lieraure as well as in reviews o

he evaluaion uncions o various humaniarian organisaions. Inerviewees commened on

heir experience o a change in leadership having a proound and posiive impac on he value and

eeciveness o evaluaions. Where leaders are no ineresed in evaluaion, or are overly deensive

abou he perormance o heir organisaions and hence relucan o accep evaluaion ndings, a

culure develops agains learning rom experience. I daa and analysis are no valued a senior level,

his can permeae hroughou he organisaion and lead o relucance even o collec he necessary

inormaion in he rs place.

Even wihou supporive leadership, an evaluaion champion can show he wider organisaion

he benes ha can arise rom evaluaion. Good qualiy evaluaions ha highligh imporan

issues wihin programmes ha would oherwise have been missed can serve as a ool o convince

managers o he imporance o evaluaion. Te board or unders o an organisaion can seek o

ensure ha recruimen o senior managers emphasises he imporance o evaluaion. A recen sudy

on srenghening learning rom research and evaluaion in DFID (Jones and Mendizabal 2010)

suggesed ha a direcor be appoined as knowledge and learning champion.

Promoe an evaluaion culure

Proudlock (2009) noes ha he evaluaion lieraure (eg Paton 2008; Mayne 2008) suggess ha

culure, as well as leadership, are a key deerminan o evaluaive capaciy. Ideally, an organisaional

culure will acively seek inormaion on perormance in order o learn how o improve he

managemen and delivery o is programmes and services, and o improve is perormance. Tere

needs o be a commimen o using evaluaion daa and hen changing behaviour based on ha daa.

Enabling credible evaluaion ha is useul and ges used requires esablishing eecive sysems and

srucures, such as organisaional policies, sraegic planning, programme reporing and inormaion

sysems, budgeing, human resources, accounabiliy rameworks and learning processes. Boh Paton

(2008) and Mayne (2008) poin ou, however, ha his is only ever secondary o esablishing he

wo key elemens o culure and leadership. According o Mayne (2008), eors o creae evaluaion

sysems wihou addressing organisaional culure or leadership are likely o end up as burdensome

and could poenially work agains a culure o seeing evaluaion (or any resuls daa) as valuable and

worh pursuing.

1 . L E A D E S H I P, C U L U EA N D S U C U E

7


8/26

How do organisaions go abou culivaing a posiive inernal culure? Alhough, as above, evaluaion

expers sae ha culure mus come rs, and hen sysems, i also seems ha developing eecivesysems and srucures or credible evaluaions is an imporan par o he process o changing he

culure. Success breeds success, so ha high-qualiy evaluaions, ha address real inormaion needs,

improve he way in which evaluaions are regarded. Tis posiively aecs he evaluaion culure,

which, in urn, increases he demand or evaluaions, and makes i more likely ha evaluaion sysems

and srucures are improved in a susainable way. A viruous circle ensues. ackling issues a each sage

o he ramework idenied in his paper can help in creaing ha viruous circle.

While many o he building blocks or increasing evaluaion capaciy wil l be similar or any

organisaion, each organisaion already has is own unique culure and way o doing hings. Tereore,

each will need o analyse is own curren siuaion, is readiness or change, and he challenges o

building evaluaion capaciy. Dieren sraegies will hen be required or each organisaion. Tose

promoing change will need o be opporunisic, seizing he easy wins rs, so ha impac is quickly

seen and suppor is gahered or he challenge o changing he evaluaion culure.

Organisaions also sand o learn rom he atemps boh successul and unsuccessul ha similar

organisaions have made o improve evaluaion uilisaion. Furher developmen o he ramework

presened in his sudy will involve workshops, case sudies and join work by small groups o

ALNAP members, o help aciliae he process o peer learning.

Increase he inernal demand or evaluaion inormaion

Increasing demand or inormaion in order o make managemen decisions will increase he demand

or evaluaions. Bu how does one increase demand or inormaion? Paton, wriing on uilisaion-

ocused evaluaion (2008), alks abou he need, when considering poenial sakeholders and

possible uses, o move rom he general and absrac o he real and specic. Very early on in he

process, one needs o ideniy primary inended users and heir explici commimens o concree,

specic uses. In Patons view, he evaluaor aciliaes judgemen and decision-making by inended

users, raher han acing as a disan, independen judge.

Inended users are more likely o use evaluaions i hey undersand and eel ownership o he

evaluaion process and ndings. Tis is more likely i users have been acively involved in every sep o

he process. Paton suggess ha evaluaion should sar wih he generaion by end users o quesions

ha need o be answered. Such quesions should ideniy inormaion ha inended users would like,

and which would make a dierence o wha hey do.

Paton reers o research ideniying he personal acor as one o wo key acors ha emerged as

consisenly imporan in explaining evaluaion uilisaion.

8


9/26

Te personal acor is he presence o an ideniable individual or

group o people who personally care abou he evaluaion and hendings i generaes. Where such a person or group was presen,

evaluaions were used; where he personal acor was absen, here was

a correspondingly marked absence o evaluaion impac. (Paton 2008)

For Paton, he evaluaion process mus discover he key sakeholders, namely hose who really wan

o know somehing. Formal posiion and auhoriy may mater less han enhusiasm and ineres.

Evaluaors need o nurure and susain ineres, and his requires skills in building relaionships,

aciliaing groups, managing confics, and eecive inerpersonal skills as well as more echnical

evaluaion skills.

Tese ndings are echoed in a recen sudy on srenghening learning rom research and evaluaion

wihin DFID evaluaions:

he lack o ownership o large evaluaions or research programmes delivered exernally can help

explain heir lack o infuence. elaed o his, direc inerpersonal relaions beween sa and boh

researchers and evaluaors, as well as o he research and evaluaion processes, mater a grea deal.

(Jones and Mendizabal 2010)

Creae organisaional srucures ha promoe evaluaion

Tere are several ways o organising he managemen o evaluaions. In some organisaions, a separae

and/or cenral evaluaion uni is responsible or carrying ou evaluaions; in ohers, responsibiliy or

evaluaion is decenralised hroughou he organisaion (Hallam 1998). Te ype o managemen

srucure infuences, or good and bad, he impac o EHA. For example, having a cenral uni

dedicaed o evaluaions migh improve he qualiy o evaluaions bu could undermine broader

insiuional learning.

Tere is signican debae abou o whom an evaluaion deparmen should repor. Someorganisaions have an evaluaion deparmen ha repors direcly o he Board, or o he Chie

Execuive. Oher organisaions place he evaluaion deparmen lower down in he hierarchy. While

here is no righ or wrong approach, i is imporan or organisaions o recognise he impac ha

insiuional srucure has on learning and accounabiliy. Te IMF has an independen evaluaion

uni answerable o he Board. However, a recen evaluaion o he uni ound ha IMF managers were

very resisan o he Evaluaion Deparmen (personal communicaion).

9


10/26

Clariy he purpose o evaluaion (accounabiliy,audi, learning)

Wheher an evaluaion can simulaneously mee boh accounabiliy and learning needs has been

he mos conenious issue arising rom he inerviews held so ar. Te lieraure ends o sugges

ha hese wo aims are in confic, and many inerviewees agree. Indeed, a brainsorming session

among experienced humaniarian evaluaors a a recen raining course on evaluaion ound ha an

accounabiliy ocus in evaluaions was a acor in heir poor uilisaion (personal communicaion).

Paton noes ha an evaluaion required by a under oen becomes an end in isel o be done

because i is mandaed, no because i will be useul. Mandaed evaluaions can undercu uiliy by

making he reason or he evaluaion compliance wih a unding requiremen, raher han genuine

ineres in being more eecive (Paton 2008). Te recen ODI sudy on learning lessons rom

research and evaluaions wihin DFID (Jones and Mendizabal 2010) quoes a similar nding: here

is generally a ension beween he independence o evaluaion deparmens and heir success in

engaging users o evaluaion (Foresi 2007).

Ohers, however, are equally adaman ha any evaluaion mus address boh aims, and ha one canonly ruly learn by being held rigorously accounable. Te ruh will vary by organisaion and culure,

and one canno be prescripive abou such hings. Noneheless, wha does seem o be lacking in some

organisaions is clariy abou he purpose o evaluaions, and recogniion ha dieren approaches

may be required or dieren evaluaion aims. For example, he peer review o he evaluaion uncion

a WFP observes ha he ension and complemenariies beween evaluaion or accounabiliy and

or learning seem no o be acknowledged everywhere in WFP (Baker e al 2007).

Some argue ha real learning and perormance improvemen requires rank and open discussion

wihin an agency, an abiliy o share ailures as well as successes, and a non-judgemenal approach.When an evaluaion has been exernally commissioned, and is carried ou by eams o ousiders, such

an approach may be paricularly dicul o achieve. Some NGOs, ha have no been advocaes o he

independen, accounabiliy-syle approach o evaluaion, have, however, ully adoped more inernal

evaluaion approaches, which can sill involve a considerable degree o criicism. Such criicism is

more readily acceped when i comes rom insiders who undersand he organisaion well, and who

have hemselves been involved in managing similar programmes. Some organisaions have ound ha

hones and rank discussion is consrained where evaluaions are o be made available o he public.

2 . E VA L U A I O N P U P O S EA N D P O L I C Y

10


11/26

One way o miigae he ension beween he diering aims o evaluaion is o separae accounabiliy

evaluaions rom learning evaluaions, and no ry o mee all agendas wih one exercise. Arguably,

i migh even be beter o call he wo ypes o evaluaion by dieren names, and have he processesmanaged and run by dieren individuals and deparmens wihin he organisaion. Few people enjoy

or welcome nancial audis, or necessarily expec o learn rom hem, bu no one hinks hey should

no happen. In a similar way, perhaps, accounabiliy evaluaions should be carried ou by he audi

and accounabiliy deparmen, whereas eld-led, learning evaluaions should be carried ou by a

deparmen o knowledge managemen and learning.

Tis separaion may already be happening o some degree. Te UK Disasers Emergency Commitee

(DEC) now has an Accounabiliy and Audi Manager, as well as someone responsible or learning.

A recen DEC repor reers o separaing ou and srenghening learning. A SIDA, here is an aid

wachdog as well as an inernal evaluaion uni, and he inernal deparmen has a major ocus on

uilisaion-based learning evaluaions. DFID may be heading in a similar direcion. Once an explici

separaion has been made beween accounabiliy and learning evaluaions, i hen becomes easier o

develop a sraegy o improve boh approaches.

Clearly ariculae evaluaion policy

EHA policies and guidelines reer o he overall purposes and objecives o he evaluaion uncion or

a paricular agency and should be where all EHA originaes. However, alhough many organisaions

have evaluaion guidelines, no all disinguish beween developmen evaluaion and EHA. Ideally,organisaions should have a disinc se o deailed policy guidelines or EHA, as agencies wihou an

agreed policy someimes experience conusion regarding he evaluaion unis mandae and/or roles

(Foresi 2007).

Tose promoing change may need o sar wih developing he appropriae policy, bu hey will

hen need o ake a sraegic approach o rolling his ou hroughou he organisaion. When wriing

EHA policy guidelines, i is imporan o consider impac and uilisaion rom he ouse. Arguably,

uilisaion should be he rs aim o any evaluaion policy. Tis will increase he likelihood ha he

enire evaluaion process is designed around end users and heir needs.

Te organisaional challenges o ranslaing policy ino pracice are no limied o high urnover

o sa and poor knowledge managemen, and are oen underesimaed by evaluaors. For many

reasons, policies are no always ollowed, and i is naive o expec anyhing dieren. Where specic

EHA guidelines do exis, hey should be made available o evaluaion eams. Failure o do his can

resul in sub-sandard and even unusable work being produced (Soddard 2005).

11


12/26

Ensure evaluaion processes are imely and orm an inegral paro he decision-making cycle

A common complain rom poenial evaluaion users is ha evaluaions oen arrive oo lae o be o

any use in decision-making. A recen repor on he way DFID learns rom research and evaluaions

noes:

Te mos common criicism o evaluaions among he inerviewees was

iming: alhough hey mark imporan rhyhms or Counry Oces,

hey were seen o generally ake oo long o be relevan o policy eams,

and insucien atenion is paid o ying hem ino policy cycles and

windows o opporuniy. ( Jones and Mendizabal 2010)

Tis is even more he case or humaniarian evaluaions. I is no unusual or evaluaions o nish

(or even sar) only aer he peak o he emergency is over. No only is signican analysis made

more dicul by he passage o ime, as he memories o key inormans wane and inormaion is

los, bu long delays beween programmes and evaluaion conclusions means ha here is hen no

opporuniy o use he ndings o change he programme concerned.

As a resul o concerns such as hese, here has been a growh in he number o real-ime evaluaions

(Es) by a range o humaniarian organisaions. A recen E o he Haii Earhquake (Groupe

UD 2010) was able o underake a eld mission wihin one monh o he earhquake, and ocapure lessons abou he immediae response. I was also able o eed back o acors in he eld, eiher

bilaerally or o he cluser groups.

Some evaluaion deparmens work o heir own imeables, planning and commissioning evaluaions

wihou regard o he managemen processes ha acually infuence programmes. Evaluaions ha

appear aer major decisions have been made are acing an uphill sruggle i hey are o have any impac.

o have a beter chance o bringing abou change, evaluaion imeabling should sar wih an analysis o

programme planning cycles, and ensure ha evaluaion producs eed ino his.

Emphasise qualiy no quaniy

Tere is nie capaciy wihin any organisaion o commission, implemen and hen learn rom

evaluaion. Some organisaions commission evaluaions or a large proporion o heir programmes,

bu hen nd hemselves sruggling o ensure he qualiy o he process. Even where qualiy is

mainained, real refecion on he ndings o an evaluaion repor akes ime, and considering and

implemening recommendaions is even more demanding.

12


13/26

ICC used o reques all ineresed paries o ideniy he range o quesions and issues hey would

like included in an evaluaion. Te evaluaion deparmen hen reramed his ino evaluable

quesions. However, i was ound ha he scope o he evaluaion always grew, unil i became diculo manage he process. o miigae his, he evaluaion deparmen now ries o ocus on jus hree

key quesions or each evaluaion.

Inerviewees repored ha hey eel ha oo many (oen poor-qualiy) evaluaions are being carried

ou, and ha his works agains he developmen o a rue evaluaive culure wihin an organisaion.

Where repors are o insucien qualiy, hey lose credibiliy, and he evaluaion process becomes less

valued in he eyes o managers and implemeners. When he same recommendaions go unheeded

ime aer ime, because o a lack o capaciy o debae and ac on hem, no one akes hem seriously.

A percepion develops ha he evaluaions are done only or appearances sake. Someimes, a qualiy-

ocused approach is made more dicul i evaluaions are required by donors as a unding condiion,

orcing organisaions o commission more evaluaions han hey can absorb. Donors can be similarly

overwhelmed by he repors ha hey commission.

13


14/26

Develop a sraegic approach o selecing wha shouldbe evaluaed

Some evaluaion deparmens review heir organisaions annual expendiure and plan evaluaions on

he basis o ensuring ha all major programmes are evaluaed every ew years. Te guiding principle

o such approaches is ha accounabiliy demands coverage o he organisaions aciviies on a

cyclical basis. However, such a mechanisic approach does no necessarily lend isel o ensuring

eecive uilisaion and impac o EHA.

Tere may be litle ha is conroversial abou large programmes ha have been running or years,

and evaluaions are unlikely o lead o any radical changes. Tere may also be all sors o barriers o

change in some areas, where evaluaion ndings are likely o be ignored. For accounabiliy reasons,

evaluaions are oen required o large-budge programmes. A mehodology ha is ligher on he

learning aspec migh be more appropriae when such programmes are unconroversial. (Tis

underlines he imporance o he need o clariy he purpose o he evaluaion a he ouse, as noed

in Secion 2 above.)

A sraegic approach o selecing wha should be evaluaed is undamenally dieren rom he cyclicapproach described above, and sars by looking a how he evaluaion process can add value. As a

good example, SIDAs evaluaion planning cycle sars wih he evaluaion uni having conversaions

wih all operaional unis, o deermine heir knowledge needs, wha hey would like o know and

how evaluaion could help. A lis o around 100 ideas or evaluaions is iniially generaed, rom which

he evaluaion deparmen chooses he 15 hey will carry ou.

Evaluaions no chosen are subjec o decenralised evaluaions, on which he evaluaion deparmen

gives eedback and advice. Sa members who proposed any o he 15 seleced evaluaions orm

reerence groups or ha evaluaion. Tey mus lis he inended use and users. I here are no enoughusers, he evaluaion is dropped. Oherwise, hese inended users are hen involved in draing

erms o reerence, and work wih he reerence group hroughou (rom a presenaion by Joakim

Molander, SIDA, a DFID workshop on lesson learning, 22 June 2010).

Involve key sakeholders hroughou he process

Evaluaions are poliical and concern various ineres groups (Carlsson e al, 1994, in Sandison,

2006) and sakeholders, boh inernal and exernal. Te mos imporan exernal sakeholders are

he aeced populaions hemselves, who should arguably be a he cenre o no jus he evaluaion

3 . E VA L U A I O N P O C E S S E SA N D S Y S E M S

14


15/26

process, bu he whole humaniarian sysem. Alhough here is sill a long way o go or his o

be achieved, a large number o humaniarian organisaions are working o improve he curren

siuaion. I is now slowly becoming he norm or beneciaries o be involved a some sage o heevaluaion process, alhough his is usually only as inormans raher han in seting he evaluaion

agenda. Noneheless, his represens an improvemen on he siuaion o a decade ago, when ew

humaniarian evaluaions involved srucured discussions wih he aeced populaion a all.

Despie recen advances in downward accounabiliy, he Humaniarian Accounabiliy Programmes

recen annual repor3 nds ha considerably more needs o be done wih respec o evaluaion

pracice. However, given ha he evoluion o EHA o dae has been driven largely by greaer demand

or upward accounabiliy o donors, eors o reorien he evaluaion process owards he concerns

o disaser-aeced sakeholders may require enirely new mechanisms: i may no be possible simply

o use exising mechanisms o ull new and dieren uncions.

Tis paper does no seek o duplicae discussions on humaniarian accounabiliy being conduced

elsewhere. However, as a minimum, all evaluaions should ensure ha beneciaries are enabled o

conribue, wih heir ideas eeding ino analysis, learning and recommendaions. A large lieraure

exiss on a range o mehods or obaining a valid specrum o beneciary views, which is no o say

ha his is easy o achieve. Tere remain very considerable challenges in ensuring rue paricipaion

in siuaions where power srucures preven evaluaors rom accessing cerain voices, ypically hose

o women and minoriy groups. Tis is exacerbaed in confic siuaions where expressing opinions

migh be no only dicul bu also dangerous.

Oher exernal evaluaion sakeholders include he general public, he media, various governance

srucures, and donors. FAO sysemaically involves sakeholders rom he beginning o hemaic

or counry evaluaions, using consulaive groups including aeced people, major donors and

programme personnel. Te sakeholders orm an advisory group, and build rus and buy-in

considered essenial by he FAO Evaluaion Uni in achieving change hrough evaluaion. Te group

ensures ha he righ quesions are asked, and has a chance o commen on dra producs.

esearch on srenghening learning rom research and evaluaion in DFID (Jones and Mendizabal,2010) ound ha increasing he sense o ownership o evaluaions by DFID sa members was

essenial. Where inerviewees had been involved acively wih he DFID Evaluaion Deparmen

in one way or anoher, his improved heir undersanding o he deparmen, breaking down any

misconcepions abou being audied, and was el o be valuable. On he oher hand, concerns

were noed ha lessons rom evaluaions relevan o DFID more generally, a a sraegic level, had

no owner, and were hus less likely o be aced upon. UNHC has a seering commitee or each

evaluaion ha includes sa members rom he relevan unis.

3. htp://www.hapinernaional.org/pool/les/a-he-2009-humaniarian-accounabiliy-repor.pd

15
http://www.hapinternational.org/pool/files/a-the-2009-humanitarian-accountability-report.pdfhttp://www.hapinternational.org/pool/files/a-the-2009-humanitarian-accountability-report.pdf


16/26

Use boh inernal and exernal personnel o encourage a culureo evaluaion

A review o he lieraure and discussions wih eld sa reveals a common heme o eld and

operaional sa being unhappy wih curren evaluaion pracices. Many eld personnel eel

ha evaluaions ake up valuable ime, and ha hey end up eaching he evaluaors abou he

programme and he issues, and hen subsequenly learn litle rom he repors, i hey ever see

and read hem. Furhermore, hey requenly nd ha nohing seems o change as a resul o he

evaluaion. Tere is also concern ha ousiders need a lo o ime o learn abou he culure and

pracice o he organisaion being evaluaed. ecommendaions can be unrealisic where his learning

has no aken place, damaging he credibiliy o he evaluaion.

A number o inerviewees menioned ha hey have sared using more inernal evaluaion, or

mixed eams o insiders and ousiders. Te involvemen o insiders means ha ndings and

recommendaions are more likely o be appropriae. In addiion, insiders are more likely o have a

beter undersanding o he concerns o eld personnel, and o heir perspecive on key issues. Using

inernal evaluaors also brings he enormous bene o reaining he experience and knowledge

gained by hose carrying ou he evaluaion. I is expensive o hire exernal evaluaors o learn huge

amouns abou he organisaion and he emergency response, only or hem o ake away his

knowledge when heir work is done.

Despie he many benes o using insiders, some organisaions are implacably opposed o doingso, earing ha his will limi independence and resric any radical recommendaions ha migh

be required on resrucuring he organisaions response. As noed above, i is essenial o clariy he

reasons or carrying ou an evaluaion, and o recognise ha rade-os are ineviable.

Improve he echnical qualiy o he evaluaion process

Te raionale behind his sudy on improving evaluaion uilisaion is ha here are many acors

aecing he impac o EHA, and ha echnical qualiy is only one o hese. However, echnical

qualiy is clearly very imporan. High-qualiy evaluaions increase he credibiliy o he wholeevaluaion process, and creae he poenial or a viruous circle o develop: i evaluaions are valued

more highly, his creaes he righ condiions or more o hem o be o higher qualiy in uure.

ALNAP is commissioning a new Guide o he Evaluaion o Humaniarian Acion as a separae

exercise, and so his imporan issue will no be discussed urher here.

16


17/26

Assign high prioriy o efecive disseminaion o ndings,including hrough new media (video, web)

Evaluaions are o poenial ineres o agency managers, beneciar y represenaives, local

governmens, donor ocials, sudens o humaniarian issues, and he press, as well as he general

public. Ideally, each audience should receive a dieren produc ailored o is needs (Hallam 1998).

However, increasing impac is also abou consrucing pahways or he evaluaion ndings o make a

dierence wihin he organisaion:

perorming a good qualiy evaluaion is only he rs sep. Te lessons

hen have o be absorbed, aken orward, and implemened in pracice

beore organisaional learning can be said o have occurred. (Soddard

2005)

I is oen assumed ha evaluaions yield inormaion or provide lessons learned ha fow direcly

back ino he policy cycle and are hus incorporaed in he planning o uure programmes and

projecs. In his way, here is a consan learning process leading o ever-improving perormance

(Frerks and Hilhors 2002). Tis undersanding presupposes a raional, scienic planning model

which has never been adoped in daily developmen pracice. In a plural, complex and disorderly

sociey, decisions on goals and programmes are oen poliical compromises ha do no necessarily

correspond wih he oucomes o evaluaion (Frerks and Hilhors 2002).

Sraegic disseminaion o ndings is key o making evaluaions more eecive. In a personal

communicaion, a senior evaluaor commened on he dierence in he impac o evaluaions he

had noed when moving rom a large and well-resourced evaluaion deparmen o a much smaller

one in a dieren organisaion. Despie he dierences in resources available, he evaluaions carried

ou by he smaller organisaion had signicanly more impac. Tere were several reasons or his,

bu he mos imporan reason was he smaller organisaions commimen o disseminaing he

resuls and argeing ha disseminaion eecively. Indeed, he evaluaion process began wih he

communicaion sraegy, raher han disseminaion being hough abou only once here was a

(perhaps inappropriae) repor in hand. Planning ahead infuences he ype o inormaion collecedhroughou he evaluaion, and ensures i mees he needs o decision-makers.

A recen ODI paper on learning lessons rom research and evaluaion ( Jones and Mendizabal 2010)

echoed he above ndings, and ound ha how and when evidence is produced, presened and

communicaed maters. Opions include argeed seminars and presenaions, one-o-one briengs

or eam leaders, an evaluaion deparmen newsleter or brieng papers, shor email producs, and

he developmen o new producs such as documens ha presen lessons rom evaluaions along

hemaic, regional/naional and programmaic/policy lines. Focusing on personal inerrelaions

raher han inermediaries was also considered imporan. Te same sudy ound ha he orma and

17


18/26

presenaion o evaluaions could be improved: Some inerviewees el ha he ull repors were oo

long and echnical... i is likely ha his is a common problem wih he evaluaion proession (Jones

and Mendizabal 2010).

Some agencies have sopped puting recommendaions ino evaluaion repors, because hey eel ha

ocus on recommendaions can derac rom analysis and learning. Te mos imporan par o he

evaluaion or hem is he analysis, because his is wha encourages learning. I is beter ha hose who

know and undersand he organisaion develop and ake ownership o he recommendaions. I is

also imporan ha evaluaion repors and relevan ndings are easily accessible. Ways o enabling his

include: good caaloguing; improving inranes o make i easier o search and browse, and making

hem SS compaible o allow sa o link o hem wihou having o access he sie every ime hey

wan inormaion; and producing global lessons learned rom mea-analyses.

Inerviews conduced or his sudy revealed some innovaive disseminaion sraegies. A number

o organisaions, including UNHC, ICC and Groupe UD, have made documenaries around

evaluaions, some reporing powerully on he key issues ound in he evaluaion solely in he words

o beneciaries. ALNAP isel produced a Lessons Paper on humaniarian aciviies aer earhquakes,

by reading and disilling he lessons rom a number o evaluaions o earhquake response (Cosgrave

2008). Tis paper was downloaded over 3,500 imes wihin days o he January 2010 earhquake in

Haii.

Ensure here is a managemen response o evaluaions

Managemen response and ollow-up o evaluaions is a key area or improving he impac o

evaluaions. Tere are numerous anecdoal repors o evaluaion repors gahering dus, largely

unread, and wih litle ormal or inormal ollow-up on ndings and recommendaions. Tis may be

parly because o problems noed above paricularly lack o clariy o purpose o he evaluaion,

and lack o involvemen by key sakeholders. Noneheless, ormal sysems o managemen response

o evaluaions serve o reduce he chance ha an evaluaion process ends wih he producion o he

repor.

Dieren organisaions ake dieren approaches o his issue. UNDP has creaed an Evaluaion

esource Cenre. Tis is a public plaorm, designed o make UNDP more accounable, where

evaluaion plans are logged along wih a managemen response. esponses and ollow-up are racked

by he evaluaion deparmen and repored o he Execuive Board o UNDP. Some DFID counry

oces hold an in day o go over perormance rameworks and evaluaion resuls, o make sure ha

key lessons are no los.

In FAO, here is a process in which senior managemen commens on he qualiy o evaluaion

repors, as well as on wha ndings hey accep and wha acions are planned o address hese.

18


19/26

Tis eedback is presened o he governing body along wih he evaluaion. In addiion, or major

evaluaions, here is a urher sep in he process: wo years aer he evaluaion, managers are required

o repor o he governing body on acion aken on he recommendaions hey acceped a he ime.For FAO, hese have proven o be powerul managemen ools, and which include he opporuniy o

revisi evaluaion ndings, and o have a dialogue abou he managemen response o hem.

Carry ou periodic mea-evaluaions and evaluaion synheses,and review recommendaions

Te huge demand ollowing he 2010 Haii earhquake or ALNAPs Lessons Paper esponding o

Earhquakes: Learning rom earhquakes relie and recovery operaions (Cosgrave 2008) has been

menioned above. I seems here is a signican demand wihin he humaniarian secor or such

mea-analysis and hemaic reviews.

Oxam has recenly sared o use he same six benchmarks or is inernal repors.

Every ew years, a mea-evaluaion is carried ou agains hese benchmarks which is

incredibly valuable (personal communicaion).

CAE has carried ou a mea-review o evaluaions and aer-acion reviews rom

24 emergency responses, and drawn ogeher key learning rom his.

Every year, NORD produces a synhesis repor o lessons rom evaluaions.

Te ODI paper on learning lessons rom research and evaluaions (Jones and

Mendizabal 2010) ound glowing reerences rom inerviewees o research andevaluaion ha oered synhesis and comparison o work rom around DFID on

paricular hemes or secoral areas.

Such mea-approaches and synheses are imporan in exracing ull value rom expensive evaluaion

processes. Tey help o ensure ha ndings across many dieren evaluaions are validaed and are

no specic o jus one projec. Greaer consisency o ndings across programmes leads o more

condence in heir credibiliy, and so o greaer poenial impac o making changes on he basis o

such ndings.

19


20/26

Improve monioring hroughou he programme cycle

Evaluaors requenly nd ha a lack o monioring daa signicanly impedes heir abiliy o make

meaningul commens on he eeciveness o a humaniarian programme. A imes, managers seem

o reques evaluaions o compensae or he lack o monioring inormaion. Tis is an expensive way

o obaining inormaion ha should be rouinely colleced and analysed.

Te lack o monioring daa is a long-sanding problem, due parly o he naure and circumsances o

humaniarian work: a mulipliciy o acors, he fuidiy o he siuaion, he diculies o working in

he conex o war and insabiliy, and he requen absence o baseline daa on he previous condiion

o he aeced/arge populaions. However, here are also atiudinal consrains, where emergency

personnel do no consider i a prioriy o se up monioring sysems rom he ouse. Tere is

also a lack o agreemen on sandardised monioring procedures and proocols among agencies,

despie widespread adopion by many humaniarian agencies o Sphere sandards. Indeed, several

inerviewees suggesed ha all humaniarian programmes should repor agains Sphere sandards.

Sandardisaion o daa collecing and reporing would make i easier or he sysem as a whole o see

where gaps exis in humaniarian provision. I agencies and donors could agree on wha daa shouldbe colleced, his would signicanly increase he likelihood o i happening, meaning ha evaluaors

could be more conden ha hey will nd he daa hey need o carry ou heir work. A disadvanage

o such an approach would be ha here migh be a endency o ocus on he measurable and pay less

atenion o conexual analysis, already a weak poin in some humaniarian evaluaions. Conex-

specic monioring would always be required as well, o ensure ha a programme was meeing is

saed objecives, alhough his would no preven sandardisaion in many areas.

Provide he necessary human resources and incenive srucures

Te incenive srucures o... agencies do no necessarily reward

hose in he evaluaion deparmens, which, as a resul, are no able

o ofer clear career opporuniies or saf members (Foresi 2007).

A WFP repor made similar commens abou career incenives or

saf in evaluaion deparmens: [evaluaion oce] poss are highly

sressul regarding relaions beween saf and oher members o he

organisaion Some el ha a posing in [he evaluaion oce] was a

bad career move and migh afec he individuals uure career. (Baker

a al 2007)

4 . S U P P O I N G P O C E S S E SA N D M E C H A N I S M S

20


21/26

Given such incenive srucures or lack o hem, i may be ha evaluaion deparmens will no

perorm a opimal levels.

As well as ormal incenive srucures, here are also many inormal incenives ha can eiher promoe

good evaluaion or ac as an obsacle o i. Such incenives can be suble o ousiders, bu play an

imporan role in deermining wheher evaluaions are likely o be used eecively. For example,

where managers demand an evidence base beore implemening new policies and programmes,

evaluaions are likely o be more highly valued.

A recen sudy o lesson learning rom DFID research and evaluaions (Jones and Mendizabal 2010)

ound ha he demand or an evidence base increased in circumsances where a rapid response was

required o an unexpeced even. . Under hese circumsances, he abiliy o personnel o use evidence

and lessons rom research and evaluaions wihin heir work undoubedly buil heir credibiliy and

infuence wihin DFID. In paricular, evidence ha proved he value or money o a paricular policy

area was highly prized, as his was useul o DFID leaders in srenghening heir case or increased

budges wih he UK reasury (underlining he imporance o messages rom he op).

However, in general, he sudy ound ha original hough and new ideas were more likely o be

rewarded han he use o proven ideas or lessons learned elsewhere. Oriening incenives owards

new hinking, whils undersandable, is likely o miliae agains he eecive use o evaluaion. Te

same sudy also ound ha ime pressures encouraged sa members o rely on experience, raher

han evidence based on research and evaluaion, o suppor heir argumens.

Te inormal incenive srucures wihin an organisaion are par o is culure, and merely changing

ormal srucures migh no ackle he underlying and less easily measured consrains o eecive

evaluaion. Srong leadership, which suppors he use o evidence and evaluaion, is imporan in

bringing abou change in such circumsances. However, i is also imporan no o neglec he ormal

incenive srucures and processes. Evaluaion sa should be rained o an appropriae degree, and

have access o coninuing educaion on evaluaion. Sa roaion hrough he evaluaion deparmen

should be encouraged and regular.

New members o sa should learn abou he evaluaion sysem when hey are induced ino he

organisaion. As discussed above, using a mix o insiders and ousiders in evaluaion eams can help

sa members o learn abou evaluaion, improve heir analyical skills and promoe heir accepance

o evaluaion. Programme planning processes should be designed so ha here is ime and space o

consider and debae evaluaion ndings, and or recommendaions (where acceped) o be adoped.

Secure adequae nancial resources

Alhough many evaluaions are driven by issues o cos-eeciveness, he coss and benes o

21


22/26

evaluaion isel are rarely analysed. esearch in his area could lead o a beter undersanding o

he concree benes o evaluaions, proving ha Doing humaniarian evaluaions beter hrough

an ongoing process o refecion and learning is he mos cos-eecive way or ensuring hahumaniarian programmes improve (Aphorpe e al 2001), and hus improving he saus o

evaluaion sa and deparmens.

Tere is a widespread percepion ha join evaluaions are prohibiively cosly, ye, provided hey

are well-designed and managed, heir benes may ouweigh hese coss. Many eld workers and

managers complain abou he coss o evaluaions, ye do no always sop o consider he poenially

huge nancial and human coss o coninuing wih unsuccessul programmes (perhaps because

evaluaions only rarely succeed in infuencing such programmes).

In he pas, some organisaions allocaed a xed percenage o programme coss o evaluaion

budges. While his was iniially useul as a way o generaing resources or evaluaions in

organisaions no used o doing hem a all, i is unlikely o be he mos cos-eecive way o using

scarce resources. Beter o ake a sraegic approach, wih proporionaely more resources going

o complex programmes, or programmes ha are more conenious and where evidence or

eeciveness is less readily available.

Undersand and ake advanage o he exernal environmen

Use peer neworks o encourage change

Peer neworks are an essenial componen o bringing abou change. Tis paper was commissioned

by ALNAP, a key nework on humaniarian evaluaion, in order o promoe peer learning around

evaluaion impac. Peers, by deniion, work in similar conexs, and so heir experiences can be used

by similar organisaions o increase heir own learning. ALNAP promoes peer learning hrough

workshops and publicaions, and visis o member organisaions.

Te evaluaion deparmens o he UN organisaions also have a peer grouping known as he UN

Evaluaion Group (UNEG). UNEG has is own peer review process, which involves visiing one ohe organisaions and assessing is evaluaion uncion agains a se o norms and sandards developed

by he group as a whole. Members have ound he peer review process useul or raising he prole o

evaluaion wihin heir organisaions, and helping evaluaions become more embedded wihin he

organisaions processes.

Some organisaions have sough o develop heir own peer groups. For example, FAO has now

insiuionalised peer reviews, heir Charer o Evaluaions demanding and budgeing or a biannual

peer review o he organisaions evaluaion uncion.

22


23/26

Engage wih media demands or inormaion

Te media have a endency o caegorise humaniarian acors as eiher sains or sinners. odaysmajor humaniarian responses are characerised by huge media ineres, wih journaliss scrambling

o be he rs on he scene and hen complaining abou he slow speed o he response, ignoring

he many consrains and challenges involved. Tis can lead o a deensive response by agencies

and unwillingness o acknowledge any problems or ear o criicism. Tis works agains a culure o

learning and improvemen.

Te UK DEC changed is policy on commissioning and publishing evaluaion sudies ollowing

negaive media repors aer he response o he Souhern Arica drough in 2002/03. Ironically, he

evaluaion repor was generally very posiive, wih relaively ew criicisms. Unorunaely, i was hese

criicisms ha were seized on by he press. An arguably more eecive response over he longer erm

would be a greaer openness and a willingness o accep ha problems occur, and o debae hem.

Tis, aer all, happens wih many oher public inervenions (healh service delivery, or example)

wihou he public losing suppor or he inervenion.

Ineresingly, he response o he 2004 sunami appeal he one immediaely aer he appeal or

he Souhern Arica drough discussed above was srongly suppored by he UK public, wih no

evidence ha any donaions were held back because o earlier media criicisms o he same agencies.

Agencies should also engage direcly wih he media o educae hem abou humaniarian response,

o avoid he unbalanced and unhelpul journalism ha can accompany a large-scale emergency.

Engage with donors on their evaluation needs

According o Frerks and Hilhors (2002), EHAs are more oen han no donor iniiaives driven

by donor values and ineress. I is, hereore, vial ha donors suppor eors o improve he

impac o EHA. A signican barrier wihin organisaions o developing an evaluaion culure

is ha evaluaions are oen seen as compliance insrumens (as discussed in Secion 2 above).

Organisaions commited o evaluaions only as a condiion or accessing unding are unlikely

o value or have ownership o he resuling producs. Consequenly, agencies may under valueevaluaions, which can represen poor value or money in erms o he acual learning and

perormance change ha hey bring abou.

A repor on DFIDs lesson learning rom research and evaluaions ( Jones and Mendizabal 2010)

ound ha, in counries ha were higher HQ and UK poliical prioriies, he prole o evaluaion was

enhanced. Tis mean ha recommendaions were more likely o be acceped (again demonsraing

he powerul infuence o clear leadership demands on pracice). I donors and agencies can agree a

sraegic approach, hen boh could bene rom carrying ou ewer, more argeed evaluaions ha

ocus on key issues raher han a predeermined lis o projecs and programmes.

23


24/26

Nex seps

Te purpose o his paper is o simulae debae on wha is required o increase he impac o

humaniarian evaluaions. From his iniial analysis, and subsequen workshops wih praciioners, we

hope o develop plans o build on his work and bring abou meaningul change.

Wihou doub, he way orward will include building sronger links beween evaluaion and

programme/managemen sa. I is undeniable ha he majoriy o he characerisics and

impedimens o organisaional change are beyond he conrol o evaluaion (Sandison 2006).

Evaluaion mus be placed and judged wihin a larger managemen ramework, or i we coninue

o expec evaluaion o cover mos o he accounabiliy and learning needs o he secor, we will be

disappoined (Sandison 2006).

Te nex ask will be o develop a sel-assessmen quesionnaire, so ha organisaions can see how

hey are doing in erms o addressing he issues ha infuence evaluaion uilisaion. Tis paper, as

well as he quesionnaire, will provide a ramework or generaing more deailed case sudies o how

organisaions have improved he uilisaion o humaniarian evaluaions. Chie execuives o ALNAP

member organisaions will be inerviewed, o nd ou wha inormaion senior decision-makers wan,

where hey ge his inormaion rom, and how evaluaions could beter serve heir needs.

I is vial ha we check evaluaors prioriies or change agains hose o operaional managers.

Wihou dialogue beween he wo groups, he poenial impac o evaluaion as a managemen ool

will no be realised. Te case sudies and he inormaion generaed rom urher in-deph inerviews

should signicanly enrich he ideas discussed in his paper, and will lead o a revised version laer in2011.

E F E E N C E S

Cosgrave, J. (2008) esponding o Earhquakes: Learning rom earhquake relie and recovery

operaions. ALNAP Lessons Paper. London: ALNAP.

Aphorpe, R., Boron J. & Woods (eds.) (2001) Evaluaing Inernaional Humaniarian

Acion: efecions rom Praciioners. London: Zed Press.

Baker, J. e al (2007) Peer eview: Evaluaion Funcion a he World Food Programme. SIDA.

Carlsson, J. e al (1994) Te Poliical Economy o Evaluaion. London: Macmillan Press.

Groupe URD (2010) eal-ime Evaluaion o he esponse o he Haii Earhquake o 12 January

2010. (available a www.urd.org).

Hallam, A. (1998) Evaluaing Humaniarian Assisance Programmes in Complex Emergencies.

elie and ehabiliaion Nework, Good Pracice eview 7. London: ODI.

24


25/26

Foresi, M. (2007)A Comparaive Sudy o Evaluaion Policies and Pracices in Developmen

Agencies. London: ODI.

Frerks, G. & Hilhors, D. (2002) Evaluaion o Humaniarian Assisance in Emergency

Siuaions, UNHC/EPAU: New Issues in eugee esearch, Working Paper 56.

Jones, H. & Mendizabal, E. (2010) Srenghening learning rom research and evaluaion: going

wih he grain. London: ODI.

Mayne, J. (2008) Building an Evaluaive Culure or Eecive Evaluaion and esuls Managemen.

Insiuional Learning and Change Iniiaive (ILAC) Brie 20. (available a www.cgiar-ilac.org).

Paton, M. (2008) Uilizaion-Focused Evaluaion. London: SAGE Publicaions.

Proudlock, K. (2009) Srenghening Evaluaion Capaciy in he Humaniarian Secor: A

Summary o Key Issues. (auhors unpublished noes).

Sandison, P. (2006) Te Uilisaion o Evaluaions in ALNAP eview o Humaniarian Acion:

Evaluaion Uilisaion. London: ODI.

Soddard, A. (2005) eview o UNICEFs Evaluaion o Humaniarian Acion. New York: Unied

Naions Childrens Fund.

WFP (2009) Closing he Learning Loop Harvesing Lessons rom Evaluaions: epor o PhaseI. ome: World Food Programme.

25


26/26

Overseas Developmen Insiue111 Wesminser Bridge oadLondon SE1 7JD, UK

el: + 44 (0)20 7922 0300Fax:+ 44 (0)20 7922 0399Email: [email protected]

ALNAP Working Paper

harnessing the power of evaluation in humanitarian action: an initiative to improve understanding...

Documents