the alnap meta-evaluation
DESCRIPTION
The ALNAP Meta-evaluation. Tony Beck Presentation for the IDEAS Conference, Delhi, 14 th April 2005. Outline. Background The ALNAP Quality Proforma Agency visits Findings from the agency visits Finding from the Quality Proforma. What is the ALNAP and its meta-evaluation?. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/1.jpg)
The ALNAP Meta-evaluation
Tony Beck
Presentation for the IDEAS Conference, Delhi, 14th April 2005
![Page 2: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/2.jpg)
Outline
1) Background
2) The ALNAP Quality Proforma
3) Agency visits
4) Findings from the agency visits
5) Finding from the Quality Proforma
![Page 3: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/3.jpg)
What is the ALNAP and its meta-evaluation?
An overview of evaluation of humanitarian action quality
Identification of strengths and weaknesses
Recommendations for improvement across the sector and in individual agencies
![Page 4: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/4.jpg)
Process
• Review of evaluation reports against a set of standards
• Visits to and interaction with agency evaluation offices
Focus:• 2001-2002: Accountability• 2003-2005: Accountability and: good practice,
dialogue, interaction
![Page 5: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/5.jpg)
The ALNAP Quality Proforma
• ALNAP’s meta-evaluation tool
• Draws on good practice in EHA and evaluation in general
• Revised and peer reviewed in 2004
![Page 6: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/6.jpg)
The ALNAP Quality Proforma
Made up of seven sections:
1. Terms of reference
2. Methods, practice and constraints
3. Contextual analysis
4. Analysis of intervention
5. Assessing the report
6. Overall comments
![Page 7: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/7.jpg)
The ALNAP Quality Proforma
4 point rating scale
A = good
B = satisfactory
C = unsatisfactory
D = poor
Guidance notes for meta-evaluators. Eg: Consideration given to confidentiality and dignity?
Guidance: The evaluation report should detail how the overall approach and methods will protect confidentiality and promote respect for stakeholders’ dignity and self-worth.
![Page 8: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/8.jpg)
The ALNAP Proforma
CoverageCoverage
2001-2005: 197 evaluations
Process•2 meta-evaluators•Reconciliation of rating•Analysis by section
![Page 9: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/9.jpg)
Mainstreaming of the Quality Proforma
• By ECHO to revise tor (lesson learning, protection, identification of users, prioritisation, time frame and users of recommendations etc)
• DEC Southern Africa evaluation (rated 7 agency report)
• Groupe URD (for planning of evaluations)
![Page 10: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/10.jpg)
Agencies included in dialogue: 2003-4
CAFOD, Danida, ECHO, ICRC, OCHA, OFDA, Oxfam, SC-UK, SIDA, UNHCR, and WHO
![Page 11: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/11.jpg)
Purpose of agency dialogue
• Agency response to initial two years of use of Quality Proforma
• To discuss Quality Proforma rating and agency strengths and weaknesses
• To discuss processes leading to good evaluation practice
• To discuss goof practice
![Page 12: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/12.jpg)
Findings from dialogue with evaluation managers
• Areas affecting evaluation quality are not currently captured by the QP, eg
Evaluation quality depends on subtle negotiations within agencies
Evaluation funds in most cases are not being allocated for follow-up
Follow-up to recommendations is complex
More agencies are using tracking matrices
![Page 13: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/13.jpg)
Findings from dialogue with evaluation managers: the EHA market
• Main constraint to improved evaluation quality is agencies accessing available evaluators with appropriate skills
• Does the EHA market need further regulation?
![Page 14: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/14.jpg)
Findings from the Proforma
Area of enquiry
Rating % attaining rating 2004
% attaining rating 2001-2003
TOR – Good practice in approach and method
Good or
Satisfactory
Unsatisfactory
or Poor
6
94
11
89
![Page 15: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/15.jpg)
Findings from the Proforma
Area of enquiry
Rating % attaining rating 2004
% attaining rating 2001-2003
TOR – Intended users and uses
Good or
Satisfactory
Unsatisfactory
or Poor
12
88
8
92
![Page 16: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/16.jpg)
Findings from the Proforma
Area of enquiry
Rating % attaining rating 2004
% attaining rating 2001-2003
Consultation with primary stakeholders
Good or
Satisfactory
Unsatisfactory
or Poor
16
84
13
87
![Page 17: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/17.jpg)
Findings from the Proforma
Area of enquiry
Rating % attaining rating 2004
% attaining rating 2001-2003
Use of the DAC criteria
Good or
Satisfactory
Unsatisfactory
or Poor
52
48
50
50
![Page 18: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/18.jpg)
Findings from the Proforma
Area of enquiry
Rating % attaining rating 2004
% attaining rating 2001-2003
HR and management
Good or
Satisfactory
Unsatisfactory
or Poor
50
50
51
49
![Page 19: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/19.jpg)
Findings from the Proforma
Area of enquiry
Rating % attaining rating 2004
% attaining rating 2001-2003
Coordination Good or
Satisfactory
Unsatisfactory
or Poor
52
48
50
50
![Page 20: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/20.jpg)
Findings from the Proforma
Area of enquiry
Rating % attaining rating 2004
% attaining rating 2001-2003
Quality of evaluation of protection issues
Good or
Satisfactory
Unsatisfactory
or Poor
32
68
10
90
![Page 21: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/21.jpg)
Findings from the Proforma - 2005
• Improvement in most areas noted above of between 10 and 30 per cent
• Too early to disaggregate or suggest why this improvement has taken place
• Still a number of areas of generic weakness
![Page 22: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/22.jpg)
Conclusions
Process:•Meta-evaluations need to include interaction with those being meta-evaluated•Agency visits have been important is discussing constraints to improved evaluation quality•Meta-evaluations need to maintain an appropriate balance between accountability functions and the need to improve evaluation quality through lesson learning
![Page 23: The ALNAP Meta-evaluation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681573a550346895dc4e18b/html5/thumbnails/23.jpg)
Conclusions: findings
• EHA demonstrates some areas of strength, and improvement over four years, eg use of most of the DAC criteria, analysis of HR
• Many evaluative areas need to be strengthened, eg gender, identification of use and users, participation of primary stakeholders, transparency of methodologies used