Update on the Multilateral Effectiveness

Initiative

James Melanson

Director of Evaluation

CIDA

DAC Network on

Development Evaluation

June 2013

The Challenge

Gap in information on development effectiveness

• Great need for performance information of MOs – Current climate for evidence-based decision making

regarding resource allocation• Variable coverage, quality and reliability of reporting among

MOs does not provide clear picture of their performance• Infrequent, lengthy and high cost of independent joint

evaluation of MOs• Existing efforts focused primarily on

organizational assessment of MOs that do not directly address development effectiveness

2

The Response

New methodology

• 2009 – DAC EVALNET Task Team established to develop new methodology that:– Generates a body of credible information on a common set of

criteria that would provide a picture of the development effectiveness of MOs

– Builds on evidence (evaluation reports) which is already available– Uses methods that are modest in time and cost requirements and

with limited burden on MOs

• 2010 – Methodology developed and pilot tested (ADB & WHO)

• 2011 – DAC EVALNET endorses as an acceptable methodology

3

Assessing Development Effectiveness

Common criteria

• Methodology focuses on a description of development effectiveness that maps onto the DAC evaluation criteria

– Achievement of development objectives and expected results– Crosscutting themes (environmental sustainability and gender

equality)– Sustainability of results/benefits– Relevance of interventions– Efficiency– Use of evaluation and monitoring to to

improve effectiveness

4

Scenarios & Options

5

Scenario AMO reporting on DE is

adequate

Scenario BMO reporting on DE is not adequate but evaluation

function is

Scenario CMO effectiveness reporting and available evaluations

inadequate for reporting on DE

Option 1Rely on MO

reporting systems

Option 2Conduct a systematic

synthesis of information from

available evaluations

Option 3 Implement actions

aimed at strengthening MO evaluation system

and DE reporting

Apply the meta-synthesis of evaluation results methodology

Preliminary ReviewEstablish Universe, Screen Reports

Methodology

Preliminary Review

• Establish universe of evaluation reports prepared by the MO over a three- to four-year time-frame

• Select sample that provides reasonable coverage of MO programming (geographic, thematic, objectives, sector, technical focus)

• Screen reports from the sample for quality using DAC and UNEG’s accepted quality standards

• Decide on Scenario A, B or C depending on screening results– In order to validate if MO falls under Scenario A, it might still be

assessed under Scenario B

6

Methodology

Meta Synthesis (Scenario B)

• Review, analyze and classify evaluation findings for each criterion (operational guidelines)

• Identify contextual factors contributing to or inhibiting effectiveness for each criterion

• Prepare report that summarizes findings and context, and establishes conclusions and recommendations

7

Experience to Date

• CIDA led, jointly with the Netherlands, the WFP and UNDP reviews Successful use of the methodology – WFP and UNDP fall under scenario B

(reporting on DE not adequate but evaluation function is) Reviews provided good understanding of the organizations’ development

effectiveness Constructive conversations took place at the boards

• CIDA prepared reports based on the pilot tests of the ADB and WHO reviews Successful use of the methodology Reviews provided good understanding of the organizations’ development

effectiveness, however, given the low number of evaluations available from WHO, no generalization at the organization level was possible

8

Experience to Date

• CIDA led the AfDB review Report in process of finalization and will be published on CIDA and

EVALNET websites

• Netherlands led the UNICEF review Report presented to the board in May 2013

9

The Findings

• Organization is effective in achieving most of its objectives and expected results

• Programs are highly relevant to the needs of the target groups and developing country governments

• Sustainability and efficiency represent areas for improvement • Challenges exist with gender equality and environmental

sustainability • Good use of evaluation, but inadequate performance

frameworks and weak monitoring

10

Scenario B

The Findings

• Organization is effective in achieving most of its objectives and expected results, and in supporting gender equality and environmental sustainability

• Programs are highly relevant to the context in which they work• Improving the sustainability of benefits remains a challenge• Efficiency is an area for improvement • Organization faces issues in strengthening decentralized

systems for evaluation, monitoring and results-based management

• Evaluation Office produces high quality evaluations

11

Scenario B

The Findings

• Insufficient evidence available to make generalizable conclusions

• The limited number of evaluation reports provide some insights into the effectiveness of those programs– Programs appear to be relevant to stakeholder needs and national

priorities and effective in achieving most of its development objectives and expected results

– Programs appear to be sustainable, but there are challenges in sustaining the capacity of its partners

– Evaluations have not regularly addressed effectiveness in supporting gender equality or environmental sustainability

– Systems for evaluation and monitoring to be unsatisfactory

12

Scenario B

The Findings

• Most programs achieve their objectives and expected results• Programs are relevant to stakeholder needs and national

priorities• Improving the sustainability of benefits remains a challenge• Efficiency represents an area for improvement • Programs contribute to gender equality and environmental

sustainability, but improvements are needed with the latter• Evaluation is effective and well used, but challenges are

highlighted in monitoring and results-based management

13

Scenario A

The Findings

• Programs are largely effective and highly relevant to the needs of target groups

• A stronger focus on gender equality and environmental sustainability is needed

• Continuity and sustainability of program benefits remain a challenge

• Efficiency appears to be a challenge• Effective use of evaluation through increased preparation of

management responses, but challenges remain with integration and dissemination of evidence

• Challenges remain with respect to monitoring and results-based management

14

Scenario B

MO Engagement

• Donors engage with multilateral organizations during the whole review process– Launch of the review– Establishment of the evaluation universe– Confirmation of evaluation sample– Preliminary findings– Draft report – Final report

• Donor-neutral version prepared for each of the reviews and published on the OECD EVALNET website For use by any partner donor

15

Utility of this approach?

• For CIDA– Allowed demonstration of accountability for results in multilateral investments– Improved the ability to substantiate positions in engagement at the Boards

• For Multilateral Organizations– Optimizes resources and reduces transaction costs of reviews by utilizing

published and publicly available evaluation reports that encapsulate evidence-based progress being made and lessons learned at country, regional and global levels (UNDP)

– Report interesting and valuable, appropriately nuanced and reasonably reflecting the evaluation reports and findings (WFP)

– Appreciate use of meta-synthesis methodology but need to ensure source of information is comprehensive and current (ADB)

– Best practice and calls for its further emulation (UNDP and WFP)– Constructively discussed at the Boards and provided impetus in areas of

needed improvement (anecdotes from Board participants)

16

Complementarity with

• A study of complementarity with MOPAN was completed during the pilot test phase (2010)– Findings included:

• The two approaches focus on different aspects of multilateral effectiveness and rely on different information sources

• Results are complementary rather than duplicative• Together, they can provide a more complete picture of multilateral

organization's overall performance

• However, does this still hold true with the new “results component” of the MOPAN assessment?– Both undertake document review – Some similarities in criteria (relevance and achievement of outcomes)– Differences in methodological approach

17

Criteria

18

• Extent of progress towards organization-wide outcomes

• Extent of contribution to country-level goals, priorities and MDGs

• Relevance of objectives and programme of work to stakeholders

• Are objectives and expected results at the national and local level in developing countries achieved?

• Are interventions relevant to needs of target groups and its members?

• Are benefits and results achieved sustainable?

• Is programming delivered in a cost-efficient manner?

• Does programming support crosscutting themes (gender equality and environmental sustainability)?

• Is evaluation and monitoring used to improve development effectiveness?

Approach

19

• Data sources• Strategic plans, performance

reports, mid-term reviews, evaluations, reviews

• Country strategies, work plans, reports, evaluations, reviews

• MDG reports, national development strategies and plans

• Key informants from HQ, donor countries and direct partners

• Methods• Document review of all available

documentation• Survey of a sample of key

informants

• Data sources• Evaluation reports• Other documents, including annual

reports (e.g., development effectiveness, strategic plans), COMPAS entries, DAC/UNEG Peer Reviews

• Methods• Preliminary review • Meta-evaluation to establish quality

and coverage of evaluation performance information

• Meta-synthesis of a sample of evaluation reports to assess development effectiveness and identify conclusions and recommendations

Possible Next Steps

• Independent evaluation of MOPAN now in progress

• Comparison of MOPAN “results component” and EVALNET development effectiveness review in the context of the 2013 IFAD assessment: – Type/level of information obtain from both approaches– Level of effort and resources required – Ability to address information gap on development effectiveness

• Explore how EVALNET can collaborate with MOPAN? – Ensure complementarity– Possibly integrate best components

of each methodology

20