world bank group impact...

1

WORLD BANK GROUP IMPACT EVALUATIONS: RELEVANCE AND EFFECTIVENESS

Approach Paper

INTRODUCTION

1. There has been a rapid expansion in recent years in the production of impact

evaluation as a method to assess the impacts of development projects, which is largely driven by an increasing demand for credible evidence of development results. Much of the

development community perceives impact evaluation (IE) as a tool that provides rigorous and

objective estimates of the effects of specific interventions. This had led to expectations that IE

may help build the knowledge base of what does and does not work in development and where

resources may be best allocated. As part of its results and knowledge agenda, the World Bank

Group (WBG) has made important efforts to expand and deepen its IE work.1 Major initiatives

include the Development Impact Evaluation Initiative (DIME), the Africa Impact Evaluation

Initiative (AIM), the Spanish Trust Fund for Impact Evaluation (SIEF), and IFC‟s Advisory

Services Unit in the Development Impact Department (previously Results Measurement Unit).

Recognizing these trends, in the fiscal year 2005, OPCS established IE as a new product line

under the analytic and advisory activities (AAA) umbrella.

2. There has been a growing base of support for applying impact evaluation to

improve development practices,2 but the contribution of impact evaluation has not been

systematically evaluated. The Bank has increased the number of IEs in Bank-supported projects

as a way to “improve the quality of Bank‟s operations, strengthen country institutions for

evidence-based policy making, and generate knowledge in 15 strategic development areas.”3

The recent DIME strategy paper for World Bank IE moreover proposes to mainstream IE as a

core instrument in the Bank‟s knowledge agenda and analytic toolkit (Box 1). The IFC has also

been conducting IEs on private sector projects, particularly for testing pilots and different

approaches of its advisory services, while an institutional strategy has not been established for

building out an IE program. Since 2005, the WBG has spent at least $44 million on IE, financed

both by internal and trust fund resources.4 Little is known about whether WBG IEs have been

conducted (i) in areas where there are strategic analytic and policy knowledge gaps, (ii) are of

high quality, and (iii) have influenced resource allocation, project design/implementation, future

evaluation, strategy or policy making. The timing of this evaluation may thus provide inputs to

the WBG‟s IE strategy moving forward.

3. This joint evaluation will examine the experience of impact evaluation at the WBG

(World Bank and IFC) and its contribution to the development practices of the WBG along

several dimensions. It aims to evaluate the relevance of IEs being conducted by the WBG

(completed and ongoing), their quality, and their use and influence from both the institution‟s

and client‟s perspectives. This includes assessing the institutional organization of IEs, the current

and ideal scope for impact evaluation in WBG-supported projects (in a selected sector), and their

relevance to operations, strategic planning, the knowledge and M&E agendas, and dialogue with

clients. On the country level, this includes the influence of IEs in policymaking and their use for

evaluation purposes. This study recognizes systematic differences in the motivation, production,

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

wb370910

Typewritten Text

59386

2

and application of IEs between the WB and IFC and will make the distinction whenever

appropriate.

4. This report is in line with the current effort of IEG to better understand and

evaluate the contribution of AAA to the development effectiveness of the WBG. This

evaluation is similar to other recent IEG evaluations of AAA: the 2010 PSIA evaluation, the

2008 ESW and TA evaluation, the 2003 Sharing Knowledge evaluation, and other evaluations of

particular AAA reports within country assistance reviews.5 It follows IEG‟s goal of evaluating

AAA products in terms of their explicit and implicit objectives and derive findings that can

enhance their relevance and effectiveness.

5. The findings will be derived from several evaluation tools: (1) desk review of impact

evaluations; (2) review of relevant documents of the evaluated projects; (3) electronic surveys of

IE authors and WBG staff who have worked on the evaluated projects; (4) interviews with

internal and external stakeholders in selected countries; and (5) sector case studies. The study

will triangulate the inputs from the perspectives of different actors (such as World Bank staff,

evaluators, government officials, and private sector representatives). Efforts will be also made to

identify information about the costs of the IEs to give a rough estimate of the resources allocated

to this type of technical analysis. The IEs included in this report must incorporate a credible

counterfactual analysis and have the WBG‟s involvement as an evaluator, funder, or technical

assistance provider. This study looks mostly at IEs of WBG projects that have been completed in

the last decade (2000-10) as well as ongoing IEs.6 The evaluation will also review the relevance

and quality of IEs of programs that have not received WBG financing in addition to the

relevance and effectiveness of IE-related activities, such as workshops, guidance notes and IE

clinics undertaken by DIME, SIEF, AIM and other initiatives at the WBG.

BACKGROUND

DEFINITION AND ROLE OF IMPACT EVALUATION

6. Impact evaluation is a method of assessing the impact attributable to an

intervention, where the outcomes are compared with a counterfactual situation – what

would have happened without the program.7 A key challenge of any evaluation is that the

counterfactual cannot be observed. IE seeks to overcome this by creating a control group (or

counterfactual) that is similar to the group which receives the treatment. The specific techniques

for identifying the control group vary according to the program setting and available data, but

can be broadly classified into two categories: experimental and quasi-experimental (i.e. non-

experimental). Experimental evaluations are possible when treatment and control groups are

selected randomly prior to, or during, program implementation. When randomization is not

possible, various quasi-experimental techniques can be used as an attempt to remove the

selection bias that could arise from observable and unobservable differences between the control

and treatment groups.8 In contrast to other evaluation approaches that, for instance, assess

whether targets have been achieved, impact evaluation is structured around counterfactual

analysis to compare what actually happened with and without the intervention.

3

7. Like any other evaluation approach, impact evaluations have strengths and

limitations in assessing the effectiveness of, and learning from, development programs. In

addition to establishing the link between an intervention and an outcome of interest, IE can

quantify the added value of variations in the interventions. It can in principle provide credible

information not only on what works but also on how, for whom, under which circumstances, and

at what cost. However, IE has also been criticized for not fully answering what factors induce the

change and in which manner, for being highly project- and context-specific, for lacking

efficiency analysis, and for their limited applicability to some sectors. Likewise, IEs may be less

appropriate to assess the effectiveness of thematic, sector and country strategies.

Box 1. Development Impact Evaluation Initiative: A World Bank-Wide Strategic Approach to

Enhance Developmental Effectiveness

Since 2009, DIME has taken on the new mandate as an organizing framework for IE at the Bank. Its

current portfolio of WB-supported IEs includes 170 completed and 280 active studies in 72 countries.

While not all evaluations in the Bank have been actively supported by DIME, DIME‟s objectives are

closely aligned with the Bank‟s strategy for making research relevant to policy decision-making and the

Bank‟s operational work. In particular, by increasing the number of Bank-supported projects with IEs,

DIME seeks to accomplish three objectives: (1) to improve the quality of the Bank‟s operations through

iterative learning, (2) to generate knowledge on critical development questions, and (3) to strengthen client

institutions for evidence-based policy-making. The model aims at increasing the share of IEs supported by

DIME so that they can benefit from the approaches, community of practice, and capacity building

activities provided by DIME. The overall effort is led by a high level steering group, coordinated by a

secretariat based in DEC, guided thematically by the networks, implemented by the regions, and supported

by the research and data groups in DEC with analytical and data services.

DIME‟s objectives are expected to be achieved through three main strategies, reflecting each of the three

functions that IEs are supposed to provide (project assessment, knowledge generation and evaluation

promotion) and that are planned to be assessed in this evaluation: (1) promoting prospective IEs, an

evaluative process planned before implementation (or expansion) of the intervention, to improve

operational quality –bringing IEs into the results agenda, providing just-in-time advice to incorporate

evidence in project designs, measuring project results, improving impacts during implementation, and

introducing formative evaluation based on experimental methods; (2) producing IEs to systematically

learn from development experience—aligning thematic IE programs with Bank‟s learning priorities and

actively disseminating the large DIME portfolio via cross-country workshops, seminars, conferences,

network weeks, web and journal publications, and thematic policy reports; (3) building evaluation

capacity and supporting results-based policy making through joint evaluations—transferring

technical skills, promoting sound data collection strategies, networking with a large community of

practitioners, and learning-by-doing through joint government-Bank evaluations.

THE WORLD BANK GROUP ’S ENGAGEMENT IN IMPACT EVALUATION

8. The WBG has been rapidly expanding its efforts in impact evaluation, increasing

the number of WBG-supported evaluations by seven fold between 2004 and 2008 through

several initiatives.9 The WBG‟s IE promotion efforts include conducting IEs, reaching out to

WBG‟s operations, supporting countries in implementing IEs of their programs and

strengthening their monitoring and evaluation (M&E) capacity, developing best practices, and

facilitating global learning from IE results on effective development interventions. The initiatives

to mainstream IE within and outside the Bank include:

4

Multi-country dedicated IE funds: (1) the Africa Impact Evaluation Initiative (AIM),

which started in 2004 in the Africa Region to support capacity development for IE, and

provides technical assistance to produce and support country-specific IEs of public

programs;10

and (2) the Spanish Trust Fund for Impact Evaluation (SIEF), established by

the Human Development Network (HDN) in 2007 and currently the largest trust fund at

the Bank that focuses on providing funding for IEs of human development interventions,

training, and result dissemination.11

Centralized efforts: (1) the Development Impact Evaluation Initiative (DIME) created in

2004 as a Bank-wide effort to increase the number of Bank projects with IE components,

enhance staff capacity, and build a learning process based on the active participation of

different stakeholders in the design of new IEs and evidence of completed evaluations;12

and (2) IFC effort since 2005 to support IEs of its advisory services projects.

Decentralized efforts: (1) the Poverty Impact Evaluation Thematic Group, managed by

PREM, focuses on IE of governance and institutional reforms;13

(2) the Finance and

Private Sector Research team in DECRG, which uses IE techniques in the areas of

finance and private sector development; and (3) collaboration between DEC researchers

and Bank‟s operations to incorporate IEs into some projects.

9. The DIME report on the Bank’s IE strategy promotes IE as playing an important

role in the WBG’s results, knowledge, and M&E agendas. It characterizes the growth in IE

as the Bank‟s response to increasing demands of donors and client countries for more meaningful

and reliable analyses that assess the effectiveness of development interventions. The potential

contributions of IE to these agendas include:

i) Results agenda: The agenda is at the core of efforts to enhance the WBG‟s development

effectiveness. By generating project performance information, the Bank identifies IEs as an

evaluation tool that contributes to improving decision making in a results-based management

framework.14

ii) Knowledge agenda: The WBG has prioritized strengthening its comparative advantage

in development knowledge.15

The pool of evidence from IEs is seen by the Bank as adding to

the knowledge base of the WBG where it could help connect local experiences, motivate

open dialogues on results and learning, and develop development solutions applicable to

different contexts.

iii) M&E effort: This agenda aims to ensure that all WBG instruments include a results

framework and that the WBG conducts self-evaluations of its work. Moreover, the Bank

assists its clients in developing and implementing M&E methodologies in all government

areas.16

The Bank deems IE to be an important component in a rigorous M&E system and,

with its emphasis on data quality and technical skills, it is argued to provide key inputs for

building and promoting capacity for results-based evidence policy-making.

EVALUATION FRAMEWORK AND QUESTIONS

10. With the goal to evaluate the impact evaluation experience at the WBG and its

contribution to enhancing development effectiveness, this study aims to address the overarching

question “To what extent and why (or why not) have impact evaluations been relevant and

5

effective to inform development practice of the WBG and its clients? The evaluation will

examine various aspects of IE relevance, quality and applicability along a process chain from

initiation through production and uptake of IEs (Figure 1). The IE process chain put forth here

proposes that factors that play a role in the initiation of IEs -such as motivation and purpose, area

of focus, timing and funding of the IE-may affect aspects of IE production, such as technical

quality, costs, analysis of impact distribution, among others. These in turn may influence the

dissemination and uptake of IEs, by affecting project decisions, knowledge generation for future

projects, and the promotion of capacity and resources for evaluation (Figure 1).17

Figure 1. Process Chain from IE Initiation to Uptake

11. There are three main evaluative questions that emerge from this overarching issue: (A)

To what extent and why (or why not) are impact evaluations relevant to close knowledge gaps

and aligned with WBG’s and clients’ priorities and strategies? (B) To what extent and why (or

why not) do impact evaluations meet the expected quality standards and address the questions of

interest? (C) To what extent and why (or why not) are impact evaluations used to influence

development practice? While there have been many guidelines and expectations of what IEs

should accomplish, there are no explicit objectives or ‟best practice‟ set out to evaluate IEs.

Therefore, to the extent possible, this report will rely on evaluation literature and other related

literature within and outside the WBG to guide the analytic framework required to approach

these questions.

Initiation of IEs

Area of focus (sector, country, and intervention)

Motivation and purpose (demand or supply driven)

Involvement of WBG staff, management, country clients, or external actors

Timing in project cycle (prospective, mid-term, ex-post)

Design and evaluative questions

Funding sources

Production of IEs

Technical elements (methods used)

Cost

Articulation of theories linking interventions to outcomes

Inclusion of other analytical issues (analysis of impact distribution, channels behind the

impacts, cost benefit analysis, etc.)

Dissemination and Uptake of IEs

Inform project decisions (design, implementation, expansion, termination, etc.)

Generate knowledge for future projects, policies, debates, and institutional strategies

Promote capacity and resources for evaluation

6

A. Relevance of Impact Evaluations

12. To what extent and why (or why not) are impact evaluations relevant to closing the

knowledge gap and alignment with WBG’s and client’s priorities and strategies? This

question refers to the first step in the process chain – initiation of the impact evaluation. It aims

to describe the areas in which IEs are (or are not) done, and if these IEs are indeed relevant in

terms of analytic and strategic priorities. Relevance will be assessed by the process and

motivation for selecting IE topics and the extent to which IEs are aligned with country, sector or

project strategies of the WBG and its clients. This will require understanding the factors behind

the adoption of IEs as explained by the following sub-questions:

Who and what reasons (operational, analytic, sector strategies) motivated the IE?

What kinds of decisions, if any, did it plan to inform (depending on the purpose, timing

and questions of the IE)?

Who and what factors determine decisions regarding the design and conduct of the IE

as well as the analytical questions examined in the evaluation?

Is the information provided by the IEs aligned with the purposes/objectives of the

projects, WBG‟s and countries‟ strategies, and learning priorities?

What is the ideal coverage of IEs? Which projects and topics for which IE would be

helpful are not being evaluated?

The answers to these questions will be obtained from IE reports as well as surveys and

interviews with the IE teams. They will then be compared with the priorities of the WBG and its

clients as outlined in the PADs/PDSs, PRSPs, CASs, and sector board strategies to detect

alignment and gaps.

B. Quality of Impact Evaluations

13. To what extent and why (or why not) do impact evaluations meet the expected quality

standards and address the questions of interest? The question refers to the second step in the

process chain – production of the impact evaluation. It aims to assess the quality of IE techniques

and scope in order to better inform the discussion on their use and influence. As mentioned

before, there are no „best-practice standards‟ and this evaluation does not attempt to rate or rank

IEs according to their technical quality. It will, however, employ existing guidelines (for

instance, the ones produced by 3IE and NONIE) as well as the guidelines employed by recent

IEG‟s meta-reviews of impact evaluations in nutrition and safety nets. For this purpose the

following sub-questions are pertinent:

What techniques did the IE use to attribute the impact to the intervention? Are these

appropriate methods to attribute causality of impacts?

Did the IE examine the robustness of their findings to methodological issues and other

aspects of the program and context that may affect the internal validity of the analysis?

Which other aspects of program impacts were captured and how (channels that explain

the impacts, distribution of impacts, cost effectiveness, sustainability of impacts)?

7

Are IEs seen to contribute much more than other analytic and evaluative tools?

14. The answers to these questions will be derived mainly from thorough desk reviews of IE

reports and documents related to the IEs (if they are ongoing). They will be linked with

information from the first question to gauge how the techniques and findings are appropriate for

the motivation of the IEs. They will also expose factors inherent to IEs that may explain the

extent to which they are relevant, utilized, and influential. Undertaking an accurate cost-

effectiveness analysis of IEs is very difficult due to the complexities of measuring both their

direct and indirect costs and benefits (multiple sources of financial support, a general lack of

data, and the difficulty of attributing benefits or financial returns to IEs). While not a full cost-

effectiveness analysis, the report will explore what sources of information about costs are

available and examine the perceived contribution of IEs relative to other evaluation approaches

from the client's and other perspectives.

C. Use and Influence of Impact Evaluations

15. To what extent and why (or why not) are impact evaluations used to influence

development practice? The question refers to the third step in the process chain – dissemination

and uptake of the impact evaluation. It aims to measure if there has been a strategic uptake of IEs

in decision making by the WBG and its clients, and, if not, what are the constraints. IEs are often

expected to be used for three broad functions: (1) as a project assessment tool to evaluate

performance, influence design and operational decisions of the evaluated program, and provide

an objective basis for shared accountability; (2) for knowledge generation beyond the program

evaluated to provide evidence about what works and does not work in development, why, for

whom, and under which conditions; and (3) to promote evaluation capacity and evidence-based

decision making mechanisms by emphasizing high-quality data collection and strong technical

skills (Annex 1).18

In assessing these three functions and the factors that may underline the

applicability of IEs, this report seeks to answer the following questions:

To what extent and how have IEs contributed to decisions throughout the life cycle of

the project, from design (choice of beneficiaries, delivery mechanisms, timing for data

collection) to implementation to completion (decisions to continue, expand, modify or

terminate, and funding decisions)?

To what extent and how have IEs encouraged learning to improve future projects and

contributed to the institutional strategies and policy debates of the WBG and its clients?

What are the „feedback loops‟ in WBG processes that support this learning function?

To what extent and how have IEs and their related activities (workshops, guidance

notes, IE clinics) contributed to promoting the capacity and demand for M&E?

What factors (motivation, methodological quality, active engagement with counterparts,

timing, focus on stakeholder‟s priority issues, dissemination, interest of decision

makers, etc.) are associated with the use and influence of IEs? 19

The answers to these questions will be acquired from project document reviews, structured

surveys with the IE teams and WBG staff related to the evaluated projects, interviews with WBG

clients in the country, and sector case studies.

8

DATA AND METHODOLOGY

16. In order to answer the above questions, this evaluation will employ five evaluation tools:

(1) desk review of impact evaluations; (2) desk review of relevant project documents; (3)

electronic surveys of evaluators and Bank staff related to the evaluated project; (4) interviews

with internal and external stakeholders in selected countries; and (5) sector case studies. (Annex

2 details how these tools serve to answer each sub-question).

Desk review of impact evaluation reports: The team has compiled a database of IEs

(completed and ongoing) supported by the WBG to serve as the basis of the evaluation. Since

IE is a decentralized effort in the WBG, it is difficult to generate a comprehensive list. The

bulk of the database is from the DIME initiative, which documents IEs across the WB, and is

complemented with the IE list from the IFC. In order to ensure that the inventory is close to

being exhaustive, the team has conducted different searches in Business Warehouse, Image

Bank, and Operations Portal/ I-docs. The compiled database reveals that there are 329 IEs of

WB projects since 2000 (119 completed and 210 ongoing) and they tend to concentrate in

certain sectors and regions (Annex 3).20

The reports of all the completed IEs will be reviewed

to extract information for the questions regarding the relevance and quality of IEs

(motivation and objectives, timeliness, involved actors, factors affecting the design,

methodology, and findings).

Desk review of project documents and policy notes: A random sample of

approximately 40 WB projects corresponding to 50-60 completed IEs and 20 WB projects

associated to 30 ongoing IEs will be selected for this in-depth document review (Annex 4

describes the sampling methodology). In the case of IFC all IEs (approximately 25) will be

included in the analysis.21

The documents reviewed will include PADs/PDS of the evaluated

and follow-up projects (to understand the objectives of the projects, if the IE was foreseen in

the project, and if the IE findings contributed to the design of the follow-up project), ISRs,

ICRs/PCRs of the evaluated projects (to check if IE findings were cited in these

assessments), as well as PPARs, PRSs, CASs, and sector board strategies (to provide context

for the priorities of stakeholders). Results from this exercise will shed light on whether the

IEs are aligned with the project, country, and/or sector strategies of the WBG and its clients,

as well as whether the IEs have been used for project assessment, knowledge generation, and

M&E promotion. Information from the review will be matched with the surveys and

interviews to create a more comprehensive picture of the IEs and related projects. Finally, a

subsample of Bank projects in a specific sector will be selected to review project documents

and determine the actual and ideal scope conceptually feasible for impact evaluation work in

connection with these projects.

Surveys: For the same randomly selected sample of projects with completed and/or

ongoing IEs, the team will distribute structured surveys to IE authors and WBG staff related

to the evaluated projects. The survey questions for IE authors seek to gain information about

the motivation and costs of a specific IE and their awareness (with concrete verifiable

examples) of the IE use and influence. The survey questions to WBG project staff will ask

their view as users (with quantitative ratings) about the usefulness and influence of the IE

related to a specific project as well as factors that may explain this (Annex 5).22

Survey

questions will be designed in a neutral way such that they do not prompt the respondents to

9

overstate the importance of IEs. If one person is related to more than one IE or project, they

will be asked about only one, which is chosen randomly. To increase a likely low response

rate, the team will actively follow up and contact senior management to ensure cooperation.

Interviews with internal and external stakeholders in selected countries: These

interviews aim to provide a country perspective (from both the WBG and clients) across IE

applications and a detailed evaluative view of their development policy impact. For this

selection of IEs, countries have been ranked according to the total number of completed IEs.

The top 10 are Nicaragua, Peru, Indonesia, Colombia, China, India, Vietnam, Brazil,

Ecuador, and Bangladesh (with at least 5 completed IEs). Due to time and budget constraints,

countries and interviewees will be purposely selected through the following process. First,

four to six of the top 10 countries will be selected to ensure coverage of most regions and that

each country offers cases of completed and ongoing IEs for which it is possible to

systematically apply a consistent set of questions. An extra project with an IFC-supported IE

with learning complementarities to WB‟s case studies will be identified. Second, within each

country, one or two main sectors will be selected to minimize overstretching the interview

team. The goal of the exercise is to interview IE authors, WBG staff and clients (for example,

government officials and private sector representatives) related to the evaluated projects

(they will be excluded from the survey) about aspects aligned with the three main questions

outlined in the previous section. In addition, the team will conduct structured interviews with

sector managers, business line leaders/product specialists, M&E staff and stakeholders

benefited from other IE-related activities supported by the WBG (e.g. workshops, guidance

notes and IE clinics) to capture their views on relevance and usefulness of IEs and related

products. The findings from these interviews will feed into the conclusions and

recommendations for the overall evaluation and provide concrete examples of cases in which

IEs are (or not) used and influential in different country contexts.

Sector Case studies: Sector case studies will complement the analysis through

examining specific sector IE experience across different country circumstances and

applications. One of the studies will be in a sector with a significant amount of evidence

(likely education) and another in a sector where the IE evidence base is starting to grow (for

instance, health and/or water and sanitation). The case study structure is outlined in Annex 6.

TEAM

17. The joint evaluation will be under the guidance of Mark Sundberg (Manager, IEGPS) and

Stoyan Tenev (Chief Evaluation Officer, IEGPR). The evaluation team will be managed by

Javier E. Baez (Economist, IEGPS) and Izlem Yenice (Evaluation Officer, IEGPR), and include

Andaleeb Alam, Carmem Domigues, and Tu Chi Nguyen (IEGPS), as well as other consultants

to be identified. The team will be assisted by Yezena Zemene Yimer.

TIMELINE

18. The approach paper was reviewed at the IEG “one stop” on November 9, 2010 and will

be submitted to CODE for review in January 2011. The evaluation will be reviewed within IEG

and by peer reviewers by July 15, 2011. It will be submitted to CODE by mid-September 2011.

10

Figure 2. Proposed Work Program

Activities 2010 2011

Oct Nov Dec Jan Feb Mar Apr May June July Aug Sep

Background work and drafting of approach

paper

One-stop review of approach paper

CODE review of approach paper

Review of impact evaluations

Desk review of project documents

Design, launch, and track of surveys

Analysis of surveys

Selection and design of country-based

interviews

Interviews in Washington DC

Field-based interviews

Drafting of sector case studies

Drafting of evaluation report

Review by IEG management and peer

reviewers

Review by WBG management

Final modification and submission to

CODE

PEER REVIEW

19. Peer reviewers for the evaluation are David McKenzie (Senior Economist, DECFP,

World Bank) and Howard White (Executive Director, International Initiative for Impact

Evaluation).

11

Annex 1

Conceptualizing Use and Influence of IE

The literature on theory of evaluation utilization, in general, identifies four types of uses: instrumental

use (making decisions about programs based on evaluation results); conceptual use (affecting how

people think about development issues); strategic use (persuading others or using evaluation findings

to reach particular outcomes); and process use (promoting evaluation activities).23

This evaluation

groups the different functions of IEs under three categories to align their potential use and influence

with the WBG‟s results, knowledge, and M&E agendas outlined above:

I. Project Assessment: Within the evaluated project, IEs are claimed to serve as an assessment

tool. This may be used by various actors, within and outside the WBG, to assess performance and

influence design and operational decisions at the following phases:

Project design: Incorporating a prospective IE into a program may affect decisions regarding

the design of a project in aspects such as targeting of beneficiaries, type of benefits, delivery

mechanisms, and timing and quality of data collection, among others. IEs may also help identify

other related development issues to be addressed, evaluate pilot projects, and test different design

features to improve subsequent phases of the project.

Project implementation: Knowledge generated along the production of an IE may be useful

for strengthening the project‟s implementation. For example, data collected for the IE could

provide information for a process review that reveals areas if and where implementation is not

carried out as planned, hence allowing needed in-time correction.

Project completion and assessment: The results from IEs could demonstrate the impacts (or

lack thereof) of the program, which could be central to operational decisions to continue, modify,

expand or terminate the evaluated project. In addition, if combined with information about costs,

IE could provide an opportunity to assess the cost-effectiveness of programs, contributing to more

efficient resource allocation.

II. Knowledge Generation: Beyond the project evaluated, evidence from IEs may also

contribute to knowledge generation for enhancing development practices. As IEs are being

conducted on an increasing number of sectors and countries, the evidence from IEs could contribute

to build the stock of knowledge, which may be argued to help inform decision makers of future

projects and the development community in general.24

Future project design and implementation: When designing and implementing a new

project, managers/policymakers may take into consideration lessons learned from IEs of other

projects (for example, what approaches have worked in similar settings).

Bank’s strategies, dialogues, and policy decisions: As the results of IEs are disseminated,

they add to the general pool of knowledge of development challenges and solutions, which may

help motivate debates and dialogues, establish best practices, influence Bank‟s strategies

(including resource allocation), and promote evidence-based policymaking.

III. Evaluation Promotion: IE has the potential role of contributing to develop capacity for and

institutionalizing evidence-based evaluations, both within the WBG and at the country/sector level.

IEs are supposed to help narrow the gap between the demand for and supply of knowledge on

development effectiveness, both directly and indirectly by supporting and supplementing the

production of other evaluations.25

Conducting IEs could promote the capacity for evaluations,

12

including the collection of systematic high quality data, specialized skills, and demand for follow-up

or new evaluations.26

Within the WBG: As the number of IEs of Bank-supported projects grows, it is expected

that a higher fraction of WBG staff will learn the skills needed to carry out sound IEs.

Furthermore, the inputs used by IEs such as rich data and information systems are often claimed

to be beneficial for other M&E and analytical purposes undertaken by the WBG.

At the country/sector level: The Bank‟s promotion of IEs produced in a collaborative

manner with local counterparts such as policymakers, private sector, local researchers, and

development institutions (as well as NGOs and civil society) may help transfer the necessary

technical skills, internalize the value of IEs, understand their limitations and costs, align M&E

and data collection strategies, and build and strengthen evaluation capacity at the country/sector

level.

13

Use & Influence Flowchart

Function 2 – Knowledge Generation

Function 1 - Evaluated Project

Function 3 - Evaluation Promotion

Project Design Project Implementation Project Completion and Assessment

Decision to Conduct

Prospective IEs

IE Production (data collection, analysis

undertaken)

IE Results (prospective or ex-

post)

Future Project Design and

Implementation

WBG’s Strategies, Countries’

Policies, Policy Dialogues

WBG’s M&E Capacity and Framework

Country’s M&E Capacity and Framework

External Stakeholders

Acquire Information

Extent and means of interaction between evaluators and project

managers; Timeliness of IE relative to decision;

Bank’s policies and incentives, stakeholder incentives;

IE quality and credibility; Relevance of results

Extent and means of result dissemination;

Institutional and external

stakeholder incentives;

IE quality and credibility;

Relevance of results

Extent and means of IE dissemination;

Bank’s policies and incentives; IE quality and credibility

Factors that may affect the use and influence of IEs

14

Annex 2 - Evaluation Design Matrix

Evaluation Questions Sub-questions Information Sources and

Collection Methods

Relevance of IEs: To what

extent and why (or why

not) are IEs relevant to

close knowledge gaps and

aligned with WBG’s and

clients’ priorities and

strategies?

Who and what motivated the IE? IE reports

Surveys with IE teams

Interviews with selected IE

teams

Priorities of the WBG and its

clients as outlined in the

PADs/PDS, PRSP, CAS, and

sector board strategies

What kinds of decisions, if any, did it plan to inform (depending on the purpose, timing

and questions of the IE)?

Who and what factors were involved in decisions regarding the design and conduct of the

IE?

Quality of IEs: To what

extent and why (or why

not) do IEs meet the

expected quality standards

and address the questions

of interest?

What techniques did the IE use to attribute the impact to the intervention? Are these

appropriate methods to attribute causality of impacts?

IE reports

Documents related to the IEs (if

they are ongoing) Do the IEs examine the robustness of their findings to methodological issues and other

aspects of the program and context that may affect the internal validity of the analysis?

Which other aspects of program impacts were captured and how (distribution of impacts,

cost effectiveness, sustainability of impacts)?

How cost-effective have IEs been? Are they seen to contribute much more than other

analytic and evaluative tools?

Use and Influence of IEs:

To what extent and why (or

why not) are IEs used to

influence development

practice?

Project assessment tool: To what extent and how have IEs contributed to decisions

throughout the life cycle of the project, from design (choice of beneficiaries, delivery

mechanisms, timing for data collection) to implementation to completion (decisions to

continue, expand, modify or terminate, and funding decisions)?

Project document reviews

Surveys with IE teams and WBG

staff related to the evaluated

projects

Interviews with WBG staff and

clients in selected countries

Sector case studies

Knowledge generation: To what extent and how have IEs encouraged learning to improve

future projects and contributed to the institutional strategies and policy debates of the

WBG and its clients?

Evaluation capacity and evidence-based decision making: To what extent and how have

IEs and their related activities (workshops, guidance notes, IE clinics) contributed to

promoting the capacity and demand of M&E for the WBG and its clients?

Factors affecting use and influence: What factors (methodological quality, active

engagement with counterparts, timing, focus on priority stakeholder issues, dissemination,

interest of decision makers, etc) are associated with the use and influence of IEs?

15

0

10

20

30

40

50

60

70

19

80

19

91

19

92

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

An

alysi

s in

pro

gre

ss

Fo

llo

w-u

p d

ata

coll

ecte

d

Bas

elin

e d

ata

coll

ecte

d

Eval

uat

ion

des

ign

ed

Un

der

dis

cuss

ion

Completed Ongoing

# o

f IE

s

WB-suported IEs of WB Projects per Completion Status

and Year

Period evaluated

Annex 3

Summary of the Preliminary Database (including WB-supported IEs of World Bank projects, not including IFC-supported IEs)

Note: The years shown are the years in which IEs were completed

0

20

40

60

80

100

FPD HDN PREM SDN

# o

f IE

s

WB-suported IEs since 2000 of WB

Projects per Network (completed and

ongoing)

Completed Ongoing

0

20

40

60

80

100

120

Africa East Asia

and the

Pacific

Europe

and

Central Asia

Latin

America

and the Caribbean

Middle

East and

North Africa

South

Asia

# o

f IE

s

WB-suported IEs since 2000 of WB

Projects per Region (completed and

ongoing)

Completed Ongoing

16

Annex 4

Sampling Methodology

Data: The WB database consists of 329 IEs linked to 260 World Bank operations. There are

120 completed IEs that correspond to 80 operations, while the 209 ongoing IEs are linked to

189 Bank operations. Of these 189 operations, 5 have already had an IE that was completed

in the past decade. Project-based breakdown of IEs is as follows:

Number of IEs Per Project 1 2 3 4 5 Number of Projects 216 27* 12 2 3**

* 4 of these projects have both one ongoing IE and one completed IE

**1 of these projects has one ongoing IE and 4 completed IEs

Sample Design: For the purposes of this evaluation, a stratified random sample of projects

will be studied. The rationale for sampling on projects as opposed to IEs is because within a

particular project, the effect of one IE cannot be isolated from another and hence, all IEs

within the project need to be analyzed. No design effects or non-response effects are assumed

in sample size calculations.

Universe: The universe consists of 260 projects linked to 329 IEs in the database

Domains: The universe of projects is classified into two mutually exclusive domains: (1) 80

projects with at least one completed IE (2) 184 projects with ongoing IEs but no prior

completed IE. Sample size for each domain is determined separately to ensure

representativity at the domain level. For the completed IEs domain, the sample is significant

at a 95 percent confidence level with ±10 percent margin of error. Given the higher number

of projects in the second domain, the focus of this evaluation on completed IEs, and the time

and resource constraints, the sample of ongoing IEs is significant at a 95 percent confidence

level with ±20 percent margin of error.

Stratification: The two domains are stratified into (1) Projects with only one IE, and (2)

Projects with multiple IEs. In the completed IE domain, there are 55 projects with single IEs

that have been completed, and 21 projects with multiple IEs at least one of which is

completed. In the ongoing IE domain, 16 projects have multiple ongoing IEs (and no prior

completed IEs), while 168 projects have single ongoing IEs (and no prior completed IEs).

Each stratum is further stratified into SDN (Sustainable Development Network), HDN

(Human Development Network), FPD (Finance and Private Sector Development Network),

and PREM (Poverty Reduction and Economic Management Network) on the basis of the

sectoral distribution of the projects and its correspondence with network. A random sample

of projects will be selected from each strata.

Sample Allocation: The sample size is allocated to each of the 8 strata based on Probability

Propotional to Size method. Although the incidence of PREM projects is low compared to

SDN, HDN and FPD projects, time and resource constraints do not allow for oversampling

PREM projects to ensure sufficient power for comparing PREM with other networks.

17

Annex 5

Surveys Structure

Surveys will target a wide range of key players to Impact Evaluation within the Bank. Since different

types of actors have different responsibilities and experiences in conducting IEs, the questions posed

to each of them will vary according to their roles. Survey questions will be grouped along themes or

topic areas, and will be asked to the relevant actors, with the possibility of a given topic area being

addressed by multiple types of actors. Some tentative groupings with a sample of questions of each

section are described below (multiple choice answers or ratings where applicable will also be

included in the actual surveys).

Actors Groups of relevant questions asked

TTLs Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q9

Evaluators Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q9

Country Directors Q1, Q4, Q5, Q6, Q7, Q9

Sector Managers Q1, Q4, Q5, Q6, Q7, Q9

Question

Group

Topic area of

questions Sample questions Information for

Q1 Decision and

support to

conduct the IE

What led/contributed to the initiation of the IE?

Who supported the IE being conducted?

What was the purpose of conducting the IE?

Relevance

Factors that influence

utilization

Q2 Production of

IEs and

linkages with

the project

Who was in charge of designing/implementing the IE?

What were the major factors that affected the design and

implementation of the IE?

What was the time lag between project implementation and the

IE?

Relevance

Function 1 (use of IE as

an assessment tool)


utilization

Q3 Funding of IE Who funded the IE?

Did WBG supervision pay any researcher time? Relevance


utilization

Q4 Results and

dissemination

When were the results available in relation to the timing of

decisions regarding the project?

Who was responsible for disseminating the results?

Through what channels were the results disseminated?

Function 2 (use of IE

for knowledge

generation)


utilization

Q5 Knowledge

generation

Did this IE contribute new information to the development

knowledge base? Function 2


utilization

Q6 Use and

influence

Have the results of the evaluation been utilized? If so, by

whom? And in what ways?

Did the IE contribute to the wider WBG Strategy? If so, how?

Did the IE contribute to the policies of the country? If so, how?

What were the factors that affected their use and influence?

Function 1

Function 2


utilization

Q7 Promotion of

evidence-based

decision

making

How have IEs contributed to improvements in data collection,

staff evaluation capacity, institutionalization of evaluation

efforts?

Function 3 (use of IE

for M&E capacity)


utilization

Q8 Relationship

among actors

How was the sharing of results between the implementers and

the WBG team?

Was the relationship between evaluators and the operational

team a smooth one?

Function 2

Function 3


utilization

Q9 Lessons from

conducting the

IE

What lessons would you point to for your colleagues who

would like to incorporate an IE into their projects?

Have you ever tried to implement an IE into another WBG

project and not succeed? If so, which one? Why did it not

succeed? What were the lessons from that?


utilization

Lessons and

recommendations

18

Annex 6

Sector Case Study Structure

In order to ensure that the team can make comparisons between case studies in an accurate fashion,

both case studies will follow a specified format (with room for certain adaptations to accommodate

inherent differences between sectors) and will be based on a structured set of questions that will guide

team members in their review of relevant documents and conduct of interviews. The final report that

will be generated for each case study is expected to follow the format specified below.

1. SECTOR BACKGROUND

This section describes the sector context including policymaking, development issues, motivation for

projects, WBG‟s engagement and M&E system in terms of the structure, resources and key actors.

The point of this section is to provide a background for how impact evaluations may fit in.

Policymaking context. This sub-section describes the broad policymaking context in the sector, such

as major actors and their interests, and what factors influence the policymaking process.

Background of the development issues/motivation for the projects. This sub-section describes the

main development issues within the sector, what the government has been doing to address them, and

the main motivations for the conduct of the projects evaluated.

Bank’s engagement. Describe the involvement of the WBG in the given sector, including the size of

its portfolio and type of WBG engagement, including with policymakers.

2. BACKGROUND ON IMPACT EVALUATION

This section describes all the completed, planned, or ongoing impact evaluations supported by the

World Bank Group in the sector. The purpose is to provide an overview of the evidence from impact

evaluations available for the sector as well as potential for the evidence to be used.

Motivation for impact evaluations. This sub-section describes what led to the conduct of the IEs,

including the purpose, key players, and alignment with Bank‟s and countries‟ priorities. It analyses

whether the results of the IEs were made available to make informed operational decisions about the

programs evaluated.

Impact evaluation design and implementation. This sub-section describes the design and

implementation of the IEs, including the kind of involvement the WBG and the country have in the

IEs, and whether there were issues in program implementation that had implications for evaluations

and vice-versa.

Impact evaluation findings and dissemination. This sub-section describes the results generated by

the IEs, as well as their dissemination both within the WBG and the sector where the IEs were

conducted.

3. ANALYSIS AND ASSESSMENT

This section analyzes the application of IEs in the WBG‟s operations and the country‟s policies and

what factors can make them more relevant and effective.

19

Applicability of IEs for WBG’s operations. This sub-section describes how IEs are used within the

WBG to influence project design and implementation.

Knowledge generation. This sub-section describes how IEs stimulate critical thinking to identify and

test the relevant and influential policy and operational questions in highest demand from governments

and the WBG.

Determinants behind the utilization of IEs. This sub-section describes what factors explain the

prevalence (or lack of) IEs in the sector.

4. LESSONS LEARNED AND RECOMMENDATIONS

This section synthesizes the findings of the case study, and makes recommendations to different

actors on how to improve the use and influence of IEs.

Lessons learned. This part draws conclusions about the extent to which IEs are relevant, utilized and

influential, and the factors that affect their relevance and applicability within the WBG and in the

sector context.

Recommendations for enhancing IE relevance and effectiveness. This sub-section makes

recommendations to evaluators, project managers, country directors, policy makers, and other

relevant actors for improving the relevance, take-up and effectiveness of IEs based on the analysis

above.

20

REFERENCES

Center for Global Development. 2006. “When Will We Ever Learn? Improving Lives through Impact

Evaluation.” Washington, DC: Center for Global Development.

Cummings, R. 2002. “Rethinking Evaluation Use”. Australasian Evaluation Society International Conference

Fiszbein, A. 2006. “Development Impact Evaluation: New Trends and Challenges.” In Evidence and Policy

2(3): 385-393.

Jones, N., C. Walsh, H. Jones, and C. Tincati. 2008. “Improving Impact Evaluation Coordination and Uptake: A

Scoping Study Commissioned by the DFID Evaluation Department on behalf of NONIE.” London, UK:

Overseas Development Institute.

Overseas Development Institute. 2009. “Improving Impact Evaluation Production and Use.” Working Paper

300. London, UK: Overseas Development Institute.

Ravallion, M. 2009. “Evaluation in the Practice of Development.” In The World Bank Research Observer 24(1).

Sandison, P. 2005. “The Utilization of Evaluations.” In ALNAP Review of Humanitarian Action. London, UK:

ALNAP.

Shadish, William R. & Cook, Thomas D. & Leviton, Laura C. 1991. Foundations of program evaluation :

theories of practice / by William R. Shadish, Jr., Thomas D. Cook, Laura C. Leviton Sage Publications,

Newbury Park, CA

Thomas, V. 2009. “Evaluation to Help Improve Development Results.” Washington, DC: The World Bank.

World Bank. 2009. “Making Smart Policy: Using Impact Evaluation for Policy Making – Case Studies on

Evaluations that Influenced Policy.” Doing Impact Evaluation No. 14, Thematic Group on Poverty Analysis,

Monitoring and Impact Evaluation. Washington, DC: The World Bank.

World Bank-DIME. 2010. “Development Impact Evaluation Initiative: A World Bank-Wide Strategic

Approach to Enhance Developmental Effectiveness.”, DIME, Mimeo. Washington, DC: The World Bank.

World Bank - IEG. 2010. “Analyzing the Effects of Policy Reforms on the Poor: An Evaluation of the

Effectiveness of World Bank Support to Poverty and Social Impact Analysis.” Washington, DC: The World

Bank.

___. 2008. “Using Knowledge to Improve Development Effectiveness: An Evaluation of World Bank

Economic and Sector Work and Technical Assistance, 2000-2006.” Washington, DC: The World Bank.

___. 2003. “Sharing Knowledge: Innovations and Remaining Challenge.” Washington, DC: The World Bank.

Worthen B., J. Sanders, and J. Fitzpatrick. 1997. “Program Evaluation: Alternative Approaches and Practical

Guidelines”. Longman, N.Y, USA.

21

ENDNOTES 1. OPCS, Impact Evaluations at the World Bank: Guidance Note

2. Thomas, 2009; CGD, 2006; Fiszbein, 2006

3. World Bank-DIME, 2010

4. This is a conservative estimated based on the DIME Strategy report (During fiscal 2000-2004, the Bank spent $160,000 a

year on impact evaluation in total. After DIME was established, spending rose to $4 million a year in fiscal 2005-2008,

evenly split between internal and trust fund resources. Since DIME was re-launched as a Bank-wide initiative and with the

maturing of the IE portfolio, spending tripled to $13-14 million a year, mostly financed by trust funds).

5. World Bank-IEG, 2010, 2008, 2003

6. The 2000-10 period covers 92 percent of all completed IEs. The team excludes IEs completed earlier because it will be

difficult to track their use and influence. IEs that do not evaluate WBG-supported projects were excluded given that the

evaluation tools available to the team will not be able to document the contribution of these IEs to the operational aspects of

projects that had no Bank involvement (function 1 of third question).

7. World Bank-IEG website

8. Quasi-experimental techniques include propensity score matching, difference-in-differences, regression discontinuity,

instrumental variable, and structural modeling.

9. World Bank-DIME, 2010

10. World Bank-AIM website

11. World Bank-SIEF website

12. James Adams, OPCS VP, Announcement July 14, 2005

13.World Bank-Poverty Impact Evaluation Website

14. Results agenda website

15. Knowledge agenda website

16. World Bank-DIME, 2010; World Bank-IEG website

17. Throughout the evaluation other factors not mentioned here may emerge as influencing one of the three parts of the

theory of change, and if this is the case, they will be duly incorporated to explain how IEs have been relevant and effective

in informing development practices.

18. World Bank-DIME, 2010; World Bank, 2009; Sandison, 2005; ODI, 2009; World Bank-IEG website

19. In fact, the literature identifies some characteristics that may hinder or facilitate IE use. Some factors include: (i) the

methodological quality of the evaluation and credibility of the international evaluators; (ii) timing and focus on priority

stakeholder issues; (iii) identifying users and change agents early in the evaluation; (iv) frequent contacts with users; (v)

providing interim results; (vi) translating findings into actions and recommendations; (vii) disseminating results through

informal meetings, oral briefs, non-technical summaries; (viii) challenging fundamental assumptions about problems and

policies and by circulating results through networks of opinion makers; (ix) extent to which incentives are aligned with

efficacy; (x) the interest of decision makers and community in the evaluation, (xi) active engagement with counterparts; (xii)

the extent to which findings can go against entrenched interests or prevailing ideology; (xiii) constraints on decision-makers

related to budget constraints, past decisions and conflicting demands.

20. The sectors that have seen the highest uptake are in human development, especially social protection and labor,

education, and health programs, followed by programs in the agriculture and rural development and finance sectors.

Geographically, sub-Saharan Africa is where the highest number of IEs has taken place, followed by Latin America, then

South and East Asia respectively. Most IEs used nonexperimental methods. It is difficult, however, to determine the degree

of Bank involvement in each of these IEs, which could vary from funding to technical assistance to implementation.

21. The sample of completed IEs is significant at a 95 percent confidence interval at ± 10 percent margin of error and the

sample of completed IEs is significant at a 95 percent confidence interval at ± 20 percent margin of error.

22. The database will provide information about the staff related to the IEs and evaluated projects, including evaluators,

TTLs, Sector Managers, Country Directors, and other key players involved in the projects or sectors being evaluated.

23. Cumming, 2002; Shadish, William, Thomas Cook and Laura Leviton, 1991; Worthen, Sanders and Fitzpatrick, 1997

24. ODI, 2009; Sandison, 2005; IEG website

25. CGD, 2006; Ravallion, 2009

26. World Bank-DIME, 2010; World Bank, 2009