How do impact evaluations improve development operations and policy?David K. Evans, World BankAfrica Chief Economist’s Office

June 1, 2016 | 3ie/IFAD devans2@worldbank.org | @tukopamoja

2

How do impact evaluations affect operations and policy?Own project• Clarify thinking about the

theory of change• Improve quality of data

systems• Improve efficiency of projects• Shifts in project design• Influence scale-up of projects

Other projects• Design of new projects• Choosing between designs• Culture of evidence• Contribute to a body of

knowledge that changes how we do business

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

3

I’m going to tell you some stories (and show some analysis).

But how do we measure impact?

Isn’t impact evaluation about careful causal inference? Where’s the counterfactual?

Image source: http://imperialcentrepaintingstudio.blogspot.com/2012/02/reading-comprehension-teach-your-child.html

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

4

Measuring the impact of impact evaluationSecond cousin to the event study

• Before-after is a poor evaluation method– Because usually many other

things are changing• But useful when no other

credible cause

Embrace the ambiguity

• The best impacts from impact evaluation can come when the resulting decisions are completely internalized by government (Goldstein 2016)

• Short of an RCT of RCTs, embrace the ambiguity.

Jensen (2007)

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

5

Indicators for IE impact3ie• Closing knowledge or evidence

gaps?• Changes in policy or programs?• Key stakeholders and key

influencers?• Plan to engage with them at

each stage of the IE?• Cleary theory of change for

policy influence?

DIME (World Bank)• Capacity building

– IE results discussed with client– Training provided for data analysis

• Quality of data– Availability of data– Improved administrative data

• Quality of policy decisions– Used to motivate scale-up/scale-down of

policy– Adopted delivery mechanism based on IE

results– Baseline informed design or

implementation

Source: Legovini 2014Source: 3ie 2012

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

6

• Impact evaluation helps clarify thinking on exactly how one gets from inputs to impacts

• By focusing on questions of – Causality– Measurement

• Impact evaluation improves theories of change

Own projectClarify thinking about the theory of change

Figure sources: Imas & Rist 2009

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

7

Own projectImprove quality of data systems• Kenya Judicial Performance IE

– Impact of incentives for greater efficiency

– Co-developed tool for measuring efficiency of courts: How quickly do they get their cases out

– Tool was adopted for all courts as official monitoring tool

• Kenya Patient Safety IE– Impact of inspections of health

facilities– Reform to regulatory framework of

inspection system for minimum patient safety.

– Design of quality of care measuresSources: World Bank 2013; i2i n.d. (1); i2i n.d. (2)

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

8

• Sample of 100 impact evaluations, 1000+ projects between 2005 and 2011

• Include time, sector, country, and project leader effects

• Significant increase in projects carrying out their activities

Own projectImprove efficiency of projects

Source: Legovini et al. 2015

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

9

• Notably, this increase in disbursements happens after impact evaluation begins

• Can’t completely control for selection into project

• BUT suggestive that IEs help projects roll out and that they don’t slow projects down (as commonly claimed)

Own projectImprove efficiency of projects

Source: Legovini et al. 2015

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

10

Own projectShifts in project design Ethiopian Nile Irrigation and Drainage Project

• Baseline survey in Ethiopia• Women have

– Lower access to extension– Greater time poverty– Much lower agricultural value-

added for women than men• Result: Pilot strategies in

Farmer’s Initiative Fund, designed for female farmers

Source: World Bank 2014

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

11

Own projectInfluence scale-up of projectsFinancial Literacy in Brazil• Pilot in high schools• Modest but positive, significant

impacts• Scale-up

– High schools around the country– Elementary schools

Other example: Land titling in Rwanda

Conditional Cash Transfers• Mexico

– Initial program showed positive results

– Expansion to urban– Survival across administrations

• PROGRESA Oportunidades Prospera

• Tanzania– Pilot showed positive results– Key evidence informing scale-up

Source: Dhaliwal & Tulloch 2012; Mkelemi & Evans 2016Source: Bruhn et al. 2013; Ali et al. 2011

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

12

Other projectsDesign of new projectsProjects reports that IEs influence design

• Education Modernization Project in Kazakhstan– Influenced by synthesis of

education impact evaluations• Adolescent Girls Empowerment

Project in India– Influenced by IEs of programs to

empower adolescent girls in Liberia and Uganda

Citations of IEs in World Bank documents ↑ 6x

05 06 07 08 09 10 11 12 0

5,000

10,000

15,000

20,000

25,000

3,691

21,271

Fiscal Year

Tota

l IE

Cita

tions

Source: Legovini 2014Source: Gender Innovation Lab 2015 Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

13

• Zambia Essential Drugs Public Pilot Program• Consistent problem with stockouts of essential malaria drugs• Randomized impact evaluation to test two different models • A: District planner• B: Direct orders from clinic to central medical stores

Other example:Early child development in the Gambia

Other projectsChoosing between designs

Source for Zambia: Vledder et al. 2015; World Bank 2010

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

Source for the Gambia: Blimpo & Carneiro 2014

14

Other projectsCulture of evidenceMexico• Progresa/Oportunidades 1997-

2000• 2000 Congress’ Degree: Annual

external evaluations for projects

• 2004 Social Development Law: National Council for Evaluation of Social Policy (CONEVAL)

Tanzania• Two donor-initiated impact evaluations

under TASAF-2– Community-based conditional cash

transfers– Vulnerable groups

• Three client-initiated impact evaluations under TASAF-3– Scaled-up cash transfers– Cash transfers on women’s empowerment– Cash transfers on transition to adulthood

• False dichotomy: Donor-initiated vs client-initiated. But a shift along the spectrum.

Source: Hernández Licona 2008

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

15

Other projectsContribute to a body of knowledge that changes how we do developmentSmall questions or big questions?• Should we give away

insecticide treated nets or charge for them?

• Higher prices – lower use– don’t improve targeting– don‘t increase use

• Between 2000 and 2015: “450 million cases of malaria and 4 million deaths averted from ITN distribution”

Many small studies add up to big answers

Pricing experiments around the world

Source: Dupas & Miguel 2016Source: Glennerster 2016

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

16

3ie. “Policy Influence Plan.” 2012. http://evaluationvigie.fr/wp-content/uploads/2014/05/2012_PIP_template.docx Ali, Daniel, Klaus Deininger, and Markus Goldstein. “Environmental and gender impacts of land tenure regularization in Africa: pilot evidence from Rwanda.” World Bank Policy Research Working Paper 5765. 2011. http://documents.worldbank.org/curated/en/2011/08/14833489/environmental-gender-impacts-land-tenure-regularization-africa-pilot-evidence-rwanda Blimpo, Moussa, and Pedro Carneiro. “Early childhood development: Some preliminary findings to guide future actions.” PowerPoint Presentation. 2014. Bruhn, Miriam, Luciana de Souza Leao, Arianna Legovini, Rogelio Marchetti, and Bilal Zia. “The impact of high school financial education: experimental evidence from Brazil.” World Bank Policy Research Working Paper 6723. 2013. http://documents.worldbank.org/curated/en/2013/12/18640673/impact-high-school-financial-education-experimental-evidence-brazil Dhaliwal, Iqbal, and Caitlin Tulloch. “From research to policy: using evidence from impact evaluations to inform development policy.” 2012. https://www.povertyactionlab.org/sites/default/files/publications/With%20JDE%20Revisions%20and%20Footnotes%202012.09.03.pdf Dupas & Miguel. “Impacts and Determinants of Health Levels in Low-Income Countries.“ Prepared for the Handbook of Field Experiments. 2016. http://emiguel.econ.berkeley.edu/assets/miguel_research/73/Health_DupasMiguel_Handbook.pdf Gender Innovation Lab. Africa Gender Innovation Lab (GIL) Influence. November 2015. Glennerster, Rachel. “Not so small.” Running Randomized Evaluations blog. May 27, 2016. http://runningres.com/blog/2016/5/27/not-so-smallGoldstein, Markus. Conversation with Dave Evans on a bus. Rome, Italy. May 31, 2016.Hernandez Licona, Gonzalo. “Impact evaluation in Mexico.” January 2008. http://siteresources.worldbank.org/INTISPMA/Resources/Training-Events-and-Materials/449365-1199828589096/GonzaloHernandezLicona.pdf

References - 1

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y

17

i2i. Changing Policy through Research: DIME IE Portfolio in Kenya. n.d.i2i. Selected Examples of DFID’s Contribution to Evidence-Based Policy through i2i. n.d.Imas, Lnda G. Morris, and Ray C. Rist. The Road to Results: Designing and Conducting Effective Development Evaluations. World Bank. 2009. https://openknowledge.worldbank.org/bitstream/handle/10986/2699/52678.pdf Jensen, Robert. “The Digital Provide: Information (Technology), Market Performance, and Welfare in the South Indian Fisheries Sector.” Quarterly Journal of Economics 122(3), August 2007. http://www.jstor.org/stable/pdf/25098864.pdf Legovini, Arianna. “The development impact of impact evaluation.” World Bank. 2014. http://www.worldbank.org/content/dam/Worldbank/Feature%20Story/japan/pdf/event/2014/120514_Arianna_Legovini.pdf Legovini, Arianna, Vincenzo di Maro, Caio Piza. “Impact evaluation helps deliver development projects.” World Bank Policy Research Working Paper 7157. 2015. http://documents.worldbank.org/curated/en/2015/01/23173058/impact-evaluation-helps-deliver-development-projects Mkelemi, Godfrey, and David K. Evans. “The Challenge and Success of Impact Evaluation for Tanzania Community-Based Conditional Cash Transfer Pilot.” Conference presentation, IFAD/3ie. May 31, 2016. http://www.slideshare.net/DavidEvans198/the-challenge-and-success-of-impact-evaluation-for-tanzania-communitybased-conditional-cash-transfer-pilot Prottas, Chris (@ChrisProttas). “I understand why, but real mental disconnect w anecdotes used to justify impact of impact evals.” May 27, 2016, 3:10am. Tweet. https://twitter.com/ChrisProttas/status/736137568954294272?lang=enVledder, Monique, Jed Friedman, Mirja Sjoblom, Thomas Brown, and Prashant Yadiv. “Optimal Supply Chain Structure for Distributing Essential Drugs in Low Income Countries: Results from a Randomized Experiment.” Ross School of Business working paper 1269. March 2015. https://deepblue.lib.umich.edu/bitstream/handle/2027.42/110800/1269_Yadav.pdf World Bank. From Evidence to Policy: The Challenge of Ensuring Adequate Stocks of Essential Drugs in Rural Health Clinics. November 2010. World Bank. “Improving patient safety in Kenya.” November 6, 2013. PowerPoint Presentation. https://www.wbginvestmentclimate.org/results/upload/Kenya-PS-BBL_November-6-2013.pdfWorld Bank. Africa Impact Evaluation Update: Agriculture and Land. March 2014.

References - 2

Evan

s |

Wor

ld B

ank|

im

pact

eva

luat

ions

, op

erat

ions

& p

olic

y