an introduction to impact evaluation and application to the ethiopia nfsp workshop on approaches to...

An introduction to An introduction to Impact Evaluation Impact Evaluation and application to the and application to the Ethiopia NFSPEthiopia NFSP

Workshop on Approaches to Workshop on Approaches to Evaluating The Impact of the National Evaluating The Impact of the National Food Security Program, Addis Ababa, Food Security Program, Addis Ababa, January 16, 2006January 16, 2006

Arianna Legovini, The World BankArianna Legovini, The World Bank

ET NFSP Arianna LegoviniET NFSP Arianna Legovini 22

ContentContent

1. Context

2. Motivation

3. Defining impact

4. The evaluation problem

5. Problems with naïve evaluations

6. Application to the NFSP


1. Context1. Context

Most social programs are targeted to a specific Most social programs are targeted to a specific group of the population. group of the population.

Within that group some units (individuals, Within that group some units (individuals, households, communities) get the program and households, communities) get the program and others do not, or some get it at the beginning of others do not, or some get it at the beginning of program implementation and others get it later, program implementation and others get it later, … depending on budget constraints, … depending on budget constraints, implementation capacity, and targeting implementation capacity, and targeting efficiency.efficiency.


2. Motivating rigorous impact 2. Motivating rigorous impact evaluationevaluation

Monitoring tells us whether things are moving Monitoring tells us whether things are moving in the right direction but cannot tell us the in the right direction but cannot tell us the cause of those movementscause of those movements

Rigorous impact evaluation causally links Rigorous impact evaluation causally links activities to results.activities to results.

There are three main functions for impact There are three main functions for impact evaluation: evaluation: It is a political tool to manage public opinion It is a fiscal tool to allocate budget resources

more efficiently, and It is a management tool to improve program

design, results and sustainability


Political toolPolitical tool

Public opinion can easily shift in Public opinion can easily shift in response to few publicized failuresresponse to few publicized failures

Empirical evidence on program impacts Empirical evidence on program impacts give governments the ability to effectively give governments the ability to effectively manage public opinion (media , manage public opinion (media , parliament)parliament)


Fiscal toolFiscal tool

Shift in budget allocation are often made on Shift in budget allocation are often made on hunches of what might deliver more resultshunches of what might deliver more results

Impact evaluation turns hunches into Impact evaluation turns hunches into quantitative estimates of the cost-quantitative estimates of the cost-effectiveness of alternative programs in effectiveness of alternative programs in reaching desired objectivesreaching desired objectives

It provides the basis for shifting resources It provides the basis for shifting resources towards more effective programstowards more effective programs


Management toolManagement tool

This is where impact evaluation can make This is where impact evaluation can make huge contributionshuge contributions

By testing alternative ways of doing things By testing alternative ways of doing things within the same program, impact evaluation within the same program, impact evaluation can measure the relative effectiveness of can measure the relative effectiveness of alternativesalternatives

Feedback can be used to modify program Feedback can be used to modify program design during implementation to ensure design during implementation to ensure success. success.

It is the ultimate managing-by-results tool.It is the ultimate managing-by-results tool.


3. Defining Impact3. Defining Impact In the language of the impact evaluation In the language of the impact evaluation

profession, the impact of a program is the profession, the impact of a program is the amount of change in any outcome (amount of change in any outcome (short, short, medium or long-termmedium or long-term) which is caused by the ) which is caused by the program. program.

Impact can be measured at Impact can be measured at any timeany time: : Even just after program announcement (e.g. change Even just after program announcement (e.g. change

in expectations may lead to changes in asset prices in expectations may lead to changes in asset prices even before a program is implemented), or even before a program is implemented), or

Any time after implementation (e.g. some changes Any time after implementation (e.g. some changes are instantaneous and some take time to realize are instantaneous and some take time to realize BUT if nothing moves in the short-run, you can rest BUT if nothing moves in the short-run, you can rest assured nothing will move in the long run either)assured nothing will move in the long run either)


4. The evaluation 4. The evaluation problemproblem

The impact of the program is the difference The impact of the program is the difference between the outcome with the program and between the outcome with the program and the outcome without the program.the outcome without the program.

BUT we only observe outcomes with OR BUT we only observe outcomes with OR without the program, not both at the same without the program, not both at the same timetime..

This means that the problem of measuring This means that the problem of measuring impact is a problem of missing data—not impact is a problem of missing data—not because we do not collect them, but because we do not collect them, but because they don’t exist.because they don’t exist.


Impact evaluation is a set of rigorous Impact evaluation is a set of rigorous analytical methods applied to solve analytical methods applied to solve the missing data problem by building the missing data problem by building a counterfactual and measuring a counterfactual and measuring observed outcomes against it. observed outcomes against it.


Observed changes and changes Observed changes and changes against a counterfactualagainst a counterfactual

Y1 (observedl)

Y1

* (counterfactual)

Y0

t=0 t=1 time


Solving the missing data Solving the missing data problemproblem

Need to build a control or comparison Need to build a control or comparison group that looks the same at time zero as group that looks the same at time zero as the group getting the program (treatment the group getting the program (treatment group)group)

What does “look the same” mean? It What does “look the same” mean? It means that, on average, the comparison means that, on average, the comparison and treatment group only differ by the and treatment group only differ by the fact that one gets the program and the fact that one gets the program and the other does not.other does not.


Building a control Building a control Experimental approachExperimental approach : Randomization : Randomization Give all units in the eligible population an Give all units in the eligible population an

equal chance to participate; and compare equal chance to participate; and compare those who win the lottery with those who did those who win the lottery with those who did not win the lotterynot win the lottery

Give all units in the eligible population an Give all units in the eligible population an equal chance to be included in either phase equal chance to be included in either phase I or phase II of the program; and then use I or phase II of the program; and then use units in phase II as controls for phase Iunits in phase II as controls for phase I


Building a comparison Building a comparison Non-experimental approachNon-experimental approach : : Matching: match characteristics of participants and Matching: match characteristics of participants and

non-participants at time zero; and then compare non-participants at time zero; and then compare changes in outcomes in treated population against changes in outcomes in treated population against changes in outcomes in matched observationschanges in outcomes in matched observations

Regression discontinuity: assume that because of Regression discontinuity: assume that because of measurement error, units just above or just below the measurement error, units just above or just below the eligibility threshold are the same on average; and eligibility threshold are the same on average; and compare participants just below the threshold with compare participants just below the threshold with non-participants just above the thresholdnon-participants just above the threshold

Pipeline comparisons: compare current participants to Pipeline comparisons: compare current participants to prospective participantsprospective participants

Instrumental variables: identify exogenous variable Instrumental variables: identify exogenous variable that influences participation but not outcomes to that influences participation but not outcomes to identify exogenous variation in outcomes due to the identify exogenous variation in outcomes due to the programprogram


Estimating impactEstimating impact

Single-difference (outcome of treatment Single-difference (outcome of treatment minus outcome of control at time t+1) can minus outcome of control at time t+1) can be used in experimental setting with be used in experimental setting with sufficiently large samplessufficiently large samples

Double-difference (before and after and Double-difference (before and after and against a control) is used in experimental against a control) is used in experimental setting with small samples and non-setting with small samples and non-experimental settings to control for experimental settings to control for selection biasselection bias


Data requirementsData requirements

Surveys with representative samples of Surveys with representative samples of beneficiaries and non-beneficiariesbeneficiaries and non-beneficiaries

Questionnaires include outcomes of Questionnaires include outcomes of interest (consumption, income, assets interest (consumption, income, assets etc), questions about the program in etc), questions about the program in question and questions about other question and questions about other programsprograms


Data requirementsData requirements

Experimental approach with large Experimental approach with large samples only requires survey at time t+1 samples only requires survey at time t+1 (single diff). In this case baseline is (single diff). In this case baseline is needed only if we are interested in needed only if we are interested in measuring targeting efficiency. measuring targeting efficiency.

Experimental with small sample and non-Experimental with small sample and non-experimental generally require baseline experimental generally require baseline and follow up panel data (double diff)and follow up panel data (double diff)


5. Problems with naive 5. Problems with naive evaluation techniquesevaluation techniques

Comparisons of beneficiaries before and after: Comparisons of beneficiaries before and after: outcomes change for many reasons, how to outcomes change for many reasons, how to determine which portion of that change is due determine which portion of that change is due to the program?to the program?

Comparisons ex-post relative to the rest of the Comparisons ex-post relative to the rest of the population: if program targets the worse off, population: if program targets the worse off, comparisons ex-post will compare beneficiaries comparisons ex-post will compare beneficiaries with non-beneficiaries that were initially with non-beneficiaries that were initially somewhat better off and wrongly conclude somewhat better off and wrongly conclude program did not have positive impactsprogram did not have positive impacts


6. Application to the 6. Application to the National Food Security National Food Security ProgramProgram

The NFSP M&E plan:The NFSP M&E plan: Is a good monitoring plan using administrative data Is a good monitoring plan using administrative data

with a good logical frameworkwith a good logical framework However, it has no plan for assessing targeting However, it has no plan for assessing targeting

efficiency and the plan for evaluation is flawed (no efficiency and the plan for evaluation is flawed (no comparison)comparison)

The sampling for the baseline only surveys The sampling for the baseline only surveys beneficiaries and the questionnaire does not permit beneficiaries and the questionnaire does not permit to estimate consumptionto estimate consumption


Current NFSP evaluation Current NFSP evaluation plan plan

Plan to track changes in outcomes before and Plan to track changes in outcomes before and after and against (unspecified) standardsafter and against (unspecified) standards

BUT program outcomes can improve or worsen BUT program outcomes can improve or worsen because of e.g. the weather. These effects because of e.g. the weather. These effects might be much larger and completely might be much larger and completely overshadow the effects of the program. overshadow the effects of the program. Adverse weather will have us conclude that the Adverse weather will have us conclude that the program is ineffective irrespectivelyprogram is ineffective irrespectively


Strengthening NFSP impact Strengthening NFSP impact evaluationevaluation

Define for which activities we want to estimate Define for which activities we want to estimate impact and ask the right questions to improve impact and ask the right questions to improve program overtimeprogram overtime

Implement framework to answer these Implement framework to answer these questions by adding variation of treatment of questions by adding variation of treatment of beneficiaries beneficiaries

Build comparison groups via randomized Build comparison groups via randomized experiments or the estimation of a experiments or the estimation of a counterfactual using statistical methodscounterfactual using statistical methods

Expand survey sample to include comparison, Expand survey sample to include comparison, include a consumption module in the include a consumption module in the questionnaire, and collect panel dataquestionnaire, and collect panel data

Estimate impacts using double difference Estimate impacts using double difference estimatorsestimators


Impact of which NFSP Impact of which NFSP activities?activities?

Productive safety-nets:Productive safety-nets: Public worksPublic works Workfare/cash or food transfersWorkfare/cash or food transfers Unconditional transfers to labor-poor householdsUnconditional transfers to labor-poor households

ResettlementResettlement Other food security activitiesOther food security activities

Financial servicesFinancial services Agricultural extensionAgricultural extension WaterWater Land conservation, etc.Land conservation, etc.


Opportunities to find a Opportunities to find a comparisoncomparison

Use phase-in of the program to compare Use phase-in of the program to compare current beneficiaries with prospective current beneficiaries with prospective beneficiaries since not all 15 M are beneficiaries since not all 15 M are beneficiaries at this time (pipeline comparison)beneficiaries at this time (pipeline comparison)

Compare woredas or kebeles with the program Compare woredas or kebeles with the program with similar woredas or kebeles without the with similar woredas or kebeles without the program (matching)program (matching)

Compare beneficiary households that barely Compare beneficiary households that barely qualified for the program with non-beneficiary qualified for the program with non-beneficiary households that barely did not (regression households that barely did not (regression discontinuity design)discontinuity design)


Data and analysisData and analysis The questionnaire should include questions to The questionnaire should include questions to

capture the outcomes of the program. capture the outcomes of the program. Currently the questionnaire includes questions Currently the questionnaire includes questions

on assets, savings and credit, and agricultural on assets, savings and credit, and agricultural activitiesactivities

Including a consumption module would provide Including a consumption module would provide the basis for measuring welfare and graduation the basis for measuring welfare and graduation in a reliable and quantitative manner.in a reliable and quantitative manner.

Collect panel data (baseline and follow up Collect panel data (baseline and follow up survey on the same sample of beneficiaries survey on the same sample of beneficiaries and non-beneficiaries).and non-beneficiaries).

Estimate impactsEstimate impacts


Yet, the most compelling Yet, the most compelling reason to do impact reason to do impact evaluation is to learn how to evaluation is to learn how to improve program improve program effectiveness over timeeffectiveness over time

This requires adding variation in program This requires adding variation in program treatment to determine which treatment to determine which mechanisms work best to achieve e.g. mechanisms work best to achieve e.g. graduationgraduation


Examples of what can be Examples of what can be learned: is farm only learned: is farm only best?best?

Is a wider menu of farm and non-farm Is a wider menu of farm and non-farm support more effective than farm support support more effective than farm support only?—Take a random sample of only?—Take a random sample of localities where we include support to localities where we include support to non-farm activities; and compare results non-farm activities; and compare results overtime.overtime.


More examples of what can More examples of what can be learned—how much is be learned—how much is enough?enough?

To secure household graduation, is it more To secure household graduation, is it more effective to spread support thin across all effective to spread support thin across all eligible households (or communities) or to eligible households (or communities) or to focus NFSP effort on a portion of the focus NFSP effort on a portion of the households (or communities) this year, another households (or communities) this year, another portion next year, etc?—randomly select a portion next year, etc?—randomly select a small group of beneficiaries, double the fiscal small group of beneficiaries, double the fiscal effort on these households, and monitor effort on these households, and monitor whether graduation rates more than double or whether graduation rates more than double or notnot


More examples of what can More examples of what can be learned—credit pick up be learned—credit pick up rates and effectiveness?rates and effectiveness? Does one size fit all? Would modifying the size of Does one size fit all? Would modifying the size of

the credit package improve credit pick up rates?—the credit package improve credit pick up rates?—randomly select some communities where randomly select some communities where household can choose the size of the credit; and household can choose the size of the credit; and compare with one size credit communitiescompare with one size credit communities

To improve the effectiveness of credit, would a To improve the effectiveness of credit, would a package of credit-training-technical assistance package of credit-training-technical assistance improve credit effectiveness? (evidence from the improve credit effectiveness? (evidence from the rest of the world says yes)—randomly select rest of the world says yes)—randomly select some communities where training and/or TA is some communities where training and/or TA is attached to credit; and compare with credit only attached to credit; and compare with credit only communitiescommunities


More examples of what can More examples of what can be learned—Quality of be learned—Quality of productive community productive community assetsassets

Does the threat of audit significantly Does the threat of audit significantly increase the quality of productive increase the quality of productive community assets?—take a random community assets?—take a random sample of communities and communicate sample of communities and communicate intention to audit project implementation intention to audit project implementation (Evidence from Indonesia says yes)(Evidence from Indonesia says yes)


Concluding remarksConcluding remarks Credible impact estimates require a Credible impact estimates require a

strengthened framework for evaluation. strengthened framework for evaluation. This requires taking the following steps:This requires taking the following steps: Agreeing on the questions that can be askedAgreeing on the questions that can be asked Designing a framework that can answer Designing a framework that can answer

these questions (program delivery, these questions (program delivery, evaluation method, sampling, questionnaire)evaluation method, sampling, questionnaire)

Implementing the framework, analyzing and Implementing the framework, analyzing and reporting periodicallyreporting periodically

Improving program overtime based on Improving program overtime based on evidenceevidence

an introduction to impact evaluation and application to the ethiopia nfsp workshop on approaches to...

Documents