An impact evaluation seeks to attribute all, or part, of the observed change in outcomes to a specific intervention.
Level Indicators
Inputs Resources: Funds and personnel
Activities Teacher trainingSchool improvementsDecentralized management and local school management committees
Outputs Trained teachersBetter school facilitiesFunctioning school management committees
Intermediate outcomes Higher school enrolments at all levelsTeacher and parent satisfactionBetter managed schools
Final outcomes Improved learning outcomes
Impact Higher productivity and earningsEmpowerment
Focus on final welfare outcomes, e.g. Infant mortality Income povertySecurity
Usually long-term, but not necessarily so (but then sustainability is an issue)
Projects (or specific interventions) Individual projects are the ‘back bone’ of impact
analysis But even then may only be able to do rigorous
impact analysis of some components Programmes
Sector wide programs can be conceived of as supporting a range of interventions, many of which can be subject to rigorous impact evaluation.
Policies In general different approaches are required,
such as CGEs – these are not being discussed today
Pick a named intervention for an impact evaluation and make a short list of indicators (using the log frame) for evaluation of this intervention
But we don’t know if they were similar before… though there are ways of doing this
Before After
Project (treatment) 66
Control 55
Ex ante design preferred to ex post: impact evaluation design is much stronger if baseline data are available (but may still be possible even if they are not)
Means collecting data before intervention starts, and can be affecting the design of the intervention
But can sometimes use secondary data, that is an existing survey
1. Confounding factors
2. Selection effects
3. Spillovers and contagion
4. Impact heterogeneity
5. Ensuring policy relevance
Other things happen – so before versus after rarely sufficient
So get a control group… but different things may happen there
So collect data on more than just outcome and impact indicators
And collect baseline data But …
Program placement and self-selection Program beneficiaries have particular
characteristics correlated with outcomes – so impact estimates are biased
Need to use experimental or quasi-experimental methods to cope with this; this is what has been meant by rigorous impact evaluation
But it is just one facet of impact evaluation design
Other things can also bias impact estimates
Experimental (randomized):Limited application, but there are
applications and it is a powerful approachMany concerns (e.g. budget and ethics) and
not valid Quasi-experimental design (regression
based):Propensity score matching is most commonRegression discontinuity Interrupted time series Regression modelling of outcomes
Spillover – positive and negative impacts on non-beneficiaries
Contagion – similar interventions in control areas
Need to collect data on these aspects and may need to revise evaluation design
WHAT ARE THE MAJOR CONFOUNDING FACTORS FOR YOUR OUTCOME AND IMPACT INDICATORS?
HOW MIGHT SELECTION BIAS, SPILLOVER AND CONTAGION AFFECT THE EVALUATION OF THE INTERVENTION YOU HAVE SELECTED?
Impact varies by intervention (design),
beneficiary and context
‘Averages’ can be misleading
Strong implications for evaluation
design
Is the impact of X and Y, bigger, equal to or less than the impacts of doing X and Y separately?
For example, hygiene promotion and sanitation facilities
Evidence suggestions they are substitutes- either one reduces incidence child diarrhea by 40-50%, but not by more if the two are combined
Irreparable damage to physical and cognitive development results from nutritional deprivation in the first two years of life
Hence interventions to infants have greater long-run impact on many outcomes than do those aimed at older children (such as school feeding programs)
What sort of differences in impact would you expect for your intervention with respect to intervention (design), context and beneficiary?
ProcessStakeholder engagementPackaging messages
DesignTheory-based approachMixed methodsCapture all costs and benefits, including
cross-sectoral effects Cost effectiveness and CBA
Make explicit underlying theory about how inputs lead to intended outcomes and impacts
Documents every step in causal chain Draws on multiple data sources and
approaches Stresses context of why or why not
working
Assumption Findings
Provide nutritional counselling to care givers
Mothers are not decision makers, especially if they live with their mother-in-law
Women know about sessions and attend
90% participation, lower in more conservative areas
Malnourished and growth faltering children correctly identified
No – community nutrition practitioners cannot interpret growth charts
Women acquire knowledge Those attending training do so
And knowledge is turned into practice
No there is a substantial knowledge-practice gap
Supplementary feeding is additional food for intended beneficiary
No, considerable evidence of substitution and leakage
Adopted changes are sufficient to improve intended outcomes
Only sometimes (not for pregnant women)
Need to collect survey data at the unit of intervention (child, firm etc)
Will need also facility/project data Need data across the log frame and for
confounding factors – and for your instrumental variables (lack of valid instruments is the major obstacle to performing IE)
Designing data collection instruments takes time and should be iterated with qualitative data
Study Data sources
Rural electrification 3 rural electrification surveys11 DHS2 LSMS
India irrigation and rural livelihoods
Own surveyDistrict-level government dataCensus data
Bangladesh maternal and child health and nutrition
DHSProject data + national nutrition survey
Ghana basic education 1988/89 GLSS (LSMS)Own follow up survey
Kenya agricultural extension 2 previous rural surveysOwn follow up survey
OUTLINE YOUR PROPOSED EVALUATION DESIGN (TIMING OF DATA COLLECTION, IDENTIFICATION OF CONTROL, IF ANY)
WHAT DATA SOURCES WOULD YOU USE FOR YOUR PROPOSED EVALUATION?