ove’s experience with impact evaluations
Post on 25-Feb-2016
33 Views
Preview:
DESCRIPTION
TRANSCRIPT
OVE’s Experience with Impact EvaluationsParisJune, 2005
Impact Evaluations
Alternative definitional models:– time elapsed since intervention– Counterfactual comparison
OVE adopted the counterfactual approach, and further limited the initial sample to programs with partial coverage.
Partial coverage allows observation of treatment effects through comparison of treated and untreated groups
Policy
The general evaluative question proposed by the IDB’s ex post policy is “…the extent to which the development objectives of IDB-financed projects have been attained.”
This questions is most convincingly answered through treatment effect evaluations
Selecting Projects
Random selection is appropriate for accountability-oriented evaluations
Purposive selection of projects of similar design across countries is better for generating learning regarding the model underlying the interventions
Clusters of like projects permit meta-evaluations of models
Projects Selected
Chose purposive cluster sampling strategy but some stand-alone projects. A total of 16 projects were selected
Clusters: (i) Neighborhood Improvement projects and (ii) Land Titling Projects.
Stand-alone: Cash Transfer (Argentina), Potable Water (Ecuador); Agricultural Subsidies and Cash Transfer Programs (Mexico’s Procampo and Opportunities programs); Social Investment Fund (Panama)
Stand-alones serve as pilots for future clusters
Both Performance Monitoring and Treatment Effect Are Required
Performance Monitoring versus Treatment Effect Evaluation Performance monitoring Treatment effect evaluation
Primary goal: accountability to stakeholders and resolution of execution problems, cost-efficiency
Analysis of outputs & gross outcome effects to improve implementation
Data collection is ongoing, relying on readily accessible and regularly collected data
Primary goal: knowledge creation (understanding and improving program treatment effects), cost-effectiveness and cost-benefit
Analysis of net effects (treatment effects) of development outcomes to improve project design (concurrently or for future similar projects)
Data collection is periodic, more intensive and requires information on both beneficiaries and non- beneficiaries over time.
Treatment Effect includes randomized design; propensity score matching, controlled comparison, discontinuous regressions.
Limits to Treatment Effect Evaluations
Comparison of Alternative Approaches to Program Evaluation Structural Econometric Approach Treatment Effect Approach Range of questions addressed
Evaluates the treatment effect of existing program. Forecasts the program’s effect in a new environment. Predicts the effects of a program never tried before.
Evaluates the treatment effect of existing program. Evaluates one program in one environment. Cannot predict effects of a new program.
Range of programs that can be evaluated
Programs with either partial or universal coverage depending on variation in data (prices and endowments
Programs with partial coverage (treatment and control groups)
Comparability Across studies
High comparability across evaluations (program invariant parameters)
Not generally comparable unless evaluations designed for a meta-evaluation of similar programs.
Source: modified from Table V in “Structural Equations, Treatment Effects And Econometric Policy Evaluations” by James J. Heckman and Edward Vytlacil, NBER Working Paper No. 11259, March 2005. Note this article proposes a synthesis of the two approaches, which is ignored in this modified table.
Experience
The required information supposedly generated through standardized performance monitoring is absent in a large majority of IDB projects examined
10 of the 16 selected projects had inadequate data for treatment effect evaluation
6 of the 16 could be retrofitted with sufficient data to attempt a treatment effect evaluation
Retrofitting implied significant data collection costs, costs that could have been avoided had adequate performance monitoring been in place over the life of the project.
The Bank’s Current Portfolio
Of 593 active projects in mid -2004: 97 (16%) claim existence of information for at least one development
outcome, of which 27 have the information in an electronic form, of which for 5 the information is held in the Bank, of which 2 appear to be collecting data for treatment effect evaluation
Experience: Limits to Retrofitting
The questions answered are dependent on the information found rather than on the relevance and usefulness of the hypotheses being tested: the tail wagging the dog .
It severely limits the set of control variables’ information thus reduces the veracity of treatment effect findings
retrofitted data may not correspond to the development outcomes declared by the projects. A project can be evaluated using intended and unintended effects, but should at least consider as a minimum the intended ones.
Experience Confirms the Value of Treatment Effect Evaluation
-0.20
-0.10
0.00
0.10
0.20
0.30
0.40
0.50
Water Sewer Rubbish Illiteracy Income Rent Child MortalityHomicide Rate
NaiveTreatment
In just one project (Neighborhood Improvement, Rio-Brazil) comparing naïve and treatment effect the following held regarding naïve and treatment effects: positive/negative, negative/positive; greater/smaller; smaller/greater; and the same.
So pictures need to be interpreted with caution
Before After
Experience
“…the six treatment effect evaluations undertaken during 2004 do show that the Bank’s interventions have a significant development effect for at least one declared development objective. These findings suggest that the Bank may be currently understating its contribution to development.”
EXPERIENCE: Findings Land Titling
“Beneficiaries of Land Regularization projects saw property values for their land increase …. However, for the other purported development effects (greater productivity, increased investment, and greater access to credit), no unambiguous treatment effects were found.
Ramifications for project design: for small and poor producers to benefit from a pro-market regime, titling alone is not sufficient
Transaction costs and market distortions that limit access to credit must be also simultaneously be addressed
EXPERIENCE: Findings Potable Water
heterogeneity of results important. a regressive relationship between treatment effect and income, where more educated (and wealthier) households did better than less educated (and poorer) households
Ramification for project design: projects should include or be coordinated with, as a hypothesis to be tested, a health education component together with potable water expansion.
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
Bottom 25% 25%-50% 50%-75% Top 25% Expenditure level
prop
ortio
nal c
hang
e
All SampleAt least Primary
Impact on infant mortality
EXPERIENCE: Findings from cash transfer and agricultural subsidy programs
Issue: Do conditions attached to cash transfers produce more change than the transfers alone
Income effect alone may be substantial, and conditionalities are costly to administer and monitor
In a comparison between two programs in Mexico with and without conditionality the following ramifications for project design were found:– Conditionality (school and clinic attendance) does result in an effect
over and above the income effect of the transfer. – Transfers to the mother as opposed to the father matters as the effects
are greater when the transfer is to the mother
EXPERIENCE: Low Costs
•Treatment effect evaluations can be done inexpensively, if attention is paid to data at the time of design and during implementation•Data collection costs can be substantial if retrofitted, but still within reasonable limits.•Costs ranged from $28,000 to $92,000 per evaluation, much lower than the “norm”: small budget high returns
Land Titling (Peru) 83 91,544 141,837 0.17%Neighbourhood Improvement (Brazil) 600 83,606 133,899 0.02%Potable Water (Ecuador) 280 89,934 140,227 0.05%Cash Transfer (Argentina) 637 37,545 87,838 0.01%Agricultural Subsidies and Cash Transfer (Mexico) 1881 50,000 100,293 0.01%Social Investment Fund (Panama) 38 27,777 78,070 0.21%
Total 3518 380406 682164 0.08%
Program Value (US$ million)
Direct Costs (US$) Total Costs ( US$) Total Costs as a Percent of
Program Value
Summary
Initial experience with treatment effect impact evaluations provided considerable knowledge relevant for future project design
Costs were moderate, and can be expected to be lower in the future if the performance monitoring system is improved
Data has value to researchers, and cost-sharing in data collection was possible in several cases
Treatment effect evaluation provides the only convincing basis for asserting development effectiveness
top related