setting the framework: designing interventions to achieve impact esteban diez roux patricia yanez...
TRANSCRIPT
![Page 1: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/1.jpg)
Setting the framework: Designing interventions to achieve impact
Esteban Diez RouxPatricia Yanez PagansInter-American Development Bank
June 22nd 2015
![Page 2: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/2.jpg)
Setting the framework: Designing
interventions to achieve impact
• Impact Evaluations and the Transport Sector
• Main Issues in Designing IEs for Transport
![Page 3: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/3.jpg)
Effectiveness: What works and what doesn’t work?
Why do we evaluate?
Efficiency: Most efficient way to achieve an outcome?
Evidence-based policy making
![Page 4: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/4.jpg)
What do we want to achieve with our projects?Is it more than reducing transportation costs?Traditional outcomes in transport:– Travel times– Vehicle operating costs
Non-traditional outcomes? – New development agenda– EI’s could be key to understand our impact on
development – But…how far can we get with our investments?– Attribution critical
![Page 5: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/5.jpg)
How can we design projects that are more effective in achieving these non-traditional outcomes and impacts?
First step…
Understand what works and what doesn’t work
How?Impact evaluation!
![Page 6: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/6.jpg)
Attribution: Let’s be careful in getting the right answers
• Traditional approaches in transport:
– Before-and-after methods– Ex-post cost-benefit analysis
• Limitations of these methods?
![Page 7: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/7.jpg)
Impact evaluations complement other approaches
They complement process evaluations and ex-post cost benefit analyses
![Page 8: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/8.jpg)
However, different evaluation instruments answer slightly different questions
8
Process evaluation
Did we achieve all products at the planned costs ?Did outcome indicators move in the right direction?
Ex-post cost-benefit analysis
Did project economic benefits exceed project costs?
Impact evaluation
What would be the situation of beneficiaries without the project?
![Page 9: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/9.jpg)
Learning, accountability, and complementarities
9
Process evaluation
Generates information about costs and trends in indicators. Helps for project accountability.
Ex–post cost-benefit analysisExtracts information of costs and benefits from monitoring. Ideally extracts benefits information from IE. Justifies investments based on their economic return.
Impact evaluationMainly for learning, helps to understand what strategies are more effective. Provides information on actual benefits for economic analysis.
![Page 10: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/10.jpg)
Using instruments in isolation might leave important open questions
Process evaluation
Are there any other external factors that influenced results?
Ex-post cost-benefit analysis
What are the real costs and benefits of the project?
Impact evaluation
Is the program cost-effective?
![Page 11: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/11.jpg)
Setting the framework: Designing
interventions to achieve impact
• Impact Evaluations and the Transport Sector
• Main Issues in Designing IEs for Transport
![Page 12: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/12.jpg)
Designing impact evaluations (IEs) for the transport sector
• Causality or attribution• Mechanisms• Spillovers
=> Need a clear understanding of the theory of change
![Page 13: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/13.jpg)
The theory of change is key for the design of projects and for IEs
Resources at the disposal of the project
Actions taken to convert inputs into outputs
Tangible goods and services project activities produce
Results likely to be achieved once beneficiaries use outputs
Final project goals
StaffBudget
ConstructionRehabilitation
Km of roads paved
Reduction in travel timesReduction in operation costs
Poverty reduction
Inputs Activities Outputs Outcomes Impacts
=> IE questions?
![Page 14: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/14.jpg)
Impact evaluation needed where program attribution is unclear
Resources at the disposal of the project
Actions taken to convert inputs into outputs
Tangible goods and services project activities produce
Results likely to be achieved once beneficiaries use outputs
Final project goals
StaffBudget
ConstructionRehabilitation
Km of roads pavedKm of new roads
Reduction in travel timesOperation costsIncreased mobility?
Poverty reductionIncreased productivity
Under project control
Inputs Activities Outputs Outcomes Impacts
![Page 15: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/15.jpg)
Household level outcomes
Firm level outcomes
Community/Regional level outcomes
What have we learned about non-traditional outcomes in transport?
Picture: Horia Varlan
![Page 16: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/16.jpg)
Some aspects we have learned about household-level outcomes
Road improvement increases…
Household investments in durable assets (Mexico - Gonzales-Navarro and Quintana-Domeneque, 2012)
Changes household production decisions(Peru - Valdivia, 2009)
Use of health and education services (Bangladesh - Khandker et al., 2009)
![Page 17: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/17.jpg)
Some aspects we have learned about firm-level outcomes
Expansion rail system
Changes hiring decisions of firms located close to stations (US - Holzer et al. 2003)
Improved highways
Helps firms reduce stock of input inventories and re-optimize suppliers choice (India - Datta, 2012)
![Page 18: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/18.jpg)
Some aspects we have learned about community & regional level outcomes
Rehabilitation of rural roads
Reduces local prices (Sierra Leone - Casaburi et al., 2013)
Railway construction
Increases wage gaps through increased specialization (Central Europe - Walker, 2015)
Electronic toll collection
Reduces vehicle emissions improving overall child health (US -Currie and Walker, 2011)
![Page 19: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/19.jpg)
Attribution: When compared to other instruments only impact evaluation seeks for causality
Process evaluationNO attribution
Ex-post cost-benefit analysisNO attribution
Impact evaluationYES attribution
![Page 20: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/20.jpg)
How do we seek for causality with an impact evaluation?
What would have happened to our beneficiaries in the absence of the
program?
=> Find the right counterfactual (comparison group)
![Page 21: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/21.jpg)
• Roads and public transportation systems are not randomly placed
• Benefits are difussed, implications on how we define the zone of intervention
• Impacts may take a long time to emerge
IEs in transport can be challenging…
![Page 22: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/22.jpg)
Mechanisms: We usually assume that roads reduce poverty, but how?
Povertyreduction
New employment opportunities?
Market prices?
Increased production?
Access to public services?
Picture by: Klaus Leidorf
Others?
![Page 23: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/23.jpg)
Mechanisms: Distribution of impacts and complementarities
• Are benefits different for men and women?
• Are benefits different for poor and less poor populations?
• How do transport investments interact with other interventions?
Picture: Bob Miller
Picture: Bob Miller
![Page 24: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/24.jpg)
Spillovers: Are we having positive/negative impacts on non-beneficiaries?
Picture: Rakesh Rocky
![Page 25: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/25.jpg)
Spillovers 1: Project externalities
Benefits or costs that are transferred directly from beneficiaries to non-beneficiaries
• Positive? Reducing travel times for users of BRT systems may reduce travel time of non-users in less congested roads
• Negative? Relocation of economic activity closer to highways may have negative impacts on areas further away
![Page 26: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/26.jpg)
Spillovers 2: General equilibrium effects
Program effects are transmitted to the local economy
• Rehabilitation of rural roads reduces local prices (Sierra Leone - Casaburi et al., 2013)
• Railway construction reduces price gaps between regions, but increased specialization increases wage gaps (Central Europe - Walker, 2015)
![Page 27: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/27.jpg)
Spillovers 3: Interactions
Non-beneficiaries may be affected by the program through social and economic
interactions with beneficiaries
• Road safety education programs might have impacts beyond those that receive education
• ICT messages that promote savings might affect the savings decisions of others (friends, family, neighbors)
![Page 28: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/28.jpg)
Spillovers 4: Behavioral effects
Program affects behavioral or social norms within the area of the intervention affecting
both beneficiaries and non-beneficiaries.
• Awareness campaigns to promote safe driving in some neighborhoods might change social norms in a city
• ICTs to promote transparency in some governments offices might promote a general culture of transparency
![Page 29: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/29.jpg)
How to successfully implement impact evaluations?
• Lots of discussion and consensus between key stakeholders
• Continued financial support • Strong local capacity for monitoring and
implementation• Local ownership to answer relevant questions and
guarantee success
Picture: Steve Davis
![Page 30: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/30.jpg)
Key messages
1) Find the right ways to identify causality
2) Design IEs that allow understanding mechanisms and spillovers
3) Work together to guarantee success
Let’s start!
![Page 31: Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez Pagans Inter-American Development Bank June 22 nd 2015](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649e9e5503460f94b9fdee/html5/thumbnails/31.jpg)
THANK YOU