8_tbilisi metro extension project and impact evaluation (cwrd)
DESCRIPTION
Conference on Impact Evaluation: Methods, Practices, and LessonsAuditorium A, ADB Headquarters, Manila 11 July 2012TRANSCRIPT
![Page 1: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/1.jpg)
TBILISI METRO EXTENSION PROJECT
IMPACT EVALUATION
July 11, 2012
Tomomi Tamaki: Principal Regional Economist, CWRD
Nina Fenton: Young Professional, CWRD
![Page 2: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/2.jpg)
THIS PRESENTATION
1. The Tbilisi Metro Extension Project
2. The Theory: Difference-in-Differences
3. The Reality: A Modified Evaluation Design
4. Some Reflections
![Page 3: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/3.jpg)
PART ONE
THE TBILISI METRO EXTENSION PROJECT
![Page 4: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/4.jpg)
THE TBILISI METRO EXTENSION PROJECT
Part of Multi-tranche Financing Facility (MFF). Four
components of first tranche:
(i) Completion of a 1.2 km Tbilisi metro extension on
Vazha Pshavela Av. to the university district,
serving 150,000 inhabitants
(ii) Redevelopment of Gorgasali embankment in Tbilisi,
including pedestrianized access to the river from the
old city
(iii) Upgrading of 20 km of the Mestia urban area road
network
(iv) Urban renewal of main avenues in Kutaisi through the
introduction of a 26-km cycle network.
![Page 5: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/5.jpg)
THE TBILISI METRO EXTENSION PROJECT
![Page 6: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/6.jpg)
THE TBILISI METRO EXTENSION PROJECT
IE study to focus on the Tbilisi metro extension.
Expected impacts
Improved urban environment
Local economic development
Better health
Poverty reduction.
Through:
Better access to employment
Lower transport cost and travel time
Reduced pollution
![Page 7: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/7.jpg)
PART TWO
THE THEORY: DIFFERENCE IN DIFFERENCES
![Page 8: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/8.jpg)
A DIFFERENCE-IN-DIFFERENCES APPROACH
Infrastructure projects are not (usually) “randomly” allocated
Difference-in-differences compares: ➙ Mean outcome value before and after the
intervention (first difference) ➙…..for beneficiaries and a comparison group
(second difference).
So, differences in outcomes between project-affected population and others cannot be attributed to the project under evaluation: selection bias
![Page 9: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/9.jpg)
THE PARALLEL TREND ASSUMPTION
The groups
develop at the
same speed in the
absence of the
project…
![Page 10: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/10.jpg)
FAILURE OF THE ASSUMPTION
Assumption fails,
groups develop at
different speeds
![Page 11: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/11.jpg)
THE TBILISI METRO EXTENSION: IE
Planned Methodology
Difference in difference of households, students
and businesses
Looking at key impact variables
The Problem
Failure of the parallel-trend assumption
![Page 12: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/12.jpg)
PARALLEL TRENDS IN TBILISI?
Widespread
development in Tbilisi:
violates parallel trend
assumption
Lack of data:
hard to
identify a
“similar” area
Different parts
of town have
very different
characteristics
![Page 13: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/13.jpg)
PART THREE
THE REALITY: A MODIFIED EVALUATION DESIGN
![Page 14: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/14.jpg)
THE STUDENT SURVEY
1. Student survey
Difference-in-differences design
Treatment group: TSU Maglivi students (2015)
Control groups:
Cohort approach. Students entering 2014 as control, students entering 2015 as treatment group (benefit from metro)
TSU downtown students (no metro access) ILIA state students
![Page 15: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/15.jpg)
HOUSEHOLDS AND BUSINESSES
Some outcomes more directly attributable to the metro than others, even in the absence of a control group: Travel time, travel cost.
2. Household survey
Single difference survey of 300 households before and after project completion. Measuring along the causal chain – time use, use of transportation, socio-economic indicators.
3. Enterprise survey
• Single difference survey of 300 businesses before and after project completion. Measuring revenue, profit, customer characteristics.
4. Air quality survey
Technical survey of air pollution by new station and in Delisi.
![Page 16: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/16.jpg)
PART FOUR
SOME REFLECTIONS
![Page 17: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/17.jpg)
BE CLEAR ABOUT THE CAUSAL CHAIN
Lay out project theory and causal chain
clearly
DMF is a starting point but, depending on
the project and the impacts we are
interested in may not be enough
Being clear about causal mechanisms can
really help with evaluation design
![Page 18: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/18.jpg)
LOGFRAME: IMPACTS ON STUDENTS
New Metro
Less travel time
More time for other activities incl. studying
Fewer delays, higher
attendance rates
Better test scores
More convenient
travel
Fewer travel costs
More money for other things
Better well-being
![Page 19: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/19.jpg)
THEORY-BASED IMPACT EVALUATION
Deciding what to measure means looking
at the Theory of Change:
”An explicit theory or model of how an intervention
contributes to a chain of intermediary results and
finally to the intended or observed outcomes”
”About the central processes or drivers by which
change comes about for individuals, groups or
communities”
![Page 20: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/20.jpg)
IMPLEMENTING IMPACT EVALUATION
Expect:
Lack of data
Project delays
Difficulty in identifying control groups, especially in
rapidly developing DMCs
Be creative
Pay close attention to survey design and
enumerator training
Consider including small-scale qualitative surveys
![Page 21: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/21.jpg)
PLANNING FOR IMPACT EVALUATION Important to consider:
→Is the budget sufficient? How will you cover the end-line? What is the project is delayed?
→Who will manage and lead the evaluation: staff
time and commitment as well as good consultants →Is there real commitment and interest from the
DMC?
→Is it likely that the results will be used to inform future project design or policymaking?
![Page 22: 8_Tbilisi Metro Extension Project and Impact Evaluation (CWRD)](https://reader033.vdocuments.net/reader033/viewer/2022051208/5450b3e3af7959bd668b4b16/html5/thumbnails/22.jpg)
Thank you!