joost de laat strategic impact evaluation fund, world...
TRANSCRIPT
![Page 1: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/1.jpg)
Joost de LaatStrategic Impact Evaluation Fund, World Bank
![Page 2: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/2.jpg)
Why bother with impact evaluations?
![Page 3: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/3.jpg)
Education example: Computers in school: good idea?
That small miracle can be replicated in every school, rich and poor, across America ... Every child in American deserves a
chance to participate in the information revolution.
President Clinton (1998)
One Laptop Per Child -Peru
![Page 4: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/4.jpg)
It depends…some rigorous impact evaluations find no impact…
• Peru: “One laptop per child” had no sizeable impact on learning.
• Colombia: “Computers for Education” program: no impact on learning
• Israel: the “tomorrow-98”, placing 35,000 computers across elementary and middles schools across Israel had no measurable impact.
…Computer-based learning was not integrated into the curriculum.
• India: but integrated computer assisted learning in India raised student performance
![Page 5: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/5.jpg)
We need more than programs
we need programs that work
![Page 6: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/6.jpg)
857 estimates of program impact from 207 evaluations
![Page 7: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/7.jpg)
We need more than programs
we need programs that work
![Page 8: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/8.jpg)
We need more than programs
we need programs that work
![Page 9: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/9.jpg)
Other conclusions…
• Human capital programs (e.g. training): Zero (or negative) effects in short-run, larger effects in long-run
• Public sector work programs: Negative to zero effects across all time horizons
• Females and long-term unemployed: on average, larger employment impacts (compared with males and short-term unemployed)
• Youth and elderly: on average, smaller impacts than middle-aged people
![Page 10: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/10.jpg)
Strong growth in impact evaluations, also in low and middle-income countries
0
50
100
150
200
250
300
350
400
2006 2007 2008 2009 2010 2011 2012
110141
170
269298
364 366
All
All
Example: International Impact Evaluation Initiative (3ie) database
![Page 11: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/11.jpg)
Using impact evaluations to make program decisions
![Page 12: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/12.jpg)
Sustain and Expand Successful Programs
Example: Mexico PROGRESA
![Page 13: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/13.jpg)
Using IE to Sustain and Expand Programs
Countries implementing CCT programs in
1997
![Page 14: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/14.jpg)
Using IE to Sustain and Expand Programs
Countries implementing CCT programs in
2011
![Page 15: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/15.jpg)
Or Cut Non-Performing Programs
• Federal US program created to prevent drug use, gang membership, and violent behavior
• Police officers visit 5th and 6th grade classrooms to educate students
Evaluations show no significant impact → funding reduced(GAO, 2003; Weiss, 2003; West and O’Neil, 2004)
D.A.R.E Program (USA)
![Page 16: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/16.jpg)
How do impact evaluations compare to other evaluations?
![Page 17: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/17.jpg)
Our Toolbox: Monitoring and Evaluation Methods
1Needs Assessment
2Process Evaluation
3Impact Evaluation
Analysis of Beneficiary Needs
Analysis of Program
Implementation
Measures How Much the Program
Impacts Beneficiaries
1. Services being delivered?
2. Clients satisfied?3. Services reaching
target population?
1. Youth skills improved?
2. Employment rates higher?
1. Why are we doing the project?2. Who is the target population?3. Why do we think this approach will work in this context?
![Page 18: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/18.jpg)
How to Go from Program to Results?
Education PROGRAM• Inputs: people,
money, supplies
• Activities: Teacher training
• Output: 200 Teacher trained
INTER-MEDIATE OUTCOMES
• Teacher’ knowledge increased?
• Teachers change classroom behavior?
OUTCOMES
• Student learning improved?
• Graduation rates increased?
• Employment rates improved?
![Page 19: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/19.jpg)
How to Go from Program to Results?
Employment
PROGRAM• Inputs: people,
money, supplies
• Activities: Job search assistance
• Output: 600 Unemployed counselled
INTER-MEDIATE OUTCOMES
• Better soft skills for searching?
• Increased search activity?
OUTCOMES
• Higher employment rate?
• Higher job satisfaction?
![Page 20: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/20.jpg)
How to measure impact?
![Page 21: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/21.jpg)
Children’s Learning Outcomes
Time
Intervention: extra tutoring
Impact?
Before-After comparison ok?
Before After
Treatment schools
![Page 22: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/22.jpg)
Time
Intervention
Counterfactual
Actual Impact
=Treatment
-Control
Need a comparison group (“control”) to measure impact
Control needs to be a good proxy for the counterfactual
Before After
Treatment schools
Control schools
Children’s learning
outcomes
![Page 23: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/23.jpg)
Quasi-experimental
methods (require more assumptions)
Randomized Assignment
Difference-in-Differences
Instrumental Variables
Regression Discontinuity Design
Matching
IE Methods
Toolbox
![Page 24: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/24.jpg)
Randomized Assignment(or “randomized control trials; RCTs”)
![Page 25: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/25.jpg)
Why randomize? Non-random program selection leads to
comparisons of apples and oranges
Often impossible to measure impact!
![Page 26: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/26.jpg)
Randomization creates the right comparison group to measure impact
Good so-called
“Internal Validity”
![Page 27: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/27.jpg)
Randomization creates the right comparison group to measure impact
“Internal Validity” still
good
But less “External Validity”
![Page 28: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/28.jpg)
Ethics and randomization:2 scenarios
![Page 29: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/29.jpg)
1: Lottery is fair when demand for program exceeds program resources
Target population (e.g. poor communities or schools, unemployed etc.)
Treatment
Control
![Page 30: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/30.jpg)
2. Lottery is fair when not everyone can be served at the same time
TimeInterventionBefore
evaluation period
After
![Page 31: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/31.jpg)
Keep in Mind
Randomized Assignment
In Randomized Assignment, large enough samples produce two statistically equivalent groups.
We have identified the perfect group match!
![Page 32: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/32.jpg)
• Sometimes we want to compare two or more different kinds of treatments– E.g., compare cost-effectiveness of efdifferent program intensities
such as 10 versus 20 weeks of training, etc.
PS.: Multi-treatment experiments
r1/3 1
T1 C
2/3
T2
1
PopulationSample
![Page 33: Joost de Laat Strategic Impact Evaluation Fund, World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/7434514464951… · • Peru: “One laptop per child” had no sizeable](https://reader035.vdocuments.net/reader035/viewer/2022071216/604726e650001447371692c4/html5/thumbnails/33.jpg)
Thank you!Who wants to learn more about impact
evaluations?