impact evaluations in education - world bank
Post on 26-Dec-2021
1 Views
Preview:
TRANSCRIPT
EDUCATION STAFF DEVELOPMENT PROGRAM
Impact evaluations in education
Evaluating effectiveness in promoting Learning for All
Deon Filmer, Lead Economist
AFTHD
The World Bank
HD Learning Week 2013
EDUCATION STAFF DEVELOPMENT PROGRAM
• Education is well suited to impact evaluation
– Intervention can be applied at school, classroom, or student level
– Clear, easy-to-grasp outcome measures
– Contrast with trade or macro policy
• Result
– Pioneering IE work among academics in education
– Most active World Bank sector for IEs
EDUCATION STAFF DEVELOPMENT PROGRAM
HOW DO WE KNOW WHICH INTERVENTIONS ARE INCREASING ENROLMENT AND COMPLETION?
EDUCATION STAFF DEVELOPMENT PROGRAM
Inequalities related to poverty: Grade 6 completion of 15-19 year olds in the richest and poorest quintiles.
Poorest quintile East Asia/Pacific Europe/Central Asia Latin America/Caribbean Middle East/North Africa South Asia Sub-Saharan Africa
Within-country inequalities are as big—if not bigger—than cross-country inequalities
Source: Filmer (2010)
EDUCATION STAFF DEVELOPMENT PROGRAM
Why are participation and completion low?
• Costs of schooling
– Poverty and cash/credit constraints
– Opportunity costs
– Exclusion
• Benefits of schooling
– Quality
– Perceived returns
EDUCATION STAFF DEVELOPMENT PROGRAM
Example #1:
Progresa/Oportunidades CCT in Mexico – RCT
How much do conditional cash transfers increase enrolment?
EDUCATION STAFF DEVELOPMENT PROGRAM
Progresa/Oportunidades CCT program in Mexico
• Large-scale program, with one of first major rigorous impact evaluations
• Multisectoral
– Cash to families based on both schooling and health conditions
• Carefully designed for rigorous impact evaluation:
– Phased in with a randomized control trial (RCT) design
– Can compare outcomes in villages with the program to villages without the program
EDUCATION STAFF DEVELOPMENT PROGRAM
• Small impact at primary grades– Baseline enrollment grades 0-5: 94%
– Impact on enrollment: 1.9
• Large impact at transition to secondary school– Baseline enrollment grade 6: 45%
– Impact on subsequent enrollment: 8.7
• Small impact at secondary level:– Baseline enrollment grades 7-9: 43%
– Impact on subsequent enrollment: 0.6
Source: Schultz (2004)
Impact evaluation evidence from a CCT program: Mexico
EDUCATION STAFF DEVELOPMENT PROGRAM
Example #2:
Cambodia scholarship program – Discontinuity design
How effective are poverty-targeted scholarships at increasing enrolment?
EDUCATION STAFF DEVELOPMENT PROGRAM
Cambodian Secondary School Scholarship Program background
• Targeted 100 lower secondary schools
• Payments conditioned on school attendance and progress
• Implementation design lends itself to rigorous impact evaluation:– Students in feeder schools ranked by a “dropout-
risk” score
– Secondary school applicants offered $60 or $45 scholarship (or none), depending on risk
– Enabled RDD design
EDUCATION STAFF DEVELOPMENT PROGRAM
Results from first follow-up: Impact of scholarships on attendance and enrollment
• Attendance from school visits • Enrollment from household survey
0
0.2
0.4
0.6
0.8
1
-25 -15 -5 5 15 25
Relative ranking
Pro
ba
bili
ty
Recipients Non-recipients
0
0.2
0.4
0.6
0.8
1
-25 -15 -5 5 15 25
Relative ranking
Pro
ba
bili
ty
Recipients Non-recipients
Estimate of impact
Estimate of impact
Large positive impact of program on attendance and enrollment (about 25 percentage point increase)
Source: Filmer and Schady (2009)
EDUCATION STAFF DEVELOPMENT PROGRAM
0
0.2
0.4
0.6
0.8
1
-25 -15 -5 5 15 25
Relative ranking
Pro
ba
bili
ty
Recipients Non-recipients
Comparing program alternatives: Impact of different scholarships amounts
• No scholarship versus $45 • $60 versus $45 scholarship
0
0.2
0.4
0.6
0.8
1
-25 -15 -5 5 15 25
Relative ranking
Pro
ba
bili
ty
Recipients Non-recipients
Estimate of impact
Estimate of impact
First $45 is much more cost effective (almost nodiscernable additional impact of $60 over $45)
EDUCATION STAFF DEVELOPMENT PROGRAM
Results from second follow-up:Girls
Enro
llme
nt
Impact estimate
Non-recipient mean
Boys
Enro
llme
nt
Impact estimate
Non-recipient mean
Sizeable impacts beyond the scholarship period itself
EDUCATION STAFF DEVELOPMENT PROGRAM
Accumulation of IE evidence:Summary of CCT/Scholarship programs
Country Age/Grade/Gender Baseline enrollment
Impact
Chile Ages 6-15 60.7 7.5***
Colombia Ages 8-13Ages 14-17
91.763.2
2.1**5.6***
Ecuador Ages 6-17 75.2 10.3**
Honduras Ages 6-13 66.4 3.3***
Jamaica Ages 7-17 18 days out of 20
0.5**
Mexico Grades 0-5Grade 6
Grades 7-9
94.045.042.5
1.98.7***
0.6
Nicaragua (1) Ages 7-15 90.5 6.6***
Nicaragua (2) Ages 7-13 72.0 12.8***
Bangladesh Ages 11-18 (Girls) 44.1 12.0**
Cambodia (1) Grades 7-9 (Girls) 65.0 31.3***
Cambodia (2) Grades 7-9 65.0 21.4***
Pakistan Ages 10-14 (Girls) 29.0 11.1***
Turkey Primary schoolSecondary school
87.939.2
-3.0*5.2
Source: Fiszbein and Schady (2009)
• Accumulation of evidence allows us to recognize patterns
• Impacts tend to …
– Be larger when baseline enrollments are lower
– Be larger at transition points
– Be larger for poorer families
EDUCATION STAFF DEVELOPMENT PROGRAM
IMPACT EVALUATIONS FOCUSED ON HOW TO IMPROVE QUALITY AND LEARNING
EDUCATION STAFF DEVELOPMENT PROGRAM
Improving learning outcomes through both demand- and supply-side programs
• Increasing the benefits of schooling through:– Targeted scholarship programs
– Increases in teacher pay
– Provision of information to parents
Increasing use of rigorously designed studies to evaluate impact on learning; e.g., these RCT examples:Cambodia targeted primary-school scholarships
India learning-based teacher bonuses
Pakistan school report cards for parents
EDUCATION STAFF DEVELOPMENT PROGRAM
Example # 1
Primary-school scholarships in Cambodia: RCT
What type of scholarship targeting best promotes learning by recipients?
EDUCATION STAFF DEVELOPMENT PROGRAM
Primary Scholarships Program
• Built on secondary school scholarships program
• 209 schools in 3 Provinces
• Two forms of targeting within schools:
– Poverty: select the poorest applicants (similar to CESSP)
– Merit: select the applicants with the best test scores
• Focus on learning
EDUCATION STAFF DEVELOPMENT PROGRAM
Evaluation design
• Schools randomly assigned to
– Treatment (poverty or merit) cohort of students that began receiving scholarships in 2008/2009 school year
– Control same cohort did not receive scholarships
– (In the subsequent school year, the next cohort in both sets of schools began receiving scholarships)
Randomization means that we can easily compare outcomes between groups and interpret difference as impact
EDUCATION STAFF DEVELOPMENT PROGRAM
Impact results of primary scholarships (1)
Impact on … Poverty targeted scholarships
Merit targeted scholarships
… probability of completing grade 6
0.170 0.120
… years of schooling achieved
0.332 0.182
Primary school scholarships increased schooling attainment
Source: Barrera-Osorio and Filmer (2012), “Incentivizing schooling for learning: Evidence from Cambodia on the impact of alternative targeting approaches”
EDUCATION STAFF DEVELOPMENT PROGRAM
(Preliminary) Impact results of primary scholarships (2)
• Impacts on assessments of cognitive skills– Reminder: Applications December 08/January 2009 (Grade 4)
– Follow-up: May-July 2011 (Grade 6)
Math test Digitspan
Merit scholarships +0.170 sd +0.149 sd
Poverty scholarships Not sig. Not sig.
“Merit” recipients performed better on tests, “poverty recipients” did not
EDUCATION STAFF DEVELOPMENT PROGRAM
Example #2
Teacher bonus pay based on student learning in India: RCT
Do learning-based teacher bonuses improve student learning?
EDUCATION STAFF DEVELOPMENT PROGRAM
APRest teacher incentives experiment: Context and rationale
• Context: poor service delivery quality and learning outcomes in AP schools
• Opportunity: AP government willing to experiment with innovative potential solutions
• Theory of change: Teachers motivated to work harder and focus on student learning results
EDUCATION STAFF DEVELOPMENT PROGRAM
India AP: Summary of results
• Incentive schools performed significantly better (0.22 SD)– Improvements for all grades, districts, baseline scores– Improvements on mechanical and conceptual
components of test, and also on non-incentive subjects
• Individual incentives had more impact than group incentives after first year
• Channels: – No reduction in teacher absence, but:– Higher levels of teaching activity among teachers at
school
EDUCATION STAFF DEVELOPMENT PROGRAM
Example #3Information for accountability through report cards in Pakistan: RCT
Does providing information on student and school performance to parents improve student learning?
EDUCATION STAFF DEVELOPMENT PROGRAM
Information for accountability: Pakistan report cards
• Context: poor and varied learning results, in an active education market
• Intervention: provide report cards to parents giving information on child’s and school’s performance
• Theory of change: competitive pressure from informed parents can lead to improved quality and/or reduced tuitions in private schools
EDUCATION STAFF DEVELOPMENT PROGRAM
Pakistan: Report Card DesignChild information Village Schools Information
Three subjects (Math; Urdu; English)
- Child score and quintile
- Child’s School score and quintile
- Child’s village score and quintile
Quintile described as “needing a lot of work” to “very good”
For all Primary schools in villages:
- School name
- Number of tested children
- School scores and quintiles in all 3 subjects
Source: Andrabi, Das, and Khwaja, “Report Cards: The Impact of Providing School and Child Test-scores on Educational Markets” (2009).
EDUCATION STAFF DEVELOPMENT PROGRAM
Pakistan: Summary of Results
• Initially low-quality private schools:– Increase in learning outcomes (by 0.15 SD)
• Initially high-quality private schools:– Decrease in school fees (by 21 percent)
• Public schools:– Increase in learning outcomes (by 0.10 SD)
• Intermediate impacts:– more likely to have textbooks – devoted around 30 more minutes per day to teaching and
learning activities
• Importance of thinking about education market
EDUCATION STAFF DEVELOPMENT PROGRAM
Some other areas for IE in education
• School inputs (textbooks, uniforms)
• Contract teachers
• Teacher training
top related