fadeout in human capital interventions: death, miracles and resurrection
TRANSCRIPT
Fadeout in human capital interventions: Death,
miracles and resurrection
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
-0.25
0.00
0.25
0.50
0.75
1.00
Age
Greg J. DuncanDrew BaileyWinnie Yu
School of Education
University of California, Irvine
IQEarnings
Fadeout in human capital interventions: Death, miracles and resurrection
ImpactDeath
IQ impacts in Perry
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
-0.25
0
0.25
0.5
0.75
1
AgeSolid marker denotes p<.05
End of program
Achievement impacts for Head Start 3 year olds
3 4 K 1st 3rd-0.10
0.00
0.10
0.20
0.30
0.40
Eff
ect
siz
e in s
d u
nit
s
Solid marker denotes p<.05
Letter-WordMath
End of program
Cognitive impacts in 67 ECE studies
0 0-1 1-2 2-4 4+0.00
0.10
0.20
0.30
0.40
0.23
0.10 0.090.05 0.06
Eff
ect
siz
e in s
d u
nit
s
Solid marker denotes p<.05
End of pro-gram
0 5 11 17 23 29 35 41 47
-10.0
0.0
10.0
20.0
30.0
Months From Random Assignment
Perc
enta
ge E
mplo
yed F
ull-
Tim
e
Program pe-riod
Employment impacts for the 36-month Canadian Self-Sufficiency Program
ImpactPersistence
IQ impacts in Perry and Abecedarian
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
-0.25
0
0.25
0.5
0.75
1
Age
Perry
Abecedarian
Solid marker denotes p<.05
IQ and Earnings impacts in Perry
3 5 7 9 11 13 15 17 19 21 23 25 27 29
-0.25
0
0.25
0.5
0.75
1
Age
Age 27 earn-ings
Solid marker denotes p<.05
IQ
A “noncognitive”
miracle occurs…
Earnings impacts for the Job Training Partnership Act Program
7-18 19-30-$500
$0
$500
$1,000
$1,500
$2,000
$2,500Adult women Adult men Female youth
Months since random assignmentSolid marker denotes p<.05
Pro-gram
period
Patterns of fade-out
A mess!
• Impacts fade out in some interventions but don’t in seemingly similar interventions
• Sometimes, for the same intervention, some impacts fade out but others emerge decades later
Outline
I. OPTIMAL CONDITIONS FOR IMPACT PERSISTENCE
II. OTHER AVENUES FOR IMPACT PERSISTENCE
III. HOW TO RECONCILE ECE PROGRAM FADEOUT ON IQ WITH IMPACTS ON ADULT OUTCOMES?
Outline
I. OPTIMAL CONDITIONS FOR IMPACT PERSISTENCE
II. OTHER AVENUES FOR IMPACT PERSISTENCE
III. HOW TO RECONCILE ECE PROGRAM FADEOUT ON IQ WITH IMPACTS ON ADULT OUTCOMES?
What conditions lead to impact persistence?
• the “right kinds” of skills or capacities, or
• the “right kinds” of environments
When interventions change:
What conditions lead to impact persistence?
• the “right kinds” of skills or capacities, or
• the “right kinds” of environments
When interventions change:
“Right kinds” of skills:
• Skills or behaviors fundamental for success in adulthood or for childhood attainments…
• that are malleable…
• and would not develop eventually in counterfactual conditions
“Right kinds” of environments:
• Malleable features of environments that are fundamental for promoting the “right kinds” of skills and behaviors
Fundamental and malleable skills
Fundamental Peripheral
More mallea
ble
Lessmallea
ble
• Conscientiousness (grit)
• g (IQ)
Who cares?
• Teaching to the test SAT test prep Flash cards• FAFSA rule knowledge?...
Fundamental AND malleable skills?
• Math• Literacy
• Implicit theories (Dweck)
• Self-concept (Cohen) • Academic motivation
• Executive function• Emotional self-
regulation
• Background knowledge
• Fixed vs. malleable intelligence (Dweck)
• Self-worth (Cohen et al.)
• Math: number line, fractions, algebra
• Literacy• Background
knowledge• Executive function• Prosocial behaviors
“Right kinds” of skills:
Skills or behaviors fundamental for success in adulthood or for childhood attainments…
that are malleable…
and would not develop eventually in counterfactual conditions
Fundamental AND malleable skills?
• Math• Literacy
• Implicit theories (Dweck)
• Self-concept (Cohen) • Academic motivation
• Executive function• Emotional self-
regulation
• Background knowledge
• Fixed vs. malleable intelligence (Dweck)
• Self-worth (Cohen et al.)
• Math: number line, fractions, algebra
• Literacy• Background
knowledge• Executive function• Prosocial behaviors
Which would develop
eventually and therefore generate impact
fadeout?
Pace of development in counterfactual conditionsNull/slow => no
fadeout?Eventually =>
fadeout?• Math• Literacy
• Implicit theories (Dweck)
• Self-concept (Cohen) • Academic motivation
• Executive function• Emotional self-
regulation
• Background knowledge
Pace of development in counterfactual conditionsNull/slow => no
fadeout?Eventually =>
fadeout?• Fractions, algebra• Large vocabulary
• Implicit theories (Dweck)
• Self-concept (Cohen) • Academic motivation
• EF working memory• Emotional self-reg for
some
• Background knowledge
• Counting• Alphabet knowledge
• Implicit theories (Dweck)
• Self-concept (Cohen) • Academic motivation
• EF impulse control• Emotional self-reg for
most• Background knowledge
Examples of impact persistence
• Algebra
– Chicago’s double-dose algebra
• Self-concept
– Value affirmation
Persistent impacts of 9th grade double-dose algebra in Chicago
Source: Cortres et al. (2011)
ImpactB or higher in 9th-grade algebra
+13% *
A in 9th-grade algebra ns
Passed geometry in 10th grade +12% *Grade 11 math scores +.24 sd *
Graduated within 5 years +12% *Enrolled in any college +11% *
Cohen values affirmation impacts on low-GPA Black students
Prein-terven-
tion
Term 1 Term 2 Term 3 Term 4 Term 1 Term 2 Term 3 Term 4
-0.25
0.00
0.25
0.50
0.75
1.00
Me
an
GPA
in
Co
re C
ou
rse
s
Year 1 Year 2
Program period
Cohen caveats
• No impacts on higher-achieving Blacks and whites
• Some replication attempts show no consistent impacts (Dee, 2014)
Conditions where “eventual” development in counterfactual may
lead to fadeout
– Impulse control by age
– Did the Canadian SSP accelerate return to labor force that would have happened anyway?
Distraction time by age
3 4 5 6 7 8 9 100
100
200
300
400
500
600
700
Age
Tim
e t
o s
olv
e c
on
-fl
ict
in m
s
Adult lev-els
Posner and Rothbart (2007)
-12 -6 1 7 13 19 25 31 37 43 49
-10
0
10
20
30
Months From Random Assignment
Perc
enta
ge E
mplo
yed F
ull-
Tim
e
Program pe-riod
Did Canadian SSP speed up employment that would have occurred anyway?
-12 -6 1 7 13 19 25 31 37 43 49
0
10
20
30
40
50
Months From Random Assignment
Perc
enta
ge E
mplo
yed F
ull-
Tim
e
Program pe-riod
Con-trol
group
Did Canadian SSP speed up employment that would have occurred anyway?
-12 -6 1 7 13 19 25 31 37 43 49
0
10
20
30
40
50
Months From Random Assignment
Perc
enta
ge E
mplo
yed F
ull-
Tim
e
Program pe-riod
Con-trol
group
Treat-ment group
Did Canadian SSP speed up employment that would have occurred anyway?
What conditions lead to impact persistence?
• the “right kinds” of skills or capacities, or
• the “right kinds” of environments
When interventions change:
“Right kinds” of environments:
• Malleable features of environments that are fundamental for promoting the “right kinds” of skills and behaviors
School quality can be a fundamental environmental feature
• Winning the lottery to enter one of NYC’s small high schools of choice (Untermann et al., 2014)
Outcomes SSC Control group Effect
Graduation
Graduated from high school 71.6 62.2 9.4 **
Regents diploma granted 50.2 43.5 6.7 **
Advanced Regents diploma granted
8.2 7.3 0.9
College readiness
Passed English Regents Exam at 75+
42.1 35.8 6.3 **
Passed Math Regents Exam at 75+
25.1 24.5 0.5
Post-secondary enrollment
Enrolled in post-secondary education
49.0 40.7 8.4 **
Winning the lottery to enter a NYC Small High School of Choice (n=14,608)
Is neighborhood quality a fundamental environmental feature?
• Moving to Opportunity suggests not for many outcomes in the US
MTO: Huge Differences in Neighborhood Poverty (Duration-Weighted)
01
23
45
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
Con Compliers (Exp) Exp Compliers
De
nsity
Neighborhood Poverty Rate
Experimental Compliers vs Control Compliers
38
Outcome
Interventions
ImpactsBaseline Ages 0 to 5
Reading Assessment ns
Math Assessment ns
Baseline Ages 6 to 11
Reading Assessment ns
Math Assessment ns
Took SAT/ACT? ns
No Impacts on School Achievement
39
What else can sustain impacts?
When interventions:
• (from before) boost the “right kinds” of skills or environment
• are supported by post-TX sustaining environment
• lead to foot-in-the-door access to sustaining environments
• are sufficiently intensive to change foundational skills for children with bad counterfactual conditions
• treat enough children to generate positive peer effects
What can sustain impacts?
When interventions:
• (from before) boost the “right kinds” of skills or environment
• are supported by post-TX sustaining environment
• lead to foot-in-the-door access to sustaining environments
• are sufficiently intensive to change foundational skills for children with bad counterfactual conditions
• treat enough children to generate positive peer effects
Building Blocks and sustaining environments
K-1 TX follow-through
No K-1 TX follow-through
High math in K-1
Low math in K-1
End of pre-K TX
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
0.17
0.19
0.66
Impact on math in sd units
Building Blocks and sustaining environments
K-1 TX follow-through
No K-1 TX follow-through
High math in K-1
Low math in K-1
End of pre-K TX
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
0.17
0.15
0.19
0.66
Impact on math in sd units
nsSimilar results for low vs. high math home environments
Building Blocks and sustaining environments
K-1 TX follow-through
No K-1 TX follow-through
High math in K-1
Low math in K-1
End of pre-K TX
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
0.32
0.17
0.15
0.19
0.66
Impact on math in sd units
ns
p=.07
Sustain environments
Building Blocks suggest that environmental supports must be tailored explicitly to the nature of the prior treatment
What can sustain impacts?
When interventions:
• (from before) boost the “right kinds” of skills or environment
• are supported by post-TX sustaining environment
• lead to foot-in-the-door access to sustaining environments
• are sufficiently intensive to change foundational skills for children with bad counterfactual conditions
• treat enough children to generate positive peer effects
Foot-in-the-door examples?
• SAT prep may affect college quality, which is known to have a positive impact on earnings
• Can FAFSA knowledge lead to college and later success?
• Was some of Abecedarian’s long-run success caused by lower rates of special ed and grade retention?
• Can pro-social behavioral interventions reduce or delay first arrests?
Foot-in-the-door
Foot-in-the-door links to the emerging literature on developmental cascades (Dodge et al. 2008)
• Sequence of positive or negative conditions that cumulate to good or bad outcomes
• Since <100% probabilities multiply, relying on cascades seems like a risky intervention strategy
What can sustain impacts?
When interventions:
• (from before) boost the “right kinds” of skills or environment
• are supported by post-TX sustaining environment
• lead to foot-in-the-door access to sustaining environments
• are sufficiently intensive to change foundational skills for children with difficult counterfactual conditions
• treat enough children to generate positive peer effects
IQ impacts in Perry and Abecedarian
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
-0.25
0
0.25
0.5
0.75
1
1.25
Age
Perry
Abecedarian
Perry vs. Abecedarian
• 1 or 2 years• Part-day ECE +
weekly home visits• Urban setting
• 5 years• Year-round full-day
ECE• Cumulative
curriculum• Rural setting
Perry Abecedarian
• African American children with low tested IQs
• High risk/low SES families
What can sustain impacts?
When interventions:
• (from before) boost the “right kinds” of skills or environment
• are supported by post-TX sustaining environment
• lead to foot-in-the-door access to sustaining environments
• are sufficiently intensive to change foundational skills for children with bad counterfactual conditions
• treat enough children to generate positive peer effects
Generating peer effects
• Measles vaccination!• Deworming treatments in
Kenya generated large benefits for untreated children in treated schools (Miguel and Kremer, 2004)
• County spending on preschool?
More county spending on ECE boosts grades 3-5 achievement
Ladd, Dodge, & Muschkin (2014, JPAM), Muschkin, Dodge, & Ladd (in press, EEPA), Dodge, Ladd, Muschkin, & Bai (under review)
Grade 3 Grade 4 Grade 50
1
2
3
4
5
6More At Four Reading Scores
More At Four Math Scores
Month
s of
Learn
ing
Gain
ed
Outline
I. OPTIMAL CONDITIONS FOR IMPACT PERSISTENCE
II. OTHER AVENUES FOR IMPACT PERSISTENCE
III. HOW TO RECONCILE ECE PROGRAM FADEOUT ON IQ WITH IMPACTS ON ADULT OUTCOMES?
Death
Resurrection
4 5 6 7 8 9 10111213141516171819202122232425262728
-0.25
0.00
0.25
0.50
0.75
1.00
.44 sd
.87 sd
Age
IQ
Earnings
Perry’s IQ Swan Dive and Earnings Resurrection
A miracle
occurs…
Perry and Abecedarian both affected adult earnings
Do they have common (statistical) mediators?
Are they “cognitive” or “non-cognitive”?
ECE treatment
MediatorsAdult
earnings
Perry’s IQ Effects by Age
4 5 6 7 8 9 1011121314151617181920
-0.25
0.00
0.25
0.50
0.75
1.00
Age
Eff
ect
Siz
e in s
d
unit
s
IQ
Source: Schweinhart et al., 2005; Effect sizes >.30 are p<.05, one-tailed test
Perry’s IQ and Achievement Effects by Age
4 5 6 7 8 9 1011121314151617181920
-0.25
0.00
0.25
0.50
0.75
1.00
Age
Eff
ect
Siz
e in s
d
unit
s
IQLan-
guage
Math
Reading Adult liter-acy
Source: Schweinhart et al., 2005; Effect sizes >.30 are p<.05, one-tailed test
Perry’s Noncognitive Effects Ages 6-9
Source: Pinto, based on Heckman et al. (2014)
Index Sample item Impact
Academic potential
Creativity .31ns
Academic motivation
Alert and interested; motivated; persists
.37ns
Classroom conduct
Disobedient; impulsive; blames others
.40*
Personal behavior Absences; swears .36ns
Teacher dependence
Seeks constant reassurance .03ns
Emotional state Depressed; withdrawn .29ns
Emotional adjustment
Trust; level of emotional adjustment
.30ns
Perry (statistical) mediators
• Perry generated a raft of potential mediational effects
– achievement, but not IQ
– a number of potentially important “noncognitive” domains
Abecedarian’s IQ and Achievement Effects by Age
4 5 6 7 8 9 101112131415161718192021
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Age
Eff
ect
Siz
e i
n s
d
un
its
IQ
Source: Campbell et al., 2001; all effect sizes are p<.05.
Abecedarian’s IQ and Achievement Effects by Age
4 5 6 7 8 9 101112131415161718192021
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Age
Eff
ect
Siz
e i
n s
d
un
its
IQMath
Reading
Source: Campbell et al., 2001; all effect sizes are p<.05.
Abecedarian’s Cognitive and Noncognitive Effects by Age
4 5 6 7 8 9 101112131415161718192021
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Age
Eff
ect
Siz
e i
n s
d
un
its
IQMath
Reading
Cognitive compe-tence
Global self-worth
Social compe-tence
Source: Campbell et al. (2001) and Campbell et al. (2002)
Abecedarian (statistical) mediators
• Abecedarian generated a number of potential mediational effects
– achievement and IQ
– but not a limited number of “noncognitive” measures
Two other relevant studies
• Both the Chicago Parent-Child program (Reynolds et al.) and kindergarten class quality (Chetty) affected adult earnings
– CPC affected reading and math achievement more than its “noncognitive” measures
– Chetty et al. found more impact on a noncognitive index than on achievement
Bottom lines on impact resurrection
An even bigger mess!
• Q: haven’t developmental psychologists invented a word for a big mess like this?• A: YES – “equifinality,” when many roads lead to the same outcomes
Summary
• Impact persistence requires treating fundamental and malleable skills that would not develop eventually in counterfactual conditions
• Other avenues are possible– Foot-in-the-door cascades from
peripheral skills are possible but risky– Environmental or intensive individual
interventions may work but are expensive
– Targeting children in the worst counterfactual conditions may be the best strategy
Bottom linesResearch priorities
• Long-run follow-ups, perhaps using administrative data
• Design better post-TX sustaining environment
• For interventions involving implicit theories, self-concept and motivation, we need more independent replications and longer-run follow-ups
That’s it!
Fundamental and malleable skills
Fundamental Peripheral
More mallea
ble
Lessmallea
ble
• Conscientiousness (grit)
• g (IQ)
Who cares?
• Teaching to the test SAT test prep Flash cards• FAFSA rule knowledge?...
Pace of development in counterfactual conditionsNull/slow => no
fadeout?Eventually =>
fadeout?• Fractions, algebra• (much) Vocabulary
• Implicit theories (Dweck)
• Self-concept (Cohen) • Academic motivation
• EF working memory• Emotional self-reg for
some
• Background knowledge
• Counting• Alphabet knowledge
• Implicit theories (Dweck)
• Self-concept (Cohen) • Academic motivation
• EF impulse control• Emotional self-reg for
most• Background knowledge