Upload File

evaluating public programs with close …crwalters/papers/kline_walters.pdfevaluating public...

  • Home
  • Documents
  • EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R
54
EVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R. Walters We use data from the Head Start Impact Study (HSIS) to evaluate the cost- effectiveness of Head Start, the largest early childhood education program in the United States. Head Start draws roughly a third of its participants from competing preschool programs, many of which receive public funds. We show that accounting for the fiscal impacts of such program substitution pushes estimates of Head Start’s benefit-cost ratio well above one under a wide range of assumptions on the structure of the market for preschool services and the dollar value of test score gains. To parse the program’s test score im- pacts relative to home care and competing preschools, we selection-correct test scores in each care environment using excluded interactions between experi- mental assignments and household characteristics. We find that Head Start generates larger test score gains for children who would not otherwise attend preschool and for children who are less likely to participate in the program. JEL Codes: I20, J24, H52, C30. I. Introduction Many government programs provide services that can be ob- tained in roughly comparable form via markets or through other public organizations. The presence of close program substitutes com- plicates the task of program evaluation by generating ambiguity re- garding which causal estimands are of interest. Standard intent-to- treat impacts from experimental demonstrations can yield unduly negative assessments of program effectiveness if most participants would receive similar services in the absence of an intervention (Heckman et al. 2000). On the other hand, experiments that artifi- cially restrict substitution alternatives may yield impacts that are not representative of the costs and benefits of actual policy changes. We thank Danny Yagan, James Heckman, Nathan Hendren, Magne Mogstad, Jesse Rothstein, Melissa Tartari, and seminar participants at UC Berkeley, the University of Chicago, Arizona State University, Harvard University, Stanford University, the NBER Public Economics Spring 2015 Meetings, Columbia University, UC San Diego, Princeton University, the NBER Labor Studies Summer Institute 2015 Meetings, Uppsala University, MIT, Vanderbilt University, and four anonymous referees for helpful comments. Raffaele Saggio provided outstanding research assistance. We also thank Research Connections for providing the data. Generous funding support for this project was provided by the Berkeley Center for Equitable Growth. ! The Author(s) 2016. Published by Oxford University Press, on behalf of President and Fellows of Harvard College. All rights reserved. For Permissions, please email: [email protected] The Quarterly Journal of Economics (2016), 1795–1848. doi:10.1093/qje/qjw027. Advance Access publication on July 11, 2016. 1795 at University of California, Berkeley on October 28, 2016 http://qje.oxfordjournals.org/ Downloaded from

Upload: others

Post on 20-May-2020

6 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

EVALUATING PUBLIC PROGRAMS WITH CLOSESUBSTITUTES THE CASE OF HEAD START

Patrick Kline and

Christopher R Walters

We use data from the Head Start Impact Study (HSIS) to evaluate the cost-effectiveness of Head Start the largest early childhood education program inthe United States Head Start draws roughly a third of its participants fromcompeting preschool programs many of which receive public funds We showthat accounting for the fiscal impacts of such program substitution pushesestimates of Head Startrsquos benefit-cost ratio well above one under a widerange of assumptions on the structure of the market for preschool servicesand the dollar value of test score gains To parse the programrsquos test score im-pacts relative to home care and competing preschools we selection-correct testscores in each care environment using excluded interactions between experi-mental assignments and household characteristics We find that Head Startgenerates larger test score gains for children who would not otherwise attendpreschool and for children who are less likely to participate in the programJEL Codes I20 J24 H52 C30

I Introduction

Many government programs provide services that can be ob-tained in roughly comparable form via markets or through otherpublic organizations The presence of close program substitutes com-plicates the task of program evaluation by generating ambiguity re-garding which causal estimands are of interest Standard intent-to-treat impacts from experimental demonstrations can yield undulynegative assessments of program effectiveness if most participantswould receive similar services in the absence of an intervention(Heckman et al 2000) On the other hand experiments that artifi-cially restrict substitution alternatives may yield impacts that arenot representative of the costs and benefits of actual policy changes

We thank Danny Yagan James Heckman Nathan Hendren Magne MogstadJesse Rothstein Melissa Tartari and seminar participants at UC Berkeley theUniversity of Chicago Arizona State University Harvard University StanfordUniversity the NBER Public Economics Spring 2015 Meetings ColumbiaUniversity UC San Diego Princeton University the NBER Labor StudiesSummer Institute 2015 Meetings Uppsala University MIT VanderbiltUniversity and four anonymous referees for helpful comments Raffaele Saggioprovided outstanding research assistance We also thank Research Connectionsfor providing the data Generous funding support for this project was provided bythe Berkeley Center for Equitable Growth

The Author(s) 2016 Published by Oxford University Press on behalf of Presidentand Fellows of Harvard College All rights reserved For Permissions please emailjournalspermissionsoupcomThe Quarterly Journal of Economics (2016) 1795ndash1848 doi101093qjeqjw027Advance Access publication on July 11 2016

1795

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

This article assesses the cost-effectiveness of Head Startmdashaprominent public education program for which close public andprivate substitutes are widely available Head Start is the largestearly childhood education program in the United StatesLaunched in 1965 as part of President Lyndon Johnsonrsquos waron poverty the program has evolved from an eight-weeksummer program into a year-round program that offers educa-tion health and nutrition services to disadvantaged children andtheir families By 2013 Head Start enrolled about 900000 three-and four-year-old children at a cost of $76 billion (US DHHS2013)

Views on the effectiveness of Head Start vary widely (Ludwigand Phillips 2007 and Gibbs Ludwig and Miller 2011 providereviews) A number of observational studies find substantialshort- and long-run impacts on test scores and other outcomes(Currie and Thomas 1995 Garces Thomas and Currie 2002Ludwig and Miller 2007 Deming 2009 Carneiro and Ginja2014) By contrast a recent randomized evaluationmdashthe HeadStart Impact Study (HSIS)mdashfinds small impacts on test scoresthat fade out quickly (Puma et al 2010 Puma Bell and Heid2012) These results have generally been interpreted as evidencethat Head Start is ineffective and in need of reform (Barnett 2011Klein 2011)

Two observations suggest that such conclusions are prema-ture First research on early childhood interventions finds long-run gains in adult outcomes despite short-run fade-out of testscore impacts (Heckman et al 2010 Heckman Pinto andSavelyev 2013 Chetty et al 2011 Chetty Friedman andRockoff 2014b) Second roughly one third of the HSIS controlgroup participated in alternate forms of preschool This suggeststhat the HSIS may have shifted many students between differentsorts of preschools without altering their exposure to preschoolservices The aim of this article is to clarify how the presence ofsubstitute preschools affects the interpretation of the HSIS re-sults and the cost-effectiveness of the Head Start program

Our study begins by revisiting the experimental impacts ofthe HSIS on student test scores We replicate the fade-out patternfound in previous work but find that adjusting for experimentalnon compliance leads to imprecise estimates of the effect of HeadStart participation beyond the first year of the experiment As aresult the conclusion of complete effect fade-out is less clear thannaive intent-to-treat estimates suggest Turning to substitution

QUARTERLY JOURNAL OF ECONOMICS1796

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

patterns we find that roughly one third of Head Start lsquolsquocompliersrsquorsquo(Angrist Imbens and Rubin 1996) in the HSIS experiment wouldhave participated in other forms of preschool had they not beenlotteried into the program These alternative preschools drawheavily on public funding which mitigates the net costs to gov-ernment of shifting children from other preschools into HeadStart

These facts motivate a theoretical analysis clarifying whichparameters are (and are not) policy relevant when publicly sub-sidized program substitutes are present We work with a stylizedmodel where test score impacts are valued according to theireffects on childrenrsquos after-tax lifetime earnings We show thatwhen competing preschool programs are not rationed thepolicy-relevant causal parameter governing the benefits of HeadStart expansion is an average effect of Head Start participationrelative to the next best alternative regardless of whether thatalternative is a competing program or home care This parametercoincides with the local average treatment effect (LATE) identi-fied by a randomized experiment with imperfect compliance whenthe experiment contains a representative sample of program com-pliers Hence imperfect compliance and program substitutionoften thought to be confounding limitations of social experimentsturn out to be virtues when the substitution patterns in the ex-periment replicate those found in the broader population

We use this result to derive an estimable benefit-cost ratioassociated with Head Start expansions This ratio scales HeadStartrsquos projected impacts on the after-tax earnings of childrenby its net costs to government inclusive of fiscal externalitiesChief among these externalities is the cost savings that arisewhen Head Start draws children away from competing subsidizedpreschool programs Although such effects are typically ignoredin cost-benefit analyses of Head Start and other similar programs(eg CEA 2015) we find via a calibration exercise that such omis-sions can be quantitatively important Head Start roughly breakseven when the cost savings associated with program substitutionare ignored but yields benefits nearly twice as large as costswhen these savings are incorporated This appears to be arobust findingmdashafter accounting for fiscal externalities HeadStartrsquos benefits exceed its costs whenever short-run test scoreimpacts yield earnings gains within the range found in therecent literature

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1797

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A limitation of our baseline analysis is that it assumeschanges in program scale do not alter the mix of program compli-ers To address this issue we also consider lsquolsquostructural reformsrsquorsquo toHead Start that change the mix of compliers without affectingtest score outcomes Examples of such reforms might include in-creased transportation services marketing efforts or spendingon program features that parents value Households who respondto structural reforms may differ from experimental compliers onunobserved dimensions including their mix of counterfactualprogram choices Assessing these reforms therefore requiresknowledge of parameters not directly identified by the HSIS ex-periment Specifically we show that such reforms require identi-fication of a variant of the marginal treatment effect (MTE)concept of Heckman and Vytlacil (1999)

To assess reforms that attract new children we develop aselection model that parameterizes variation in treatment effectswith respect to counterfactual care alternatives as well as ob-served and unobserved child characteristics We prove that themodel parameters are identified and propose a two-step controlfunction estimator that exploits heterogeneity in the response toHead Start offers across sites and demographic groups to inferrelationships between unobserved factors driving preschool en-rollment and potential outcomes The estimator is shown to passa variety of specification tests and accurately reproduce patternsof treatment effect heterogeneity found in the experiment Themodel estimates indicate that Head Start has large positiveshort-run effects on the test scores of children who would haveotherwise been cared for at home and insignificant effects on chil-dren who would otherwise attend other preschoolsmdasha finding cor-roborated by Feller et al (forthcoming) who reach similarconclusions using principal stratification methods (Frangakisand Rubin 2002) Our estimates also reveal a lsquolsquoreverse Royrsquorsquo pat-tern of selection whereby children with unobserved characteris-tics that make them less likely to enroll in Head Start experiencelarger test score gains

We conclude with an assessment of prospects for increasingHead Startrsquos rate of return via outreach to new populations Ourestimates suggest that expansions of Head Start could boost theprogramrsquos rate of return provided that the proposed technologyfor increasing enrollment (eg improved transportation services)is not too costly We also use our estimated selection model toexamine the robustness of our results to rationing of competing

QUARTERLY JOURNAL OF ECONOMICS1798

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschools Rationing implies that competing subsidized pre-schools do not contract when Head Start expands which shutsdown a form of public savings On the other hand expandingHead Start generates opportunities for new children to fill va-cated seats in substitute programs Our estimates indicate thatthe effect on test scores (and therefore earnings) of moving chil-dren from home care to competing preschools is substantial lead-ing us to conclude that rationing is in fact likely to increase thefavorable estimated rates of return found in our baselineanalysis

The rest of the article is structured as follows Section II pro-vides background on Head Start Section III describes the HSISdata and basic experimental impacts Section IV presents evi-dence on substitution patterns Section V introduces a theoreticalframework for assessing public programs with close substitutesSection VI provides a cost-benefit analysis of Head Start SectionVII develops our econometric selection model and discusses iden-tification and estimation Section VIII reports estimates of themodel Section IX simulates the effects of structural program re-forms Section X concludes

II Background on Head Start

Head Start provides preschool for disadvantaged children inthe United States The program is funded by federal grantsawarded to local public or private organizations Grantees arerequired to match at least 20 of their Head Start awards fromother sources and must meet a set of program-wide performancecriteria Eligibility for Head Start is generally limited to childrenfrom households below the federal poverty line though familiesabove this threshold may be eligible if they meet other criteriasuch as participation in the Temporary Aid for Needy Families(TANF) program Up to 10 of a Head Start centerrsquos enrollmentcan also come from higher-income families The program is freeHead Start grantees are prohibited from charging families feesfor services (US DHHS 2014) It is also oversubscribed in 200285 of Head Start participants attended programs with moreapplicants than available seats (Puma et al 2010)

Head Start is not the only form of subsidized preschool avail-able to poor families Preschool participation rates for disadvan-taged children have risen over time as cities and states expanded

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1799

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

their public preschool offerings (Cascio and Schanzenbach 2013)Moreover the Child Care Development Fund program providesblock grants that finance childcare subsidies for low-income fam-ilies often in the form of child care vouchers that can be used forcenter-based preschool (US DHHS 2012) Most states also useTANF funds to finance additional child care subsidies(Schumacher Greenberg and Duffy 2001) Because Head Startservices are provided by local organizations who themselves mustraise outside funds it is unclear to what extent Head Start andother public preschool programs actually differ in their educationtechnology

A large nonexperimental literature suggests that Head Startproduced large short- and long-run benefits for early cohorts ofprogram participants Several studies estimate the effects ofHead Start by comparing program participants to their nonpar-ticipant siblings (Currie and Thomas 1995 Garces Thomas andCurrie 2002 Deming 2009) Results from this research designshow positive short-run effects on test scores and long-run effectson educational attainment earnings and crime Other studiesexploit discontinuities in Head Start program rules to infer pro-gram effects (Ludwig and Miller 2007 Carneiro and Ginja 2014)These studies show longer run improvements in health outcomesand criminal activity

In contrast to these nonexperimental estimates results froma recent randomized controlled trial reveal smaller less persis-tent effects The 1998 Head Start reauthorization bill included acongressional mandate to determine the effects of the programThis mandate resulted in the HSIS an experiment in which morethan 4000 applicants were randomly assigned via lottery toeither a treatment group with access to Head Start or a controlgroup without access in fall 2002 The experimental resultsshowed that a Head Start offer increased measures of cognitiveachievement by roughly 01 standard deviations during pre-school but these gains faded out by kindergarten Moreoverthe experiment showed little evidence of effects on noncognitiveor health outcomes (Puma et al 2010 Puma Bell and Heid2012) These results suggest both smaller short-run effects andfaster fade-out than nonexperimental estimates for earlier co-horts Scholars and policy makers have generally interpretedthe HSIS results as evidence that Head Start is ineffective andin need of reform (Barnett 2011) The experimental results havealso been cited in the popular media to motivate calls for dramatic

QUARTERLY JOURNAL OF ECONOMICS1800

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

restructuring or elimination of the program (Klein 2011 Stossel2014)1

Differences between the HSIS results and the nonexperi-mental literature could be due to changes in program effective-ness over time or to selection bias in nonexperimental siblingcomparisons Another explanation however is that these tworesearch designs identify different parameters Most nonexperi-mental analyses have focused on recovering the effect of HeadStart relative to home care In contrast the HSIS measures theeffect of Head Start relative to a mix of alternative care environ-ments including other preschools

III Data and Experimental Impacts

Before turning to an analysis of program substitution issueswe describe the HSIS data and report experimental impacts ontest scores and program compliance

IIIA Data

Our core analysis sample includes 3571 HSIS applicantswith nonmissing baseline characteristics and spring 2003 testscores Online Appendix A describes construction of this sampleThe outcome of interest is a summary index of cognitive testscores that averages Woodcock Johnson III (WJIII) test scoreswith Peabody Picture and Vocabulary Test (PPVT) scores witheach test normed to have mean 0 and variance 1 in the controlgroup by cohort and year We use WJIII and PPVT scores becausethese are among the most reliable tests in the HSIS data both areavailable in each year of the experiment allowing us to producecomparable estimates over time

1 Subsequent analyses of the HSIS data suggest caveats to this negative in-terpretation but do not overturn the finding of modest mean test score impactsaccompanied by rapid fade-out Gelber and Isen (2013) find persistent effects onparental engagement with children Bitler Domina and Hoynes (2014) find largerexperimental impacts at low quantiles of the test score distribution These quantiletreatment effects fade out by first grade though there is some evidence of persistenteffects at the bottom of the distribution for Spanish speakers Walters (2015) findsevidence of substantial heterogeneity in impacts across experimental sites and in-vestigates the relationship between this heterogeneity and observed program char-acteristics Walters finds smaller effects for Head Start centers that draw morechildren from other preschools rather than home care a finding we explore inmore detail here

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1801

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Table I provides summary statistics for our analysis sampleThe HSIS experiment included two age cohorts 55 of applicantswere randomized at age three and could attend Head Start for upto two years while the remaining 45 were randomized at age fourand could attend for up to one year The demographic informationin Table I shows that the Head Start population is disadvantagedLess than half of Head Start applicants live in two-parent house-holds and the average applicantrsquos household earns about 90 ofthe federal poverty line Column (2) of Table I compares these andother baseline characteristics for the HSIS treatment and controlgroups to check balance in randomization The results here indi-cate that randomization was successful baseline characteristicswere similar for offered and non offered applicants2

Columns (3) through (5) of Table I report summary statisticsfor children attending Head Start other preschool centers andno preschool3 Children in other preschools tend to be less disad-vantaged than children in Head Start or no preschool thoughmost differences between these groups are modest The otherpreschool group has a lower share of high school dropout mothersa higher share of mothers who attended college and higher av-erage household income than the Head Start and no preschoolgroups Children in other preschools outscore the other groups byabout 01 standard deviations on a baseline summary index ofcognitive skills The other preschool group also includes a rela-tively large share of four-year-olds likely reflecting the fact thatalternative preschool options are more widely available for four-year-olds (Cascio and Schanzenbach 2013)

IIIB Experimental Impacts

Table II reports experimental impacts on test scoresColumns (1) (4) and (7) report intent-to-treat impacts of the

2 Random assignment in the HSIS occurred at the Head Start center leveland offer probabilities differed across centers We weight all models by the inverseprobability of a childrsquos assignment calculated as the site-specific fraction of chil-dren assigned to the treatment group

3 Preschool attendance is measured from the HSIS lsquolsquofocal arrangement typersquorsquovariable which combines information from parent interviews and teachercareprovider interviews to construct a summary measure of the childcare setting SeeOnline Appendix A for details

QUARTERLY JOURNAL OF ECONOMICS1802

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

DE

SC

RIP

TIV

ES

TA

TIS

TIC

S

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

Male

04

94

05

05

00

11

05

01

05

06

04

92

(00

19)

Bla

ck03

08

02

98

00

10

03

17

03

53

02

50

(00

10)

His

pan

ic03

76

03

69

00

07

03

80

03

54

03

73

(00

10)

Tee

nm

oth

er01

59

01

74

00

15

01

59

01

69

01

76

(00

14)

Mot

her

marr

ied

04

36

04

48

00

11

04

39

04

20

04

60

(00

17)

Bot

hp

are

nts

inh

ouse

hol

d04

97

04

88

00

09

04

97

04

68

04

99

(00

17)

Mot

her

ish

igh

sch

ool

dro

pou

t03

68

03

97

00

29

03

77

03

22

04

26

(00

17)

Mot

her

att

end

edso

me

coll

ege

02

98

02

81

00

17

02

93

03

42

02

53

(00

16)

Sp

an

ish

spea

ker

02

87

02

73

00

14

02

96

02

74

02

60

(00

11)

Sp

ecia

led

uca

tion

01

36

01

08

00

28

01

34

01

45

00

91

(00

11)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1803

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

(CO

NT

INU

ED)

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

On

lych

ild

01

61

01

39

00

22

01

51

01

90

01

23

(00

12)

Inco

me

(fra

ctio

nof

FP

L)

08

96

08

96

00

00

08

92

09

83

08

51

(00

24)

Age

4co

hor

t04

48

04

51

00

03

04

26

05

67

04

13

(00

12)

Base

lin

esu

mm

ary

ind

ex00

03

00

12

00

09

00

01

01

06

00

40

(00

27)

Urb

an

08

33

08

35

00

02

08

34

08

59

08

19

(00

03)

Cen

ter

pro

vid

estr

an

spor

tati

on06

06

06

04

00

02

05

86

06

14

06

28

(00

05)

Cen

ter

qu

ali

tyin

dex

04

65

04

70

00

05

04

52

04

74

04

88

(00

05)

Joi

nt

p-v

alu

e05

06

N22

56

13

15

35

71

20

43

598

930

Not

es

All

stati

stic

sw

eigh

tby

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tS

tan

dard

erro

rsare

clu

ster

edat

the

cen

ter

level

T

he

tran

spor

tati

onan

dqu

ali

tyin

dex

vari

able

sre

fer

toa

chil

drsquos

Hea

dS

tart

cen

ter

ofra

nd

omass

ign

men

tT

he

qu

ali

tyvari

able

com

bin

esin

form

ati

onon

cen

ter

chara

cter

isti

cs(t

each

eran

dce

nte

rd

irec

tor

edu

cati

onan

dqu

ali

fica

tion

scl

ass

size

)an

dp

ract

ices

(vari

ety

ofli

tera

cyan

dm

ath

act

ivit

ies

hom

evis

itin

g

hea

lth

an

dn

utr

itio

n)

Inco

me

ism

issi

ng

for

19

ofob

serv

ati

ons

Mis

sin

gvalu

esare

excl

ud

edin

stati

stic

sfo

rin

com

eT

he

base

lin

esu

mm

ary

ind

exis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

infa

ll2002

wit

hea

chsc

ore

stan

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tT

he

join

tp

-valu

eis

from

ate

stof

the

hyp

oth

esis

that

all

coef

fici

ents

equ

al

0

QUARTERLY JOURNAL OF ECONOMICS1804

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

EX

PE

RIM

EN

TA

LIM

PA

CT

SO

NT

ES

TS

CO

RE

S

Th

ree-

yea

r-ol

dco

hor

tF

our-

yea

r-ol

dco

hor

tC

ohor

tsp

oole

d

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

Tim

ep

erio

dR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IV

Yea

r1

01

94

06

99

02

78

01

41

06

63

02

13

01

68

06

82

02

47

(00

29)

(00

25)

(00

41)

(00

29)

(00

22)

(00

44)

(00

21)

(00

18)

(00

31)

N19

70

16

01

35

71

Yea

r2

00

87

03

56

02

45

00

15

06

70

00

22

00

46

04

97

00

93

(00

29)

(00

28)

(00

80)

(00

37)

(00

23)

(00

54)

(00

24)

(00

20)

(00

49)

N17

60

14

16

31

76

Yea

r3

00

10

03

65

00

27

00

54

06

66

00

81

00

19

05

00

00

38

(00

31)

(00

28)

(00

85)

(00

40)

(00

25)

(00

60)

(00

25)

(00

20)

(00

50)

N16

59

13

36

29

95

Yea

r4

00

38

03

44

01

10

mdashmdash

(00

34)

(00

29)

(00

98)

N15

99

Not

es

Th

ista

ble

rep

orts

exp

erim

enta

les

tim

ate

sof

the

effe

cts

ofH

ead

Sta

rton

test

scor

es

Th

eou

tcom

eis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

w

ith

each

scor

est

an

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tan

dyea

rC

olu

mn

s(1

)(4

)an

d(7

)re

por

tco

effi

cien

tsfr

omre

gre

ssio

ns

ofte

stsc

ores

onan

ind

icato

rfo

rass

ign

men

tto

Hea

dS

tart

C

olu

mn

s(2

)(5

)an

d(8

)re

por

tco

effi

cien

tsfr

omfi

rst-

stage

regre

ssio

ns

ofH

ead

Sta

rtatt

end

an

ceon

Hea

dS

tart

ass

ign

men

tT

he

att

end

an

cevari

able

isan

ind

icato

req

ual

to1

ifa

chil

datt

end

sH

ead

Sta

rtat

an

yti

me

pri

orto

the

test

C

olu

mn

s(3

)(6

)an

d(9

)re

por

tco

effi

cien

tsfr

omtw

o-st

age

least

squ

are

s(2

SL

S)

mod

els

that

inst

rum

ent

Hea

dS

tart

att

end

an

cew

ith

Hea

dS

tart

ass

ign

men

tA

llm

odel

sw

eigh

tby

the

reci

pro

cal

ofa

chil

drsquos

exp

erim

enta

lass

ign

-m

ent

an

dco

ntr

olfo

rse

x

race

S

pan

ish

lan

gu

age

teen

mot

her

m

oth

errsquos

mari

tal

statu

sp

rese

nce

ofbot

hp

are

nts

inth

eh

ome

fam

ily

size

sp

ecia

led

uca

tion

statu

sin

com

equ

art

ile

du

mm

ies

urb

an

an

da

cubic

pol

yn

omia

lin

base

lin

esc

ore

Mis

sin

gvalu

esfo

rco

vari

ate

sare

set

to0

an

dd

um

mie

sfo

rm

issi

ng

are

incl

ud

ed

Sta

nd

ard

erro

rsare

clu

ster

edby

cen

ter

ofra

nd

omass

ign

men

t

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1805

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Head Start offer separately by year and age cohort To increaseprecision we regression-adjust these treatmentcontrol differ-ences using the baseline characteristics in Table I4 Theintent-to-treat estimates mirror those previously reported inthe literature (eg Puma et al 2010) In the first year of theexperiment children offered Head Start scored higher on thesummary index For example three-year-olds offered HeadStart gained 019 standard deviations in test score outcomesrelative to those denied Head Start The corresponding effectfor four-year-olds is 014 standard deviations However thesegains diminish rapidly the pooled impact falls to a statisticallyinsignificant 002 standard deviations by year 3 Our data in-clude a fourth year of follow-up for the three-year-old cohortHere too the intent-to-treat is small and statistically insignif-icant (0038 standard deviations)

Interpretation of these intent-to-treat impacts is clouded bynoncompliance with random assignment Columns (2) (5) and(8) of Table II report first-stage effects of assignment to HeadStart on the probability of participating in Head Start and col-umns (3) (6) and (9) report instrumental variables (IV) esti-mates which scale the intent-to-treat estimates by the first-stage estimates5 These estimates can be interpreted as local av-erage treatment effects (LATEs) for lsquolsquocompliersrsquorsquomdashchildren whorespond to the Head Start offer by enrolling in Head StartAssignment to Head Start increases the probability of participa-tion by two thirds in the first year after random assignment The

4 The control vector includes sex race assignment cohort teen mothermotherrsquos education motherrsquos marital status presence of both parents an onlychild dummy a Spanish language indicator dummies for quartiles of familyincome and missing income urban status an indicator for whether the HeadStart center provides transportation an index of Head Start center quality and athird-order polynomial in baseline test scores

5 Here we define Head Start participation as enrollment at any time prior tothe test This definition includes attendance at Head Start centers outside the ex-perimental sample An experimental offer may cause some children to switch froman out-of-sample center to an experimental center if the quality of these centersdiffers the exclusion restriction required for our IV approach is violated OnlineAppendix Table AI compares characteristics of centers attended by children in thecontrol group (always takers) to those of the experimental centers to which thesechildren applied These two groups of centers are very similar suggesting thatsubstitution between Head Start centers is unlikely to bias our estimates

QUARTERLY JOURNAL OF ECONOMICS1806

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

corresponding IV estimate implies that Head Start attendanceboosts first-year test scores by 0247 standard deviations

Compliance for the three-year-old cohort falls after the firstyear as members of the control group reapply for Head Startresulting in larger standard errors for estimates in later yearsof the experiment The first stage for three-year-olds falls to 036in the second year whereas the intent-to-treat falls roughly inproportion generating a second-year IV estimate of 0245 for thiscohort Estimates in years 3 and 4 are statistically insignificantand imprecise The fourth-year estimate for the three-year-oldcohort (corresponding to first grade) is 0110 standard deviationswith a standard error of 0098 The corresponding first grade es-timate for four-year-olds is 0081 with a standard error of 0060Notably the 95 confidence intervals for first-grade impacts in-clude effects as large as 02 standard deviations for four-year-oldsand 03 standard deviations for three-year-olds These resultsshow that although the longer run estimates are insignificantthey are also imprecise due to experimental noncomplianceEvidence for fade-out is therefore less definitive than the naiveintent-to-treat estimates suggest This observation helps to rec-oncile the HSIS results with observational studies based on sib-ling comparisons which show effects that partially fade out butare still detectable in elementary school (Currie and Thomas1995 Deming 2009)6

IV Program Substitution

We turn now to documenting program substitution in theHSIS and how it influences our results It is helpful to developsome notation to describe the role of alternative care environ-ments Each Head Start applicant participates in one of three pos-sible treatments Head Start which we label h other center-based

6 One might also be interested in the effects of Head Start on noncognitiveoutcomes which appear to be important mediators of the effects of early childhoodprograms in other contexts (Chetty et al 2011 Heckman Pinto and Savelyev2013) The HSIS includes short-run parent-reported measures of behavior andteacher-reported measures of teacherndashstudent relationships and Head Start ap-pears to have no impact on these outcomes (Puma et al 2010 Walters 2015) TheHSIS noncognitive outcomes differ significantly from those analyzed in previousstudies however and it is unclear whether they capture the same skills

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1807

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 2: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

This article assesses the cost-effectiveness of Head Startmdashaprominent public education program for which close public andprivate substitutes are widely available Head Start is the largestearly childhood education program in the United StatesLaunched in 1965 as part of President Lyndon Johnsonrsquos waron poverty the program has evolved from an eight-weeksummer program into a year-round program that offers educa-tion health and nutrition services to disadvantaged children andtheir families By 2013 Head Start enrolled about 900000 three-and four-year-old children at a cost of $76 billion (US DHHS2013)

Views on the effectiveness of Head Start vary widely (Ludwigand Phillips 2007 and Gibbs Ludwig and Miller 2011 providereviews) A number of observational studies find substantialshort- and long-run impacts on test scores and other outcomes(Currie and Thomas 1995 Garces Thomas and Currie 2002Ludwig and Miller 2007 Deming 2009 Carneiro and Ginja2014) By contrast a recent randomized evaluationmdashthe HeadStart Impact Study (HSIS)mdashfinds small impacts on test scoresthat fade out quickly (Puma et al 2010 Puma Bell and Heid2012) These results have generally been interpreted as evidencethat Head Start is ineffective and in need of reform (Barnett 2011Klein 2011)

Two observations suggest that such conclusions are prema-ture First research on early childhood interventions finds long-run gains in adult outcomes despite short-run fade-out of testscore impacts (Heckman et al 2010 Heckman Pinto andSavelyev 2013 Chetty et al 2011 Chetty Friedman andRockoff 2014b) Second roughly one third of the HSIS controlgroup participated in alternate forms of preschool This suggeststhat the HSIS may have shifted many students between differentsorts of preschools without altering their exposure to preschoolservices The aim of this article is to clarify how the presence ofsubstitute preschools affects the interpretation of the HSIS re-sults and the cost-effectiveness of the Head Start program

Our study begins by revisiting the experimental impacts ofthe HSIS on student test scores We replicate the fade-out patternfound in previous work but find that adjusting for experimentalnon compliance leads to imprecise estimates of the effect of HeadStart participation beyond the first year of the experiment As aresult the conclusion of complete effect fade-out is less clear thannaive intent-to-treat estimates suggest Turning to substitution

QUARTERLY JOURNAL OF ECONOMICS1796

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

patterns we find that roughly one third of Head Start lsquolsquocompliersrsquorsquo(Angrist Imbens and Rubin 1996) in the HSIS experiment wouldhave participated in other forms of preschool had they not beenlotteried into the program These alternative preschools drawheavily on public funding which mitigates the net costs to gov-ernment of shifting children from other preschools into HeadStart

These facts motivate a theoretical analysis clarifying whichparameters are (and are not) policy relevant when publicly sub-sidized program substitutes are present We work with a stylizedmodel where test score impacts are valued according to theireffects on childrenrsquos after-tax lifetime earnings We show thatwhen competing preschool programs are not rationed thepolicy-relevant causal parameter governing the benefits of HeadStart expansion is an average effect of Head Start participationrelative to the next best alternative regardless of whether thatalternative is a competing program or home care This parametercoincides with the local average treatment effect (LATE) identi-fied by a randomized experiment with imperfect compliance whenthe experiment contains a representative sample of program com-pliers Hence imperfect compliance and program substitutionoften thought to be confounding limitations of social experimentsturn out to be virtues when the substitution patterns in the ex-periment replicate those found in the broader population

We use this result to derive an estimable benefit-cost ratioassociated with Head Start expansions This ratio scales HeadStartrsquos projected impacts on the after-tax earnings of childrenby its net costs to government inclusive of fiscal externalitiesChief among these externalities is the cost savings that arisewhen Head Start draws children away from competing subsidizedpreschool programs Although such effects are typically ignoredin cost-benefit analyses of Head Start and other similar programs(eg CEA 2015) we find via a calibration exercise that such omis-sions can be quantitatively important Head Start roughly breakseven when the cost savings associated with program substitutionare ignored but yields benefits nearly twice as large as costswhen these savings are incorporated This appears to be arobust findingmdashafter accounting for fiscal externalities HeadStartrsquos benefits exceed its costs whenever short-run test scoreimpacts yield earnings gains within the range found in therecent literature

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1797

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A limitation of our baseline analysis is that it assumeschanges in program scale do not alter the mix of program compli-ers To address this issue we also consider lsquolsquostructural reformsrsquorsquo toHead Start that change the mix of compliers without affectingtest score outcomes Examples of such reforms might include in-creased transportation services marketing efforts or spendingon program features that parents value Households who respondto structural reforms may differ from experimental compliers onunobserved dimensions including their mix of counterfactualprogram choices Assessing these reforms therefore requiresknowledge of parameters not directly identified by the HSIS ex-periment Specifically we show that such reforms require identi-fication of a variant of the marginal treatment effect (MTE)concept of Heckman and Vytlacil (1999)

To assess reforms that attract new children we develop aselection model that parameterizes variation in treatment effectswith respect to counterfactual care alternatives as well as ob-served and unobserved child characteristics We prove that themodel parameters are identified and propose a two-step controlfunction estimator that exploits heterogeneity in the response toHead Start offers across sites and demographic groups to inferrelationships between unobserved factors driving preschool en-rollment and potential outcomes The estimator is shown to passa variety of specification tests and accurately reproduce patternsof treatment effect heterogeneity found in the experiment Themodel estimates indicate that Head Start has large positiveshort-run effects on the test scores of children who would haveotherwise been cared for at home and insignificant effects on chil-dren who would otherwise attend other preschoolsmdasha finding cor-roborated by Feller et al (forthcoming) who reach similarconclusions using principal stratification methods (Frangakisand Rubin 2002) Our estimates also reveal a lsquolsquoreverse Royrsquorsquo pat-tern of selection whereby children with unobserved characteris-tics that make them less likely to enroll in Head Start experiencelarger test score gains

We conclude with an assessment of prospects for increasingHead Startrsquos rate of return via outreach to new populations Ourestimates suggest that expansions of Head Start could boost theprogramrsquos rate of return provided that the proposed technologyfor increasing enrollment (eg improved transportation services)is not too costly We also use our estimated selection model toexamine the robustness of our results to rationing of competing

QUARTERLY JOURNAL OF ECONOMICS1798

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschools Rationing implies that competing subsidized pre-schools do not contract when Head Start expands which shutsdown a form of public savings On the other hand expandingHead Start generates opportunities for new children to fill va-cated seats in substitute programs Our estimates indicate thatthe effect on test scores (and therefore earnings) of moving chil-dren from home care to competing preschools is substantial lead-ing us to conclude that rationing is in fact likely to increase thefavorable estimated rates of return found in our baselineanalysis

The rest of the article is structured as follows Section II pro-vides background on Head Start Section III describes the HSISdata and basic experimental impacts Section IV presents evi-dence on substitution patterns Section V introduces a theoreticalframework for assessing public programs with close substitutesSection VI provides a cost-benefit analysis of Head Start SectionVII develops our econometric selection model and discusses iden-tification and estimation Section VIII reports estimates of themodel Section IX simulates the effects of structural program re-forms Section X concludes

II Background on Head Start

Head Start provides preschool for disadvantaged children inthe United States The program is funded by federal grantsawarded to local public or private organizations Grantees arerequired to match at least 20 of their Head Start awards fromother sources and must meet a set of program-wide performancecriteria Eligibility for Head Start is generally limited to childrenfrom households below the federal poverty line though familiesabove this threshold may be eligible if they meet other criteriasuch as participation in the Temporary Aid for Needy Families(TANF) program Up to 10 of a Head Start centerrsquos enrollmentcan also come from higher-income families The program is freeHead Start grantees are prohibited from charging families feesfor services (US DHHS 2014) It is also oversubscribed in 200285 of Head Start participants attended programs with moreapplicants than available seats (Puma et al 2010)

Head Start is not the only form of subsidized preschool avail-able to poor families Preschool participation rates for disadvan-taged children have risen over time as cities and states expanded

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1799

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

their public preschool offerings (Cascio and Schanzenbach 2013)Moreover the Child Care Development Fund program providesblock grants that finance childcare subsidies for low-income fam-ilies often in the form of child care vouchers that can be used forcenter-based preschool (US DHHS 2012) Most states also useTANF funds to finance additional child care subsidies(Schumacher Greenberg and Duffy 2001) Because Head Startservices are provided by local organizations who themselves mustraise outside funds it is unclear to what extent Head Start andother public preschool programs actually differ in their educationtechnology

A large nonexperimental literature suggests that Head Startproduced large short- and long-run benefits for early cohorts ofprogram participants Several studies estimate the effects ofHead Start by comparing program participants to their nonpar-ticipant siblings (Currie and Thomas 1995 Garces Thomas andCurrie 2002 Deming 2009) Results from this research designshow positive short-run effects on test scores and long-run effectson educational attainment earnings and crime Other studiesexploit discontinuities in Head Start program rules to infer pro-gram effects (Ludwig and Miller 2007 Carneiro and Ginja 2014)These studies show longer run improvements in health outcomesand criminal activity

In contrast to these nonexperimental estimates results froma recent randomized controlled trial reveal smaller less persis-tent effects The 1998 Head Start reauthorization bill included acongressional mandate to determine the effects of the programThis mandate resulted in the HSIS an experiment in which morethan 4000 applicants were randomly assigned via lottery toeither a treatment group with access to Head Start or a controlgroup without access in fall 2002 The experimental resultsshowed that a Head Start offer increased measures of cognitiveachievement by roughly 01 standard deviations during pre-school but these gains faded out by kindergarten Moreoverthe experiment showed little evidence of effects on noncognitiveor health outcomes (Puma et al 2010 Puma Bell and Heid2012) These results suggest both smaller short-run effects andfaster fade-out than nonexperimental estimates for earlier co-horts Scholars and policy makers have generally interpretedthe HSIS results as evidence that Head Start is ineffective andin need of reform (Barnett 2011) The experimental results havealso been cited in the popular media to motivate calls for dramatic

QUARTERLY JOURNAL OF ECONOMICS1800

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

restructuring or elimination of the program (Klein 2011 Stossel2014)1

Differences between the HSIS results and the nonexperi-mental literature could be due to changes in program effective-ness over time or to selection bias in nonexperimental siblingcomparisons Another explanation however is that these tworesearch designs identify different parameters Most nonexperi-mental analyses have focused on recovering the effect of HeadStart relative to home care In contrast the HSIS measures theeffect of Head Start relative to a mix of alternative care environ-ments including other preschools

III Data and Experimental Impacts

Before turning to an analysis of program substitution issueswe describe the HSIS data and report experimental impacts ontest scores and program compliance

IIIA Data

Our core analysis sample includes 3571 HSIS applicantswith nonmissing baseline characteristics and spring 2003 testscores Online Appendix A describes construction of this sampleThe outcome of interest is a summary index of cognitive testscores that averages Woodcock Johnson III (WJIII) test scoreswith Peabody Picture and Vocabulary Test (PPVT) scores witheach test normed to have mean 0 and variance 1 in the controlgroup by cohort and year We use WJIII and PPVT scores becausethese are among the most reliable tests in the HSIS data both areavailable in each year of the experiment allowing us to producecomparable estimates over time

1 Subsequent analyses of the HSIS data suggest caveats to this negative in-terpretation but do not overturn the finding of modest mean test score impactsaccompanied by rapid fade-out Gelber and Isen (2013) find persistent effects onparental engagement with children Bitler Domina and Hoynes (2014) find largerexperimental impacts at low quantiles of the test score distribution These quantiletreatment effects fade out by first grade though there is some evidence of persistenteffects at the bottom of the distribution for Spanish speakers Walters (2015) findsevidence of substantial heterogeneity in impacts across experimental sites and in-vestigates the relationship between this heterogeneity and observed program char-acteristics Walters finds smaller effects for Head Start centers that draw morechildren from other preschools rather than home care a finding we explore inmore detail here

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1801

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Table I provides summary statistics for our analysis sampleThe HSIS experiment included two age cohorts 55 of applicantswere randomized at age three and could attend Head Start for upto two years while the remaining 45 were randomized at age fourand could attend for up to one year The demographic informationin Table I shows that the Head Start population is disadvantagedLess than half of Head Start applicants live in two-parent house-holds and the average applicantrsquos household earns about 90 ofthe federal poverty line Column (2) of Table I compares these andother baseline characteristics for the HSIS treatment and controlgroups to check balance in randomization The results here indi-cate that randomization was successful baseline characteristicswere similar for offered and non offered applicants2

Columns (3) through (5) of Table I report summary statisticsfor children attending Head Start other preschool centers andno preschool3 Children in other preschools tend to be less disad-vantaged than children in Head Start or no preschool thoughmost differences between these groups are modest The otherpreschool group has a lower share of high school dropout mothersa higher share of mothers who attended college and higher av-erage household income than the Head Start and no preschoolgroups Children in other preschools outscore the other groups byabout 01 standard deviations on a baseline summary index ofcognitive skills The other preschool group also includes a rela-tively large share of four-year-olds likely reflecting the fact thatalternative preschool options are more widely available for four-year-olds (Cascio and Schanzenbach 2013)

IIIB Experimental Impacts

Table II reports experimental impacts on test scoresColumns (1) (4) and (7) report intent-to-treat impacts of the

2 Random assignment in the HSIS occurred at the Head Start center leveland offer probabilities differed across centers We weight all models by the inverseprobability of a childrsquos assignment calculated as the site-specific fraction of chil-dren assigned to the treatment group

3 Preschool attendance is measured from the HSIS lsquolsquofocal arrangement typersquorsquovariable which combines information from parent interviews and teachercareprovider interviews to construct a summary measure of the childcare setting SeeOnline Appendix A for details

QUARTERLY JOURNAL OF ECONOMICS1802

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

DE

SC

RIP

TIV

ES

TA

TIS

TIC

S

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

Male

04

94

05

05

00

11

05

01

05

06

04

92

(00

19)

Bla

ck03

08

02

98

00

10

03

17

03

53

02

50

(00

10)

His

pan

ic03

76

03

69

00

07

03

80

03

54

03

73

(00

10)

Tee

nm

oth

er01

59

01

74

00

15

01

59

01

69

01

76

(00

14)

Mot

her

marr

ied

04

36

04

48

00

11

04

39

04

20

04

60

(00

17)

Bot

hp

are

nts

inh

ouse

hol

d04

97

04

88

00

09

04

97

04

68

04

99

(00

17)

Mot

her

ish

igh

sch

ool

dro

pou

t03

68

03

97

00

29

03

77

03

22

04

26

(00

17)

Mot

her

att

end

edso

me

coll

ege

02

98

02

81

00

17

02

93

03

42

02

53

(00

16)

Sp

an

ish

spea

ker

02

87

02

73

00

14

02

96

02

74

02

60

(00

11)

Sp

ecia

led

uca

tion

01

36

01

08

00

28

01

34

01

45

00

91

(00

11)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1803

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

(CO

NT

INU

ED)

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

On

lych

ild

01

61

01

39

00

22

01

51

01

90

01

23

(00

12)

Inco

me

(fra

ctio

nof

FP

L)

08

96

08

96

00

00

08

92

09

83

08

51

(00

24)

Age

4co

hor

t04

48

04

51

00

03

04

26

05

67

04

13

(00

12)

Base

lin

esu

mm

ary

ind

ex00

03

00

12

00

09

00

01

01

06

00

40

(00

27)

Urb

an

08

33

08

35

00

02

08

34

08

59

08

19

(00

03)

Cen

ter

pro

vid

estr

an

spor

tati

on06

06

06

04

00

02

05

86

06

14

06

28

(00

05)

Cen

ter

qu

ali

tyin

dex

04

65

04

70

00

05

04

52

04

74

04

88

(00

05)

Joi

nt

p-v

alu

e05

06

N22

56

13

15

35

71

20

43

598

930

Not

es

All

stati

stic

sw

eigh

tby

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tS

tan

dard

erro

rsare

clu

ster

edat

the

cen

ter

level

T

he

tran

spor

tati

onan

dqu

ali

tyin

dex

vari

able

sre

fer

toa

chil

drsquos

Hea

dS

tart

cen

ter

ofra

nd

omass

ign

men

tT

he

qu

ali

tyvari

able

com

bin

esin

form

ati

onon

cen

ter

chara

cter

isti

cs(t

each

eran

dce

nte

rd

irec

tor

edu

cati

onan

dqu

ali

fica

tion

scl

ass

size

)an

dp

ract

ices

(vari

ety

ofli

tera

cyan

dm

ath

act

ivit

ies

hom

evis

itin

g

hea

lth

an

dn

utr

itio

n)

Inco

me

ism

issi

ng

for

19

ofob

serv

ati

ons

Mis

sin

gvalu

esare

excl

ud

edin

stati

stic

sfo

rin

com

eT

he

base

lin

esu

mm

ary

ind

exis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

infa

ll2002

wit

hea

chsc

ore

stan

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tT

he

join

tp

-valu

eis

from

ate

stof

the

hyp

oth

esis

that

all

coef

fici

ents

equ

al

0

QUARTERLY JOURNAL OF ECONOMICS1804

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

EX

PE

RIM

EN

TA

LIM

PA

CT

SO

NT

ES

TS

CO

RE

S

Th

ree-

yea

r-ol

dco

hor

tF

our-

yea

r-ol

dco

hor

tC

ohor

tsp

oole

d

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

Tim

ep

erio

dR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IV

Yea

r1

01

94

06

99

02

78

01

41

06

63

02

13

01

68

06

82

02

47

(00

29)

(00

25)

(00

41)

(00

29)

(00

22)

(00

44)

(00

21)

(00

18)

(00

31)

N19

70

16

01

35

71

Yea

r2

00

87

03

56

02

45

00

15

06

70

00

22

00

46

04

97

00

93

(00

29)

(00

28)

(00

80)

(00

37)

(00

23)

(00

54)

(00

24)

(00

20)

(00

49)

N17

60

14

16

31

76

Yea

r3

00

10

03

65

00

27

00

54

06

66

00

81

00

19

05

00

00

38

(00

31)

(00

28)

(00

85)

(00

40)

(00

25)

(00

60)

(00

25)

(00

20)

(00

50)

N16

59

13

36

29

95

Yea

r4

00

38

03

44

01

10

mdashmdash

(00

34)

(00

29)

(00

98)

N15

99

Not

es

Th

ista

ble

rep

orts

exp

erim

enta

les

tim

ate

sof

the

effe

cts

ofH

ead

Sta

rton

test

scor

es

Th

eou

tcom

eis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

w

ith

each

scor

est

an

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tan

dyea

rC

olu

mn

s(1

)(4

)an

d(7

)re

por

tco

effi

cien

tsfr

omre

gre

ssio

ns

ofte

stsc

ores

onan

ind

icato

rfo

rass

ign

men

tto

Hea

dS

tart

C

olu

mn

s(2

)(5

)an

d(8

)re

por

tco

effi

cien

tsfr

omfi

rst-

stage

regre

ssio

ns

ofH

ead

Sta

rtatt

end

an

ceon

Hea

dS

tart

ass

ign

men

tT

he

att

end

an

cevari

able

isan

ind

icato

req

ual

to1

ifa

chil

datt

end

sH

ead

Sta

rtat

an

yti

me

pri

orto

the

test

C

olu

mn

s(3

)(6

)an

d(9

)re

por

tco

effi

cien

tsfr

omtw

o-st

age

least

squ

are

s(2

SL

S)

mod

els

that

inst

rum

ent

Hea

dS

tart

att

end

an

cew

ith

Hea

dS

tart

ass

ign

men

tA

llm

odel

sw

eigh

tby

the

reci

pro

cal

ofa

chil

drsquos

exp

erim

enta

lass

ign

-m

ent

an

dco

ntr

olfo

rse

x

race

S

pan

ish

lan

gu

age

teen

mot

her

m

oth

errsquos

mari

tal

statu

sp

rese

nce

ofbot

hp

are

nts

inth

eh

ome

fam

ily

size

sp

ecia

led

uca

tion

statu

sin

com

equ

art

ile

du

mm

ies

urb

an

an

da

cubic

pol

yn

omia

lin

base

lin

esc

ore

Mis

sin

gvalu

esfo

rco

vari

ate

sare

set

to0

an

dd

um

mie

sfo

rm

issi

ng

are

incl

ud

ed

Sta

nd

ard

erro

rsare

clu

ster

edby

cen

ter

ofra

nd

omass

ign

men

t

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1805

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Head Start offer separately by year and age cohort To increaseprecision we regression-adjust these treatmentcontrol differ-ences using the baseline characteristics in Table I4 Theintent-to-treat estimates mirror those previously reported inthe literature (eg Puma et al 2010) In the first year of theexperiment children offered Head Start scored higher on thesummary index For example three-year-olds offered HeadStart gained 019 standard deviations in test score outcomesrelative to those denied Head Start The corresponding effectfor four-year-olds is 014 standard deviations However thesegains diminish rapidly the pooled impact falls to a statisticallyinsignificant 002 standard deviations by year 3 Our data in-clude a fourth year of follow-up for the three-year-old cohortHere too the intent-to-treat is small and statistically insignif-icant (0038 standard deviations)

Interpretation of these intent-to-treat impacts is clouded bynoncompliance with random assignment Columns (2) (5) and(8) of Table II report first-stage effects of assignment to HeadStart on the probability of participating in Head Start and col-umns (3) (6) and (9) report instrumental variables (IV) esti-mates which scale the intent-to-treat estimates by the first-stage estimates5 These estimates can be interpreted as local av-erage treatment effects (LATEs) for lsquolsquocompliersrsquorsquomdashchildren whorespond to the Head Start offer by enrolling in Head StartAssignment to Head Start increases the probability of participa-tion by two thirds in the first year after random assignment The

4 The control vector includes sex race assignment cohort teen mothermotherrsquos education motherrsquos marital status presence of both parents an onlychild dummy a Spanish language indicator dummies for quartiles of familyincome and missing income urban status an indicator for whether the HeadStart center provides transportation an index of Head Start center quality and athird-order polynomial in baseline test scores

5 Here we define Head Start participation as enrollment at any time prior tothe test This definition includes attendance at Head Start centers outside the ex-perimental sample An experimental offer may cause some children to switch froman out-of-sample center to an experimental center if the quality of these centersdiffers the exclusion restriction required for our IV approach is violated OnlineAppendix Table AI compares characteristics of centers attended by children in thecontrol group (always takers) to those of the experimental centers to which thesechildren applied These two groups of centers are very similar suggesting thatsubstitution between Head Start centers is unlikely to bias our estimates

QUARTERLY JOURNAL OF ECONOMICS1806

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

corresponding IV estimate implies that Head Start attendanceboosts first-year test scores by 0247 standard deviations

Compliance for the three-year-old cohort falls after the firstyear as members of the control group reapply for Head Startresulting in larger standard errors for estimates in later yearsof the experiment The first stage for three-year-olds falls to 036in the second year whereas the intent-to-treat falls roughly inproportion generating a second-year IV estimate of 0245 for thiscohort Estimates in years 3 and 4 are statistically insignificantand imprecise The fourth-year estimate for the three-year-oldcohort (corresponding to first grade) is 0110 standard deviationswith a standard error of 0098 The corresponding first grade es-timate for four-year-olds is 0081 with a standard error of 0060Notably the 95 confidence intervals for first-grade impacts in-clude effects as large as 02 standard deviations for four-year-oldsand 03 standard deviations for three-year-olds These resultsshow that although the longer run estimates are insignificantthey are also imprecise due to experimental noncomplianceEvidence for fade-out is therefore less definitive than the naiveintent-to-treat estimates suggest This observation helps to rec-oncile the HSIS results with observational studies based on sib-ling comparisons which show effects that partially fade out butare still detectable in elementary school (Currie and Thomas1995 Deming 2009)6

IV Program Substitution

We turn now to documenting program substitution in theHSIS and how it influences our results It is helpful to developsome notation to describe the role of alternative care environ-ments Each Head Start applicant participates in one of three pos-sible treatments Head Start which we label h other center-based

6 One might also be interested in the effects of Head Start on noncognitiveoutcomes which appear to be important mediators of the effects of early childhoodprograms in other contexts (Chetty et al 2011 Heckman Pinto and Savelyev2013) The HSIS includes short-run parent-reported measures of behavior andteacher-reported measures of teacherndashstudent relationships and Head Start ap-pears to have no impact on these outcomes (Puma et al 2010 Walters 2015) TheHSIS noncognitive outcomes differ significantly from those analyzed in previousstudies however and it is unclear whether they capture the same skills

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1807

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 3: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

patterns we find that roughly one third of Head Start lsquolsquocompliersrsquorsquo(Angrist Imbens and Rubin 1996) in the HSIS experiment wouldhave participated in other forms of preschool had they not beenlotteried into the program These alternative preschools drawheavily on public funding which mitigates the net costs to gov-ernment of shifting children from other preschools into HeadStart

These facts motivate a theoretical analysis clarifying whichparameters are (and are not) policy relevant when publicly sub-sidized program substitutes are present We work with a stylizedmodel where test score impacts are valued according to theireffects on childrenrsquos after-tax lifetime earnings We show thatwhen competing preschool programs are not rationed thepolicy-relevant causal parameter governing the benefits of HeadStart expansion is an average effect of Head Start participationrelative to the next best alternative regardless of whether thatalternative is a competing program or home care This parametercoincides with the local average treatment effect (LATE) identi-fied by a randomized experiment with imperfect compliance whenthe experiment contains a representative sample of program com-pliers Hence imperfect compliance and program substitutionoften thought to be confounding limitations of social experimentsturn out to be virtues when the substitution patterns in the ex-periment replicate those found in the broader population

We use this result to derive an estimable benefit-cost ratioassociated with Head Start expansions This ratio scales HeadStartrsquos projected impacts on the after-tax earnings of childrenby its net costs to government inclusive of fiscal externalitiesChief among these externalities is the cost savings that arisewhen Head Start draws children away from competing subsidizedpreschool programs Although such effects are typically ignoredin cost-benefit analyses of Head Start and other similar programs(eg CEA 2015) we find via a calibration exercise that such omis-sions can be quantitatively important Head Start roughly breakseven when the cost savings associated with program substitutionare ignored but yields benefits nearly twice as large as costswhen these savings are incorporated This appears to be arobust findingmdashafter accounting for fiscal externalities HeadStartrsquos benefits exceed its costs whenever short-run test scoreimpacts yield earnings gains within the range found in therecent literature

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1797

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A limitation of our baseline analysis is that it assumeschanges in program scale do not alter the mix of program compli-ers To address this issue we also consider lsquolsquostructural reformsrsquorsquo toHead Start that change the mix of compliers without affectingtest score outcomes Examples of such reforms might include in-creased transportation services marketing efforts or spendingon program features that parents value Households who respondto structural reforms may differ from experimental compliers onunobserved dimensions including their mix of counterfactualprogram choices Assessing these reforms therefore requiresknowledge of parameters not directly identified by the HSIS ex-periment Specifically we show that such reforms require identi-fication of a variant of the marginal treatment effect (MTE)concept of Heckman and Vytlacil (1999)

To assess reforms that attract new children we develop aselection model that parameterizes variation in treatment effectswith respect to counterfactual care alternatives as well as ob-served and unobserved child characteristics We prove that themodel parameters are identified and propose a two-step controlfunction estimator that exploits heterogeneity in the response toHead Start offers across sites and demographic groups to inferrelationships between unobserved factors driving preschool en-rollment and potential outcomes The estimator is shown to passa variety of specification tests and accurately reproduce patternsof treatment effect heterogeneity found in the experiment Themodel estimates indicate that Head Start has large positiveshort-run effects on the test scores of children who would haveotherwise been cared for at home and insignificant effects on chil-dren who would otherwise attend other preschoolsmdasha finding cor-roborated by Feller et al (forthcoming) who reach similarconclusions using principal stratification methods (Frangakisand Rubin 2002) Our estimates also reveal a lsquolsquoreverse Royrsquorsquo pat-tern of selection whereby children with unobserved characteris-tics that make them less likely to enroll in Head Start experiencelarger test score gains

We conclude with an assessment of prospects for increasingHead Startrsquos rate of return via outreach to new populations Ourestimates suggest that expansions of Head Start could boost theprogramrsquos rate of return provided that the proposed technologyfor increasing enrollment (eg improved transportation services)is not too costly We also use our estimated selection model toexamine the robustness of our results to rationing of competing

QUARTERLY JOURNAL OF ECONOMICS1798

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschools Rationing implies that competing subsidized pre-schools do not contract when Head Start expands which shutsdown a form of public savings On the other hand expandingHead Start generates opportunities for new children to fill va-cated seats in substitute programs Our estimates indicate thatthe effect on test scores (and therefore earnings) of moving chil-dren from home care to competing preschools is substantial lead-ing us to conclude that rationing is in fact likely to increase thefavorable estimated rates of return found in our baselineanalysis

The rest of the article is structured as follows Section II pro-vides background on Head Start Section III describes the HSISdata and basic experimental impacts Section IV presents evi-dence on substitution patterns Section V introduces a theoreticalframework for assessing public programs with close substitutesSection VI provides a cost-benefit analysis of Head Start SectionVII develops our econometric selection model and discusses iden-tification and estimation Section VIII reports estimates of themodel Section IX simulates the effects of structural program re-forms Section X concludes

II Background on Head Start

Head Start provides preschool for disadvantaged children inthe United States The program is funded by federal grantsawarded to local public or private organizations Grantees arerequired to match at least 20 of their Head Start awards fromother sources and must meet a set of program-wide performancecriteria Eligibility for Head Start is generally limited to childrenfrom households below the federal poverty line though familiesabove this threshold may be eligible if they meet other criteriasuch as participation in the Temporary Aid for Needy Families(TANF) program Up to 10 of a Head Start centerrsquos enrollmentcan also come from higher-income families The program is freeHead Start grantees are prohibited from charging families feesfor services (US DHHS 2014) It is also oversubscribed in 200285 of Head Start participants attended programs with moreapplicants than available seats (Puma et al 2010)

Head Start is not the only form of subsidized preschool avail-able to poor families Preschool participation rates for disadvan-taged children have risen over time as cities and states expanded

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1799

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

their public preschool offerings (Cascio and Schanzenbach 2013)Moreover the Child Care Development Fund program providesblock grants that finance childcare subsidies for low-income fam-ilies often in the form of child care vouchers that can be used forcenter-based preschool (US DHHS 2012) Most states also useTANF funds to finance additional child care subsidies(Schumacher Greenberg and Duffy 2001) Because Head Startservices are provided by local organizations who themselves mustraise outside funds it is unclear to what extent Head Start andother public preschool programs actually differ in their educationtechnology

A large nonexperimental literature suggests that Head Startproduced large short- and long-run benefits for early cohorts ofprogram participants Several studies estimate the effects ofHead Start by comparing program participants to their nonpar-ticipant siblings (Currie and Thomas 1995 Garces Thomas andCurrie 2002 Deming 2009) Results from this research designshow positive short-run effects on test scores and long-run effectson educational attainment earnings and crime Other studiesexploit discontinuities in Head Start program rules to infer pro-gram effects (Ludwig and Miller 2007 Carneiro and Ginja 2014)These studies show longer run improvements in health outcomesand criminal activity

In contrast to these nonexperimental estimates results froma recent randomized controlled trial reveal smaller less persis-tent effects The 1998 Head Start reauthorization bill included acongressional mandate to determine the effects of the programThis mandate resulted in the HSIS an experiment in which morethan 4000 applicants were randomly assigned via lottery toeither a treatment group with access to Head Start or a controlgroup without access in fall 2002 The experimental resultsshowed that a Head Start offer increased measures of cognitiveachievement by roughly 01 standard deviations during pre-school but these gains faded out by kindergarten Moreoverthe experiment showed little evidence of effects on noncognitiveor health outcomes (Puma et al 2010 Puma Bell and Heid2012) These results suggest both smaller short-run effects andfaster fade-out than nonexperimental estimates for earlier co-horts Scholars and policy makers have generally interpretedthe HSIS results as evidence that Head Start is ineffective andin need of reform (Barnett 2011) The experimental results havealso been cited in the popular media to motivate calls for dramatic

QUARTERLY JOURNAL OF ECONOMICS1800

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

restructuring or elimination of the program (Klein 2011 Stossel2014)1

Differences between the HSIS results and the nonexperi-mental literature could be due to changes in program effective-ness over time or to selection bias in nonexperimental siblingcomparisons Another explanation however is that these tworesearch designs identify different parameters Most nonexperi-mental analyses have focused on recovering the effect of HeadStart relative to home care In contrast the HSIS measures theeffect of Head Start relative to a mix of alternative care environ-ments including other preschools

III Data and Experimental Impacts

Before turning to an analysis of program substitution issueswe describe the HSIS data and report experimental impacts ontest scores and program compliance

IIIA Data

Our core analysis sample includes 3571 HSIS applicantswith nonmissing baseline characteristics and spring 2003 testscores Online Appendix A describes construction of this sampleThe outcome of interest is a summary index of cognitive testscores that averages Woodcock Johnson III (WJIII) test scoreswith Peabody Picture and Vocabulary Test (PPVT) scores witheach test normed to have mean 0 and variance 1 in the controlgroup by cohort and year We use WJIII and PPVT scores becausethese are among the most reliable tests in the HSIS data both areavailable in each year of the experiment allowing us to producecomparable estimates over time

1 Subsequent analyses of the HSIS data suggest caveats to this negative in-terpretation but do not overturn the finding of modest mean test score impactsaccompanied by rapid fade-out Gelber and Isen (2013) find persistent effects onparental engagement with children Bitler Domina and Hoynes (2014) find largerexperimental impacts at low quantiles of the test score distribution These quantiletreatment effects fade out by first grade though there is some evidence of persistenteffects at the bottom of the distribution for Spanish speakers Walters (2015) findsevidence of substantial heterogeneity in impacts across experimental sites and in-vestigates the relationship between this heterogeneity and observed program char-acteristics Walters finds smaller effects for Head Start centers that draw morechildren from other preschools rather than home care a finding we explore inmore detail here

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1801

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Table I provides summary statistics for our analysis sampleThe HSIS experiment included two age cohorts 55 of applicantswere randomized at age three and could attend Head Start for upto two years while the remaining 45 were randomized at age fourand could attend for up to one year The demographic informationin Table I shows that the Head Start population is disadvantagedLess than half of Head Start applicants live in two-parent house-holds and the average applicantrsquos household earns about 90 ofthe federal poverty line Column (2) of Table I compares these andother baseline characteristics for the HSIS treatment and controlgroups to check balance in randomization The results here indi-cate that randomization was successful baseline characteristicswere similar for offered and non offered applicants2

Columns (3) through (5) of Table I report summary statisticsfor children attending Head Start other preschool centers andno preschool3 Children in other preschools tend to be less disad-vantaged than children in Head Start or no preschool thoughmost differences between these groups are modest The otherpreschool group has a lower share of high school dropout mothersa higher share of mothers who attended college and higher av-erage household income than the Head Start and no preschoolgroups Children in other preschools outscore the other groups byabout 01 standard deviations on a baseline summary index ofcognitive skills The other preschool group also includes a rela-tively large share of four-year-olds likely reflecting the fact thatalternative preschool options are more widely available for four-year-olds (Cascio and Schanzenbach 2013)

IIIB Experimental Impacts

Table II reports experimental impacts on test scoresColumns (1) (4) and (7) report intent-to-treat impacts of the

2 Random assignment in the HSIS occurred at the Head Start center leveland offer probabilities differed across centers We weight all models by the inverseprobability of a childrsquos assignment calculated as the site-specific fraction of chil-dren assigned to the treatment group

3 Preschool attendance is measured from the HSIS lsquolsquofocal arrangement typersquorsquovariable which combines information from parent interviews and teachercareprovider interviews to construct a summary measure of the childcare setting SeeOnline Appendix A for details

QUARTERLY JOURNAL OF ECONOMICS1802

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

DE

SC

RIP

TIV

ES

TA

TIS

TIC

S

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

Male

04

94

05

05

00

11

05

01

05

06

04

92

(00

19)

Bla

ck03

08

02

98

00

10

03

17

03

53

02

50

(00

10)

His

pan

ic03

76

03

69

00

07

03

80

03

54

03

73

(00

10)

Tee

nm

oth

er01

59

01

74

00

15

01

59

01

69

01

76

(00

14)

Mot

her

marr

ied

04

36

04

48

00

11

04

39

04

20

04

60

(00

17)

Bot

hp

are

nts

inh

ouse

hol

d04

97

04

88

00

09

04

97

04

68

04

99

(00

17)

Mot

her

ish

igh

sch

ool

dro

pou

t03

68

03

97

00

29

03

77

03

22

04

26

(00

17)

Mot

her

att

end

edso

me

coll

ege

02

98

02

81

00

17

02

93

03

42

02

53

(00

16)

Sp

an

ish

spea

ker

02

87

02

73

00

14

02

96

02

74

02

60

(00

11)

Sp

ecia

led

uca

tion

01

36

01

08

00

28

01

34

01

45

00

91

(00

11)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1803

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

(CO

NT

INU

ED)

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

On

lych

ild

01

61

01

39

00

22

01

51

01

90

01

23

(00

12)

Inco

me

(fra

ctio

nof

FP

L)

08

96

08

96

00

00

08

92

09

83

08

51

(00

24)

Age

4co

hor

t04

48

04

51

00

03

04

26

05

67

04

13

(00

12)

Base

lin

esu

mm

ary

ind

ex00

03

00

12

00

09

00

01

01

06

00

40

(00

27)

Urb

an

08

33

08

35

00

02

08

34

08

59

08

19

(00

03)

Cen

ter

pro

vid

estr

an

spor

tati

on06

06

06

04

00

02

05

86

06

14

06

28

(00

05)

Cen

ter

qu

ali

tyin

dex

04

65

04

70

00

05

04

52

04

74

04

88

(00

05)

Joi

nt

p-v

alu

e05

06

N22

56

13

15

35

71

20

43

598

930

Not

es

All

stati

stic

sw

eigh

tby

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tS

tan

dard

erro

rsare

clu

ster

edat

the

cen

ter

level

T

he

tran

spor

tati

onan

dqu

ali

tyin

dex

vari

able

sre

fer

toa

chil

drsquos

Hea

dS

tart

cen

ter

ofra

nd

omass

ign

men

tT

he

qu

ali

tyvari

able

com

bin

esin

form

ati

onon

cen

ter

chara

cter

isti

cs(t

each

eran

dce

nte

rd

irec

tor

edu

cati

onan

dqu

ali

fica

tion

scl

ass

size

)an

dp

ract

ices

(vari

ety

ofli

tera

cyan

dm

ath

act

ivit

ies

hom

evis

itin

g

hea

lth

an

dn

utr

itio

n)

Inco

me

ism

issi

ng

for

19

ofob

serv

ati

ons

Mis

sin

gvalu

esare

excl

ud

edin

stati

stic

sfo

rin

com

eT

he

base

lin

esu

mm

ary

ind

exis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

infa

ll2002

wit

hea

chsc

ore

stan

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tT

he

join

tp

-valu

eis

from

ate

stof

the

hyp

oth

esis

that

all

coef

fici

ents

equ

al

0

QUARTERLY JOURNAL OF ECONOMICS1804

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

EX

PE

RIM

EN

TA

LIM

PA

CT

SO

NT

ES

TS

CO

RE

S

Th

ree-

yea

r-ol

dco

hor

tF

our-

yea

r-ol

dco

hor

tC

ohor

tsp

oole

d

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

Tim

ep

erio

dR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IV

Yea

r1

01

94

06

99

02

78

01

41

06

63

02

13

01

68

06

82

02

47

(00

29)

(00

25)

(00

41)

(00

29)

(00

22)

(00

44)

(00

21)

(00

18)

(00

31)

N19

70

16

01

35

71

Yea

r2

00

87

03

56

02

45

00

15

06

70

00

22

00

46

04

97

00

93

(00

29)

(00

28)

(00

80)

(00

37)

(00

23)

(00

54)

(00

24)

(00

20)

(00

49)

N17

60

14

16

31

76

Yea

r3

00

10

03

65

00

27

00

54

06

66

00

81

00

19

05

00

00

38

(00

31)

(00

28)

(00

85)

(00

40)

(00

25)

(00

60)

(00

25)

(00

20)

(00

50)

N16

59

13

36

29

95

Yea

r4

00

38

03

44

01

10

mdashmdash

(00

34)

(00

29)

(00

98)

N15

99

Not

es

Th

ista

ble

rep

orts

exp

erim

enta

les

tim

ate

sof

the

effe

cts

ofH

ead

Sta

rton

test

scor

es

Th

eou

tcom

eis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

w

ith

each

scor

est

an

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tan

dyea

rC

olu

mn

s(1

)(4

)an

d(7

)re

por

tco

effi

cien

tsfr

omre

gre

ssio

ns

ofte

stsc

ores

onan

ind

icato

rfo

rass

ign

men

tto

Hea

dS

tart

C

olu

mn

s(2

)(5

)an

d(8

)re

por

tco

effi

cien

tsfr

omfi

rst-

stage

regre

ssio

ns

ofH

ead

Sta

rtatt

end

an

ceon

Hea

dS

tart

ass

ign

men

tT

he

att

end

an

cevari

able

isan

ind

icato

req

ual

to1

ifa

chil

datt

end

sH

ead

Sta

rtat

an

yti

me

pri

orto

the

test

C

olu

mn

s(3

)(6

)an

d(9

)re

por

tco

effi

cien

tsfr

omtw

o-st

age

least

squ

are

s(2

SL

S)

mod

els

that

inst

rum

ent

Hea

dS

tart

att

end

an

cew

ith

Hea

dS

tart

ass

ign

men

tA

llm

odel

sw

eigh

tby

the

reci

pro

cal

ofa

chil

drsquos

exp

erim

enta

lass

ign

-m

ent

an

dco

ntr

olfo

rse

x

race

S

pan

ish

lan

gu

age

teen

mot

her

m

oth

errsquos

mari

tal

statu

sp

rese

nce

ofbot

hp

are

nts

inth

eh

ome

fam

ily

size

sp

ecia

led

uca

tion

statu

sin

com

equ

art

ile

du

mm

ies

urb

an

an

da

cubic

pol

yn

omia

lin

base

lin

esc

ore

Mis

sin

gvalu

esfo

rco

vari

ate

sare

set

to0

an

dd

um

mie

sfo

rm

issi

ng

are

incl

ud

ed

Sta

nd

ard

erro

rsare

clu

ster

edby

cen

ter

ofra

nd

omass

ign

men

t

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1805

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Head Start offer separately by year and age cohort To increaseprecision we regression-adjust these treatmentcontrol differ-ences using the baseline characteristics in Table I4 Theintent-to-treat estimates mirror those previously reported inthe literature (eg Puma et al 2010) In the first year of theexperiment children offered Head Start scored higher on thesummary index For example three-year-olds offered HeadStart gained 019 standard deviations in test score outcomesrelative to those denied Head Start The corresponding effectfor four-year-olds is 014 standard deviations However thesegains diminish rapidly the pooled impact falls to a statisticallyinsignificant 002 standard deviations by year 3 Our data in-clude a fourth year of follow-up for the three-year-old cohortHere too the intent-to-treat is small and statistically insignif-icant (0038 standard deviations)

Interpretation of these intent-to-treat impacts is clouded bynoncompliance with random assignment Columns (2) (5) and(8) of Table II report first-stage effects of assignment to HeadStart on the probability of participating in Head Start and col-umns (3) (6) and (9) report instrumental variables (IV) esti-mates which scale the intent-to-treat estimates by the first-stage estimates5 These estimates can be interpreted as local av-erage treatment effects (LATEs) for lsquolsquocompliersrsquorsquomdashchildren whorespond to the Head Start offer by enrolling in Head StartAssignment to Head Start increases the probability of participa-tion by two thirds in the first year after random assignment The

4 The control vector includes sex race assignment cohort teen mothermotherrsquos education motherrsquos marital status presence of both parents an onlychild dummy a Spanish language indicator dummies for quartiles of familyincome and missing income urban status an indicator for whether the HeadStart center provides transportation an index of Head Start center quality and athird-order polynomial in baseline test scores

5 Here we define Head Start participation as enrollment at any time prior tothe test This definition includes attendance at Head Start centers outside the ex-perimental sample An experimental offer may cause some children to switch froman out-of-sample center to an experimental center if the quality of these centersdiffers the exclusion restriction required for our IV approach is violated OnlineAppendix Table AI compares characteristics of centers attended by children in thecontrol group (always takers) to those of the experimental centers to which thesechildren applied These two groups of centers are very similar suggesting thatsubstitution between Head Start centers is unlikely to bias our estimates

QUARTERLY JOURNAL OF ECONOMICS1806

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

corresponding IV estimate implies that Head Start attendanceboosts first-year test scores by 0247 standard deviations

Compliance for the three-year-old cohort falls after the firstyear as members of the control group reapply for Head Startresulting in larger standard errors for estimates in later yearsof the experiment The first stage for three-year-olds falls to 036in the second year whereas the intent-to-treat falls roughly inproportion generating a second-year IV estimate of 0245 for thiscohort Estimates in years 3 and 4 are statistically insignificantand imprecise The fourth-year estimate for the three-year-oldcohort (corresponding to first grade) is 0110 standard deviationswith a standard error of 0098 The corresponding first grade es-timate for four-year-olds is 0081 with a standard error of 0060Notably the 95 confidence intervals for first-grade impacts in-clude effects as large as 02 standard deviations for four-year-oldsand 03 standard deviations for three-year-olds These resultsshow that although the longer run estimates are insignificantthey are also imprecise due to experimental noncomplianceEvidence for fade-out is therefore less definitive than the naiveintent-to-treat estimates suggest This observation helps to rec-oncile the HSIS results with observational studies based on sib-ling comparisons which show effects that partially fade out butare still detectable in elementary school (Currie and Thomas1995 Deming 2009)6

IV Program Substitution

We turn now to documenting program substitution in theHSIS and how it influences our results It is helpful to developsome notation to describe the role of alternative care environ-ments Each Head Start applicant participates in one of three pos-sible treatments Head Start which we label h other center-based

6 One might also be interested in the effects of Head Start on noncognitiveoutcomes which appear to be important mediators of the effects of early childhoodprograms in other contexts (Chetty et al 2011 Heckman Pinto and Savelyev2013) The HSIS includes short-run parent-reported measures of behavior andteacher-reported measures of teacherndashstudent relationships and Head Start ap-pears to have no impact on these outcomes (Puma et al 2010 Walters 2015) TheHSIS noncognitive outcomes differ significantly from those analyzed in previousstudies however and it is unclear whether they capture the same skills

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1807

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 4: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

A limitation of our baseline analysis is that it assumeschanges in program scale do not alter the mix of program compli-ers To address this issue we also consider lsquolsquostructural reformsrsquorsquo toHead Start that change the mix of compliers without affectingtest score outcomes Examples of such reforms might include in-creased transportation services marketing efforts or spendingon program features that parents value Households who respondto structural reforms may differ from experimental compliers onunobserved dimensions including their mix of counterfactualprogram choices Assessing these reforms therefore requiresknowledge of parameters not directly identified by the HSIS ex-periment Specifically we show that such reforms require identi-fication of a variant of the marginal treatment effect (MTE)concept of Heckman and Vytlacil (1999)

To assess reforms that attract new children we develop aselection model that parameterizes variation in treatment effectswith respect to counterfactual care alternatives as well as ob-served and unobserved child characteristics We prove that themodel parameters are identified and propose a two-step controlfunction estimator that exploits heterogeneity in the response toHead Start offers across sites and demographic groups to inferrelationships between unobserved factors driving preschool en-rollment and potential outcomes The estimator is shown to passa variety of specification tests and accurately reproduce patternsof treatment effect heterogeneity found in the experiment Themodel estimates indicate that Head Start has large positiveshort-run effects on the test scores of children who would haveotherwise been cared for at home and insignificant effects on chil-dren who would otherwise attend other preschoolsmdasha finding cor-roborated by Feller et al (forthcoming) who reach similarconclusions using principal stratification methods (Frangakisand Rubin 2002) Our estimates also reveal a lsquolsquoreverse Royrsquorsquo pat-tern of selection whereby children with unobserved characteris-tics that make them less likely to enroll in Head Start experiencelarger test score gains

We conclude with an assessment of prospects for increasingHead Startrsquos rate of return via outreach to new populations Ourestimates suggest that expansions of Head Start could boost theprogramrsquos rate of return provided that the proposed technologyfor increasing enrollment (eg improved transportation services)is not too costly We also use our estimated selection model toexamine the robustness of our results to rationing of competing

QUARTERLY JOURNAL OF ECONOMICS1798

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschools Rationing implies that competing subsidized pre-schools do not contract when Head Start expands which shutsdown a form of public savings On the other hand expandingHead Start generates opportunities for new children to fill va-cated seats in substitute programs Our estimates indicate thatthe effect on test scores (and therefore earnings) of moving chil-dren from home care to competing preschools is substantial lead-ing us to conclude that rationing is in fact likely to increase thefavorable estimated rates of return found in our baselineanalysis

The rest of the article is structured as follows Section II pro-vides background on Head Start Section III describes the HSISdata and basic experimental impacts Section IV presents evi-dence on substitution patterns Section V introduces a theoreticalframework for assessing public programs with close substitutesSection VI provides a cost-benefit analysis of Head Start SectionVII develops our econometric selection model and discusses iden-tification and estimation Section VIII reports estimates of themodel Section IX simulates the effects of structural program re-forms Section X concludes

II Background on Head Start

Head Start provides preschool for disadvantaged children inthe United States The program is funded by federal grantsawarded to local public or private organizations Grantees arerequired to match at least 20 of their Head Start awards fromother sources and must meet a set of program-wide performancecriteria Eligibility for Head Start is generally limited to childrenfrom households below the federal poverty line though familiesabove this threshold may be eligible if they meet other criteriasuch as participation in the Temporary Aid for Needy Families(TANF) program Up to 10 of a Head Start centerrsquos enrollmentcan also come from higher-income families The program is freeHead Start grantees are prohibited from charging families feesfor services (US DHHS 2014) It is also oversubscribed in 200285 of Head Start participants attended programs with moreapplicants than available seats (Puma et al 2010)

Head Start is not the only form of subsidized preschool avail-able to poor families Preschool participation rates for disadvan-taged children have risen over time as cities and states expanded

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1799

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

their public preschool offerings (Cascio and Schanzenbach 2013)Moreover the Child Care Development Fund program providesblock grants that finance childcare subsidies for low-income fam-ilies often in the form of child care vouchers that can be used forcenter-based preschool (US DHHS 2012) Most states also useTANF funds to finance additional child care subsidies(Schumacher Greenberg and Duffy 2001) Because Head Startservices are provided by local organizations who themselves mustraise outside funds it is unclear to what extent Head Start andother public preschool programs actually differ in their educationtechnology

A large nonexperimental literature suggests that Head Startproduced large short- and long-run benefits for early cohorts ofprogram participants Several studies estimate the effects ofHead Start by comparing program participants to their nonpar-ticipant siblings (Currie and Thomas 1995 Garces Thomas andCurrie 2002 Deming 2009) Results from this research designshow positive short-run effects on test scores and long-run effectson educational attainment earnings and crime Other studiesexploit discontinuities in Head Start program rules to infer pro-gram effects (Ludwig and Miller 2007 Carneiro and Ginja 2014)These studies show longer run improvements in health outcomesand criminal activity

In contrast to these nonexperimental estimates results froma recent randomized controlled trial reveal smaller less persis-tent effects The 1998 Head Start reauthorization bill included acongressional mandate to determine the effects of the programThis mandate resulted in the HSIS an experiment in which morethan 4000 applicants were randomly assigned via lottery toeither a treatment group with access to Head Start or a controlgroup without access in fall 2002 The experimental resultsshowed that a Head Start offer increased measures of cognitiveachievement by roughly 01 standard deviations during pre-school but these gains faded out by kindergarten Moreoverthe experiment showed little evidence of effects on noncognitiveor health outcomes (Puma et al 2010 Puma Bell and Heid2012) These results suggest both smaller short-run effects andfaster fade-out than nonexperimental estimates for earlier co-horts Scholars and policy makers have generally interpretedthe HSIS results as evidence that Head Start is ineffective andin need of reform (Barnett 2011) The experimental results havealso been cited in the popular media to motivate calls for dramatic

QUARTERLY JOURNAL OF ECONOMICS1800

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

restructuring or elimination of the program (Klein 2011 Stossel2014)1

Differences between the HSIS results and the nonexperi-mental literature could be due to changes in program effective-ness over time or to selection bias in nonexperimental siblingcomparisons Another explanation however is that these tworesearch designs identify different parameters Most nonexperi-mental analyses have focused on recovering the effect of HeadStart relative to home care In contrast the HSIS measures theeffect of Head Start relative to a mix of alternative care environ-ments including other preschools

III Data and Experimental Impacts

Before turning to an analysis of program substitution issueswe describe the HSIS data and report experimental impacts ontest scores and program compliance

IIIA Data

Our core analysis sample includes 3571 HSIS applicantswith nonmissing baseline characteristics and spring 2003 testscores Online Appendix A describes construction of this sampleThe outcome of interest is a summary index of cognitive testscores that averages Woodcock Johnson III (WJIII) test scoreswith Peabody Picture and Vocabulary Test (PPVT) scores witheach test normed to have mean 0 and variance 1 in the controlgroup by cohort and year We use WJIII and PPVT scores becausethese are among the most reliable tests in the HSIS data both areavailable in each year of the experiment allowing us to producecomparable estimates over time

1 Subsequent analyses of the HSIS data suggest caveats to this negative in-terpretation but do not overturn the finding of modest mean test score impactsaccompanied by rapid fade-out Gelber and Isen (2013) find persistent effects onparental engagement with children Bitler Domina and Hoynes (2014) find largerexperimental impacts at low quantiles of the test score distribution These quantiletreatment effects fade out by first grade though there is some evidence of persistenteffects at the bottom of the distribution for Spanish speakers Walters (2015) findsevidence of substantial heterogeneity in impacts across experimental sites and in-vestigates the relationship between this heterogeneity and observed program char-acteristics Walters finds smaller effects for Head Start centers that draw morechildren from other preschools rather than home care a finding we explore inmore detail here

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1801

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Table I provides summary statistics for our analysis sampleThe HSIS experiment included two age cohorts 55 of applicantswere randomized at age three and could attend Head Start for upto two years while the remaining 45 were randomized at age fourand could attend for up to one year The demographic informationin Table I shows that the Head Start population is disadvantagedLess than half of Head Start applicants live in two-parent house-holds and the average applicantrsquos household earns about 90 ofthe federal poverty line Column (2) of Table I compares these andother baseline characteristics for the HSIS treatment and controlgroups to check balance in randomization The results here indi-cate that randomization was successful baseline characteristicswere similar for offered and non offered applicants2

Columns (3) through (5) of Table I report summary statisticsfor children attending Head Start other preschool centers andno preschool3 Children in other preschools tend to be less disad-vantaged than children in Head Start or no preschool thoughmost differences between these groups are modest The otherpreschool group has a lower share of high school dropout mothersa higher share of mothers who attended college and higher av-erage household income than the Head Start and no preschoolgroups Children in other preschools outscore the other groups byabout 01 standard deviations on a baseline summary index ofcognitive skills The other preschool group also includes a rela-tively large share of four-year-olds likely reflecting the fact thatalternative preschool options are more widely available for four-year-olds (Cascio and Schanzenbach 2013)

IIIB Experimental Impacts

Table II reports experimental impacts on test scoresColumns (1) (4) and (7) report intent-to-treat impacts of the

2 Random assignment in the HSIS occurred at the Head Start center leveland offer probabilities differed across centers We weight all models by the inverseprobability of a childrsquos assignment calculated as the site-specific fraction of chil-dren assigned to the treatment group

3 Preschool attendance is measured from the HSIS lsquolsquofocal arrangement typersquorsquovariable which combines information from parent interviews and teachercareprovider interviews to construct a summary measure of the childcare setting SeeOnline Appendix A for details

QUARTERLY JOURNAL OF ECONOMICS1802

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

DE

SC

RIP

TIV

ES

TA

TIS

TIC

S

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

Male

04

94

05

05

00

11

05

01

05

06

04

92

(00

19)

Bla

ck03

08

02

98

00

10

03

17

03

53

02

50

(00

10)

His

pan

ic03

76

03

69

00

07

03

80

03

54

03

73

(00

10)

Tee

nm

oth

er01

59

01

74

00

15

01

59

01

69

01

76

(00

14)

Mot

her

marr

ied

04

36

04

48

00

11

04

39

04

20

04

60

(00

17)

Bot

hp

are

nts

inh

ouse

hol

d04

97

04

88

00

09

04

97

04

68

04

99

(00

17)

Mot

her

ish

igh

sch

ool

dro

pou

t03

68

03

97

00

29

03

77

03

22

04

26

(00

17)

Mot

her

att

end

edso

me

coll

ege

02

98

02

81

00

17

02

93

03

42

02

53

(00

16)

Sp

an

ish

spea

ker

02

87

02

73

00

14

02

96

02

74

02

60

(00

11)

Sp

ecia

led

uca

tion

01

36

01

08

00

28

01

34

01

45

00

91

(00

11)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1803

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

(CO

NT

INU

ED)

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

On

lych

ild

01

61

01

39

00

22

01

51

01

90

01

23

(00

12)

Inco

me

(fra

ctio

nof

FP

L)

08

96

08

96

00

00

08

92

09

83

08

51

(00

24)

Age

4co

hor

t04

48

04

51

00

03

04

26

05

67

04

13

(00

12)

Base

lin

esu

mm

ary

ind

ex00

03

00

12

00

09

00

01

01

06

00

40

(00

27)

Urb

an

08

33

08

35

00

02

08

34

08

59

08

19

(00

03)

Cen

ter

pro

vid

estr

an

spor

tati

on06

06

06

04

00

02

05

86

06

14

06

28

(00

05)

Cen

ter

qu

ali

tyin

dex

04

65

04

70

00

05

04

52

04

74

04

88

(00

05)

Joi

nt

p-v

alu

e05

06

N22

56

13

15

35

71

20

43

598

930

Not

es

All

stati

stic

sw

eigh

tby

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tS

tan

dard

erro

rsare

clu

ster

edat

the

cen

ter

level

T

he

tran

spor

tati

onan

dqu

ali

tyin

dex

vari

able

sre

fer

toa

chil

drsquos

Hea

dS

tart

cen

ter

ofra

nd

omass

ign

men

tT

he

qu

ali

tyvari

able

com

bin

esin

form

ati

onon

cen

ter

chara

cter

isti

cs(t

each

eran

dce

nte

rd

irec

tor

edu

cati

onan

dqu

ali

fica

tion

scl

ass

size

)an

dp

ract

ices

(vari

ety

ofli

tera

cyan

dm

ath

act

ivit

ies

hom

evis

itin

g

hea

lth

an

dn

utr

itio

n)

Inco

me

ism

issi

ng

for

19

ofob

serv

ati

ons

Mis

sin

gvalu

esare

excl

ud

edin

stati

stic

sfo

rin

com

eT

he

base

lin

esu

mm

ary

ind

exis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

infa

ll2002

wit

hea

chsc

ore

stan

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tT

he

join

tp

-valu

eis

from

ate

stof

the

hyp

oth

esis

that

all

coef

fici

ents

equ

al

0

QUARTERLY JOURNAL OF ECONOMICS1804

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

EX

PE

RIM

EN

TA

LIM

PA

CT

SO

NT

ES

TS

CO

RE

S

Th

ree-

yea

r-ol

dco

hor

tF

our-

yea

r-ol

dco

hor

tC

ohor

tsp

oole

d

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

Tim

ep

erio

dR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IV

Yea

r1

01

94

06

99

02

78

01

41

06

63

02

13

01

68

06

82

02

47

(00

29)

(00

25)

(00

41)

(00

29)

(00

22)

(00

44)

(00

21)

(00

18)

(00

31)

N19

70

16

01

35

71

Yea

r2

00

87

03

56

02

45

00

15

06

70

00

22

00

46

04

97

00

93

(00

29)

(00

28)

(00

80)

(00

37)

(00

23)

(00

54)

(00

24)

(00

20)

(00

49)

N17

60

14

16

31

76

Yea

r3

00

10

03

65

00

27

00

54

06

66

00

81

00

19

05

00

00

38

(00

31)

(00

28)

(00

85)

(00

40)

(00

25)

(00

60)

(00

25)

(00

20)

(00

50)

N16

59

13

36

29

95

Yea

r4

00

38

03

44

01

10

mdashmdash

(00

34)

(00

29)

(00

98)

N15

99

Not

es

Th

ista

ble

rep

orts

exp

erim

enta

les

tim

ate

sof

the

effe

cts

ofH

ead

Sta

rton

test

scor

es

Th

eou

tcom

eis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

w

ith

each

scor

est

an

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tan

dyea

rC

olu

mn

s(1

)(4

)an

d(7

)re

por

tco

effi

cien

tsfr

omre

gre

ssio

ns

ofte

stsc

ores

onan

ind

icato

rfo

rass

ign

men

tto

Hea

dS

tart

C

olu

mn

s(2

)(5

)an

d(8

)re

por

tco

effi

cien

tsfr

omfi

rst-

stage

regre

ssio

ns

ofH

ead

Sta

rtatt

end

an

ceon

Hea

dS

tart

ass

ign

men

tT

he

att

end

an

cevari

able

isan

ind

icato

req

ual

to1

ifa

chil

datt

end

sH

ead

Sta

rtat

an

yti

me

pri

orto

the

test

C

olu

mn

s(3

)(6

)an

d(9

)re

por

tco

effi

cien

tsfr

omtw

o-st

age

least

squ

are

s(2

SL

S)

mod

els

that

inst

rum

ent

Hea

dS

tart

att

end

an

cew

ith

Hea

dS

tart

ass

ign

men

tA

llm

odel

sw

eigh

tby

the

reci

pro

cal

ofa

chil

drsquos

exp

erim

enta

lass

ign

-m

ent

an

dco

ntr

olfo

rse

x

race

S

pan

ish

lan

gu

age

teen

mot

her

m

oth

errsquos

mari

tal

statu

sp

rese

nce

ofbot

hp

are

nts

inth

eh

ome

fam

ily

size

sp

ecia

led

uca

tion

statu

sin

com

equ

art

ile

du

mm

ies

urb

an

an

da

cubic

pol

yn

omia

lin

base

lin

esc

ore

Mis

sin

gvalu

esfo

rco

vari

ate

sare

set

to0

an

dd

um

mie

sfo

rm

issi

ng

are

incl

ud

ed

Sta

nd

ard

erro

rsare

clu

ster

edby

cen

ter

ofra

nd

omass

ign

men

t

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1805

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Head Start offer separately by year and age cohort To increaseprecision we regression-adjust these treatmentcontrol differ-ences using the baseline characteristics in Table I4 Theintent-to-treat estimates mirror those previously reported inthe literature (eg Puma et al 2010) In the first year of theexperiment children offered Head Start scored higher on thesummary index For example three-year-olds offered HeadStart gained 019 standard deviations in test score outcomesrelative to those denied Head Start The corresponding effectfor four-year-olds is 014 standard deviations However thesegains diminish rapidly the pooled impact falls to a statisticallyinsignificant 002 standard deviations by year 3 Our data in-clude a fourth year of follow-up for the three-year-old cohortHere too the intent-to-treat is small and statistically insignif-icant (0038 standard deviations)

Interpretation of these intent-to-treat impacts is clouded bynoncompliance with random assignment Columns (2) (5) and(8) of Table II report first-stage effects of assignment to HeadStart on the probability of participating in Head Start and col-umns (3) (6) and (9) report instrumental variables (IV) esti-mates which scale the intent-to-treat estimates by the first-stage estimates5 These estimates can be interpreted as local av-erage treatment effects (LATEs) for lsquolsquocompliersrsquorsquomdashchildren whorespond to the Head Start offer by enrolling in Head StartAssignment to Head Start increases the probability of participa-tion by two thirds in the first year after random assignment The

4 The control vector includes sex race assignment cohort teen mothermotherrsquos education motherrsquos marital status presence of both parents an onlychild dummy a Spanish language indicator dummies for quartiles of familyincome and missing income urban status an indicator for whether the HeadStart center provides transportation an index of Head Start center quality and athird-order polynomial in baseline test scores

5 Here we define Head Start participation as enrollment at any time prior tothe test This definition includes attendance at Head Start centers outside the ex-perimental sample An experimental offer may cause some children to switch froman out-of-sample center to an experimental center if the quality of these centersdiffers the exclusion restriction required for our IV approach is violated OnlineAppendix Table AI compares characteristics of centers attended by children in thecontrol group (always takers) to those of the experimental centers to which thesechildren applied These two groups of centers are very similar suggesting thatsubstitution between Head Start centers is unlikely to bias our estimates

QUARTERLY JOURNAL OF ECONOMICS1806

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

corresponding IV estimate implies that Head Start attendanceboosts first-year test scores by 0247 standard deviations

Compliance for the three-year-old cohort falls after the firstyear as members of the control group reapply for Head Startresulting in larger standard errors for estimates in later yearsof the experiment The first stage for three-year-olds falls to 036in the second year whereas the intent-to-treat falls roughly inproportion generating a second-year IV estimate of 0245 for thiscohort Estimates in years 3 and 4 are statistically insignificantand imprecise The fourth-year estimate for the three-year-oldcohort (corresponding to first grade) is 0110 standard deviationswith a standard error of 0098 The corresponding first grade es-timate for four-year-olds is 0081 with a standard error of 0060Notably the 95 confidence intervals for first-grade impacts in-clude effects as large as 02 standard deviations for four-year-oldsand 03 standard deviations for three-year-olds These resultsshow that although the longer run estimates are insignificantthey are also imprecise due to experimental noncomplianceEvidence for fade-out is therefore less definitive than the naiveintent-to-treat estimates suggest This observation helps to rec-oncile the HSIS results with observational studies based on sib-ling comparisons which show effects that partially fade out butare still detectable in elementary school (Currie and Thomas1995 Deming 2009)6

IV Program Substitution

We turn now to documenting program substitution in theHSIS and how it influences our results It is helpful to developsome notation to describe the role of alternative care environ-ments Each Head Start applicant participates in one of three pos-sible treatments Head Start which we label h other center-based

6 One might also be interested in the effects of Head Start on noncognitiveoutcomes which appear to be important mediators of the effects of early childhoodprograms in other contexts (Chetty et al 2011 Heckman Pinto and Savelyev2013) The HSIS includes short-run parent-reported measures of behavior andteacher-reported measures of teacherndashstudent relationships and Head Start ap-pears to have no impact on these outcomes (Puma et al 2010 Walters 2015) TheHSIS noncognitive outcomes differ significantly from those analyzed in previousstudies however and it is unclear whether they capture the same skills

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1807

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 5: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

preschools Rationing implies that competing subsidized pre-schools do not contract when Head Start expands which shutsdown a form of public savings On the other hand expandingHead Start generates opportunities for new children to fill va-cated seats in substitute programs Our estimates indicate thatthe effect on test scores (and therefore earnings) of moving chil-dren from home care to competing preschools is substantial lead-ing us to conclude that rationing is in fact likely to increase thefavorable estimated rates of return found in our baselineanalysis

The rest of the article is structured as follows Section II pro-vides background on Head Start Section III describes the HSISdata and basic experimental impacts Section IV presents evi-dence on substitution patterns Section V introduces a theoreticalframework for assessing public programs with close substitutesSection VI provides a cost-benefit analysis of Head Start SectionVII develops our econometric selection model and discusses iden-tification and estimation Section VIII reports estimates of themodel Section IX simulates the effects of structural program re-forms Section X concludes

II Background on Head Start

Head Start provides preschool for disadvantaged children inthe United States The program is funded by federal grantsawarded to local public or private organizations Grantees arerequired to match at least 20 of their Head Start awards fromother sources and must meet a set of program-wide performancecriteria Eligibility for Head Start is generally limited to childrenfrom households below the federal poverty line though familiesabove this threshold may be eligible if they meet other criteriasuch as participation in the Temporary Aid for Needy Families(TANF) program Up to 10 of a Head Start centerrsquos enrollmentcan also come from higher-income families The program is freeHead Start grantees are prohibited from charging families feesfor services (US DHHS 2014) It is also oversubscribed in 200285 of Head Start participants attended programs with moreapplicants than available seats (Puma et al 2010)

Head Start is not the only form of subsidized preschool avail-able to poor families Preschool participation rates for disadvan-taged children have risen over time as cities and states expanded

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1799

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

their public preschool offerings (Cascio and Schanzenbach 2013)Moreover the Child Care Development Fund program providesblock grants that finance childcare subsidies for low-income fam-ilies often in the form of child care vouchers that can be used forcenter-based preschool (US DHHS 2012) Most states also useTANF funds to finance additional child care subsidies(Schumacher Greenberg and Duffy 2001) Because Head Startservices are provided by local organizations who themselves mustraise outside funds it is unclear to what extent Head Start andother public preschool programs actually differ in their educationtechnology

A large nonexperimental literature suggests that Head Startproduced large short- and long-run benefits for early cohorts ofprogram participants Several studies estimate the effects ofHead Start by comparing program participants to their nonpar-ticipant siblings (Currie and Thomas 1995 Garces Thomas andCurrie 2002 Deming 2009) Results from this research designshow positive short-run effects on test scores and long-run effectson educational attainment earnings and crime Other studiesexploit discontinuities in Head Start program rules to infer pro-gram effects (Ludwig and Miller 2007 Carneiro and Ginja 2014)These studies show longer run improvements in health outcomesand criminal activity

In contrast to these nonexperimental estimates results froma recent randomized controlled trial reveal smaller less persis-tent effects The 1998 Head Start reauthorization bill included acongressional mandate to determine the effects of the programThis mandate resulted in the HSIS an experiment in which morethan 4000 applicants were randomly assigned via lottery toeither a treatment group with access to Head Start or a controlgroup without access in fall 2002 The experimental resultsshowed that a Head Start offer increased measures of cognitiveachievement by roughly 01 standard deviations during pre-school but these gains faded out by kindergarten Moreoverthe experiment showed little evidence of effects on noncognitiveor health outcomes (Puma et al 2010 Puma Bell and Heid2012) These results suggest both smaller short-run effects andfaster fade-out than nonexperimental estimates for earlier co-horts Scholars and policy makers have generally interpretedthe HSIS results as evidence that Head Start is ineffective andin need of reform (Barnett 2011) The experimental results havealso been cited in the popular media to motivate calls for dramatic

QUARTERLY JOURNAL OF ECONOMICS1800

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

restructuring or elimination of the program (Klein 2011 Stossel2014)1

Differences between the HSIS results and the nonexperi-mental literature could be due to changes in program effective-ness over time or to selection bias in nonexperimental siblingcomparisons Another explanation however is that these tworesearch designs identify different parameters Most nonexperi-mental analyses have focused on recovering the effect of HeadStart relative to home care In contrast the HSIS measures theeffect of Head Start relative to a mix of alternative care environ-ments including other preschools

III Data and Experimental Impacts

Before turning to an analysis of program substitution issueswe describe the HSIS data and report experimental impacts ontest scores and program compliance

IIIA Data

Our core analysis sample includes 3571 HSIS applicantswith nonmissing baseline characteristics and spring 2003 testscores Online Appendix A describes construction of this sampleThe outcome of interest is a summary index of cognitive testscores that averages Woodcock Johnson III (WJIII) test scoreswith Peabody Picture and Vocabulary Test (PPVT) scores witheach test normed to have mean 0 and variance 1 in the controlgroup by cohort and year We use WJIII and PPVT scores becausethese are among the most reliable tests in the HSIS data both areavailable in each year of the experiment allowing us to producecomparable estimates over time

1 Subsequent analyses of the HSIS data suggest caveats to this negative in-terpretation but do not overturn the finding of modest mean test score impactsaccompanied by rapid fade-out Gelber and Isen (2013) find persistent effects onparental engagement with children Bitler Domina and Hoynes (2014) find largerexperimental impacts at low quantiles of the test score distribution These quantiletreatment effects fade out by first grade though there is some evidence of persistenteffects at the bottom of the distribution for Spanish speakers Walters (2015) findsevidence of substantial heterogeneity in impacts across experimental sites and in-vestigates the relationship between this heterogeneity and observed program char-acteristics Walters finds smaller effects for Head Start centers that draw morechildren from other preschools rather than home care a finding we explore inmore detail here

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1801

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Table I provides summary statistics for our analysis sampleThe HSIS experiment included two age cohorts 55 of applicantswere randomized at age three and could attend Head Start for upto two years while the remaining 45 were randomized at age fourand could attend for up to one year The demographic informationin Table I shows that the Head Start population is disadvantagedLess than half of Head Start applicants live in two-parent house-holds and the average applicantrsquos household earns about 90 ofthe federal poverty line Column (2) of Table I compares these andother baseline characteristics for the HSIS treatment and controlgroups to check balance in randomization The results here indi-cate that randomization was successful baseline characteristicswere similar for offered and non offered applicants2

Columns (3) through (5) of Table I report summary statisticsfor children attending Head Start other preschool centers andno preschool3 Children in other preschools tend to be less disad-vantaged than children in Head Start or no preschool thoughmost differences between these groups are modest The otherpreschool group has a lower share of high school dropout mothersa higher share of mothers who attended college and higher av-erage household income than the Head Start and no preschoolgroups Children in other preschools outscore the other groups byabout 01 standard deviations on a baseline summary index ofcognitive skills The other preschool group also includes a rela-tively large share of four-year-olds likely reflecting the fact thatalternative preschool options are more widely available for four-year-olds (Cascio and Schanzenbach 2013)

IIIB Experimental Impacts

Table II reports experimental impacts on test scoresColumns (1) (4) and (7) report intent-to-treat impacts of the

2 Random assignment in the HSIS occurred at the Head Start center leveland offer probabilities differed across centers We weight all models by the inverseprobability of a childrsquos assignment calculated as the site-specific fraction of chil-dren assigned to the treatment group

3 Preschool attendance is measured from the HSIS lsquolsquofocal arrangement typersquorsquovariable which combines information from parent interviews and teachercareprovider interviews to construct a summary measure of the childcare setting SeeOnline Appendix A for details

QUARTERLY JOURNAL OF ECONOMICS1802

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

DE

SC

RIP

TIV

ES

TA

TIS

TIC

S

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

Male

04

94

05

05

00

11

05

01

05

06

04

92

(00

19)

Bla

ck03

08

02

98

00

10

03

17

03

53

02

50

(00

10)

His

pan

ic03

76

03

69

00

07

03

80

03

54

03

73

(00

10)

Tee

nm

oth

er01

59

01

74

00

15

01

59

01

69

01

76

(00

14)

Mot

her

marr

ied

04

36

04

48

00

11

04

39

04

20

04

60

(00

17)

Bot

hp

are

nts

inh

ouse

hol

d04

97

04

88

00

09

04

97

04

68

04

99

(00

17)

Mot

her

ish

igh

sch

ool

dro

pou

t03

68

03

97

00

29

03

77

03

22

04

26

(00

17)

Mot

her

att

end

edso

me

coll

ege

02

98

02

81

00

17

02

93

03

42

02

53

(00

16)

Sp

an

ish

spea

ker

02

87

02

73

00

14

02

96

02

74

02

60

(00

11)

Sp

ecia

led

uca

tion

01

36

01

08

00

28

01

34

01

45

00

91

(00

11)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1803

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

(CO

NT

INU

ED)

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

On

lych

ild

01

61

01

39

00

22

01

51

01

90

01

23

(00

12)

Inco

me

(fra

ctio

nof

FP

L)

08

96

08

96

00

00

08

92

09

83

08

51

(00

24)

Age

4co

hor

t04

48

04

51

00

03

04

26

05

67

04

13

(00

12)

Base

lin

esu

mm

ary

ind

ex00

03

00

12

00

09

00

01

01

06

00

40

(00

27)

Urb

an

08

33

08

35

00

02

08

34

08

59

08

19

(00

03)

Cen

ter

pro

vid

estr

an

spor

tati

on06

06

06

04

00

02

05

86

06

14

06

28

(00

05)

Cen

ter

qu

ali

tyin

dex

04

65

04

70

00

05

04

52

04

74

04

88

(00

05)

Joi

nt

p-v

alu

e05

06

N22

56

13

15

35

71

20

43

598

930

Not

es

All

stati

stic

sw

eigh

tby

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tS

tan

dard

erro

rsare

clu

ster

edat

the

cen

ter

level

T

he

tran

spor

tati

onan

dqu

ali

tyin

dex

vari

able

sre

fer

toa

chil

drsquos

Hea

dS

tart

cen

ter

ofra

nd

omass

ign

men

tT

he

qu

ali

tyvari

able

com

bin

esin

form

ati

onon

cen

ter

chara

cter

isti

cs(t

each

eran

dce

nte

rd

irec

tor

edu

cati

onan

dqu

ali

fica

tion

scl

ass

size

)an

dp

ract

ices

(vari

ety

ofli

tera

cyan

dm

ath

act

ivit

ies

hom

evis

itin

g

hea

lth

an

dn

utr

itio

n)

Inco

me

ism

issi

ng

for

19

ofob

serv

ati

ons

Mis

sin

gvalu

esare

excl

ud

edin

stati

stic

sfo

rin

com

eT

he

base

lin

esu

mm

ary

ind

exis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

infa

ll2002

wit

hea

chsc

ore

stan

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tT

he

join

tp

-valu

eis

from

ate

stof

the

hyp

oth

esis

that

all

coef

fici

ents

equ

al

0

QUARTERLY JOURNAL OF ECONOMICS1804

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

EX

PE

RIM

EN

TA

LIM

PA

CT

SO

NT

ES

TS

CO

RE

S

Th

ree-

yea

r-ol

dco

hor

tF

our-

yea

r-ol

dco

hor

tC

ohor

tsp

oole

d

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

Tim

ep

erio

dR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IV

Yea

r1

01

94

06

99

02

78

01

41

06

63

02

13

01

68

06

82

02

47

(00

29)

(00

25)

(00

41)

(00

29)

(00

22)

(00

44)

(00

21)

(00

18)

(00

31)

N19

70

16

01

35

71

Yea

r2

00

87

03

56

02

45

00

15

06

70

00

22

00

46

04

97

00

93

(00

29)

(00

28)

(00

80)

(00

37)

(00

23)

(00

54)

(00

24)

(00

20)

(00

49)

N17

60

14

16

31

76

Yea

r3

00

10

03

65

00

27

00

54

06

66

00

81

00

19

05

00

00

38

(00

31)

(00

28)

(00

85)

(00

40)

(00

25)

(00

60)

(00

25)

(00

20)

(00

50)

N16

59

13

36

29

95

Yea

r4

00

38

03

44

01

10

mdashmdash

(00

34)

(00

29)

(00

98)

N15

99

Not

es

Th

ista

ble

rep

orts

exp

erim

enta

les

tim

ate

sof

the

effe

cts

ofH

ead

Sta

rton

test

scor

es

Th

eou

tcom

eis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

w

ith

each

scor

est

an

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tan

dyea

rC

olu

mn

s(1

)(4

)an

d(7

)re

por

tco

effi

cien

tsfr

omre

gre

ssio

ns

ofte

stsc

ores

onan

ind

icato

rfo

rass

ign

men

tto

Hea

dS

tart

C

olu

mn

s(2

)(5

)an

d(8

)re

por

tco

effi

cien

tsfr

omfi

rst-

stage

regre

ssio

ns

ofH

ead

Sta

rtatt

end

an

ceon

Hea

dS

tart

ass

ign

men

tT

he

att

end

an

cevari

able

isan

ind

icato

req

ual

to1

ifa

chil

datt

end

sH

ead

Sta

rtat

an

yti

me

pri

orto

the

test

C

olu

mn

s(3

)(6

)an

d(9

)re

por

tco

effi

cien

tsfr

omtw

o-st

age

least

squ

are

s(2

SL

S)

mod

els

that

inst

rum

ent

Hea

dS

tart

att

end

an

cew

ith

Hea

dS

tart

ass

ign

men

tA

llm

odel

sw

eigh

tby

the

reci

pro

cal

ofa

chil

drsquos

exp

erim

enta

lass

ign

-m

ent

an

dco

ntr

olfo

rse

x

race

S

pan

ish

lan

gu

age

teen

mot

her

m

oth

errsquos

mari

tal

statu

sp

rese

nce

ofbot

hp

are

nts

inth

eh

ome

fam

ily

size

sp

ecia

led

uca

tion

statu

sin

com

equ

art

ile

du

mm

ies

urb

an

an

da

cubic

pol

yn

omia

lin

base

lin

esc

ore

Mis

sin

gvalu

esfo

rco

vari

ate

sare

set

to0

an

dd

um

mie

sfo

rm

issi

ng

are

incl

ud

ed

Sta

nd

ard

erro

rsare

clu

ster

edby

cen

ter

ofra

nd

omass

ign

men

t

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1805

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Head Start offer separately by year and age cohort To increaseprecision we regression-adjust these treatmentcontrol differ-ences using the baseline characteristics in Table I4 Theintent-to-treat estimates mirror those previously reported inthe literature (eg Puma et al 2010) In the first year of theexperiment children offered Head Start scored higher on thesummary index For example three-year-olds offered HeadStart gained 019 standard deviations in test score outcomesrelative to those denied Head Start The corresponding effectfor four-year-olds is 014 standard deviations However thesegains diminish rapidly the pooled impact falls to a statisticallyinsignificant 002 standard deviations by year 3 Our data in-clude a fourth year of follow-up for the three-year-old cohortHere too the intent-to-treat is small and statistically insignif-icant (0038 standard deviations)

Interpretation of these intent-to-treat impacts is clouded bynoncompliance with random assignment Columns (2) (5) and(8) of Table II report first-stage effects of assignment to HeadStart on the probability of participating in Head Start and col-umns (3) (6) and (9) report instrumental variables (IV) esti-mates which scale the intent-to-treat estimates by the first-stage estimates5 These estimates can be interpreted as local av-erage treatment effects (LATEs) for lsquolsquocompliersrsquorsquomdashchildren whorespond to the Head Start offer by enrolling in Head StartAssignment to Head Start increases the probability of participa-tion by two thirds in the first year after random assignment The

4 The control vector includes sex race assignment cohort teen mothermotherrsquos education motherrsquos marital status presence of both parents an onlychild dummy a Spanish language indicator dummies for quartiles of familyincome and missing income urban status an indicator for whether the HeadStart center provides transportation an index of Head Start center quality and athird-order polynomial in baseline test scores

5 Here we define Head Start participation as enrollment at any time prior tothe test This definition includes attendance at Head Start centers outside the ex-perimental sample An experimental offer may cause some children to switch froman out-of-sample center to an experimental center if the quality of these centersdiffers the exclusion restriction required for our IV approach is violated OnlineAppendix Table AI compares characteristics of centers attended by children in thecontrol group (always takers) to those of the experimental centers to which thesechildren applied These two groups of centers are very similar suggesting thatsubstitution between Head Start centers is unlikely to bias our estimates

QUARTERLY JOURNAL OF ECONOMICS1806

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

corresponding IV estimate implies that Head Start attendanceboosts first-year test scores by 0247 standard deviations

Compliance for the three-year-old cohort falls after the firstyear as members of the control group reapply for Head Startresulting in larger standard errors for estimates in later yearsof the experiment The first stage for three-year-olds falls to 036in the second year whereas the intent-to-treat falls roughly inproportion generating a second-year IV estimate of 0245 for thiscohort Estimates in years 3 and 4 are statistically insignificantand imprecise The fourth-year estimate for the three-year-oldcohort (corresponding to first grade) is 0110 standard deviationswith a standard error of 0098 The corresponding first grade es-timate for four-year-olds is 0081 with a standard error of 0060Notably the 95 confidence intervals for first-grade impacts in-clude effects as large as 02 standard deviations for four-year-oldsand 03 standard deviations for three-year-olds These resultsshow that although the longer run estimates are insignificantthey are also imprecise due to experimental noncomplianceEvidence for fade-out is therefore less definitive than the naiveintent-to-treat estimates suggest This observation helps to rec-oncile the HSIS results with observational studies based on sib-ling comparisons which show effects that partially fade out butare still detectable in elementary school (Currie and Thomas1995 Deming 2009)6

IV Program Substitution

We turn now to documenting program substitution in theHSIS and how it influences our results It is helpful to developsome notation to describe the role of alternative care environ-ments Each Head Start applicant participates in one of three pos-sible treatments Head Start which we label h other center-based

6 One might also be interested in the effects of Head Start on noncognitiveoutcomes which appear to be important mediators of the effects of early childhoodprograms in other contexts (Chetty et al 2011 Heckman Pinto and Savelyev2013) The HSIS includes short-run parent-reported measures of behavior andteacher-reported measures of teacherndashstudent relationships and Head Start ap-pears to have no impact on these outcomes (Puma et al 2010 Walters 2015) TheHSIS noncognitive outcomes differ significantly from those analyzed in previousstudies however and it is unclear whether they capture the same skills

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1807

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 6: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

their public preschool offerings (Cascio and Schanzenbach 2013)Moreover the Child Care Development Fund program providesblock grants that finance childcare subsidies for low-income fam-ilies often in the form of child care vouchers that can be used forcenter-based preschool (US DHHS 2012) Most states also useTANF funds to finance additional child care subsidies(Schumacher Greenberg and Duffy 2001) Because Head Startservices are provided by local organizations who themselves mustraise outside funds it is unclear to what extent Head Start andother public preschool programs actually differ in their educationtechnology

A large nonexperimental literature suggests that Head Startproduced large short- and long-run benefits for early cohorts ofprogram participants Several studies estimate the effects ofHead Start by comparing program participants to their nonpar-ticipant siblings (Currie and Thomas 1995 Garces Thomas andCurrie 2002 Deming 2009) Results from this research designshow positive short-run effects on test scores and long-run effectson educational attainment earnings and crime Other studiesexploit discontinuities in Head Start program rules to infer pro-gram effects (Ludwig and Miller 2007 Carneiro and Ginja 2014)These studies show longer run improvements in health outcomesand criminal activity

In contrast to these nonexperimental estimates results froma recent randomized controlled trial reveal smaller less persis-tent effects The 1998 Head Start reauthorization bill included acongressional mandate to determine the effects of the programThis mandate resulted in the HSIS an experiment in which morethan 4000 applicants were randomly assigned via lottery toeither a treatment group with access to Head Start or a controlgroup without access in fall 2002 The experimental resultsshowed that a Head Start offer increased measures of cognitiveachievement by roughly 01 standard deviations during pre-school but these gains faded out by kindergarten Moreoverthe experiment showed little evidence of effects on noncognitiveor health outcomes (Puma et al 2010 Puma Bell and Heid2012) These results suggest both smaller short-run effects andfaster fade-out than nonexperimental estimates for earlier co-horts Scholars and policy makers have generally interpretedthe HSIS results as evidence that Head Start is ineffective andin need of reform (Barnett 2011) The experimental results havealso been cited in the popular media to motivate calls for dramatic

QUARTERLY JOURNAL OF ECONOMICS1800

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

restructuring or elimination of the program (Klein 2011 Stossel2014)1

Differences between the HSIS results and the nonexperi-mental literature could be due to changes in program effective-ness over time or to selection bias in nonexperimental siblingcomparisons Another explanation however is that these tworesearch designs identify different parameters Most nonexperi-mental analyses have focused on recovering the effect of HeadStart relative to home care In contrast the HSIS measures theeffect of Head Start relative to a mix of alternative care environ-ments including other preschools

III Data and Experimental Impacts

Before turning to an analysis of program substitution issueswe describe the HSIS data and report experimental impacts ontest scores and program compliance

IIIA Data

Our core analysis sample includes 3571 HSIS applicantswith nonmissing baseline characteristics and spring 2003 testscores Online Appendix A describes construction of this sampleThe outcome of interest is a summary index of cognitive testscores that averages Woodcock Johnson III (WJIII) test scoreswith Peabody Picture and Vocabulary Test (PPVT) scores witheach test normed to have mean 0 and variance 1 in the controlgroup by cohort and year We use WJIII and PPVT scores becausethese are among the most reliable tests in the HSIS data both areavailable in each year of the experiment allowing us to producecomparable estimates over time

1 Subsequent analyses of the HSIS data suggest caveats to this negative in-terpretation but do not overturn the finding of modest mean test score impactsaccompanied by rapid fade-out Gelber and Isen (2013) find persistent effects onparental engagement with children Bitler Domina and Hoynes (2014) find largerexperimental impacts at low quantiles of the test score distribution These quantiletreatment effects fade out by first grade though there is some evidence of persistenteffects at the bottom of the distribution for Spanish speakers Walters (2015) findsevidence of substantial heterogeneity in impacts across experimental sites and in-vestigates the relationship between this heterogeneity and observed program char-acteristics Walters finds smaller effects for Head Start centers that draw morechildren from other preschools rather than home care a finding we explore inmore detail here

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1801

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Table I provides summary statistics for our analysis sampleThe HSIS experiment included two age cohorts 55 of applicantswere randomized at age three and could attend Head Start for upto two years while the remaining 45 were randomized at age fourand could attend for up to one year The demographic informationin Table I shows that the Head Start population is disadvantagedLess than half of Head Start applicants live in two-parent house-holds and the average applicantrsquos household earns about 90 ofthe federal poverty line Column (2) of Table I compares these andother baseline characteristics for the HSIS treatment and controlgroups to check balance in randomization The results here indi-cate that randomization was successful baseline characteristicswere similar for offered and non offered applicants2

Columns (3) through (5) of Table I report summary statisticsfor children attending Head Start other preschool centers andno preschool3 Children in other preschools tend to be less disad-vantaged than children in Head Start or no preschool thoughmost differences between these groups are modest The otherpreschool group has a lower share of high school dropout mothersa higher share of mothers who attended college and higher av-erage household income than the Head Start and no preschoolgroups Children in other preschools outscore the other groups byabout 01 standard deviations on a baseline summary index ofcognitive skills The other preschool group also includes a rela-tively large share of four-year-olds likely reflecting the fact thatalternative preschool options are more widely available for four-year-olds (Cascio and Schanzenbach 2013)

IIIB Experimental Impacts

Table II reports experimental impacts on test scoresColumns (1) (4) and (7) report intent-to-treat impacts of the

2 Random assignment in the HSIS occurred at the Head Start center leveland offer probabilities differed across centers We weight all models by the inverseprobability of a childrsquos assignment calculated as the site-specific fraction of chil-dren assigned to the treatment group

3 Preschool attendance is measured from the HSIS lsquolsquofocal arrangement typersquorsquovariable which combines information from parent interviews and teachercareprovider interviews to construct a summary measure of the childcare setting SeeOnline Appendix A for details

QUARTERLY JOURNAL OF ECONOMICS1802

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

DE

SC

RIP

TIV

ES

TA

TIS

TIC

S

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

Male

04

94

05

05

00

11

05

01

05

06

04

92

(00

19)

Bla

ck03

08

02

98

00

10

03

17

03

53

02

50

(00

10)

His

pan

ic03

76

03

69

00

07

03

80

03

54

03

73

(00

10)

Tee

nm

oth

er01

59

01

74

00

15

01

59

01

69

01

76

(00

14)

Mot

her

marr

ied

04

36

04

48

00

11

04

39

04

20

04

60

(00

17)

Bot

hp

are

nts

inh

ouse

hol

d04

97

04

88

00

09

04

97

04

68

04

99

(00

17)

Mot

her

ish

igh

sch

ool

dro

pou

t03

68

03

97

00

29

03

77

03

22

04

26

(00

17)

Mot

her

att

end

edso

me

coll

ege

02

98

02

81

00

17

02

93

03

42

02

53

(00

16)

Sp

an

ish

spea

ker

02

87

02

73

00

14

02

96

02

74

02

60

(00

11)

Sp

ecia

led

uca

tion

01

36

01

08

00

28

01

34

01

45

00

91

(00

11)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1803

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

(CO

NT

INU

ED)

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

On

lych

ild

01

61

01

39

00

22

01

51

01

90

01

23

(00

12)

Inco

me

(fra

ctio

nof

FP

L)

08

96

08

96

00

00

08

92

09

83

08

51

(00

24)

Age

4co

hor

t04

48

04

51

00

03

04

26

05

67

04

13

(00

12)

Base

lin

esu

mm

ary

ind

ex00

03

00

12

00

09

00

01

01

06

00

40

(00

27)

Urb

an

08

33

08

35

00

02

08

34

08

59

08

19

(00

03)

Cen

ter

pro

vid

estr

an

spor

tati

on06

06

06

04

00

02

05

86

06

14

06

28

(00

05)

Cen

ter

qu

ali

tyin

dex

04

65

04

70

00

05

04

52

04

74

04

88

(00

05)

Joi

nt

p-v

alu

e05

06

N22

56

13

15

35

71

20

43

598

930

Not

es

All

stati

stic

sw

eigh

tby

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tS

tan

dard

erro

rsare

clu

ster

edat

the

cen

ter

level

T

he

tran

spor

tati

onan

dqu

ali

tyin

dex

vari

able

sre

fer

toa

chil

drsquos

Hea

dS

tart

cen

ter

ofra

nd

omass

ign

men

tT

he

qu

ali

tyvari

able

com

bin

esin

form

ati

onon

cen

ter

chara

cter

isti

cs(t

each

eran

dce

nte

rd

irec

tor

edu

cati

onan

dqu

ali

fica

tion

scl

ass

size

)an

dp

ract

ices

(vari

ety

ofli

tera

cyan

dm

ath

act

ivit

ies

hom

evis

itin

g

hea

lth

an

dn

utr

itio

n)

Inco

me

ism

issi

ng

for

19

ofob

serv

ati

ons

Mis

sin

gvalu

esare

excl

ud

edin

stati

stic

sfo

rin

com

eT

he

base

lin

esu

mm

ary

ind

exis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

infa

ll2002

wit

hea

chsc

ore

stan

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tT

he

join

tp

-valu

eis

from

ate

stof

the

hyp

oth

esis

that

all

coef

fici

ents

equ

al

0

QUARTERLY JOURNAL OF ECONOMICS1804

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

EX

PE

RIM

EN

TA

LIM

PA

CT

SO

NT

ES

TS

CO

RE

S

Th

ree-

yea

r-ol

dco

hor

tF

our-

yea

r-ol

dco

hor

tC

ohor

tsp

oole

d

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

Tim

ep

erio

dR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IV

Yea

r1

01

94

06

99

02

78

01

41

06

63

02

13

01

68

06

82

02

47

(00

29)

(00

25)

(00

41)

(00

29)

(00

22)

(00

44)

(00

21)

(00

18)

(00

31)

N19

70

16

01

35

71

Yea

r2

00

87

03

56

02

45

00

15

06

70

00

22

00

46

04

97

00

93

(00

29)

(00

28)

(00

80)

(00

37)

(00

23)

(00

54)

(00

24)

(00

20)

(00

49)

N17

60

14

16

31

76

Yea

r3

00

10

03

65

00

27

00

54

06

66

00

81

00

19

05

00

00

38

(00

31)

(00

28)

(00

85)

(00

40)

(00

25)

(00

60)

(00

25)

(00

20)

(00

50)

N16

59

13

36

29

95

Yea

r4

00

38

03

44

01

10

mdashmdash

(00

34)

(00

29)

(00

98)

N15

99

Not

es

Th

ista

ble

rep

orts

exp

erim

enta

les

tim

ate

sof

the

effe

cts

ofH

ead

Sta

rton

test

scor

es

Th

eou

tcom

eis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

w

ith

each

scor

est

an

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tan

dyea

rC

olu

mn

s(1

)(4

)an

d(7

)re

por

tco

effi

cien

tsfr

omre

gre

ssio

ns

ofte

stsc

ores

onan

ind

icato

rfo

rass

ign

men

tto

Hea

dS

tart

C

olu

mn

s(2

)(5

)an

d(8

)re

por

tco

effi

cien

tsfr

omfi

rst-

stage

regre

ssio

ns

ofH

ead

Sta

rtatt

end

an

ceon

Hea

dS

tart

ass

ign

men

tT

he

att

end

an

cevari

able

isan

ind

icato

req

ual

to1

ifa

chil

datt

end

sH

ead

Sta

rtat

an

yti

me

pri

orto

the

test

C

olu

mn

s(3

)(6

)an

d(9

)re

por

tco

effi

cien

tsfr

omtw

o-st

age

least

squ

are

s(2

SL

S)

mod

els

that

inst

rum

ent

Hea

dS

tart

att

end

an

cew

ith

Hea

dS

tart

ass

ign

men

tA

llm

odel

sw

eigh

tby

the

reci

pro

cal

ofa

chil

drsquos

exp

erim

enta

lass

ign

-m

ent

an

dco

ntr

olfo

rse

x

race

S

pan

ish

lan

gu

age

teen

mot

her

m

oth

errsquos

mari

tal

statu

sp

rese

nce

ofbot

hp

are

nts

inth

eh

ome

fam

ily

size

sp

ecia

led

uca

tion

statu

sin

com

equ

art

ile

du

mm

ies

urb

an

an

da

cubic

pol

yn

omia

lin

base

lin

esc

ore

Mis

sin

gvalu

esfo

rco

vari

ate

sare

set

to0

an

dd

um

mie

sfo

rm

issi

ng

are

incl

ud

ed

Sta

nd

ard

erro

rsare

clu

ster

edby

cen

ter

ofra

nd

omass

ign

men

t

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1805

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Head Start offer separately by year and age cohort To increaseprecision we regression-adjust these treatmentcontrol differ-ences using the baseline characteristics in Table I4 Theintent-to-treat estimates mirror those previously reported inthe literature (eg Puma et al 2010) In the first year of theexperiment children offered Head Start scored higher on thesummary index For example three-year-olds offered HeadStart gained 019 standard deviations in test score outcomesrelative to those denied Head Start The corresponding effectfor four-year-olds is 014 standard deviations However thesegains diminish rapidly the pooled impact falls to a statisticallyinsignificant 002 standard deviations by year 3 Our data in-clude a fourth year of follow-up for the three-year-old cohortHere too the intent-to-treat is small and statistically insignif-icant (0038 standard deviations)

Interpretation of these intent-to-treat impacts is clouded bynoncompliance with random assignment Columns (2) (5) and(8) of Table II report first-stage effects of assignment to HeadStart on the probability of participating in Head Start and col-umns (3) (6) and (9) report instrumental variables (IV) esti-mates which scale the intent-to-treat estimates by the first-stage estimates5 These estimates can be interpreted as local av-erage treatment effects (LATEs) for lsquolsquocompliersrsquorsquomdashchildren whorespond to the Head Start offer by enrolling in Head StartAssignment to Head Start increases the probability of participa-tion by two thirds in the first year after random assignment The

4 The control vector includes sex race assignment cohort teen mothermotherrsquos education motherrsquos marital status presence of both parents an onlychild dummy a Spanish language indicator dummies for quartiles of familyincome and missing income urban status an indicator for whether the HeadStart center provides transportation an index of Head Start center quality and athird-order polynomial in baseline test scores

5 Here we define Head Start participation as enrollment at any time prior tothe test This definition includes attendance at Head Start centers outside the ex-perimental sample An experimental offer may cause some children to switch froman out-of-sample center to an experimental center if the quality of these centersdiffers the exclusion restriction required for our IV approach is violated OnlineAppendix Table AI compares characteristics of centers attended by children in thecontrol group (always takers) to those of the experimental centers to which thesechildren applied These two groups of centers are very similar suggesting thatsubstitution between Head Start centers is unlikely to bias our estimates

QUARTERLY JOURNAL OF ECONOMICS1806

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

corresponding IV estimate implies that Head Start attendanceboosts first-year test scores by 0247 standard deviations

Compliance for the three-year-old cohort falls after the firstyear as members of the control group reapply for Head Startresulting in larger standard errors for estimates in later yearsof the experiment The first stage for three-year-olds falls to 036in the second year whereas the intent-to-treat falls roughly inproportion generating a second-year IV estimate of 0245 for thiscohort Estimates in years 3 and 4 are statistically insignificantand imprecise The fourth-year estimate for the three-year-oldcohort (corresponding to first grade) is 0110 standard deviationswith a standard error of 0098 The corresponding first grade es-timate for four-year-olds is 0081 with a standard error of 0060Notably the 95 confidence intervals for first-grade impacts in-clude effects as large as 02 standard deviations for four-year-oldsand 03 standard deviations for three-year-olds These resultsshow that although the longer run estimates are insignificantthey are also imprecise due to experimental noncomplianceEvidence for fade-out is therefore less definitive than the naiveintent-to-treat estimates suggest This observation helps to rec-oncile the HSIS results with observational studies based on sib-ling comparisons which show effects that partially fade out butare still detectable in elementary school (Currie and Thomas1995 Deming 2009)6

IV Program Substitution

We turn now to documenting program substitution in theHSIS and how it influences our results It is helpful to developsome notation to describe the role of alternative care environ-ments Each Head Start applicant participates in one of three pos-sible treatments Head Start which we label h other center-based

6 One might also be interested in the effects of Head Start on noncognitiveoutcomes which appear to be important mediators of the effects of early childhoodprograms in other contexts (Chetty et al 2011 Heckman Pinto and Savelyev2013) The HSIS includes short-run parent-reported measures of behavior andteacher-reported measures of teacherndashstudent relationships and Head Start ap-pears to have no impact on these outcomes (Puma et al 2010 Walters 2015) TheHSIS noncognitive outcomes differ significantly from those analyzed in previousstudies however and it is unclear whether they capture the same skills

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1807

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 7: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

restructuring or elimination of the program (Klein 2011 Stossel2014)1

Differences between the HSIS results and the nonexperi-mental literature could be due to changes in program effective-ness over time or to selection bias in nonexperimental siblingcomparisons Another explanation however is that these tworesearch designs identify different parameters Most nonexperi-mental analyses have focused on recovering the effect of HeadStart relative to home care In contrast the HSIS measures theeffect of Head Start relative to a mix of alternative care environ-ments including other preschools

III Data and Experimental Impacts

Before turning to an analysis of program substitution issueswe describe the HSIS data and report experimental impacts ontest scores and program compliance

IIIA Data

Our core analysis sample includes 3571 HSIS applicantswith nonmissing baseline characteristics and spring 2003 testscores Online Appendix A describes construction of this sampleThe outcome of interest is a summary index of cognitive testscores that averages Woodcock Johnson III (WJIII) test scoreswith Peabody Picture and Vocabulary Test (PPVT) scores witheach test normed to have mean 0 and variance 1 in the controlgroup by cohort and year We use WJIII and PPVT scores becausethese are among the most reliable tests in the HSIS data both areavailable in each year of the experiment allowing us to producecomparable estimates over time

1 Subsequent analyses of the HSIS data suggest caveats to this negative in-terpretation but do not overturn the finding of modest mean test score impactsaccompanied by rapid fade-out Gelber and Isen (2013) find persistent effects onparental engagement with children Bitler Domina and Hoynes (2014) find largerexperimental impacts at low quantiles of the test score distribution These quantiletreatment effects fade out by first grade though there is some evidence of persistenteffects at the bottom of the distribution for Spanish speakers Walters (2015) findsevidence of substantial heterogeneity in impacts across experimental sites and in-vestigates the relationship between this heterogeneity and observed program char-acteristics Walters finds smaller effects for Head Start centers that draw morechildren from other preschools rather than home care a finding we explore inmore detail here

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1801

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Table I provides summary statistics for our analysis sampleThe HSIS experiment included two age cohorts 55 of applicantswere randomized at age three and could attend Head Start for upto two years while the remaining 45 were randomized at age fourand could attend for up to one year The demographic informationin Table I shows that the Head Start population is disadvantagedLess than half of Head Start applicants live in two-parent house-holds and the average applicantrsquos household earns about 90 ofthe federal poverty line Column (2) of Table I compares these andother baseline characteristics for the HSIS treatment and controlgroups to check balance in randomization The results here indi-cate that randomization was successful baseline characteristicswere similar for offered and non offered applicants2

Columns (3) through (5) of Table I report summary statisticsfor children attending Head Start other preschool centers andno preschool3 Children in other preschools tend to be less disad-vantaged than children in Head Start or no preschool thoughmost differences between these groups are modest The otherpreschool group has a lower share of high school dropout mothersa higher share of mothers who attended college and higher av-erage household income than the Head Start and no preschoolgroups Children in other preschools outscore the other groups byabout 01 standard deviations on a baseline summary index ofcognitive skills The other preschool group also includes a rela-tively large share of four-year-olds likely reflecting the fact thatalternative preschool options are more widely available for four-year-olds (Cascio and Schanzenbach 2013)

IIIB Experimental Impacts

Table II reports experimental impacts on test scoresColumns (1) (4) and (7) report intent-to-treat impacts of the

2 Random assignment in the HSIS occurred at the Head Start center leveland offer probabilities differed across centers We weight all models by the inverseprobability of a childrsquos assignment calculated as the site-specific fraction of chil-dren assigned to the treatment group

3 Preschool attendance is measured from the HSIS lsquolsquofocal arrangement typersquorsquovariable which combines information from parent interviews and teachercareprovider interviews to construct a summary measure of the childcare setting SeeOnline Appendix A for details

QUARTERLY JOURNAL OF ECONOMICS1802

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

DE

SC

RIP

TIV

ES

TA

TIS

TIC

S

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

Male

04

94

05

05

00

11

05

01

05

06

04

92

(00

19)

Bla

ck03

08

02

98

00

10

03

17

03

53

02

50

(00

10)

His

pan

ic03

76

03

69

00

07

03

80

03

54

03

73

(00

10)

Tee

nm

oth

er01

59

01

74

00

15

01

59

01

69

01

76

(00

14)

Mot

her

marr

ied

04

36

04

48

00

11

04

39

04

20

04

60

(00

17)

Bot

hp

are

nts

inh

ouse

hol

d04

97

04

88

00

09

04

97

04

68

04

99

(00

17)

Mot

her

ish

igh

sch

ool

dro

pou

t03

68

03

97

00

29

03

77

03

22

04

26

(00

17)

Mot

her

att

end

edso

me

coll

ege

02

98

02

81

00

17

02

93

03

42

02

53

(00

16)

Sp

an

ish

spea

ker

02

87

02

73

00

14

02

96

02

74

02

60

(00

11)

Sp

ecia

led

uca

tion

01

36

01

08

00

28

01

34

01

45

00

91

(00

11)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1803

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

(CO

NT

INU

ED)

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

On

lych

ild

01

61

01

39

00

22

01

51

01

90

01

23

(00

12)

Inco

me

(fra

ctio

nof

FP

L)

08

96

08

96

00

00

08

92

09

83

08

51

(00

24)

Age

4co

hor

t04

48

04

51

00

03

04

26

05

67

04

13

(00

12)

Base

lin

esu

mm

ary

ind

ex00

03

00

12

00

09

00

01

01

06

00

40

(00

27)

Urb

an

08

33

08

35

00

02

08

34

08

59

08

19

(00

03)

Cen

ter

pro

vid

estr

an

spor

tati

on06

06

06

04

00

02

05

86

06

14

06

28

(00

05)

Cen

ter

qu

ali

tyin

dex

04

65

04

70

00

05

04

52

04

74

04

88

(00

05)

Joi

nt

p-v

alu

e05

06

N22

56

13

15

35

71

20

43

598

930

Not

es

All

stati

stic

sw

eigh

tby

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tS

tan

dard

erro

rsare

clu

ster

edat

the

cen

ter

level

T

he

tran

spor

tati

onan

dqu

ali

tyin

dex

vari

able

sre

fer

toa

chil

drsquos

Hea

dS

tart

cen

ter

ofra

nd

omass

ign

men

tT

he

qu

ali

tyvari

able

com

bin

esin

form

ati

onon

cen

ter

chara

cter

isti

cs(t

each

eran

dce

nte

rd

irec

tor

edu

cati

onan

dqu

ali

fica

tion

scl

ass

size

)an

dp

ract

ices

(vari

ety

ofli

tera

cyan

dm

ath

act

ivit

ies

hom

evis

itin

g

hea

lth

an

dn

utr

itio

n)

Inco

me

ism

issi

ng

for

19

ofob

serv

ati

ons

Mis

sin

gvalu

esare

excl

ud

edin

stati

stic

sfo

rin

com

eT

he

base

lin

esu

mm

ary

ind

exis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

infa

ll2002

wit

hea

chsc

ore

stan

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tT

he

join

tp

-valu

eis

from

ate

stof

the

hyp

oth

esis

that

all

coef

fici

ents

equ

al

0

QUARTERLY JOURNAL OF ECONOMICS1804

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

EX

PE

RIM

EN

TA

LIM

PA

CT

SO

NT

ES

TS

CO

RE

S

Th

ree-

yea

r-ol

dco

hor

tF

our-

yea

r-ol

dco

hor

tC

ohor

tsp

oole

d

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

Tim

ep

erio

dR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IV

Yea

r1

01

94

06

99

02

78

01

41

06

63

02

13

01

68

06

82

02

47

(00

29)

(00

25)

(00

41)

(00

29)

(00

22)

(00

44)

(00

21)

(00

18)

(00

31)

N19

70

16

01

35

71

Yea

r2

00

87

03

56

02

45

00

15

06

70

00

22

00

46

04

97

00

93

(00

29)

(00

28)

(00

80)

(00

37)

(00

23)

(00

54)

(00

24)

(00

20)

(00

49)

N17

60

14

16

31

76

Yea

r3

00

10

03

65

00

27

00

54

06

66

00

81

00

19

05

00

00

38

(00

31)

(00

28)

(00

85)

(00

40)

(00

25)

(00

60)

(00

25)

(00

20)

(00

50)

N16

59

13

36

29

95

Yea

r4

00

38

03

44

01

10

mdashmdash

(00

34)

(00

29)

(00

98)

N15

99

Not

es

Th

ista

ble

rep

orts

exp

erim

enta

les

tim

ate

sof

the

effe

cts

ofH

ead

Sta

rton

test

scor

es

Th

eou

tcom

eis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

w

ith

each

scor

est

an

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tan

dyea

rC

olu

mn

s(1

)(4

)an

d(7

)re

por

tco

effi

cien

tsfr

omre

gre

ssio

ns

ofte

stsc

ores

onan

ind

icato

rfo

rass

ign

men

tto

Hea

dS

tart

C

olu

mn

s(2

)(5

)an

d(8

)re

por

tco

effi

cien

tsfr

omfi

rst-

stage

regre

ssio

ns

ofH

ead

Sta

rtatt

end

an

ceon

Hea

dS

tart

ass

ign

men

tT

he

att

end

an

cevari

able

isan

ind

icato

req

ual

to1

ifa

chil

datt

end

sH

ead

Sta

rtat

an

yti

me

pri

orto

the

test

C

olu

mn

s(3

)(6

)an

d(9

)re

por

tco

effi

cien

tsfr

omtw

o-st

age

least

squ

are

s(2

SL

S)

mod

els

that

inst

rum

ent

Hea

dS

tart

att

end

an

cew

ith

Hea

dS

tart

ass

ign

men

tA

llm

odel

sw

eigh

tby

the

reci

pro

cal

ofa

chil

drsquos

exp

erim

enta

lass

ign

-m

ent

an

dco

ntr

olfo

rse

x

race

S

pan

ish

lan

gu

age

teen

mot

her

m

oth

errsquos

mari

tal

statu

sp

rese

nce

ofbot

hp

are

nts

inth

eh

ome

fam

ily

size

sp

ecia

led

uca

tion

statu

sin

com

equ

art

ile

du

mm

ies

urb

an

an

da

cubic

pol

yn

omia

lin

base

lin

esc

ore

Mis

sin

gvalu

esfo

rco

vari

ate

sare

set

to0

an

dd

um

mie

sfo

rm

issi

ng

are

incl

ud

ed

Sta

nd

ard

erro

rsare

clu

ster

edby

cen

ter

ofra

nd

omass

ign

men

t

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1805

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Head Start offer separately by year and age cohort To increaseprecision we regression-adjust these treatmentcontrol differ-ences using the baseline characteristics in Table I4 Theintent-to-treat estimates mirror those previously reported inthe literature (eg Puma et al 2010) In the first year of theexperiment children offered Head Start scored higher on thesummary index For example three-year-olds offered HeadStart gained 019 standard deviations in test score outcomesrelative to those denied Head Start The corresponding effectfor four-year-olds is 014 standard deviations However thesegains diminish rapidly the pooled impact falls to a statisticallyinsignificant 002 standard deviations by year 3 Our data in-clude a fourth year of follow-up for the three-year-old cohortHere too the intent-to-treat is small and statistically insignif-icant (0038 standard deviations)

Interpretation of these intent-to-treat impacts is clouded bynoncompliance with random assignment Columns (2) (5) and(8) of Table II report first-stage effects of assignment to HeadStart on the probability of participating in Head Start and col-umns (3) (6) and (9) report instrumental variables (IV) esti-mates which scale the intent-to-treat estimates by the first-stage estimates5 These estimates can be interpreted as local av-erage treatment effects (LATEs) for lsquolsquocompliersrsquorsquomdashchildren whorespond to the Head Start offer by enrolling in Head StartAssignment to Head Start increases the probability of participa-tion by two thirds in the first year after random assignment The

4 The control vector includes sex race assignment cohort teen mothermotherrsquos education motherrsquos marital status presence of both parents an onlychild dummy a Spanish language indicator dummies for quartiles of familyincome and missing income urban status an indicator for whether the HeadStart center provides transportation an index of Head Start center quality and athird-order polynomial in baseline test scores

5 Here we define Head Start participation as enrollment at any time prior tothe test This definition includes attendance at Head Start centers outside the ex-perimental sample An experimental offer may cause some children to switch froman out-of-sample center to an experimental center if the quality of these centersdiffers the exclusion restriction required for our IV approach is violated OnlineAppendix Table AI compares characteristics of centers attended by children in thecontrol group (always takers) to those of the experimental centers to which thesechildren applied These two groups of centers are very similar suggesting thatsubstitution between Head Start centers is unlikely to bias our estimates

QUARTERLY JOURNAL OF ECONOMICS1806

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

corresponding IV estimate implies that Head Start attendanceboosts first-year test scores by 0247 standard deviations

Compliance for the three-year-old cohort falls after the firstyear as members of the control group reapply for Head Startresulting in larger standard errors for estimates in later yearsof the experiment The first stage for three-year-olds falls to 036in the second year whereas the intent-to-treat falls roughly inproportion generating a second-year IV estimate of 0245 for thiscohort Estimates in years 3 and 4 are statistically insignificantand imprecise The fourth-year estimate for the three-year-oldcohort (corresponding to first grade) is 0110 standard deviationswith a standard error of 0098 The corresponding first grade es-timate for four-year-olds is 0081 with a standard error of 0060Notably the 95 confidence intervals for first-grade impacts in-clude effects as large as 02 standard deviations for four-year-oldsand 03 standard deviations for three-year-olds These resultsshow that although the longer run estimates are insignificantthey are also imprecise due to experimental noncomplianceEvidence for fade-out is therefore less definitive than the naiveintent-to-treat estimates suggest This observation helps to rec-oncile the HSIS results with observational studies based on sib-ling comparisons which show effects that partially fade out butare still detectable in elementary school (Currie and Thomas1995 Deming 2009)6

IV Program Substitution

We turn now to documenting program substitution in theHSIS and how it influences our results It is helpful to developsome notation to describe the role of alternative care environ-ments Each Head Start applicant participates in one of three pos-sible treatments Head Start which we label h other center-based

6 One might also be interested in the effects of Head Start on noncognitiveoutcomes which appear to be important mediators of the effects of early childhoodprograms in other contexts (Chetty et al 2011 Heckman Pinto and Savelyev2013) The HSIS includes short-run parent-reported measures of behavior andteacher-reported measures of teacherndashstudent relationships and Head Start ap-pears to have no impact on these outcomes (Puma et al 2010 Walters 2015) TheHSIS noncognitive outcomes differ significantly from those analyzed in previousstudies however and it is unclear whether they capture the same skills

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1807

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 8: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

Table I provides summary statistics for our analysis sampleThe HSIS experiment included two age cohorts 55 of applicantswere randomized at age three and could attend Head Start for upto two years while the remaining 45 were randomized at age fourand could attend for up to one year The demographic informationin Table I shows that the Head Start population is disadvantagedLess than half of Head Start applicants live in two-parent house-holds and the average applicantrsquos household earns about 90 ofthe federal poverty line Column (2) of Table I compares these andother baseline characteristics for the HSIS treatment and controlgroups to check balance in randomization The results here indi-cate that randomization was successful baseline characteristicswere similar for offered and non offered applicants2

Columns (3) through (5) of Table I report summary statisticsfor children attending Head Start other preschool centers andno preschool3 Children in other preschools tend to be less disad-vantaged than children in Head Start or no preschool thoughmost differences between these groups are modest The otherpreschool group has a lower share of high school dropout mothersa higher share of mothers who attended college and higher av-erage household income than the Head Start and no preschoolgroups Children in other preschools outscore the other groups byabout 01 standard deviations on a baseline summary index ofcognitive skills The other preschool group also includes a rela-tively large share of four-year-olds likely reflecting the fact thatalternative preschool options are more widely available for four-year-olds (Cascio and Schanzenbach 2013)

IIIB Experimental Impacts

Table II reports experimental impacts on test scoresColumns (1) (4) and (7) report intent-to-treat impacts of the

2 Random assignment in the HSIS occurred at the Head Start center leveland offer probabilities differed across centers We weight all models by the inverseprobability of a childrsquos assignment calculated as the site-specific fraction of chil-dren assigned to the treatment group

3 Preschool attendance is measured from the HSIS lsquolsquofocal arrangement typersquorsquovariable which combines information from parent interviews and teachercareprovider interviews to construct a summary measure of the childcare setting SeeOnline Appendix A for details

QUARTERLY JOURNAL OF ECONOMICS1802

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

DE

SC

RIP

TIV

ES

TA

TIS

TIC

S

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

Male

04

94

05

05

00

11

05

01

05

06

04

92

(00

19)

Bla

ck03

08

02

98

00

10

03

17

03

53

02

50

(00

10)

His

pan

ic03

76

03

69

00

07

03

80

03

54

03

73

(00

10)

Tee

nm

oth

er01

59

01

74

00

15

01

59

01

69

01

76

(00

14)

Mot

her

marr

ied

04

36

04

48

00

11

04

39

04

20

04

60

(00

17)

Bot

hp

are

nts

inh

ouse

hol

d04

97

04

88

00

09

04

97

04

68

04

99

(00

17)

Mot

her

ish

igh

sch

ool

dro

pou

t03

68

03

97

00

29

03

77

03

22

04

26

(00

17)

Mot

her

att

end

edso

me

coll

ege

02

98

02

81

00

17

02

93

03

42

02

53

(00

16)

Sp

an

ish

spea

ker

02

87

02

73

00

14

02

96

02

74

02

60

(00

11)

Sp

ecia

led

uca

tion

01

36

01

08

00

28

01

34

01

45

00

91

(00

11)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1803

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

(CO

NT

INU

ED)

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

On

lych

ild

01

61

01

39

00

22

01

51

01

90

01

23

(00

12)

Inco

me

(fra

ctio

nof

FP

L)

08

96

08

96

00

00

08

92

09

83

08

51

(00

24)

Age

4co

hor

t04

48

04

51

00

03

04

26

05

67

04

13

(00

12)

Base

lin

esu

mm

ary

ind

ex00

03

00

12

00

09

00

01

01

06

00

40

(00

27)

Urb

an

08

33

08

35

00

02

08

34

08

59

08

19

(00

03)

Cen

ter

pro

vid

estr

an

spor

tati

on06

06

06

04

00

02

05

86

06

14

06

28

(00

05)

Cen

ter

qu

ali

tyin

dex

04

65

04

70

00

05

04

52

04

74

04

88

(00

05)

Joi

nt

p-v

alu

e05

06

N22

56

13

15

35

71

20

43

598

930

Not

es

All

stati

stic

sw

eigh

tby

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tS

tan

dard

erro

rsare

clu

ster

edat

the

cen

ter

level

T

he

tran

spor

tati

onan

dqu

ali

tyin

dex

vari

able

sre

fer

toa

chil

drsquos

Hea

dS

tart

cen

ter

ofra

nd

omass

ign

men

tT

he

qu

ali

tyvari

able

com

bin

esin

form

ati

onon

cen

ter

chara

cter

isti

cs(t

each

eran

dce

nte

rd

irec

tor

edu

cati

onan

dqu

ali

fica

tion

scl

ass

size

)an

dp

ract

ices

(vari

ety

ofli

tera

cyan

dm

ath

act

ivit

ies

hom

evis

itin

g

hea

lth

an

dn

utr

itio

n)

Inco

me

ism

issi

ng

for

19

ofob

serv

ati

ons

Mis

sin

gvalu

esare

excl

ud

edin

stati

stic

sfo

rin

com

eT

he

base

lin

esu

mm

ary

ind

exis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

infa

ll2002

wit

hea

chsc

ore

stan

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tT

he

join

tp

-valu

eis

from

ate

stof

the

hyp

oth

esis

that

all

coef

fici

ents

equ

al

0

QUARTERLY JOURNAL OF ECONOMICS1804

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

EX

PE

RIM

EN

TA

LIM

PA

CT

SO

NT

ES

TS

CO

RE

S

Th

ree-

yea

r-ol

dco

hor

tF

our-

yea

r-ol

dco

hor

tC

ohor

tsp

oole

d

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

Tim

ep

erio

dR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IV

Yea

r1

01

94

06

99

02

78

01

41

06

63

02

13

01

68

06

82

02

47

(00

29)

(00

25)

(00

41)

(00

29)

(00

22)

(00

44)

(00

21)

(00

18)

(00

31)

N19

70

16

01

35

71

Yea

r2

00

87

03

56

02

45

00

15

06

70

00

22

00

46

04

97

00

93

(00

29)

(00

28)

(00

80)

(00

37)

(00

23)

(00

54)

(00

24)

(00

20)

(00

49)

N17

60

14

16

31

76

Yea

r3

00

10

03

65

00

27

00

54

06

66

00

81

00

19

05

00

00

38

(00

31)

(00

28)

(00

85)

(00

40)

(00

25)

(00

60)

(00

25)

(00

20)

(00

50)

N16

59

13

36

29

95

Yea

r4

00

38

03

44

01

10

mdashmdash

(00

34)

(00

29)

(00

98)

N15

99

Not

es

Th

ista

ble

rep

orts

exp

erim

enta

les

tim

ate

sof

the

effe

cts

ofH

ead

Sta

rton

test

scor

es

Th

eou

tcom

eis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

w

ith

each

scor

est

an

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tan

dyea

rC

olu

mn

s(1

)(4

)an

d(7

)re

por

tco

effi

cien

tsfr

omre

gre

ssio

ns

ofte

stsc

ores

onan

ind

icato

rfo

rass

ign

men

tto

Hea

dS

tart

C

olu

mn

s(2

)(5

)an

d(8

)re

por

tco

effi

cien

tsfr

omfi

rst-

stage

regre

ssio

ns

ofH

ead

Sta

rtatt

end

an

ceon

Hea

dS

tart

ass

ign

men

tT

he

att

end

an

cevari

able

isan

ind

icato

req

ual

to1

ifa

chil

datt

end

sH

ead

Sta

rtat

an

yti

me

pri

orto

the

test

C

olu

mn

s(3

)(6

)an

d(9

)re

por

tco

effi

cien

tsfr

omtw

o-st

age

least

squ

are

s(2

SL

S)

mod

els

that

inst

rum

ent

Hea

dS

tart

att

end

an

cew

ith

Hea

dS

tart

ass

ign

men

tA

llm

odel

sw

eigh

tby

the

reci

pro

cal

ofa

chil

drsquos

exp

erim

enta

lass

ign

-m

ent

an

dco

ntr

olfo

rse

x

race

S

pan

ish

lan

gu

age

teen

mot

her

m

oth

errsquos

mari

tal

statu

sp

rese

nce

ofbot

hp

are

nts

inth

eh

ome

fam

ily

size

sp

ecia

led

uca

tion

statu

sin

com

equ

art

ile

du

mm

ies

urb

an

an

da

cubic

pol

yn

omia

lin

base

lin

esc

ore

Mis

sin

gvalu

esfo

rco

vari

ate

sare

set

to0

an

dd

um

mie

sfo

rm

issi

ng

are

incl

ud

ed

Sta

nd

ard

erro

rsare

clu

ster

edby

cen

ter

ofra

nd

omass

ign

men

t

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1805

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Head Start offer separately by year and age cohort To increaseprecision we regression-adjust these treatmentcontrol differ-ences using the baseline characteristics in Table I4 Theintent-to-treat estimates mirror those previously reported inthe literature (eg Puma et al 2010) In the first year of theexperiment children offered Head Start scored higher on thesummary index For example three-year-olds offered HeadStart gained 019 standard deviations in test score outcomesrelative to those denied Head Start The corresponding effectfor four-year-olds is 014 standard deviations However thesegains diminish rapidly the pooled impact falls to a statisticallyinsignificant 002 standard deviations by year 3 Our data in-clude a fourth year of follow-up for the three-year-old cohortHere too the intent-to-treat is small and statistically insignif-icant (0038 standard deviations)

Interpretation of these intent-to-treat impacts is clouded bynoncompliance with random assignment Columns (2) (5) and(8) of Table II report first-stage effects of assignment to HeadStart on the probability of participating in Head Start and col-umns (3) (6) and (9) report instrumental variables (IV) esti-mates which scale the intent-to-treat estimates by the first-stage estimates5 These estimates can be interpreted as local av-erage treatment effects (LATEs) for lsquolsquocompliersrsquorsquomdashchildren whorespond to the Head Start offer by enrolling in Head StartAssignment to Head Start increases the probability of participa-tion by two thirds in the first year after random assignment The

4 The control vector includes sex race assignment cohort teen mothermotherrsquos education motherrsquos marital status presence of both parents an onlychild dummy a Spanish language indicator dummies for quartiles of familyincome and missing income urban status an indicator for whether the HeadStart center provides transportation an index of Head Start center quality and athird-order polynomial in baseline test scores

5 Here we define Head Start participation as enrollment at any time prior tothe test This definition includes attendance at Head Start centers outside the ex-perimental sample An experimental offer may cause some children to switch froman out-of-sample center to an experimental center if the quality of these centersdiffers the exclusion restriction required for our IV approach is violated OnlineAppendix Table AI compares characteristics of centers attended by children in thecontrol group (always takers) to those of the experimental centers to which thesechildren applied These two groups of centers are very similar suggesting thatsubstitution between Head Start centers is unlikely to bias our estimates

QUARTERLY JOURNAL OF ECONOMICS1806

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

corresponding IV estimate implies that Head Start attendanceboosts first-year test scores by 0247 standard deviations

Compliance for the three-year-old cohort falls after the firstyear as members of the control group reapply for Head Startresulting in larger standard errors for estimates in later yearsof the experiment The first stage for three-year-olds falls to 036in the second year whereas the intent-to-treat falls roughly inproportion generating a second-year IV estimate of 0245 for thiscohort Estimates in years 3 and 4 are statistically insignificantand imprecise The fourth-year estimate for the three-year-oldcohort (corresponding to first grade) is 0110 standard deviationswith a standard error of 0098 The corresponding first grade es-timate for four-year-olds is 0081 with a standard error of 0060Notably the 95 confidence intervals for first-grade impacts in-clude effects as large as 02 standard deviations for four-year-oldsand 03 standard deviations for three-year-olds These resultsshow that although the longer run estimates are insignificantthey are also imprecise due to experimental noncomplianceEvidence for fade-out is therefore less definitive than the naiveintent-to-treat estimates suggest This observation helps to rec-oncile the HSIS results with observational studies based on sib-ling comparisons which show effects that partially fade out butare still detectable in elementary school (Currie and Thomas1995 Deming 2009)6

IV Program Substitution

We turn now to documenting program substitution in theHSIS and how it influences our results It is helpful to developsome notation to describe the role of alternative care environ-ments Each Head Start applicant participates in one of three pos-sible treatments Head Start which we label h other center-based

6 One might also be interested in the effects of Head Start on noncognitiveoutcomes which appear to be important mediators of the effects of early childhoodprograms in other contexts (Chetty et al 2011 Heckman Pinto and Savelyev2013) The HSIS includes short-run parent-reported measures of behavior andteacher-reported measures of teacherndashstudent relationships and Head Start ap-pears to have no impact on these outcomes (Puma et al 2010 Walters 2015) TheHSIS noncognitive outcomes differ significantly from those analyzed in previousstudies however and it is unclear whether they capture the same skills

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1807

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 9: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

TA

BL

EI

DE

SC

RIP

TIV

ES

TA

TIS

TIC

S

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

Male

04

94

05

05

00

11

05

01

05

06

04

92

(00

19)

Bla

ck03

08

02

98

00

10

03

17

03

53

02

50

(00

10)

His

pan

ic03

76

03

69

00

07

03

80

03

54

03

73

(00

10)

Tee

nm

oth

er01

59

01

74

00

15

01

59

01

69

01

76

(00

14)

Mot

her

marr

ied

04

36

04

48

00

11

04

39

04

20

04

60

(00

17)

Bot

hp

are

nts

inh

ouse

hol

d04

97

04

88

00

09

04

97

04

68

04

99

(00

17)

Mot

her

ish

igh

sch

ool

dro

pou

t03

68

03

97

00

29

03

77

03

22

04

26

(00

17)

Mot

her

att

end

edso

me

coll

ege

02

98

02

81

00

17

02

93

03

42

02

53

(00

16)

Sp

an

ish

spea

ker

02

87

02

73

00

14

02

96

02

74

02

60

(00

11)

Sp

ecia

led

uca

tion

01

36

01

08

00

28

01

34

01

45

00

91

(00

11)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1803

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EI

(CO

NT

INU

ED)

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

On

lych

ild

01

61

01

39

00

22

01

51

01

90

01

23

(00

12)

Inco

me

(fra

ctio

nof

FP

L)

08

96

08

96

00

00

08

92

09

83

08

51

(00

24)

Age

4co

hor

t04

48

04

51

00

03

04

26

05

67

04

13

(00

12)

Base

lin

esu

mm

ary

ind

ex00

03

00

12

00

09

00

01

01

06

00

40

(00

27)

Urb

an

08

33

08

35

00

02

08

34

08

59

08

19

(00

03)

Cen

ter

pro

vid

estr

an

spor

tati

on06

06

06

04

00

02

05

86

06

14

06

28

(00

05)

Cen

ter

qu

ali

tyin

dex

04

65

04

70

00

05

04

52

04

74

04

88

(00

05)

Joi

nt

p-v

alu

e05

06

N22

56

13

15

35

71

20

43

598

930

Not

es

All

stati

stic

sw

eigh

tby

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tS

tan

dard

erro

rsare

clu

ster

edat

the

cen

ter

level

T

he

tran

spor

tati

onan

dqu

ali

tyin

dex

vari

able

sre

fer

toa

chil

drsquos

Hea

dS

tart

cen

ter

ofra

nd

omass

ign

men

tT

he

qu

ali

tyvari

able

com

bin

esin

form

ati

onon

cen

ter

chara

cter

isti

cs(t

each

eran

dce

nte

rd

irec

tor

edu

cati

onan

dqu

ali

fica

tion

scl

ass

size

)an

dp

ract

ices

(vari

ety

ofli

tera

cyan

dm

ath

act

ivit

ies

hom

evis

itin

g

hea

lth

an

dn

utr

itio

n)

Inco

me

ism

issi

ng

for

19

ofob

serv

ati

ons

Mis

sin

gvalu

esare

excl

ud

edin

stati

stic

sfo

rin

com

eT

he

base

lin

esu

mm

ary

ind

exis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

infa

ll2002

wit

hea

chsc

ore

stan

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tT

he

join

tp

-valu

eis

from

ate

stof

the

hyp

oth

esis

that

all

coef

fici

ents

equ

al

0

QUARTERLY JOURNAL OF ECONOMICS1804

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

EX

PE

RIM

EN

TA

LIM

PA

CT

SO

NT

ES

TS

CO

RE

S

Th

ree-

yea

r-ol

dco

hor

tF

our-

yea

r-ol

dco

hor

tC

ohor

tsp

oole

d

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

Tim

ep

erio

dR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IV

Yea

r1

01

94

06

99

02

78

01

41

06

63

02

13

01

68

06

82

02

47

(00

29)

(00

25)

(00

41)

(00

29)

(00

22)

(00

44)

(00

21)

(00

18)

(00

31)

N19

70

16

01

35

71

Yea

r2

00

87

03

56

02

45

00

15

06

70

00

22

00

46

04

97

00

93

(00

29)

(00

28)

(00

80)

(00

37)

(00

23)

(00

54)

(00

24)

(00

20)

(00

49)

N17

60

14

16

31

76

Yea

r3

00

10

03

65

00

27

00

54

06

66

00

81

00

19

05

00

00

38

(00

31)

(00

28)

(00

85)

(00

40)

(00

25)

(00

60)

(00

25)

(00

20)

(00

50)

N16

59

13

36

29

95

Yea

r4

00

38

03

44

01

10

mdashmdash

(00

34)

(00

29)

(00

98)

N15

99

Not

es

Th

ista

ble

rep

orts

exp

erim

enta

les

tim

ate

sof

the

effe

cts

ofH

ead

Sta

rton

test

scor

es

Th

eou

tcom

eis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

w

ith

each

scor

est

an

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tan

dyea

rC

olu

mn

s(1

)(4

)an

d(7

)re

por

tco

effi

cien

tsfr

omre

gre

ssio

ns

ofte

stsc

ores

onan

ind

icato

rfo

rass

ign

men

tto

Hea

dS

tart

C

olu

mn

s(2

)(5

)an

d(8

)re

por

tco

effi

cien

tsfr

omfi

rst-

stage

regre

ssio

ns

ofH

ead

Sta

rtatt

end

an

ceon

Hea

dS

tart

ass

ign

men

tT

he

att

end

an

cevari

able

isan

ind

icato

req

ual

to1

ifa

chil

datt

end

sH

ead

Sta

rtat

an

yti

me

pri

orto

the

test

C

olu

mn

s(3

)(6

)an

d(9

)re

por

tco

effi

cien

tsfr

omtw

o-st

age

least

squ

are

s(2

SL

S)

mod

els

that

inst

rum

ent

Hea

dS

tart

att

end

an

cew

ith

Hea

dS

tart

ass

ign

men

tA

llm

odel

sw

eigh

tby

the

reci

pro

cal

ofa

chil

drsquos

exp

erim

enta

lass

ign

-m

ent

an

dco

ntr

olfo

rse

x

race

S

pan

ish

lan

gu

age

teen

mot

her

m

oth

errsquos

mari

tal

statu

sp

rese

nce

ofbot

hp

are

nts

inth

eh

ome

fam

ily

size

sp

ecia

led

uca

tion

statu

sin

com

equ

art

ile

du

mm

ies

urb

an

an

da

cubic

pol

yn

omia

lin

base

lin

esc

ore

Mis

sin

gvalu

esfo

rco

vari

ate

sare

set

to0

an

dd

um

mie

sfo

rm

issi

ng

are

incl

ud

ed

Sta

nd

ard

erro

rsare

clu

ster

edby

cen

ter

ofra

nd

omass

ign

men

t

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1805

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Head Start offer separately by year and age cohort To increaseprecision we regression-adjust these treatmentcontrol differ-ences using the baseline characteristics in Table I4 Theintent-to-treat estimates mirror those previously reported inthe literature (eg Puma et al 2010) In the first year of theexperiment children offered Head Start scored higher on thesummary index For example three-year-olds offered HeadStart gained 019 standard deviations in test score outcomesrelative to those denied Head Start The corresponding effectfor four-year-olds is 014 standard deviations However thesegains diminish rapidly the pooled impact falls to a statisticallyinsignificant 002 standard deviations by year 3 Our data in-clude a fourth year of follow-up for the three-year-old cohortHere too the intent-to-treat is small and statistically insignif-icant (0038 standard deviations)

Interpretation of these intent-to-treat impacts is clouded bynoncompliance with random assignment Columns (2) (5) and(8) of Table II report first-stage effects of assignment to HeadStart on the probability of participating in Head Start and col-umns (3) (6) and (9) report instrumental variables (IV) esti-mates which scale the intent-to-treat estimates by the first-stage estimates5 These estimates can be interpreted as local av-erage treatment effects (LATEs) for lsquolsquocompliersrsquorsquomdashchildren whorespond to the Head Start offer by enrolling in Head StartAssignment to Head Start increases the probability of participa-tion by two thirds in the first year after random assignment The

4 The control vector includes sex race assignment cohort teen mothermotherrsquos education motherrsquos marital status presence of both parents an onlychild dummy a Spanish language indicator dummies for quartiles of familyincome and missing income urban status an indicator for whether the HeadStart center provides transportation an index of Head Start center quality and athird-order polynomial in baseline test scores

5 Here we define Head Start participation as enrollment at any time prior tothe test This definition includes attendance at Head Start centers outside the ex-perimental sample An experimental offer may cause some children to switch froman out-of-sample center to an experimental center if the quality of these centersdiffers the exclusion restriction required for our IV approach is violated OnlineAppendix Table AI compares characteristics of centers attended by children in thecontrol group (always takers) to those of the experimental centers to which thesechildren applied These two groups of centers are very similar suggesting thatsubstitution between Head Start centers is unlikely to bias our estimates

QUARTERLY JOURNAL OF ECONOMICS1806

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

corresponding IV estimate implies that Head Start attendanceboosts first-year test scores by 0247 standard deviations

Compliance for the three-year-old cohort falls after the firstyear as members of the control group reapply for Head Startresulting in larger standard errors for estimates in later yearsof the experiment The first stage for three-year-olds falls to 036in the second year whereas the intent-to-treat falls roughly inproportion generating a second-year IV estimate of 0245 for thiscohort Estimates in years 3 and 4 are statistically insignificantand imprecise The fourth-year estimate for the three-year-oldcohort (corresponding to first grade) is 0110 standard deviationswith a standard error of 0098 The corresponding first grade es-timate for four-year-olds is 0081 with a standard error of 0060Notably the 95 confidence intervals for first-grade impacts in-clude effects as large as 02 standard deviations for four-year-oldsand 03 standard deviations for three-year-olds These resultsshow that although the longer run estimates are insignificantthey are also imprecise due to experimental noncomplianceEvidence for fade-out is therefore less definitive than the naiveintent-to-treat estimates suggest This observation helps to rec-oncile the HSIS results with observational studies based on sib-ling comparisons which show effects that partially fade out butare still detectable in elementary school (Currie and Thomas1995 Deming 2009)6

IV Program Substitution

We turn now to documenting program substitution in theHSIS and how it influences our results It is helpful to developsome notation to describe the role of alternative care environ-ments Each Head Start applicant participates in one of three pos-sible treatments Head Start which we label h other center-based

6 One might also be interested in the effects of Head Start on noncognitiveoutcomes which appear to be important mediators of the effects of early childhoodprograms in other contexts (Chetty et al 2011 Heckman Pinto and Savelyev2013) The HSIS includes short-run parent-reported measures of behavior andteacher-reported measures of teacherndashstudent relationships and Head Start ap-pears to have no impact on these outcomes (Puma et al 2010 Walters 2015) TheHSIS noncognitive outcomes differ significantly from those analyzed in previousstudies however and it is unclear whether they capture the same skills

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1807

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 10: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

TA

BL

EI

(CO

NT

INU

ED)

By

offe

rst

atu

sB

yp

resc

hoo

lch

oice

(1)

(2)

(3)

(4)

(5)

(6)

Vari

able

Off

ered

mea

nN

onof

fere

dm

ean

Dif

fere

nti

al

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

l

On

lych

ild

01

61

01

39

00

22

01

51

01

90

01

23

(00

12)

Inco

me

(fra

ctio

nof

FP

L)

08

96

08

96

00

00

08

92

09

83

08

51

(00

24)

Age

4co

hor

t04

48

04

51

00

03

04

26

05

67

04

13

(00

12)

Base

lin

esu

mm

ary

ind

ex00

03

00

12

00

09

00

01

01

06

00

40

(00

27)

Urb

an

08

33

08

35

00

02

08

34

08

59

08

19

(00

03)

Cen

ter

pro

vid

estr

an

spor

tati

on06

06

06

04

00

02

05

86

06

14

06

28

(00

05)

Cen

ter

qu

ali

tyin

dex

04

65

04

70

00

05

04

52

04

74

04

88

(00

05)

Joi

nt

p-v

alu

e05

06

N22

56

13

15

35

71

20

43

598

930

Not

es

All

stati

stic

sw

eigh

tby

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tS

tan

dard

erro

rsare

clu

ster

edat

the

cen

ter

level

T

he

tran

spor

tati

onan

dqu

ali

tyin

dex

vari

able

sre

fer

toa

chil

drsquos

Hea

dS

tart

cen

ter

ofra

nd

omass

ign

men

tT

he

qu

ali

tyvari

able

com

bin

esin

form

ati

onon

cen

ter

chara

cter

isti

cs(t

each

eran

dce

nte

rd

irec

tor

edu

cati

onan

dqu

ali

fica

tion

scl

ass

size

)an

dp

ract

ices

(vari

ety

ofli

tera

cyan

dm

ath

act

ivit

ies

hom

evis

itin

g

hea

lth

an

dn

utr

itio

n)

Inco

me

ism

issi

ng

for

19

ofob

serv

ati

ons

Mis

sin

gvalu

esare

excl

ud

edin

stati

stic

sfo

rin

com

eT

he

base

lin

esu

mm

ary

ind

exis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

infa

ll2002

wit

hea

chsc

ore

stan

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tT

he

join

tp

-valu

eis

from

ate

stof

the

hyp

oth

esis

that

all

coef

fici

ents

equ

al

0

QUARTERLY JOURNAL OF ECONOMICS1804

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

EX

PE

RIM

EN

TA

LIM

PA

CT

SO

NT

ES

TS

CO

RE

S

Th

ree-

yea

r-ol

dco

hor

tF

our-

yea

r-ol

dco

hor

tC

ohor

tsp

oole

d

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

Tim

ep

erio

dR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IV

Yea

r1

01

94

06

99

02

78

01

41

06

63

02

13

01

68

06

82

02

47

(00

29)

(00

25)

(00

41)

(00

29)

(00

22)

(00

44)

(00

21)

(00

18)

(00

31)

N19

70

16

01

35

71

Yea

r2

00

87

03

56

02

45

00

15

06

70

00

22

00

46

04

97

00

93

(00

29)

(00

28)

(00

80)

(00

37)

(00

23)

(00

54)

(00

24)

(00

20)

(00

49)

N17

60

14

16

31

76

Yea

r3

00

10

03

65

00

27

00

54

06

66

00

81

00

19

05

00

00

38

(00

31)

(00

28)

(00

85)

(00

40)

(00

25)

(00

60)

(00

25)

(00

20)

(00

50)

N16

59

13

36

29

95

Yea

r4

00

38

03

44

01

10

mdashmdash

(00

34)

(00

29)

(00

98)

N15

99

Not

es

Th

ista

ble

rep

orts

exp

erim

enta

les

tim

ate

sof

the

effe

cts

ofH

ead

Sta

rton

test

scor

es

Th

eou

tcom

eis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

w

ith

each

scor

est

an

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tan

dyea

rC

olu

mn

s(1

)(4

)an

d(7

)re

por

tco

effi

cien

tsfr

omre

gre

ssio

ns

ofte

stsc

ores

onan

ind

icato

rfo

rass

ign

men

tto

Hea

dS

tart

C

olu

mn

s(2

)(5

)an

d(8

)re

por

tco

effi

cien

tsfr

omfi

rst-

stage

regre

ssio

ns

ofH

ead

Sta

rtatt

end

an

ceon

Hea

dS

tart

ass

ign

men

tT

he

att

end

an

cevari

able

isan

ind

icato

req

ual

to1

ifa

chil

datt

end

sH

ead

Sta

rtat

an

yti

me

pri

orto

the

test

C

olu

mn

s(3

)(6

)an

d(9

)re

por

tco

effi

cien

tsfr

omtw

o-st

age

least

squ

are

s(2

SL

S)

mod

els

that

inst

rum

ent

Hea

dS

tart

att

end

an

cew

ith

Hea

dS

tart

ass

ign

men

tA

llm

odel

sw

eigh

tby

the

reci

pro

cal

ofa

chil

drsquos

exp

erim

enta

lass

ign

-m

ent

an

dco

ntr

olfo

rse

x

race

S

pan

ish

lan

gu

age

teen

mot

her

m

oth

errsquos

mari

tal

statu

sp

rese

nce

ofbot

hp

are

nts

inth

eh

ome

fam

ily

size

sp

ecia

led

uca

tion

statu

sin

com

equ

art

ile

du

mm

ies

urb

an

an

da

cubic

pol

yn

omia

lin

base

lin

esc

ore

Mis

sin

gvalu

esfo

rco

vari

ate

sare

set

to0

an

dd

um

mie

sfo

rm

issi

ng

are

incl

ud

ed

Sta

nd

ard

erro

rsare

clu

ster

edby

cen

ter

ofra

nd

omass

ign

men

t

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1805

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Head Start offer separately by year and age cohort To increaseprecision we regression-adjust these treatmentcontrol differ-ences using the baseline characteristics in Table I4 Theintent-to-treat estimates mirror those previously reported inthe literature (eg Puma et al 2010) In the first year of theexperiment children offered Head Start scored higher on thesummary index For example three-year-olds offered HeadStart gained 019 standard deviations in test score outcomesrelative to those denied Head Start The corresponding effectfor four-year-olds is 014 standard deviations However thesegains diminish rapidly the pooled impact falls to a statisticallyinsignificant 002 standard deviations by year 3 Our data in-clude a fourth year of follow-up for the three-year-old cohortHere too the intent-to-treat is small and statistically insignif-icant (0038 standard deviations)

Interpretation of these intent-to-treat impacts is clouded bynoncompliance with random assignment Columns (2) (5) and(8) of Table II report first-stage effects of assignment to HeadStart on the probability of participating in Head Start and col-umns (3) (6) and (9) report instrumental variables (IV) esti-mates which scale the intent-to-treat estimates by the first-stage estimates5 These estimates can be interpreted as local av-erage treatment effects (LATEs) for lsquolsquocompliersrsquorsquomdashchildren whorespond to the Head Start offer by enrolling in Head StartAssignment to Head Start increases the probability of participa-tion by two thirds in the first year after random assignment The

4 The control vector includes sex race assignment cohort teen mothermotherrsquos education motherrsquos marital status presence of both parents an onlychild dummy a Spanish language indicator dummies for quartiles of familyincome and missing income urban status an indicator for whether the HeadStart center provides transportation an index of Head Start center quality and athird-order polynomial in baseline test scores

5 Here we define Head Start participation as enrollment at any time prior tothe test This definition includes attendance at Head Start centers outside the ex-perimental sample An experimental offer may cause some children to switch froman out-of-sample center to an experimental center if the quality of these centersdiffers the exclusion restriction required for our IV approach is violated OnlineAppendix Table AI compares characteristics of centers attended by children in thecontrol group (always takers) to those of the experimental centers to which thesechildren applied These two groups of centers are very similar suggesting thatsubstitution between Head Start centers is unlikely to bias our estimates

QUARTERLY JOURNAL OF ECONOMICS1806

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

corresponding IV estimate implies that Head Start attendanceboosts first-year test scores by 0247 standard deviations

Compliance for the three-year-old cohort falls after the firstyear as members of the control group reapply for Head Startresulting in larger standard errors for estimates in later yearsof the experiment The first stage for three-year-olds falls to 036in the second year whereas the intent-to-treat falls roughly inproportion generating a second-year IV estimate of 0245 for thiscohort Estimates in years 3 and 4 are statistically insignificantand imprecise The fourth-year estimate for the three-year-oldcohort (corresponding to first grade) is 0110 standard deviationswith a standard error of 0098 The corresponding first grade es-timate for four-year-olds is 0081 with a standard error of 0060Notably the 95 confidence intervals for first-grade impacts in-clude effects as large as 02 standard deviations for four-year-oldsand 03 standard deviations for three-year-olds These resultsshow that although the longer run estimates are insignificantthey are also imprecise due to experimental noncomplianceEvidence for fade-out is therefore less definitive than the naiveintent-to-treat estimates suggest This observation helps to rec-oncile the HSIS results with observational studies based on sib-ling comparisons which show effects that partially fade out butare still detectable in elementary school (Currie and Thomas1995 Deming 2009)6

IV Program Substitution

We turn now to documenting program substitution in theHSIS and how it influences our results It is helpful to developsome notation to describe the role of alternative care environ-ments Each Head Start applicant participates in one of three pos-sible treatments Head Start which we label h other center-based

6 One might also be interested in the effects of Head Start on noncognitiveoutcomes which appear to be important mediators of the effects of early childhoodprograms in other contexts (Chetty et al 2011 Heckman Pinto and Savelyev2013) The HSIS includes short-run parent-reported measures of behavior andteacher-reported measures of teacherndashstudent relationships and Head Start ap-pears to have no impact on these outcomes (Puma et al 2010 Walters 2015) TheHSIS noncognitive outcomes differ significantly from those analyzed in previousstudies however and it is unclear whether they capture the same skills

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1807

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 11: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

TA

BL

EII

EX

PE

RIM

EN

TA

LIM

PA

CT

SO

NT

ES

TS

CO

RE

S

Th

ree-

yea

r-ol

dco

hor

tF

our-

yea

r-ol

dco

hor

tC

ohor

tsp

oole

d

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

Tim

ep

erio

dR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IVR

edu

ced

form

Fir

stst

age

IV

Yea

r1

01

94

06

99

02

78

01

41

06

63

02

13

01

68

06

82

02

47

(00

29)

(00

25)

(00

41)

(00

29)

(00

22)

(00

44)

(00

21)

(00

18)

(00

31)

N19

70

16

01

35

71

Yea

r2

00

87

03

56

02

45

00

15

06

70

00

22

00

46

04

97

00

93

(00

29)

(00

28)

(00

80)

(00

37)

(00

23)

(00

54)

(00

24)

(00

20)

(00

49)

N17

60

14

16

31

76

Yea

r3

00

10

03

65

00

27

00

54

06

66

00

81

00

19

05

00

00

38

(00

31)

(00

28)

(00

85)

(00

40)

(00

25)

(00

60)

(00

25)

(00

20)

(00

50)

N16

59

13

36

29

95

Yea

r4

00

38

03

44

01

10

mdashmdash

(00

34)

(00

29)

(00

98)

N15

99

Not

es

Th

ista

ble

rep

orts

exp

erim

enta

les

tim

ate

sof

the

effe

cts

ofH

ead

Sta

rton

test

scor

es

Th

eou

tcom

eis

the

aver

age

ofst

an

dard

ized

PP

VT

an

dW

JII

Isc

ores

w

ith

each

scor

est

an

dard

ized

toh

ave

mea

n0

an

dst

an

dard

dev

iati

on1

inth

eco

ntr

olgro

up

sep

ara

tely

by

ap

pli

can

tco

hor

tan

dyea

rC

olu

mn

s(1

)(4

)an

d(7

)re

por

tco

effi

cien

tsfr

omre

gre

ssio

ns

ofte

stsc

ores

onan

ind

icato

rfo

rass

ign

men

tto

Hea

dS

tart

C

olu

mn

s(2

)(5

)an

d(8

)re

por

tco

effi

cien

tsfr

omfi

rst-

stage

regre

ssio

ns

ofH

ead

Sta

rtatt

end

an

ceon

Hea

dS

tart

ass

ign

men

tT

he

att

end

an

cevari

able

isan

ind

icato

req

ual

to1

ifa

chil

datt

end

sH

ead

Sta

rtat

an

yti

me

pri

orto

the

test

C

olu

mn

s(3

)(6

)an

d(9

)re

por

tco

effi

cien

tsfr

omtw

o-st

age

least

squ

are

s(2

SL

S)

mod

els

that

inst

rum

ent

Hea

dS

tart

att

end

an

cew

ith

Hea

dS

tart

ass

ign

men

tA

llm

odel

sw

eigh

tby

the

reci

pro

cal

ofa

chil

drsquos

exp

erim

enta

lass

ign

-m

ent

an

dco

ntr

olfo

rse

x

race

S

pan

ish

lan

gu

age

teen

mot

her

m

oth

errsquos

mari

tal

statu

sp

rese

nce

ofbot

hp

are

nts

inth

eh

ome

fam

ily

size

sp

ecia

led

uca

tion

statu

sin

com

equ

art

ile

du

mm

ies

urb

an

an

da

cubic

pol

yn

omia

lin

base

lin

esc

ore

Mis

sin

gvalu

esfo

rco

vari

ate

sare

set

to0

an

dd

um

mie

sfo

rm

issi

ng

are

incl

ud

ed

Sta

nd

ard

erro

rsare

clu

ster

edby

cen

ter

ofra

nd

omass

ign

men

t

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1805

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Head Start offer separately by year and age cohort To increaseprecision we regression-adjust these treatmentcontrol differ-ences using the baseline characteristics in Table I4 Theintent-to-treat estimates mirror those previously reported inthe literature (eg Puma et al 2010) In the first year of theexperiment children offered Head Start scored higher on thesummary index For example three-year-olds offered HeadStart gained 019 standard deviations in test score outcomesrelative to those denied Head Start The corresponding effectfor four-year-olds is 014 standard deviations However thesegains diminish rapidly the pooled impact falls to a statisticallyinsignificant 002 standard deviations by year 3 Our data in-clude a fourth year of follow-up for the three-year-old cohortHere too the intent-to-treat is small and statistically insignif-icant (0038 standard deviations)

Interpretation of these intent-to-treat impacts is clouded bynoncompliance with random assignment Columns (2) (5) and(8) of Table II report first-stage effects of assignment to HeadStart on the probability of participating in Head Start and col-umns (3) (6) and (9) report instrumental variables (IV) esti-mates which scale the intent-to-treat estimates by the first-stage estimates5 These estimates can be interpreted as local av-erage treatment effects (LATEs) for lsquolsquocompliersrsquorsquomdashchildren whorespond to the Head Start offer by enrolling in Head StartAssignment to Head Start increases the probability of participa-tion by two thirds in the first year after random assignment The

4 The control vector includes sex race assignment cohort teen mothermotherrsquos education motherrsquos marital status presence of both parents an onlychild dummy a Spanish language indicator dummies for quartiles of familyincome and missing income urban status an indicator for whether the HeadStart center provides transportation an index of Head Start center quality and athird-order polynomial in baseline test scores

5 Here we define Head Start participation as enrollment at any time prior tothe test This definition includes attendance at Head Start centers outside the ex-perimental sample An experimental offer may cause some children to switch froman out-of-sample center to an experimental center if the quality of these centersdiffers the exclusion restriction required for our IV approach is violated OnlineAppendix Table AI compares characteristics of centers attended by children in thecontrol group (always takers) to those of the experimental centers to which thesechildren applied These two groups of centers are very similar suggesting thatsubstitution between Head Start centers is unlikely to bias our estimates

QUARTERLY JOURNAL OF ECONOMICS1806

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

corresponding IV estimate implies that Head Start attendanceboosts first-year test scores by 0247 standard deviations

Compliance for the three-year-old cohort falls after the firstyear as members of the control group reapply for Head Startresulting in larger standard errors for estimates in later yearsof the experiment The first stage for three-year-olds falls to 036in the second year whereas the intent-to-treat falls roughly inproportion generating a second-year IV estimate of 0245 for thiscohort Estimates in years 3 and 4 are statistically insignificantand imprecise The fourth-year estimate for the three-year-oldcohort (corresponding to first grade) is 0110 standard deviationswith a standard error of 0098 The corresponding first grade es-timate for four-year-olds is 0081 with a standard error of 0060Notably the 95 confidence intervals for first-grade impacts in-clude effects as large as 02 standard deviations for four-year-oldsand 03 standard deviations for three-year-olds These resultsshow that although the longer run estimates are insignificantthey are also imprecise due to experimental noncomplianceEvidence for fade-out is therefore less definitive than the naiveintent-to-treat estimates suggest This observation helps to rec-oncile the HSIS results with observational studies based on sib-ling comparisons which show effects that partially fade out butare still detectable in elementary school (Currie and Thomas1995 Deming 2009)6

IV Program Substitution

We turn now to documenting program substitution in theHSIS and how it influences our results It is helpful to developsome notation to describe the role of alternative care environ-ments Each Head Start applicant participates in one of three pos-sible treatments Head Start which we label h other center-based

6 One might also be interested in the effects of Head Start on noncognitiveoutcomes which appear to be important mediators of the effects of early childhoodprograms in other contexts (Chetty et al 2011 Heckman Pinto and Savelyev2013) The HSIS includes short-run parent-reported measures of behavior andteacher-reported measures of teacherndashstudent relationships and Head Start ap-pears to have no impact on these outcomes (Puma et al 2010 Walters 2015) TheHSIS noncognitive outcomes differ significantly from those analyzed in previousstudies however and it is unclear whether they capture the same skills

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1807

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 12: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

Head Start offer separately by year and age cohort To increaseprecision we regression-adjust these treatmentcontrol differ-ences using the baseline characteristics in Table I4 Theintent-to-treat estimates mirror those previously reported inthe literature (eg Puma et al 2010) In the first year of theexperiment children offered Head Start scored higher on thesummary index For example three-year-olds offered HeadStart gained 019 standard deviations in test score outcomesrelative to those denied Head Start The corresponding effectfor four-year-olds is 014 standard deviations However thesegains diminish rapidly the pooled impact falls to a statisticallyinsignificant 002 standard deviations by year 3 Our data in-clude a fourth year of follow-up for the three-year-old cohortHere too the intent-to-treat is small and statistically insignif-icant (0038 standard deviations)

Interpretation of these intent-to-treat impacts is clouded bynoncompliance with random assignment Columns (2) (5) and(8) of Table II report first-stage effects of assignment to HeadStart on the probability of participating in Head Start and col-umns (3) (6) and (9) report instrumental variables (IV) esti-mates which scale the intent-to-treat estimates by the first-stage estimates5 These estimates can be interpreted as local av-erage treatment effects (LATEs) for lsquolsquocompliersrsquorsquomdashchildren whorespond to the Head Start offer by enrolling in Head StartAssignment to Head Start increases the probability of participa-tion by two thirds in the first year after random assignment The

4 The control vector includes sex race assignment cohort teen mothermotherrsquos education motherrsquos marital status presence of both parents an onlychild dummy a Spanish language indicator dummies for quartiles of familyincome and missing income urban status an indicator for whether the HeadStart center provides transportation an index of Head Start center quality and athird-order polynomial in baseline test scores

5 Here we define Head Start participation as enrollment at any time prior tothe test This definition includes attendance at Head Start centers outside the ex-perimental sample An experimental offer may cause some children to switch froman out-of-sample center to an experimental center if the quality of these centersdiffers the exclusion restriction required for our IV approach is violated OnlineAppendix Table AI compares characteristics of centers attended by children in thecontrol group (always takers) to those of the experimental centers to which thesechildren applied These two groups of centers are very similar suggesting thatsubstitution between Head Start centers is unlikely to bias our estimates

QUARTERLY JOURNAL OF ECONOMICS1806

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

corresponding IV estimate implies that Head Start attendanceboosts first-year test scores by 0247 standard deviations

Compliance for the three-year-old cohort falls after the firstyear as members of the control group reapply for Head Startresulting in larger standard errors for estimates in later yearsof the experiment The first stage for three-year-olds falls to 036in the second year whereas the intent-to-treat falls roughly inproportion generating a second-year IV estimate of 0245 for thiscohort Estimates in years 3 and 4 are statistically insignificantand imprecise The fourth-year estimate for the three-year-oldcohort (corresponding to first grade) is 0110 standard deviationswith a standard error of 0098 The corresponding first grade es-timate for four-year-olds is 0081 with a standard error of 0060Notably the 95 confidence intervals for first-grade impacts in-clude effects as large as 02 standard deviations for four-year-oldsand 03 standard deviations for three-year-olds These resultsshow that although the longer run estimates are insignificantthey are also imprecise due to experimental noncomplianceEvidence for fade-out is therefore less definitive than the naiveintent-to-treat estimates suggest This observation helps to rec-oncile the HSIS results with observational studies based on sib-ling comparisons which show effects that partially fade out butare still detectable in elementary school (Currie and Thomas1995 Deming 2009)6

IV Program Substitution

We turn now to documenting program substitution in theHSIS and how it influences our results It is helpful to developsome notation to describe the role of alternative care environ-ments Each Head Start applicant participates in one of three pos-sible treatments Head Start which we label h other center-based

6 One might also be interested in the effects of Head Start on noncognitiveoutcomes which appear to be important mediators of the effects of early childhoodprograms in other contexts (Chetty et al 2011 Heckman Pinto and Savelyev2013) The HSIS includes short-run parent-reported measures of behavior andteacher-reported measures of teacherndashstudent relationships and Head Start ap-pears to have no impact on these outcomes (Puma et al 2010 Walters 2015) TheHSIS noncognitive outcomes differ significantly from those analyzed in previousstudies however and it is unclear whether they capture the same skills

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1807

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 13: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

corresponding IV estimate implies that Head Start attendanceboosts first-year test scores by 0247 standard deviations

Compliance for the three-year-old cohort falls after the firstyear as members of the control group reapply for Head Startresulting in larger standard errors for estimates in later yearsof the experiment The first stage for three-year-olds falls to 036in the second year whereas the intent-to-treat falls roughly inproportion generating a second-year IV estimate of 0245 for thiscohort Estimates in years 3 and 4 are statistically insignificantand imprecise The fourth-year estimate for the three-year-oldcohort (corresponding to first grade) is 0110 standard deviationswith a standard error of 0098 The corresponding first grade es-timate for four-year-olds is 0081 with a standard error of 0060Notably the 95 confidence intervals for first-grade impacts in-clude effects as large as 02 standard deviations for four-year-oldsand 03 standard deviations for three-year-olds These resultsshow that although the longer run estimates are insignificantthey are also imprecise due to experimental noncomplianceEvidence for fade-out is therefore less definitive than the naiveintent-to-treat estimates suggest This observation helps to rec-oncile the HSIS results with observational studies based on sib-ling comparisons which show effects that partially fade out butare still detectable in elementary school (Currie and Thomas1995 Deming 2009)6

IV Program Substitution

We turn now to documenting program substitution in theHSIS and how it influences our results It is helpful to developsome notation to describe the role of alternative care environ-ments Each Head Start applicant participates in one of three pos-sible treatments Head Start which we label h other center-based

6 One might also be interested in the effects of Head Start on noncognitiveoutcomes which appear to be important mediators of the effects of early childhoodprograms in other contexts (Chetty et al 2011 Heckman Pinto and Savelyev2013) The HSIS includes short-run parent-reported measures of behavior andteacher-reported measures of teacherndashstudent relationships and Head Start ap-pears to have no impact on these outcomes (Puma et al 2010 Walters 2015) TheHSIS noncognitive outcomes differ significantly from those analyzed in previousstudies however and it is unclear whether they capture the same skills

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1807

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 14: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

preschool programs which we label c and no preschool (ie homecare) which we label n Let Zi 2 f01g indicate whether household ihas a Head Start offer and DiethzTHORN 2 fhcng denote household irsquospotential treatment status as a function of the offer Then observedtreatment status can be written Di frac14 DiethZiTHORN

The structure of the HSIS leads to natural theoretical restric-tions on substitution patterns We expect a Head Start offer toinduce some children who would otherwise participate in c or n toenroll in Head Start By revealed preference no child shouldswitch between c and n in response to a Head Start offer andno child should be induced by an offer to leave Head Start Theserestrictions can be expressed succinctly by the followingcondition

Dieth1THORN 6frac14 Dieth0THORN ) Dieth1THORN frac14 heth1THORN

which extends the monotonicity assumption of Imbens andAngrist (1994) to a setting with multiple counterfactual treat-ments This restriction states that anyone who changes behav-ior as a result of the Head Start offer does so to attend HeadStart7

Under restriction (1) the population of Head Start applicantscan be partitioned into five groups defined by their values of Dieth1THORNand Dieth0THORN

(i) n-compliers Dieth1THORN frac14 h Dieth0THORN frac14 n(ii) c-compliers Dieth1THORN frac14 h Dieth0THORN frac14 c

(iii) n-never takers Dieth1THORN frac14 Dieth0THORN frac14 n(iv) c-never takers Dieth1THORN frac14 Dieth0THORN frac14 c(v) always takers Dieth1THORN frac14 Dieth0THORN frac14 h

The n- and c-compliers switch to Head Start from home care andcompeting preschools respectively when offered a seat The twogroups of never takers choose not to attend Head Start regard-less of the offer Always takers manage to enroll in Head Starteven when denied an offer presumably by applying to otherHead Start centers outside the HSIS sample

Using this rubric the group of children enrolled in alterna-tive preschool programs is a mixture of c-never takers and c-com-pliers denied Head Start offers Similarly the group of children in

7 See Engberg et al (2014) for discussion of related restrictions in the contextof attrition from experimental data

QUARTERLY JOURNAL OF ECONOMICS1808

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 15: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

home care includes n-never takers and n-compliers withoutoffers The two complier subgroups switch into Head Startwhen offered admission as a result the set of children enrolledin Head Start is a mixture of always takers and the two groups ofoffered compliers

IVA Substitution Patterns

Table III presents empirical evidence on substitution pat-terns by comparing program participation choices for offeredand nonoffered households In the first year of the experiment83 of households decline Head Start offers in favor of otherpreschool centers this is the share of c-never takers Similarlycolumn (3) shows that 95 of households are n-never takers Ascan be seen in column (4) 136 of households manage to attendHead Start without an offer which is the share of always takersThe Head Start offer reduces the share of children in other cen-ters from 315 to 83 and reduces the share of children inhome care from 55 to 95 This implies that 232 of house-holds are c-compliers and 455 are n-compliers

Notably in the first year of the experiment three-year-oldshave uniformly higher participation rates in Head Start andlower participation rates in competing centers which likely re-flects the fact that many state-provided programs only acceptfour-year-olds In the second year of the experiment participa-tion in Head Start drops among children in the three-year-oldcohort with a program offer suggesting that many families en-rolled in the first year decided that Head Start was a bad matchfor their child We also see that Head Start enrollment risesamong those families that did not obtain an offer in the firstround which reflects reapplication behavior

IVB Interpreting IV

How do the substitution patterns displayed in Table III affectthe interpretation of the HSIS test score impacts Let YiethdTHORNdenote child irsquos potential test score if he or she participates intreatment d 2 fhcng Observed scores are given by Yi frac14 YiethDiTHORNWe assume that Head Start offers affect test scores only throughprogram participation choices Under assumption (1) IV identi-fies a variant of the LATE of Imbens and Angrist (1994) givingthe average effect of Head Start participation for compliers

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1809

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 16: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

TA

BL

EII

I

PR

ES

CH

OO

LC

HO

ICE

SB

YY

EA

R

CO

HO

RT

AN

DO

FF

ER

ST

AT

US

Off

ered

Not

offe

red

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Tim

ep

erio

dC

ohor

tH

ead

Sta

rtO

ther

cen

ters

No

pre

sch

ool

Hea

dS

tart

Oth

erce

nte

rsN

op

resc

hoo

lC

-com

pli

ersh

are

Yea

r1

3-y

ear-

old

s08

51

00

58

00

92

01

47

02

56

05

97

02

82

4-y

ear-

old

s07

87

01

14

00

99

01

22

03

86

04

92

04

10

Poo

led

08

22

00

83

00

95

01

36

03

15

05

50

03

38

Yea

r2

3-y

ear-

old

s06

57

02

62

00

81

04

94

03

79

01

27

07

19

Not

es

Th

ista

ble

rep

orts

share

sof

offe

red

an

dn

onof

fere

dst

ud

ents

att

end

ing

Hea

dS

tart

ot

her

cen

ter-

base

dp

resc

hoo

ls

an

dn

op

resc

hoo

lse

para

tely

by

yea

ran

dage

coh

ort

All

stati

stic

sare

wei

gh

ted

by

the

reci

pro

cal

ofth

ep

robabil

ity

ofa

chil

drsquos

exp

erim

enta

lass

ign

men

tC

olu

mn

(7)

rep

orts

esti

mate

sof

the

share

ofco

mp

lier

sd

raw

nfr

omot

her

pre

sch

ools

giv

enby

min

us

the

rati

oof

the

offe

rrsquos

effe

cton

att

end

an

ceat

oth

erp

resc

hoo

lsto

its

effe

cton

Hea

dS

tart

att

end

an

ce

QUARTERLY JOURNAL OF ECONOMICS1810

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 17: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

relative to a mix of program alternatives Specifically underequation (1) and excludability of Head Start offers

E YijZi frac14 1frac12 E YijZi frac14 0frac12

E 1 Di frac14 hf gjZi frac14 1frac12 E 1 Di frac14 hf gjZi frac14 0frac12

frac14 E YiethhTHORN YiethDieth0THORNTHORNjDieth1THORN frac14 hDieth0THORN 6frac14 hfrac12

LATEh

eth2THORN

The left-hand side of equation (2) is the population coefficientfrom a model that instruments Head Start attendance with theHead Start offer This equation implies that the IV strategyemployed in Table II yields the average effect of Head Startfor compliers relative to their own counterfactual care choicesa quantity we label LATEh

We can decompose LATEh into a weighted average oflsquolsquosubLATEsrsquorsquo measuring the effects of Head Start for compliersdrawn from specific counterfactual alternatives as follows

LATEh frac14 ScLATEch thorn eth1 ScTHORNLATEnheth3THORN

where LATEdh E YiethhTHORN YiethdTHORNjDieth1THORN frac14 hDieth0THORN frac14 dfrac12 gives theaverage treatment effect on d-compliers for d 2 fcng and the

weight Sc P Di 1eth THORNfrac14hDi 0eth THORNfrac14ceth THORN

P Di 1eth THORNfrac14hDi 0eth THORN6frac14heth THORNgives the fraction of compliers drawn

from other preschoolsColumn (7) of Table III reports estimates of Sc by year and

cohort computed as minus the ratio of the Head Start offerrsquoseffect on other preschool attendance to its effect on Head Startattendance (see Online Appendix B) In the first year of the HSISexperiment 34 of compliers would have otherwise attendedcompeting preschools IV estimates combine effects for these com-pliers with effects for compliers who would not otherwise attendpreschool

As detailed in Online Appendix D the competing preschoolsattended by c-compliers are largely publicly funded and provideservices roughly comparable to Head Start The modal alterna-tive preschool is a state-provided preschool program whereasothers receive funding from a mix of public sources (see OnlineAppendix Table AII) Moreover it is likely that even Head Startndasheligible children attending private preschool centers receivepublic funding (eg through CCDF or TANF subsidies) Nextwe consider the implications of substitution from these alterna-tive preschools for assessments of Head Startrsquos cost-effectiveness

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1811

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 18: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

V A Model of Head Start Provision

In this section we develop a model of Head Start participationwith the goal of conducting a cost-benefit analysis that acknowl-edges the presence of publicly subsidized program substitutes Ourmodel is highly stylized and focuses on obtaining an estimablelower bound on the rate of return to potential reforms of HeadStart measured in terms of lifetime earnings The analysis ignoresredistributional motives and any effects of human capital invest-ment on criminal activity (Lochner and Moretti 2004 Heckmanet al 2010) health (Deming 2009 Carneiro and Ginja 2014) orgrade repetition (Currie 2001) Adding such features would tend toraise the implied return to Head Start We also abstract from pa-rental labor supply decisions because prior analyses of the HSISfind no short-term impacts on parentsrsquo work decisions (Puma et al2010)8 Again incorporating parental labor supply responseswould likely raise the programrsquos rate of return

Our discussion emphasizes that the cost-effectiveness of HeadStart is contingent on assumptions regarding the structure of themarket for preschool services and the nature of the specific policyreforms under consideration Building on the heterogeneous ef-fects framework of the previous section we derive expressionsfor policy relevant lsquolsquosufficient statisticsrsquorsquo (Chetty 2009) in terms ofcausal effects on student outcomes Specifically we show that avariant of the LATE concept of Imbens and Angrist (1994) is policyrelevant when considering program expansions in an environmentwhere slots in competing preschools are not rationed With ration-ing a mix of LATEs becomes relevant which poses challenges toidentification with the HSIS data When considering reforms toHead Start program features that change selection into the pro-gram the policy-relevant parameter is shown to be a variant of theMTE concept of Heckman and Vytlacil (1999)

VA Setup

Consider a population of households indexed by i each witha single preschool-aged child Each household can enroll itschild in Head Start enroll in a competing preschool program

8 We replicate this analysis for our sample in Online Appendix Table AIIIwhich shows that a Head Start offer has no effect on the probability that a childrsquosmother works or on the likelihood of working full- versus part-time Recent work byLong (2015) suggests that Head Start may have small effects on full- versus part-time work for mothers of three-year-olds

QUARTERLY JOURNAL OF ECONOMICS1812

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 19: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

(eg state-subsidized preschool) or care for the child at home Thegovernment rations Head Start participation via program offersZi which arrive at random via lottery with probability P Zi frac14 1eth THORN Offers are distributed in a first period In a secondperiod households make enrollment decisions Tenacious appli-cants who have not received an offer can enroll in Head Start byexerting additional effort We begin by assuming that competingprograms are not rationed and then relax this assumption later

Each household has utility over its enrollment options givenby the function Ui dzeth THORN The argument d 2 hcnf g indexes childcare environments and the argument z 2 0 1f g indexes offerstatus Head Start offers raise the value of Head Start and haveno effect on the value of other options so that

Ui h1eth THORN gt Ui h0eth THORNUi czeth THORN frac14 Ui ceth THORNUi nzeth THORN frac14 Ui neth THORN

Household irsquos enrollment choice as a function of its offer statusz is given by

Di zeth THORN frac14 arg maxd2 hcnf g

Ui dzeth THORN

It is straightforward to show that this model satisfies the mono-tonicity restriction (1) Because offers are assigned at randommarket shares for the three care environments can be writtenP Di frac14 deth THORN frac14 P Di 1eth THORN frac14 deth THORN thorn 1 eth THORNP Di 0eth THORN frac14 deth THORN

VB Benefits and Costs

Debate over the effectiveness of educational programs oftencenters on their test score impacts A standard means of valuingsuch impacts is in terms of their effects on later life earnings(Heckman et al 2010 Chetty et al 2011 Heckman Pinto andSavelyev 2013 Chetty Friedman and Rockoff 2014b)9 Let thesymbol B denote the total after-tax lifetime income of a cohort ofchildren We assume that B is linked to test scores by theequation

B frac14 B0 thorn 1 eth THORNpE Yifrac12 eth4THORN

where p gives the market price of human capital is the taxrate faced by the children of eligible households and B0 is anintercept reflecting how test scores are normed Our focus on

9 Online Appendix C considers how such valuations should be adjusted whentest score impacts yield labor supply responses

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1813

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 20: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

mean test scores neglects distributional concerns that may leadus to undervalue Head Startrsquos test score impacts (see BitlerDomina and Hoynes 2014)

The net costs to government of financing preschool are given by

C frac14 C0 thorn rsquohP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth5THORN

where the term C0 reflects fixed costs of administering the pro-gram and rsquoh gives the administrative cost of providing HeadStart services to an additional child Likewise rsquoc gives theadministrative cost to government of providing competing pre-school services (which often receive public subsidies) to anotherstudent The term pE Yifrac12 captures the revenue generated bytaxes on the adult earnings of Head Startndasheligible childrenThis formulation abstracts from the fact that program outlaysmust be determined before the children enter the labor marketand begin paying taxes a complication we adjust for in ourempirical work via discounting

VC Changing Offer Probabilities

Consider the effects of adjusting Head Start enrollment bychanging the rationing probability An increase in draws ad-ditional households into Head Start from competing programsand home care As shown in Online Appendix C the effect of achange in the offer rate on average test scores is given by

qE Yifrac12

qfrac14 LATEh|fflfflfflfflzfflfflfflffl

Effect on compliers

qP Di frac14 heth THORN

q|fflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflComplier density

eth6THORN

In words the aggregate impact on test scores of a small in-crease in the offer rate equals the average impact of HeadStart on complier test scores times the measure of compliersBy the arguments in Section IV both LATEh and q

qP Di frac14 heth THORN

frac14 P Di 1eth THORN frac14 hDi 0eth THORN 6frac14 heth THORN are identified by the HSIS experimentHence equation (6) implies that the hypothetical effects of amarket-level change in offer probabilities can be inferred froman individual-level randomized trial with a fixed offer probabil-ity This convenient result follows from the assumption thatHead Start offers are distributed at random and that doesnot directly enter the alternative specific choice utilitieswhich in turn implies that the composition of compliers (andhence LATEh) does not change with Later we explore how

QUARTERLY JOURNAL OF ECONOMICS1814

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 21: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

this expression changes when the composition of compliers re-sponds to a policy change

From equation (4) the marginal benefit of an increase in isgiven by

qB

qfrac14 1 eth THORNpLATEh

qP Di frac14 heth THORN

q

The offsetting marginal cost to government of financing such anexpansion can be written

qC

qfrac14 rsquoh|z

Provision Cost

rsquocSc|fflzfflPublic Savings

pLATEh|fflfflfflfflfflfflfflzfflfflfflfflfflfflfflAdded Revenue

0BBB

1CCCA qP Di frac14 heth THORN

qeth7THORN

This cost consists of the measure of compliers times the admin-istrative cost rsquoh of enrolling them in Head Start minus theprobability Sc that a complying household comes from a substi-tute preschool times the expected government savings rsquoc asso-ciated with reduced enrollment in substitute preschools Thequantity rsquoh rsquocSc can be viewed as a LATE of Head Start ongovernment spending for compliers Subtracted from this effectis any extra revenue the government gets from raising the pro-ductivity of the children of complying households

The ratio of marginal impacts on after-tax income and gov-ernment costs gives the marginal value of public funds (Mayshar1990 Hendren 2016) which we can write

MVPF qBqqCq

frac141 eth THORNpLATEh

rsquoh rsquocSc pLATEheth8THORN

The MVPF gives the value of an extra dollar spent on HeadStart net of fiscal externalities These fiscal externalities in-clude reduced spending on competing subsidized programs (cap-tured by the term rsquocSc) and additional tax revenue generatedby higher earnings (captured by pLATEh) As emphasized byHendren (2016) the MVPF is a metric that can easily be com-pared across programs without specifying exactly how programexpenditures are to be funded In our case if MVPF gt 1 $1 ofgovernment spending can raise the after-tax incomes of chil-dren by more than $1 which is a robust indicator that programexpansions are likely to be welfare improving

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1815

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 22: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

An important lesson of the foregoing analysis is that iden-tifying costs and benefits of changes to offer probabilities doesnot require identification of treatment effects relative to partic-ular counterfactual care states Specifically it is not necessaryto separately identify the subLATEs This result shows thatprogram substitution is not a design flaw of evaluationsRather it is a feature of the policy environment that needs tobe considered when computing the likely effects of changes topolicy parameters Here program substitution alters the usuallogic of program evaluation only by requiring identification ofthe complier share Sc which governs the degree of public sav-ings realized as a result of reducing subsidies to competingprograms

VD Rationed Substitutes

The foregoing analysis presumes that Head Start expansionsyield reductions in the enrollment of competing preschoolsHowever if competing programs are also oversubscribed the slotsvacated by c-compliers may be filled by other households This willreduce the public savings associated with Head Start expansionsbut also generate the potential for additional test score gains

With rationing in substitute preschool programs the utilityof enrollment in c can be written UiethcZicTHORN where Zic indicates anoffered slot in the competing program Household irsquos enrollmentchoice DiethZihZicTHORN depends on both the Head Start offer Zih andthe competing program offer Assume these offers are assignedindependently with probabilities h and c but that c adjusts tochanges in h to keep total enrollment in c constant In additionassume that all children induced to move into c as a result of anincrease in c come from n rather than h

We show in Online Appendix C that under these assumptionsthe marginal impact of expanding Head Start becomes

qE Yifrac12

qhfrac14 LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

where LATEnc E Yi ceth THORN Yi neth THORNjDi Zih1eth THORN frac14 cDi Zih0eth THORN frac14 nfrac12 Intuitively every c-complier now spawns a corresponding n-to-c complier who fills the vacated preschool slot

The marginal cost to government of inducing this change intest scores can be written

QUARTERLY JOURNAL OF ECONOMICS1816

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 23: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

qC

qhfrac14 rsquoh p LATEh thorn LATEnc Sceth THORN

qP Di frac14 heth THORN

qh

Relative to equation (7) rationing eliminates the public savingsfrom reduced enrollment in substitute programs but adds an-other fiscal externality in its place the tax revenue associatedwith any test score gains of shifting children from home care tocompeting preschools The resulting marginal value of publicfunds can be written

MVPFrat frac14eth1 THORNp LATEh thorn LATEnc Sceth THORN

rsquoh p LATEh thorn LATEnc Sceth THORNeth9THORN

While the impact of rationed substitutes on the marginal valueof public funds is theoretically ambiguous there is good reasonto expect MVPFrat gt MVPF in practice Specifically ignoringrationing of competing programs yields a lower bound onthe rate of return to Head Start expansions if Head Start andother forms of center-based care have roughly comparable effectson test scores and competing programs are cheaper (see OnlineAppendix C) Unfortunately effects for n-to-c compliers are notnonparametrically identified by the HSIS experiment becauseone cannot know which households that care for their childrenat home would otherwise choose to enroll them in competingpreschools We return to this issue in Section IX

VE Structural Reforms

An important assumption in the previous analyses is thatchanging lottery probabilities does not alter the mix of programcompliers Consider now the effects of altering some structuralfeature f of the Head Start program that households value buthas no impact on test scores For example Executive Order13330 issued by President George W Bush in February 2004mandated enhancements to the transportation services providedby Head Start and other federal programs (Federal Register2004) Expanding Head Start transportation services should notdirectly influence educational outcomes but may yield a composi-tional effect by drawing in households from a different mix ofcounterfactual care environments10 By shifting the composition

10 This presumes that peer effects are not an important determinant of testscore outcomes Large changes in the student composition of Head Start classroomscould potentially change the effectiveness of Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1817

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 24: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

of program participants changes in f may boost the programrsquos rateof return

To establish notation we assume that households now valueHead Start participation as

UiethhZi f THORN frac14 Ui hZieth THORN thorn f

Utilities for other preschools and home care are assumed to beunaffected by changes in f This implies that increases in fmake Head Start more attractive for all households Forsimplicity we return to our prior assumption that competingprograms are not rationed As shown in Online Appendix Cthe assumption that f has no effect on potential outcomesimplies

qE Yifrac12

qffrac14MTEh

qP Di frac14 heth THORN

qf

where

MTEh E YiethhTHORN Yi ceth THORNjUiethhZiTHORN thorn f frac14 UiethcTHORNUiethcTHORN gt UiethnTHORNfrac12 ~Sc

thorn E YiethhTHORN YiethnTHORNjUiethhZiTHORN thorn f frac14 UiethnTHORNUiethnTHORN gt UiethcTHORNfrac12 eth1 ~ScTHORN

and ~Sc gives the share of children on the margin of participat-ing in Head Start who prefer the competing program topreschool nonparticipation Following the terminology inHeckman Urzua and Vytlacil (2008) the marginal treatmenteffect MTEh is the average effect of Head Start on test scoresamong households indifferent between Head Start and the nextbest alternative This is a marginal version of the result inequation (6) where integration is now over a set of childrenwho may differ from current program compliers in their meanimpacts Like LATEh MTEh is a weighted average oflsquolsquosubMTEsrsquorsquo corresponding to whether the next best alternativeis home care or a competing preschool program The weight ~Sc

may differ from Sc if inframarginal participants are drawn fromdifferent sources than marginal ones

The test score effects of improvements to the program featuremust be balanced against the costs We suppose that changingprogram features changes the average cost rsquoh feth THORN of Head Startservices so that the net costs to government of financing pre-school are now

QUARTERLY JOURNAL OF ECONOMICS1818

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 25: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

C feth THORN frac14 C0 thorn rsquoh feth THORNP Di frac14 heth THORN thorn rsquocP Di frac14 ceth THORN pE Yifrac12 eth10THORN

where qrsquoh feth THORNqf 0 The marginal costs to government (per program

complier) of a change in the program feature can be written

qC feth THORN

qfqP Di frac14 heth THORN

qf

frac14 rsquoh|zMarginal Provision Cost

thorn

qrsquoh feth THORN

qfqln P Di frac14 heth THORN

qf|fflfflfflfflfflfflfflfflfflfflfflzfflfflfflfflfflfflfflfflfflfflfflInframarginal Provision Cost

rsquoc~Sc|fflzffl

Public Savings

pMTEh|fflfflfflfflfflfflzfflfflfflfflfflfflAdded Revenue

eth11THORN

The first term on the right-hand side of equation (11) gives theadministrative cost of enrolling another child The second termgives the increased cost of providing inframarginal familieswith the improved program feature The third term is the ex-pected savings in reduced funding to competing preschool pro-grams The final term gives the additional tax revenue raisedby the boost in the marginal enrolleersquos human capital

Letting qln rsquoethf THORN

qfqln PethDi frac14 hTHORN

qf

be the elasticity of costs with respect to

enrollment we can write the marginal value of public funds as-sociated with a change in program features as

MVPFf

qBqf

qC feth THORNqf

frac141 eth THORNpMTEh

rsquoh 1thorn eth THORN rsquoc~Sc pMTEh

eth12THORN

As in our analysis of optimal program scale equation (11) showsthat it is not necessary to separately identify the subMTEs thatcompose MTEh to determine the optimal value of f Rather it issufficient to identify the average causal effect of Head Start forchildren on the margin of participation along with the averagenet cost of an additional seat in this population

VI A Cost-Benefit Analysis of Program Expansion

We use the HSIS data to conduct a formal cost-benefit anal-ysis of changes to Head Startrsquos offer rate under the assumptionthat competing programs are not rationed (we consider the casewith rationing in Section IX) Our analysis focuses on the costs

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1819

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 26: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

and benefits associated with one year of Head Start atten-dance11 This exercise requires estimates of each term in equa-tion (7) We estimate LATEh and Sc from the HSIS and calibratethe remaining parameters using estimates from the literatureCalibrated parameters are listed in Table IV Panel A To beconservative we deliberately bias our calibrations towardunderstating Head Startrsquos benefits and overstating its costs toarrive at a lower bound rate of return Further details of thecalibration exercise are provided in Online Appendix D

Table IV Panel B reports estimates of the marginal value ofpublic funds associated with an expansion of Head Start offers(MVPF) To account for sampling uncertainty in our estimates ofLATEh and Sc we report standard errors calculated via the deltamethod Because asymptotic delta method approximations can beinaccurate when the statistic of interest is highly nonlinear(Lafontaine and White 1986) we also report bootstrap p-valuesfrom one-tailed tests of the null hypothesis that the benefit-costratio is less than 112

The results show that accounting for the public savings as-sociated with enrollment in substitute preschools has a largeeffect on the estimated social value of Head Start We conductcost-benefit analyses under three assumptions rsquoc is either 050 or 75 of rsquoh Our preferred calibration uses rsquoc frac14 075rsquohreflecting that fact that roughly 75 of competing centers arepublicly funded (see Online Appendix D) Setting rsquoc frac14 0 yields aMVPF of 110 Setting rsquoc equal to 05rsquoh and 075rsquoh raises theMVPF to 150 and 184 respectively This indicates that the fiscalexternality generated by program substitution has an importanteffect on the social value of Head Start Bootstrap tests decisivelyreject values of MVPF less than 1 when rsquoc frac14 05rsquoh or 075rsquoh

11 Children in the three-year-old cohort who enroll for two years generate ad-ditional costs As shown in Table III a Head Start offer raises the probability ofenrollment in the second year by only 016 implying that first-year offers havemodest net effects on second-year costs Enrollment for two years may also generateadditional benefits but these cannot be estimated without strong assumptions onthe Head Start dose-response function We therefore consider only first-year ben-efits and costs

12 This test is computed by a nonparametric block bootstrap of the Studentizedt-statistic that resamples Head Start sites We have found in Monte Carlo exercisesthat delta method confidence intervals for MVPF tend to overcover whereas boot-strap-t tests have approximately correct size This is in accord with theoreticalresults from Hall (1992) that show bootstrap-t methods yield a higher-order refine-ment to p-values based on the standard delta method approximation

QUARTERLY JOURNAL OF ECONOMICS1820

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 27: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

TA

BL

EIV

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elA

P

ara

met

ervalu

esp

Eff

ect

ofa

1st

d

dev

in

crea

sein

test

scor

eson

earn

ings

01

eT

able

AI

V

e US

US

aver

age

pre

sen

td

isco

un

ted

valu

eof

life

-ti

me

earn

ings

at

age

34

$4380

00

Ch

etty

etal

(2011)

wit

h3

dis

cou

nt

rate

e pa

ren

te U

SA

ver

age

earn

ings

ofH

ead

Sta

rtp

are

nts

rela

tive

toU

S

aver

age

04

6H

ead

Sta

rtP

rogra

mF

act

s

IGE

Inte

rgen

erati

onal

inco

me

elast

icit

y04

0L

eean

dS

olon

(2009)

eA

ver

age

pre

sen

td

isco

un

ted

valu

eof

life

tim

eea

rnin

gs

for

Hea

dS

tart

ap

pli

can

ts$3433

92

[1

(1

e pa

ren

te U

S)I

GE

]eU

S

01

eE

ffec

tof

a1

std

d

ev

incr

ease

inte

stsc

ores

onea

rnin

gs

ofH

ead

Sta

rtap

pli

can

ts$343

39

LA

TE

hL

ocal

aver

age

trea

tmen

tef

fect

02

47

HS

IS

Marg

inal

tax

rate

for

Hea

dS

tart

pop

ula

tion

03

5C

BO

(2012)

Sc

Sh

are

ofH

ead

Sta

rtp

opu

lati

ond

raw

nfr

omot

her

pre

sch

ools

03

4H

SIS

uh

Marg

inal

cost

ofen

roll

men

tin

Hea

dS

tart

$80

00

Hea

dS

tart

Pro

gra

mF

act

su

cM

arg

inal

cost

ofen

roll

men

tin

oth

erp

resc

hoo

ls$0

Naiv

eass

um

pti

on

uc

=0

$40

00

Pes

sim

isti

cass

um

pti

on

uc

=05

uh

$60

00

Pre

ferr

edass

um

pti

on

uc

=07

5u

h

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1821

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 28: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

TA

BL

EIV

(CO

NT

INU

ED)

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

Pan

elB

M

arg

inal

valu

eof

pu

bli

cfu

nd

sN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

lati

onn

etof

taxes

$55

13

(1)

pL

AT

Eh

MF

CM

arg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uh

ucS

cp

LA

TE

h

naiv

eass

um

pti

on$36

71

Pes

sim

isti

cass

um

pti

on$29

91

Pre

ferr

edass

um

pti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0(0

22)

NM

BM

FC

(std

er

r)

naiv

eass

um

pti

onp

-valu

e=

1B

reak

even

p= efrac14

00

9eth00

1THORN

15

0(0

34)

Pes

sim

isti

cass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

8eth00

1THORN

18

4(0

47)

Pre

ferr

edass

um

pti

onp

-valu

e=

00

Bre

ak

even

p=efrac14

00

7eth00

1THORN

Not

es

Th

ista

ble

rep

orts

resu

lts

ofco

st-b

enefi

tca

lcu

lati

ons

for

Hea

dS

tart

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

Sta

nd

ard

erro

rsfo

rM

VP

Fra

tios

are

calc

ula

ted

usi

ng

the

del

tam

eth

od

P-v

alu

esare

from

one-

tail

edte

sts

ofth

en

ull

hyp

oth

eses

that

the

MV

PF

isle

ssth

an

1

Th

ese

test

sare

per

form

edvia

non

para

met

ric

blo

ckboo

tstr

ap

ofth

et-

stati

stic

cl

ust

ered

at

the

Hea

dS

tart

cen

ter

level

B

reak

even

sgiv

ep

erce

nta

ge

effe

cts

ofa

stan

dard

dev

iati

onof

test

scor

eson

earn

ings

that

set

MV

PF

equ

al

to1

QUARTERLY JOURNAL OF ECONOMICS1822

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 29: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

Notably our preferred estimate of 184 is well above the esti-mated rates of return of comparable expenditure programs sum-marized in Hendren (2016) and comparable to the marginalvalue of public funds associated with increases in the top mar-ginal tax rate (between 133 and 20)

To assess the sensitivity of our results to alternative assump-tions regarding the relationship between test score effects andearnings Table IV also reports breakeven relationships betweentest scores and earnings that set MVPF equal to 1 for each valueof rsquoc When rsquoc frac14 0 the breakeven earnings effect is 9 per testscore standard deviation only slightly below our calibrated valueof 10 This indicates that when substitution is ignored HeadStart is close to breaking even and small changes in assumptionswill yield values of MVPF below 1 Increasing rsquoc to 05rsquoh or 075rsquoh

reduces the breakeven earnings effect to 8 or 7 respectivelyThe latter figure is well below comparable estimates in the recentliterature such as estimates from Chetty et alrsquos (2011) study ofthe Tennessee STAR class size experiment (13 see AppendixTable AIV) Therefore after accounting for fiscal externalitiesHead Startrsquos costs are estimated to exceed its benefits only if itstest score impacts translate into earnings gains at a lower ratethan similar interventions for which earnings data are available

VII Beyond LATE

Thus far we have evaluated the return to a marginal expan-sion of Head Start under the assumption that the mix of compli-ers can be held constant However it is likely that major reformsto Head Start would entail changes to program features such asaccessibility that could in turn change the mix of program com-pliers To evaluate such reforms it is necessary to predict howselection into Head Start is likely to change and how this affectsthe programrsquos rate of return

VIIA IV Estimates of SubLATEs

A first way in which selection into Head Start could change isif the mix of compliers drawn from home care and competingpreschools were altered while holding the composition of thosetwo groups constant To predict the effects of such a change onthe programrsquos rate of return we need to estimate the subLATEs inequation (3)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1823

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 30: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

One approach to identifying subLATEs is to conduct two-stage least squares (2SLS) estimation treating Head Start enroll-ment and enrollment in other preschools as separate endogenousvariables A common strategy for generating instruments in suchsettings is to interact an experimentally assigned program offerwith observed covariates or site indicators (eg Kling Liebmanand Katz 2007 Abdulkadiroglu Angrist and Pathak 2014) Suchapproaches can secure identification in a constant effects frame-work but as we demonstrate in Online Appendix E will typicallyfail to identify interpretable parameters if the subLATEs them-selves vary across the interacting groups (see Hull 2015 andKirkeboen Leuven and Mogstad 2016 for related results)

Table V reports 2SLS estimates of the separate effects ofHead Start and competing preschools using as instruments theHead Start offer indicator and its interaction with eight student-and site-level covariates likely to capture heterogeneity in com-pliance patterns13 These instruments strongly predict HeadStart enrollment but induce relatively weak independent varia-tion in enrollment in other preschools with a partial first-stage F-statistic of only 18 The 2SLS estimates indicate that Head Startand other centers yield large and roughly equivalent effects ontest scores of approximately 04 standard deviations This findingis roughly in line with the view that preschool effects are homo-geneous and that program substitution simply attenuates IV es-timates of the effect of Head Start relative to home careCautioning against this interpretation is the 2SLS overidentifica-tion test which strongly rejects the constant effects model indi-cating the presence of substantial effect heterogeneity acrosscovariate groups

A separate source of variation comes from experimental sitesthe HSIS was implemented at hundreds of individual Head Startcenters and previous studies have shown substantial variation intreatment effects across these centers (Bloom and Weiland 2015

13 Previous analyses of the HSIS have shown important effect heterogeneitywith respect to baseline scores and first language (Bitler Domina and Hoynes2014 Bloom and Weiland 2015) so we include these in the list of student-levelinteractions We also allow interactions with variables measuring whether achildrsquos center of random assignment offers transportation to Head Start whetherthe center of random assignment is above the median of the Head Start qualitymeasure the education level of the childrsquos mother whether the child is age fourwhether the child is black and an indicator for family income above the povertyline

QUARTERLY JOURNAL OF ECONOMICS1824

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 31: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

Walters 2015) Using site interactions as instruments againyields much more independent variation in Head Start enroll-ment than in competing preschools14 However the estimatedimpact of Head Start is smaller and competing centers are esti-mated to yield no gains relative to home care While these site-based estimates are nominally more precise than those obtainedfrom the covariate interactions with 183 instruments the

TABLE V

TWO-STAGE LEAST SQUARES ESTIMATES WITH INTERACTION INSTRUMENTS

One endogenousvariable Two endogenous

variables

Head Start Head Start Other centersInstruments (1) (2) (3)

Offer 0247(1 instrument) (0031)

Offer covariates 0241 0384 0419(9 instruments) (0030) (0127) (0359)First-stage F 2762 177 18Overid p-value 007 006

Offer sites 0210 0213 0008(183 instruments) (0026) (0039) (0095)First-stage F 2151 900 27Overid p-value 002 002

Offer site groups 0229 0265 0110(6 instruments) (0029) (0056) (0146)First-stage F 10152 3391 326Overid p-value 077 050

Offer covariates and 0229 0302 0225offer site groups

(14 instruments)(0029) (0054) (0134)

First-stage F 3402 1212 133Overid p-value 012 010

Notes This table reports two-stage least squares estimates of the effects of Head Start and otherpreschool centers in spring 2003 The model in the first row instruments Head Start attendance with theHead Start offer Models in the second row instrument Head Start and other preschool attendance withinteractions of the offer and transportation above-median quality race Spanish language motherrsquos ed-ucation an indicator for income above the federal poverty line and baseline score The third row uses theHead Start offer interacted with 183 experimental site indicators as instruments The fourth row usesinteractions of the offer and indicators for groups of experimental sites obtained from a multinomial probitmodel with unobserved group fixed effects as described in Online Appendix G The fifth row uses bothcovariate and site group interactions All models control for main effects of the interacting variables andbaseline covariates First-stage F-statistics are Angrist and Pischke (2009) partial Frsquos Standard errors areclustered at the center level

14 To avoid extreme imbalance in site size we grouped the 356 sites in our datainto 183 sites with 10 or more observations See Online Appendix G for details

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1825

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 32: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

asymptotic standard errors may provide a poor guide to thedegree of uncertainty in the parameter estimates (BoundJaeger and Baker 1995) We explore this issue in OnlineAppendix Table AV which reports limited information maxi-mum likelihood and jackknife IV estimates of the same modelThese approaches yield much larger standard errors and verydifferent point estimates suggesting that weak instrumentbiases are at play here

To deal with these statistical problems we use a choice modelwith discrete unobserved heterogeneity (described in more detaillater) to aggregate Head Start sites together into six groups withsimilar substitution patterns Using the site group interactions asinstruments yields significant independent variation in bothHead Start and competing preschool enrollment and producesresults more in line with those obtained from the covariate inter-actions Pooling the site group and covariate interaction instru-ments together yields the most precise estimates which indicatethat both preschool types increase scores relative to home careand that Head Start is slightly more effective than competingpreschools However the overidentification test continues toreject the constant effects model suggesting that these estimatesare still likely to provide a misleading guide to the underlyingsubLATEs Another important limitation of the interacted 2SLSapproach is that it conditions on realized selection patterns andtherefore cannot be used to predict the effects of reforms thatchange the underlying composition of n- and c-compliers Wenow turn to developing an econometric selection model thatallows us to address both of these limitations

VIIB Selection Model

Our selection model parameterizes the preferences and po-tential outcomes introduced in the model of Section V to motivatea two-step control function estimator Like the interacted 2SLSapproach the proposed estimator exploits interactions of theHead Start offer with covariates and site groups to separatelyidentify the causal effects of care alternatives Unlike the inter-acted 2SLS approach the control function estimator allows theinteracting groups to have different subLATEs that vary para-metrically with the probability of enrolling in Head Start andcompeting preschools

QUARTERLY JOURNAL OF ECONOMICS1826

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 33: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

Normalizing the value of preschool nonparticipation to zerowe assume households have utilities over program alternativesgiven by

Ui hZieth THORN frac14 h XiZieth THORN thorn vih

Ui ceth THORN frac14 c Xieth THORN thorn vic

Ui neth THORN frac14 0

eth13THORN

where Xi denotes the vector of baseline household and experi-mental site characteristics listed in Table I and Zi again de-notes the Head Start offer dummy The stochastic componentsof utility vih viceth THORN reflect unobserved differences in householddemand for Head Start and competing preschools relative tohome care In addition to pure preference heterogeneity theseterms may capture unobserved constraints such as whetherfamily members are available to help with child care We sup-pose these components obey a multinomial probit specification

vih viceth THORNjXiZi N 01 Xieth THORN

Xieth THORN 1

which allows for violations of the independence from irrelevantalternatives (IIA) condition that underlies multinomial logit se-lection models such as that of Dubin and McFadden (1984)

As in the Heckman (1979) selection framework we modelendogeneity in participation decisions by allowing linear depen-dence of mean potential outcomes on the unobservables that in-fluence choices Specifically for each program alternatived 2 hcnf g we assume

E Yi deth THORNjXiZi vih vicfrac12 frac14 d Xieth THORN thorn dhvih thorn dcviceth14THORN

The dh dc

coefficients in equation (14) describe the nature of

selection on unobservables This specification can accommodate avariety of selection schemes For example if dh frac14 h gt 0 thenconditional on observables selection into Head Start is governedby potential outcome levelsmdashthose most likely to participate inHead Start have higher test scores in all care environments Butif hh gt 0 and nh frac14 hh then households engage in Roy (1951)-style selection into Head Start based on test score gainsmdashthosemost likely to participate in Head Start receive larger test scorebenefits when they switch from home care to Head Start

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1827

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 34: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

By iterated expectations equation (14) implies the condi-tional expectation of realized outcomes can be written

E YijXiZiDi frac14 dfrac12 frac14 d Xieth THORN thorn dhh XiZideth THORN

thorn dcc XiZideth THORNeth15THORN

where h XiZiDieth THORN E vihjXiZiDifrac12 and c XiZiDieth THORN

E vicjXiZiDifrac12 are generalizations of the standard inverseMills ratio terms used in the two-step Heckman (1979) selectioncorrection (see Online Appendix F for details) These termsdepend on Xi and Zi only through the conditional probabilitiesof enrolling in Head Start and other preschools

VIIC Identification

To demonstrate identification of the selection coefficientsdh dc

it is useful to eliminate the main effect of the covariates

by differencing equation (15) across values of the program offer Zi

as follows

E YijXiZi frac14 1Di frac14 dfrac12 E YijXiZi frac14 0Di frac14 dfrac12

frac14 dh h Xi1deth THORN h Xi0deth THORNfrac12 thorn dc c Xi1deth THORN c Xi0deth THORNfrac12 eth16THORN

This difference measures how selected test score outcomes in aparticular care alternative respond to a Head Start offerResponses in selected outcomes are driven entirely by composi-tional changesmdashthat is from compliers switching betweenalternatives

With two values of the covariates Xi equation (16) can beevaluated twice yielding two equations in the two unknown se-lection coefficients Online Appendix F details the conditionsunder which this system can be solved and provides expressionsfor the selection coefficients in terms of population momentsAdditive separability of the potential outcomes in observablesand unobservables is essential for identification If the selectioncoefficients in equation (16) were allowed to depend on Xi therewould be two unknowns for every value of the covariates andidentification would fail Heuristically then our key assumptionis that selection on unobservables works lsquolsquothe same wayrsquorsquo for everyvalue of the covariates which allows us to exploit variation inselected outcome responses across subgroups to infer the param-eters governing the selection process

QUARTERLY JOURNAL OF ECONOMICS1828

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 35: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

To understand this restriction suppose (as turns out to bethe case) that college-educated mothers are more likely to enrolltheir children in competing preschools when denied access toHead Start Our model allows Head Start and other preschoolsto have different average treatment effects on the children ofmore and less educated mothers However it rules out the possi-bility that children with college-educated mothers sort into HeadStart on the basis of potential test score gains while children ofless educated mothers exhibit no sorting on these gains As inBrinch Mogstad and Wiswall (forthcoming) this restriction istestable when Xi takes more than two values because it implieswe should obtain similar estimates of the selection coefficientsbased on variation in different subsets of the covariates We pro-vide evidence along these lines by contrasting estimates that ex-ploit site variation with estimates based on household covariates

VIID Estimation

To make estimation tractable we approximate h XZeth THORN

and cethXTHORN with flexible linear functions The nonseparabilityof hethXZTHORN is captured by linear interactions between Z andthe eight covariates used in our earlier 2SLS analysis We alsoallow interactions with the 183 experimental site indicatorsbut to avoid incidental parameters problems we constrain thecoefficients on those dummies to belong to one of K discrete cat-egories Results from Bonhomme and Manresa (2015) and Saggio(2012) suggest that this lsquolsquogrouped fixed effectsrsquorsquo approach shouldyield good finite sample performance even when some sites haveas few as 10 observations As described in Online Appendix Gwe choose the number of site groups K using the Bayesian in-formation criterion (BIC) Finally all of the interacting variables(both site groups and covariates) are allowed to influence thecorrelation parameter Xeth THORN We assume that arctanhethXTHORN frac14 1

2 ln1thorn Xeth THORN1 Xeth THORN

is linear in these variables a standard transformation

that ensures the correlation is between 1 and 1 (Cox 2008)The model is fit in two steps First we estimate the param-

eters of the probit model via simulated maximum likelihood eval-uating choice probabilities with the Geweke-Hajivassiliou-Keane(GHK) simulator (Geweke 1989 Keane 1994 Hajivassiliou andMcFadden 1998) Models including site groups are estimatedwith an algorithm that alternates between maximizing the like-lihood and reassigning groups described in detail in Online

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1829

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 36: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

Appendix G Second we use the parameters of the choice model toform control function estimates h XiZiDieth THORN c XiZiDieth THORN

which are then used in a second step regression of the form

Yifrac14n0thornX 0nxthornnhh XiZiDieth THORNthornncc XiZiDieth THORN

thorn1 Difrac14cf gc0n0eth THORNthornXi

0 cxnxeth THORNthorn chnh

h XiZiceth THORN

thorn ccnc

c XiZiceth THORN

thorn1 Difrac14hf gh0n0eth THORNthornXi

0 hxnxeth THORNthorn hhnh

h XiZiheth THORN

thorn hcnc

c XiZiheth THORN

24

35thornei

(17)

The covariate vector Xi is normed to have unconditional meanzero so the intercepts d0 can be interpreted as average potentialoutcomes Hence the differences h0 n0 and h0 c0 captureaverage treatment effects of Head Start and other preschools rel-ative to no preschool To avoid overfitting we restrict variablesother than the site types and eight key covariates to havecommon coefficients across care alternatives15 Inference on thesecond-step parameters is conducted via the nonparametricblock bootstrap clustered by experimental site

VIII Model Estimates

VIIIA Model Parameters

Table VI reports estimates of the full choice model obtainedfrom exploiting both covariates and site heterogeneity The BICselects a specification with six site groups for the full model (seeOnline Appendix Table AVI) with group shares that vary be-tween 12 and 21 of the sample These assignments make upthe site groups used in the earlier 2SLS analysis of Table V

Columns (1) and (2) of Table VI show the coefficients govern-ing the mean utility of enrollment in Head Start We easily rejectthe null hypothesis that the program offer interaction effects inthe Head Start utility equation are homogeneous Panel A col-umn (2) indicates that the effects of an offer are greater at high-quality centers and lower among nonpoor children that would

15 This restriction cannot be statistically rejected and has minimal effects onthe point estimates

QUARTERLY JOURNAL OF ECONOMICS1830

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 37: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

TABLE VI

MULTINOMIAL PROBIT ESTIMATES

Head Start utility

(1) (2) (3) (4)Maineffect

Offerinteraction

Other centerutility Arctanh

Panel A CovariatesCenter provides 0022 0111 0054 0096transportation (0114) (0142) (0087) (0178)Above-median 0233 0425 0115 0007center quality (0091) (0102) (0082) (0153)Black 0095 0282 0206 0185

(0108) (0127) (0100) (0166)Spanish speaker 0049 0273 0213 0262

(0136) (0122) (0169) (0224)Motherrsquos education 0106 0021 0105 0219

(0056) (0060) (0064) (0110)Income above FPL 0216 0305 0173 0097

(0128) (0121) (0126) (0192)Baseline score 0080 0025 0292 0026

(0094) (0108) (0069) (0094)Age 4 0164 0277 0518 0010

(0142) (0166) (0104) (0170)p-values no heterogeneity 015 000 000 666

Panel B Experimental site groupsGroup 1 0644 2095 0424 0435(share = 0215) (0136) (0153) (0085) (0128)Group 2 4847 6760 0577 0496(share = 0183) (0076) (0158) (0045) (0172)Group 3 2148 2912 0768 0530(share = 0183) (0312) (0340) (0081) (0159)Group 4 0488 0541 0139 0483(share = 0151) (0130) (0150) (0226) (0322)Group 5 1243 2849 1643 0772(share = 0145) (0108) (0171) (0164) (0359)Group 6 0072 1191 0110 2988(share = 0124) (0127) (0183) (0106) (0925)p-values no heterogeneity 000 000 000 000

Notes This table reports simulated maximum likelihood estimates of a multinomial probit model ofpreschool choice The model includes fixed effects for six unobserved groups of experimental sites esti-mated as described in Online Appendix G The Head Start and other center utilities also include the maineffects of sex test language teen mother motherrsquos marital status presence of both parents family sizespecial education family income categories and second- and third-order terms in baseline test scores Thelikelihood is evaluated using the GHK simulator and likelihood contributions are weighted by the recip-rocal of the probability of experimental assignments P-values for site heterogeneity are from tests that allgroup-specific constants are equal P-values for covariate heterogeneity are from tests that all covariatecoefficients in a column are 0 Standard errors are clustered at the Head Start center level

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1831

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 38: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

typically be ineligible for Head Start enrollment16 Panel B col-umn (2) reveals the presence of significant heterogeneity acrosssite groups in the response to a program offer which likely re-flects unobserved market features such as the presence or ab-sence of state provided preschool

Column (4) of Table VI reports the parameters governing thecorrelation in unobserved tastes for Head Start and competingprograms The correlation is positive for four of six site groupsindicating that most households view preschool alternatives asmore similar to each other than to home care This establishesthat the IIA condition underlying logit-based choice models isempirically violated Although there is some evidence of hetero-geneity in the correlation based on motherrsquos education we cannotreject the joint null hypothesis that the correlation is constantacross covariate groups However we easily reject that the cor-relation is constant across site groups

The many sources of heterogeneity captured by the choicemodel yield substantial variation in predicted enrollment sharesfor Head Start and competing preschools Online Appendix FigureAI shows that these predictions match variation in choice proba-bilities across subgroups Moreover diagnostics indicate this vari-ation is adequate to secure separate identification of the secondstage control function coefficients From equation (16) the modelis underidentified if for any alternative d the control functiondifference h Xi1deth THORN h Xi0deth THORN is linearly dependent on the cor-responding difference c Xi1deth THORN c Xi0deth THORN Online AppendixFigure AII shows that the deviations from linear dependence arevisually apparent and strongly statistically significant

Table VII reports second-step estimates of the parameters inequation (17) Column (1) omits all controls and simply reportsdifferences in mean test scores across care alternatives (the omit-ted category is home care) Head Start students score 02 stan-dard deviations higher than students in home care and thecorresponding difference for students in competing preschools is026 standard deviations Column (2) adds controls for baseline

16 The quality variable aggregates information on center characteristics (tea-cher and center director education and qualifications class size) and practices (va-riety of literacy and math activities home visiting health and nutrition) measuredin interviews with center directors teachers and parents of children enrolled in thepreschool center

QUARTERLY JOURNAL OF ECONOMICS1832

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 39: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

TABLE VII

SELECTION-CORRECTED ESTIMATES OF PRESCHOOL EFFECTS

Least squares Control function

(1) (2) (3) (4) (5)No controls Covariates Covariates Site groups Full model

Head Start 0202 0218 0483 0380 0470(0037) (0022) (0117) (0121) (0101)

Other preschools 0262 0151 0183 0065 0109(0052) (0035) (0269) (0991) (0253)

lh 0015 0004 0019(0053) (0063) (0053)

Head Start lh 0167 0137 0158(0080) (0126) (0091)

Otherpreschools lh

0030 0047 0000(0109) (0366) (0115)

lc 0333 0174 0293(0203) (0187) (0115)

Head Start lc 0224 0065 0131(0306) (0453) (0172)

Otherpreschools lc

0488 0440 0486(0248) (0926) (0197)

p-valuesNo selection 016 510 046No selection

on gains133 560 084

Additiveseparability

261 452 349

Notes This table reports selection-corrected estimates of the effects of Head Start and other preschoolcenters in spring 2003 Each column shows coefficients from regressions of test scores on an intercept aHead Start indicator an other preschool indicator and controls Column (1) shows estimates with nocontrols Column (2) adds controls for sex race home language test language motherrsquos education teenmother motherrsquos marital status presence of both parents family size special education income catego-ries experimental site characteristics (transportation above-median quality and urban status) and athird-order polynomial in baseline test score This column interacts the preschool variables with trans-portation above-median quality race Spanish language motherrsquos education an indicator for incomeabove the federal poverty line and the main effect of baseline score Covariates are demeaned in theestimation sample so that main effects can be interpreted as estimates of average treatment effectsColumn (3) adds control function terms constructed from a multinomial probit model using the covariatesfrom column (2) and the Head Start offer The interacting variables from column (2) are allowed tointeract with the Head Start offer and enter the preschool taste correlation equation in column (3)Column (4) omits observed covariates and includes indicators for experimental site groups constructedusing the algorithm described in Online Appendix G The multinomial probit model is saturated in thesesite group indicators and the second-step regression interacts site groups with preschool alternativesColumn (5) combines the variables used in columns (3) and (4) Standard errors are bootstrapped andclustered at the center level The bottom row shows p-values from a score test of the hypothesis thatinteractions between the control functions and covariates are 0 in each preschool alternative (see OnlineAppendix F for details)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1833

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 40: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

characteristics Because the controls include a third-order poly-nomial in baseline test scores this column can be thought of asreporting lsquolsquovalue-addedrsquorsquo estimates of the sort that have receivedrenewed attention in the education literature (Kane Rockoff andStaiger 2008 Rothstein 2010 Chetty Friedman and Rockoff2014a) Surprisingly adding these controls does little to the esti-mated effect of Head Start relative to home care but improvesprecision By contrast the estimated impact of competing pre-schools relative to home care falls significantly once controls areadded

Columns (3)ndash(5) add control functions adjusting for selectionon unobservables based on choice models with covariates sitegroups or both Unlike the specifications in previous columnsthese control function terms exploit experimental variation inoffer assignment Adjusting for selection on unobservables dra-matically raises the estimated average impact of Head Start rel-ative to home care However the estimates are fairly impreciseImprecision in estimates of average treatment effects is to be ex-pected given that these quantities are only identified via para-metric restrictions that allow us to infer the counterfactualoutcomes of always takers and never takers Below we consideraverage treatment effects on compliers which are estimatedmore precisely

Although some of the control function coefficient estimatesare also imprecise we reject the hypotheses of no selection onlevels (kd frac14 0 8ethkdTHORNTHORN and no selection on gains (dk frac14 jk ford 6frac14 j k 2 fhcg) in our most precise specification The selectioncoefficient estimates exhibit some interesting patterns One reg-ularity is that estimates of hh nh are negative in all specifica-tions (though insignificant in the model using site groups only) Inother words children who are more likely to attend Head Startreceive smaller achievement benefits when shifted from homecare to Head Start This lsquolsquoreverse Royrsquorsquo pattern of negative selec-tion on test score gains suggests large benefits for children withunobservables making them less likely to attend the program17

Other preschool programs by contrast seem to exhibit positiveselection on gains the estimated difference cc nc is always

17 Walters (2014) finds a related pattern of negative selection in the context ofcharter schools though in his setting the fall-back potential outcome (as opposed tothe charter school outcome) appears to respond positively to unobserved character-istics driving program participation

QUARTERLY JOURNAL OF ECONOMICS1834

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 41: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

positive and is significant in the full model A possible interpre-tation of these patterns is that Head Start is viewed by parents asa preschool of last resort leading to enrollment by the familiesmost desperate to get help with child care Such householdscannot be selective about whether the local Head Start center isa good match for their child which results in lower test scoregains By contrast households considering enrollment in substi-tute preschools may have greater resources that afford them theluxury of being more selective about whether such programs are agood match for their child

Estimates of the control function coefficients are very similarin columns (3) and (4) though the estimates are less precise whenonly site group interactions are used This indicates that the im-plied nature of selection is the same regardless of whether iden-tification is based on site or covariate interactions lendingcredibility to our assumption that selection works the same wayacross subgroups Also supporting this assumption are the re-sults of score tests for the additive separability of control func-tions and covariates reported in the bottom row of Table VIIThese tests are conducted by regressing residuals from the two-step models on interactions of the control functions with covari-ates and site groups along with the main effects from equation(15) In all specifications we fail to reject additive separability atconventional levels (see Online Appendix F for some additionalgoodness-of-fit tests) Although these tests do not have the powerto detect all forms of nonseparability the correspondence be-tween estimates based on covariate and site variation suggeststhat our key identifying assumption is reasonable

VIIIB Treatment Effects

Table VIII reports average treatment effects on compliers foreach of our selection-corrected models The first row uses themodel parameters to compute the pooled LATEh which is non-parametrically identified by the experiment The model estimatesline up closely with the nonparametric estimate obtained via IVOnline Appendix Figure AIII shows that this close correspon-dence between model and nonparametric LATEh holds evenacross covariate groups with very different average treatmenteffects The remaining rows of Table VIII report estimates of av-erage effects for compliers relative to specific care alternatives

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1835

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 42: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

(ie subLATEs)18 Estimates of the subLATE for n-compliersLATEnh are stable across specifications and indicate that theimpact of moving from home care to Head Start is largemdashon theorder of 037 standard deviations By contrast estimates ofLATEch though more variable across specifications never differsignificantly from zero

Our estimates of LATEnh are somewhat smaller than theaverage treatment effects of Head Start relative to home caredisplayed in Table VII This is a consequence of the reverse Roypattern captured by the control function coefficients familieswilling to switch from home care to Head Start in response toan offer have stronger than average tastes for Head Start imply-ing smaller than average gains We can reject that predicted

TABLE VIII

TREATMENT EFFECTS FOR SUBPOPULATIONS

Control function

(1) (2) (3) (4)Parameter IV Covariates Sites Full model

LATEh 0247 0261 0190 0214(0031) (0032) (0076) (0042)

LATEnh 0386 0341 0370(0143) (0219) (0088)

LATEch 0023 0122 0093(0251) (0469) (0154)

Lowest predictedquintile

LATEh 0095 0114 0027(0061) (0112) (0067)

LATEh with fixed Sc 0125 0125 0130(0060) (0434) (0119)

Highest predictedquintile

LATEh 0402 0249 0472(0042) (0173) (0079)

LATEh with fixed Sc 0364 0289 0350(0056) (1049) (0126)

Notes This table reports estimates of treatment effects for subpopulations Column (1) reports an IVestimate of the effect of Head Start Columns (2)ndash(4) show estimates of treatment effects computed fromthe control function models displayed in Table VII The bottom rows show effects in the lowest and highestquintiles of model-predicted LATE Rows with fixed c-complier shares weight subLATEs using the full-sample estimate of this share (034) Standard errors are bootstrapped and clustered at the center level

18 We compute the subLATEs by integrating over the relevant regions of Xivih and vic as described in Online Appendix F

QUARTERLY JOURNAL OF ECONOMICS1836

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 43: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

effects of moving from home care to Head Start are equal for n-compliers and n-never takers implying that this pattern is sta-tistically significant (p = 038) Likewise LATEhc is slightly neg-ative whereas the average treatment effect of Head Start relativeto other preschools is positive (047 011) In other wordsswitching from c to h reduces test scores for c-compliers butwould improve the score of an average student This reflects acombination of above average tastes for competing preschoolsamong c-compliers and positive selection on gains into other pre-schools Note that the control function coefficients in Table VIIcapture selection conditional on covariates and sites while thetreatment effects in Table VIII average over the distribution ofobservables for each subgroup The subLATE estimates showthat the selection patterns discussed here still hold when varia-tion in effects across covariate and site groups is taken intoaccount

Another interesting point of comparison is to the 2SLS esti-mates of Table V The 2SLS approach found a somewhat smallerLATEnh than our two-step estimator It also found that HeadStart preschools were slightly more effective at raising testscores than were competing programs (LATEch gt 0) whereasour full control function estimates suggest the oppositeImportantly the control function estimates corroborate thefailed overidentification tests of Table V by detecting substantialheterogeneity in the underlying subLATEs This can be seen inthe last four rows of Table VIII which report estimates for the topand bottom quintiles of the model-predicted distribution ofLATEh (see Online Appendix F for details) Fixing each grouprsquosSc at the population average brings estimates for the top andbottom quintiles closer together but a large gap remains due tosubLATE heterogeneity

Finally it is worth comparing our findings with those ofFeller et al (forthcoming) who use the principal stratificationframework of Frangakis and Rubin (2002) to estimate effects onn- and c-compliers in the HSIS They also find large effects forcompliers drawn from home and negligible effects for compliersdrawn from other preschools though their point estimate ofLATEnh is somewhat smaller than ours (021 versus 037) Thisdifference reflects a combination of different test score outcomes(Feller et al look only at PPVT scores) and different modelingassumptions Since neither estimation approach nests theother it is reassuring that we find qualitatively similar results

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1837

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 44: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

TA

BL

EIX

BE

NE

FIT

SA

ND

CO

ST

SO

FH

EA

DS

TA

RT

WH

EN

CO

MP

ET

ING

PR

ES

CH

OO

LS

AR

ER

AT

ION

ED

(1)

(2)

(3)

(4)

Para

met

erD

escr

ipti

onV

alu

eS

ourc

e

LA

TE

hH

ead

Sta

rtlo

cal

aver

age

trea

tmen

tef

fect

02

47

HS

ISL

AT

En

cE

ffec

tof

oth

erce

nte

rsfo

rm

arg

inal

chil

dre

n0

Naiv

eass

um

pti

on

no

effe

ctof

com

pet

ing

pre

sch

ools

03

70

Hom

ogen

eity

ass

um

pti

on

LA

TE

nc

equ

als

LA

TE

nh

02

94

Mod

el-b

ase

dp

red

icti

onN

MB

Marg

inal

ben

efit

toH

ead

Sta

rtp

opu

la-

tion

net

ofta

xes

$55

13

(1)

p(L

AT

Eh

+L

AT

En

cS

c)

naiv

eass

um

pti

on$83

21

Hom

ogen

eity

ass

um

pti

on$77

44

Mod

el-b

ase

dp

red

icti

onM

FC

Marg

inal

fisc

al

cost

ofH

ead

Sta

rten

roll

men

t$50

31

uhp

(LA

TE

h+

LA

TE

nc

Sc)

n

aiv

eass

um

pti

on$35

19

Hom

ogen

eity

ass

um

pti

on$38

30

Mod

el-b

ase

dp

red

icti

onM

VP

FM

arg

inal

valu

eof

pu

bli

cfu

nd

s11

0N

aiv

eass

um

pti

on23

6H

omog

enei

tyass

um

pti

on20

2M

odel

-base

dp

red

icti

on

Not

es

Th

ista

ble

rep

orts

resu

lts

ofa

rate

ofre

turn

calc

ula

tion

for

Hea

dS

tart

ass

um

ing

that

com

pet

ing

pre

sch

ools

are

rati

oned

an

dth

at

marg

inal

stu

den

tsof

fere

dse

ats

inth

ese

pro

gra

ms

as

are

sult

ofH

ead

Sta

rtex

pan

sion

wou

ldot

her

wis

ere

ceiv

eh

ome

care

P

ara

met

ervalu

esare

obta

ined

from

the

sou

rces

list

edin

colu

mn

(4)

QUARTERLY JOURNAL OF ECONOMICS1838

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 45: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

IX Policy Counterfactuals

We now use our model estimates to consider policy counter-factuals that are not nonparametrically identified by the HSISexperiment

IXA Rationed Substitutes

In the cost-benefit analysis of Section VI we assumed thatseats at competing preschools were not rationed Although thisassumption is reasonable in states with universal preschool man-dates other areas may have preschool programs that face rela-tively fixed budgets and offer any vacated seats to new childrenIn this case increases in Head Start enrollment will create op-portunities for new children to attend substitute preschoolsrather than generating cost savings in these programs Ourmodel-based estimates allow us to assess the sensitivity of ourcost-benefit results to the possibility of rationing in competingprograms

From equation (9) the marginal value of public funds underrationing depends on LATEncmdashthe average treatment effect ofcompeting preschools on lsquolsquon- to c-compliersrsquorsquo who would movefrom home care to a competing preschool program in responseto an offered seat We compute the MVPFrat under three alter-native assumptions regarding this parameter First we considerthe case where LATEnc = 0 Next we consider the case where theaverage test score effect of competing preschools for marginalstudents equals the corresponding effect for Head Start compliersdrawn from home care (ie LATEnc = LATEnh) Finally we useour model to construct an estimate for LATEnc Specifically wecompute average treatment effects of competing preschools rela-tive to home care for students who would be induced to movealong this margin by an increase in UiethcTHORN equal to the utilityvalue of the Head Start offer coefficient19 This calculation as-sumes that the utility value households place on an offered seat

19 Ideally we would compute LATEnc for students who do not receive offers tocompeting programs but would attend these programs if offered Because we do notobserve offers to substitute preschools it is not possible to distinguish betweennonoffered children and children who decline offers Our estimate of LATEnc there-fore captures a mix of effects for compliers who would respond to offers and childrenwho currently decline offers but would be induced to attend competing programs ifthese programs became more attractive

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1839

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 46: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

at a competing program is comparable to the value of a HeadStart offer

Table IX shows the results of this analysis Setting LATEnc =0 yields an MVPFrat of 110 This replicates the naive analysiswith rsquoc frac14 0 in the nonrationed analysis Both of these casesignore costs and benefits due to substitution from competing pro-grams Assuming that LATEnc = LATEnh produces a benefit-costratio of 236 Finally our preferred model estimates from SectionVIII predict that LATEnc = 0294 which produces a ratio of 202These results suggest that under plausible assumptions aboutthe effects of competing programs relative to home care account-ing for the benefits generated by vacated seats in these programsyields estimated social returns larger than those displayed inTable IV Panel B

IXB Structural Reforms

We next predict the social benefits of a reform that expandsHead Start by making it more attractive rather than by extendingoffers to additional households This reform is modeled as an im-provement in the structural program feature f as described inSection V Examples of such reforms might include increases intransportation services outreach efforts or spending on otherservices that make Head Start attractive to parents Increasesin f are assumed to draw additional households into Head Startbut to have no effect on potential outcomes which rules out peereffects generated by changes in student composition We use theestimates from our preferred model to compute marginal treat-ment effects and marginal values of public funds for such reformstreating changes in f as shifts in the mean Head Start utility hethXZTHORN

Figure I Panel A displays predicted effects of structural re-forms on test scores Since the program feature has no intrinsicscale the horizontal axis is scaled in terms of the Head Startattendance rate with a vertical line indicating the current rate(f = 0) The right axis measures ~Scmdashthe share of marginal stu-dents drawn from other preschools The left axis measures testscore effects The figure plots average treatment effects for sub-groups of marginal students drawn from home care and otherpreschools along with MTEh a weighted average of alterna-tive-specific effects

QUARTERLY JOURNAL OF ECONOMICS1840

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 47: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

A

B

FIGURE I

Effects of Structural Reforms

This figure plots predicted test score effects and marginal values of publicfunds for various values of the program feature f which shifts the utility of HeadStart attendance Horizontal axes shows the Head Start attendance rate at each fand a vertical line indicates the HSIS attendance rate (f = 0) Panel A showsmarginal treatment effects and competing preschool compliance shares The leftaxis measures test score effects MTEh is the average effect for marginal studentsand MTEnh and MTEch are effects for subgroups of marginal students drawn fromhome care and other preschools The right axis measures the share of marginalstudents drawn from other preschools The shaded region shows a 90 symmetricbootstrap confidence interval for MTEh Panel B shows predicted marginal valuesof public funds for structural reforms using the same parameter calibrations asTable IV P-values come from bootstrap tests of the hypothesis that the marginalvalue of public funds is less than or equal to 1 at f = 0

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1841

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 48: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

Figure I shows that Head Startrsquos effects on marginal homecompliers increase modestly with enrollment and then level outin the neighborhood of the current program scale (f = 0) Thispattern is driven by reverse Roy selection for children drawnfrom home care increases in f attract children with weakertastes for Head Start leading to increases in effects for complierswho would otherwise stay home Predicted effects for childrendrawn from other preschools are slightly negative for all valuesof f At the current program scale the model predicts that theshare of marginal students drawn from other preschools is largerfor structural reforms than for an increase in the offer rate (044versus 035) This implies that marginal compliers are more likelyto be drawn from other preschools than are inframarginal com-pliers As a result the value of MTEh is comparable to the exper-imental LATEh despite very large effects on marginal childrendrawn from home care (roughly 05 standard deviations)

To investigate the consequences of this pattern for the socialreturn to Head Start Panel B plots MVPFf the marginal value ofpublic funds for structural reforms This figure relies on the sameparameter calibrations as Table IV Calculations of MVPFf mustaccount for the fact that changes in structural program featuresmay increase the direct costs of the program This effect is cap-tured in equation (12) by the term which gives the elasticity ofthe per child cost of Head Start with respect to the scale of theprogram Without specifying the program feature being manipu-lated there is no natural value for We start with the extremecase where = 0 which allows us to characterize costs and ben-efits associated with reforms that draw in children on the marginwithout changing the per capita cost of the program We thenconsider how the cost-benefit calculus changes when gt 0

As in our basic cost-benefit analysis the results in Figure IPanel B show that accounting for the public savings associatedwith program substitution has an important effect on the mar-ginal value of public funds The short-dashed curve plots MVPFf

setting rsquoc frac14 0 This calibration suggests a marginal value ofpublic funds slightly above 1 at the current program scale similarto the naive calibration in Table IV The solid curve accounts forpublic savings by setting rsquoc equal to our preferred value of 075rsquohThis generates an upward shift and steepens the MVPFf sched-ule indicating that both marginal and average social returns in-crease with program scale The implied marginal value of publicfunds at the current program scale (f = 0) is above 2 This is larger

QUARTERLY JOURNAL OF ECONOMICS1842

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 49: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

than the MVPF of 184 reported in Table IV which indicates thesocial returns to marginal expansions that shift the compositionof compliers are greater than those for expansions that simplyraise the offer rate

The final scenario in Panel B shows MVPFf when rsquoc frac14 075rsquoh

and frac14 0520 This scenario implies sharply rising marginal costsof Head Start provision an increase in f that doubles enrollmentraises per capita costs by 50 In this simulation the marginalvalue of public funds is roughly equal to 1 when f = 0 and fallsbelow 1 for higher values Hence if is at least 05 a $1 increasein Head Start spending generated by structural reform will resultin less than $1 transferred to Head Start applicants This exerciseillustrates the quantitative importance of determining provisioncosts when evaluating specific policy changes such as improve-ments to transportation services or marketing

Our analysis of structural reforms suggests increasing re-turns to the expansion of Head Start in the neighborhood of thecurrent program scalemdashexpansions will draw in households withweaker tastes for preschool with above average potential gainsThese findings imply that structural reforms targeting childrenwho are currently unlikely to attend Head Start and children thatare likely to be drawn from nonparticipation will generate largereffects than reforms that simply create more seats Our resultsalso echo other recent studies finding increasing returns to earlychildhood investments though the mechanism generating in-creasing returns in these studies is typically dynamic comple-mentarity in human capital investments rather than selectionand effect heterogeneity (see eg Cunha Heckman andSchennach 2010)

X Conclusion

Our analysis suggests that Head Start in its current incar-nation passes a strict cost-benefit test predicated only on proj-ected effects on adult earnings It is reasonable to expect that thisconclusion would be strengthened by incorporating the value ofany impacts on crime (as in Lochner and Moretti 2004 and

20 For this case marginal costs are obtained by solving the differential equa-tion rsquo

0

hethf THORN frac14 rsquoethf THORNqln P Difrac14heth THORN

qf

with the initial condition rsquoheth0THORN frac14 8000 This yields the

solution rsquohethf THORN frac14 8000exp ln P Di frac14 heth THORN ln P0eth THORNeth THORN where P0 is the initial HeadStart attendance rate

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1843

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 50: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

Heckman et al 2010) or other externalities such as civic engage-ment (Milligan Moretti and Oreopoulos 2004) or by incorporat-ing the value to parents of subsidized care (as in Aaberge et al2010) We find evidence that Head Start generates especiallylarge benefits for children who would not otherwise attend pre-school and for children with weak unobserved tastes for the pro-gram This suggests that the programrsquos rate of return can beboosted by reforms that target new populations though this ne-cessitates the existence of a cost-effective technology for attract-ing these children

The finding that returns are on average greater for nonpar-ticipants is potentially informative for the debate over calls foruniversal preschool which might reach high-return householdsHowever it is important to note that if competing state-level pre-school programs become ubiquitous the rationale for expansionsto federal preschool programs could be undermined To see thisconsider how the marginal value of expanding Head Startchanges as the compliance share Sc approaches 1 so thatnearly all denied Head Start applicants would otherwise enrollin competing programs If Head Start and competing programshave equivalent effects on test scores then equation (8) indicatesthat we should decide between federal and state-level provisionbased entirely on cost criteria Since state programs are oftencheaper (CEA 2014) and are expanding rapidly the case for fed-eral preschool may actually be weaker now than at the time of theHSIS

It is important to note some other limitations to our analysisFirst our cost-benefit calculations rely on literature estimates ofthe link between test score effects and earnings gains These cal-culations are necessarily speculative as the only way to be sure ofHead Startrsquos long-run effects is to directly measure long-run out-comes for HSIS participants Second we have ignored the possi-bility that substantial changes to program features or scale couldin equilibrium change the education production technology Forexample implementing recent proposals for universal preschoolcould generate a shortage of qualified teachers (Rothstein 2015)Finally we have ignored the possibility that administrative pro-gram costs might change with program scale choosing instead toequate average with marginal provision costs

Despite these caveats our analysis has shown that account-ing for program substitution in the HSIS experiment is crucial foran assessment of the Head Start programrsquos costs and benefits

QUARTERLY JOURNAL OF ECONOMICS1844

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 51: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

Similar issues arise in the evaluation of job training programs(Heckman et al 2000) health insurance (Finkelstein et al2012) and housing subsidies (Kling Liebman and Katz 2007Jacob and Ludwig 2012) The tools developed here are potentiallyapplicable to a wide variety of evaluation settings where data onenrollment in competing programs are available

Supplementary Material

An Online Appendix for this article can be found at QJEonline (qjeoxfordjournalsorg)

University of California Berkeley and National

Bureau of Economic Research

University of California Berkeley and National

Bureau of Economic Research

References

Aaberge Rolf Manudeep Bhuller Audun Langoslashrgen and Magne Mogstad lsquolsquoTheDistributional Impact of Public Services When Needs Differrsquorsquo Journal ofPublic Economics 94 (2010) 549ndash562

Abdulkadiroglu Atila Joshua Angrist and Parag Pathak lsquolsquoThe Elite IllusionAchievement Effects at Boston and New York Exam SchoolsrsquorsquoEconometrica 82 (2014) 137ndash196

Angrist Joshua Guido Imbens and Donald Rubin lsquolsquoIdentification of CausalEffects Using Instrumental Variablesrsquorsquo Journal of the American StatisticalAssociation 91 (1996) 444ndash455

Angrist Joshua and Jorn-Steffen Pischke Mostly Harmless Econometrics(Princeton NJ Princeton University Press 2009)

Barnett W Steven lsquolsquoEffectiveness of Early Educational Interventionrsquorsquo Science333 (2011) 975ndash978

Bitler Marianne Thurston Domina and Hilary Hoynes lsquolsquoExperimental Evidenceon Distributional Effects of Head Startrsquorsquo NBER Working Paper 20434 2014

Bloom Howard and Christina Weiland lsquolsquoQuantifying Variation in Head StartEffects on Young Childrenrsquos Cognitive and Socio-Emotional Skills UsingData from the National Head Start Impact Studyrsquorsquo MDRC Report 2015

Bonhomme Stephane and Elena Manresa lsquolsquoGrouped Patterns of Heterogeneityin Panel Datarsquorsquo Econometrica 83 (2015) 1147ndash1184

Bound John David Jaeger and Regina Baker lsquolsquoProblems with InstrumentalVariables Estimation When the Correlation Between the Instruments andthe Endogenous Explanatory Variable Is Weakrsquorsquo Journal of the AmericanStatistical Association 90 (1995) 443ndash450

Brinch Christian Magne Mogstad and Matthew Wiswall lsquolsquoBeyond LATE with aDiscrete Instrumentrsquorsquo Journal of Political Economy forthcoming

Carneiro Pedro and Rita Ginja lsquolsquoLong-Term Impacts of Compensatory Preschoolon Health and Behavior Evidence from Head Startrsquorsquo American EconomicJournal Economic Policy 6 (2014) 135ndash173

Cascio Elizabeth and Diane Whitmore Schanzenbach lsquolsquoThe Impacts ofExpanding Access to High-Quality Preschool Educationrsquorsquo Brookings Paperson Economic Activity Fall (2013) 127ndash192

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1845

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 52: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

Chetty Raj lsquolsquoSufficient Statistics for Welfare Analysis A Bridge betweenStructural and Reduced-Form Methodsrsquorsquo Annual Review of Economics 1(2009) 451ndash488

Chetty Raj John Friedman Nathaniel Hilger Emmanuel Saez DianeWhitmore Schanzenbach and Danny Yagan lsquolsquoHow Does YourKindergarten Classroom Affect Your Earnings Evidence from ProjectSTARrsquorsquo Quarterly Journal of Economics 126 (2011) 1593ndash1660

Chetty Raj John Friedman and Jonah Rockoff lsquolsquoMeasuring the Impacts ofTeachers I Measuring Bias in Teacher Value-added Estimatesrsquorsquo AmericanEconomic Review 104 (2014a) 2593ndash2632

mdashmdashmdash lsquolsquoMeasuring the Impacts of Teachers II Teacher Value-Added and StudentOutcomes in Adulthoodrsquorsquo American Economic Review 104 (2014b) 2633ndash2679

Congressional Budget Office (CBO) lsquolsquoEffective Marginal Tax Rates for Low- andModerate-Income Workersrsquorsquo 2012 available at httpswwwcbogovsitesde-faultfiles11-15-2012-MarginalTaxRatespdf

Council of Economic Advisers (CEA) lsquolsquoThe Economics of Early ChildhoodInvestmentsrsquorsquo 2014 available at httpswwwwhitehousegovsitesdefaultfilesdocsearly_childhood_report1pdf

Cox Nicholas lsquolsquoSpeaking Stata Correlation with Confidence or Fisherrsquos ZRevisitedrsquorsquo Stata Journal 8 (2008) 413ndash439

Cunha Flavio James Heckman and Susanne Schennach lsquolsquoEstimating theTechnology of Cognitive and Noncognitive Skill Formationrsquorsquo Econometrica78 (2010) 883ndash931

Currie Janet lsquolsquoEarly Childhood Education Programsrsquorsquo Journal of EconomicPerspectives 15 (2001) 213ndash238

Currie Janet and Duncan Thomas lsquolsquoDoes Head Start Make a DifferencersquorsquoAmerican Economic Review 85 (1995) 341ndash364

Deming David lsquolsquoEarly Childhood Intervention and Life-Cycle Skill DevelopmentEvidence from Head Startrsquorsquo American Economic Journal AppliedEconomics 1 (2009) 111ndash134

Dubin Jeffrey and Daniel McFadden lsquolsquoAn Econometric Analysis of ResidentialElectric Appliance Holdingsrsquorsquo Econometrica 52 (1984) 345ndash362

Engberg John Dennis Epple Jason Imbrogno Holger Sieg and Ron ZimmerlsquolsquoEvaluating Education Programs That Have Lotteried Admission andSelective Attritionrsquorsquo Journal of Labor Economics 32 (2014) 27ndash63

Federal Register lsquolsquoExecutive Order 13330 of February 24 2004rsquorsquo Federal Register69 (38) (2004) 9185ndash9187

Feller Avi Todd Grindal Luke Miratrix and Lindsay Page lsquolsquoCompared to WhatVariation in the Impact of Early Childhood Education by Alternative Care-Type Settingsrsquorsquo Annals of Applied Statistics forthcoming

Finkelstein Amy Sarah Taubman Bill Wright Mira BernsteinJonathan Gruber Joseph Newhouse Heidi Allen Katherine Baicker andthe Oregon Health Study Group lsquolsquoThe Oregon Health InsuranceExperiment Evidence from the First Yearrsquorsquo Quarterly Journal ofEconomics 127 (2012) 1057ndash1106

Frangakis Constantine and Donald Rubin lsquolsquoPrincipal Stratification in CausalInferencersquorsquo Biometrics 58 (2002) 21ndash29

Garces Eliana Duncan Thomas and Janet Currie lsquolsquoLonger-Term Effects of HeadStartrsquorsquo American Economic Review 92 (2002) 999ndash1012

Gelber Alexander and Adam Isen lsquolsquoChildrenrsquos Schooling and ParentsrsquoInvestment in Children Evidence from the Head Start Impact StudyrsquorsquoJournal of Public Economics 101 (2013) 25ndash38

Geweke John lsquolsquoBayesian Inference in Econometric Models Using Monte CarloIntegrationrsquorsquo Econometrica 57 (1989) 1317ndash1339

Gibbs Chloe Jens Ludwig and Douglas Miller lsquolsquoDoes Head Start Do Any LastingGoodrsquorsquo NBER Working Paper 17452 2011

Hajivassiliou Vassilis and Daniel McFadden lsquolsquoThe Method of Simulated Scoresfor the Estimation of LDV Modelsrsquorsquo Econometrica 66 (1998) 863ndash898

Hall Peter The Bootstrap and Edgeworth Expansion (New York Springer 1992)Heckman James lsquolsquoSample Selection Bias as a Specification Errorrsquorsquo Econometrica

47 (1979) 153ndash161

QUARTERLY JOURNAL OF ECONOMICS1846

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 53: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

Heckman James Neil Hohmann Jeffrey Smith and Michael Khoo lsquolsquoSubstitutionand Dropout Bias in Social Experiments A Study of an Influential SocialExperimentrsquorsquo Quarterly Journal of Economics 115 (2000) 651ndash694

Heckman James Seong Hyeok Moon Rodrigo Pinto Peter Savelyev andAdam Yavitz lsquolsquoThe Rate of Return to the HighScope Perry PreschoolProgramrsquorsquo Journal of Public Economics 94 (2010) 114ndash128

Heckman James Rodrigo Pinto and Peter Savelyev lsquolsquoUnderstanding theMechanisms through Which an Influential Early Childhood ProgramBoosted Adult Outcomesrsquorsquo American Economic Review 103 (2013) 2052ndash2086

Heckman James Sergio Urzua and Edward Vytlacil lsquolsquoInstrumental Variables inModels with Multiple Outcomes The General Unordered Casersquorsquo AnnalesdrsquoEconomie et de Statistique 9192 (2008) 151ndash174

Heckman James and Edward Vytlacil lsquolsquoLocal Instrumental Variables andLatent Variable Models for Identifying and Bounding Treatment EffectsrsquorsquoProceedings of the National Academy of Sciences 96 (1999) 4730ndash4734

Hendren Nathaniel lsquolsquoThe Policy Elasticityrsquorsquo Tax Policy and the Economy 30(2016)

Hull Peter lsquolsquoIsoLATEing Identifying Counterfactual-Specific Treatment Effectsby Stratified Comparisonrsquorsquo working paper 2015

Imbens Guido and Joshua Angrist lsquolsquoIdentification and Estimation of LocalAverage Treatment Effectsrsquorsquo Econometrica 62 (1994) 467ndash475

Jacob Brian and Jens Ludwig lsquolsquoThe Effects of Housing Assistance on LaborSupply Evidence from a Voucher Lotteryrsquorsquo American Economic Review102 (2012) 272ndash304

Kane Thomas Jonah Rockoff and Douglas Staiger lsquolsquoWhat Does Certification TellUs about Teacher Effectiveness Evidence from New York Cityrsquorsquo Economicsof Education Review 27 (2008) 615ndash631

Keane Michael lsquolsquoA Computationally Practical Simulation Estimator for PanelDatarsquorsquo Econometrica 62 (1994) 95ndash116

Kirkeboen Lars Edwin Leuven and Magne Mogstad lsquolsquoField of Study Earningsand Self-Selectionrsquorsquo Quarterly Journal of Economics 131 (2016) doi101093qjeqjw019 (this issue)

Klein Joe lsquolsquoTime to Ax Public Programs that Donrsquot Yield Resultsrsquorsquo Time July 72011

Kling Jeffrey Jeffrey Liebman and Lawrence Katz lsquolsquoExperimental Analysis ofNeighborhood Effectsrsquorsquo Econometrica 75 (2007) 83ndash119

Lafontaine Francine and Kenneth White lsquolsquoObtaining Any Wald Statistic YouWantrsquorsquo Economics Letters 21 (1986) 35ndash40

Lee Chul-In and Gary Solon lsquolsquoTrends in Intergenerational Income MobilityrsquorsquoReview of Economics and Statistics 91 (2009) 766ndash772

Lochner Lance and Enrico Moretti lsquolsquoThe Effect of Education on Crime Evidencefrom Prison Inmates Arrests and Self-Reportsrsquorsquo American Economic Review94 (2004) 155ndash189

Long Cuiping lsquolsquoExperimental Evidence of the Effect of Head Start on MaternalHuman Capital Investmentrsquorsquo working paper 2015

Ludwig Jens and Douglas Miller lsquolsquoDoes Head Start Improve Childrenrsquos LifeChances Evidence from a Regression Discontinuity Designrsquorsquo QuarterlyJournal of Economics 122 (2007) 159ndash208

Ludwig Jens and Deborah Phillips lsquolsquoThe Benefits and Costs of Head StartrsquorsquoNBER Working Paper 12973 2007

Mayshar Joram lsquolsquoOn Measures of Excess Burden and Their ApplicationrsquorsquoJournal of Public Economics 43 (1990) 263ndash289

Milligan Kevin Enrico Moretti and Philip Oreopoulos lsquolsquoDoes Education ImproveCitizenship Evidence from the United States and the United KingdomrsquorsquoJournal of Public Economics 88 (2004) 1667ndash1695

Puma Michael Stephen Bell Ronna Cook and Camilla Heid Head Start ImpactStudy Final Report (Washington DC US Department of Health andServices Administration for Children and Families 2010)

Puma Michael Stephen Bell and Camilla Heid Third Grade Follow-Up to theHead Start Impact Study (Washington DC US Department of Health andHuman Services Administration for Children and Families 2012)

PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES 1847

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from

Page 54: EVALUATING PUBLIC PROGRAMS WITH CLOSE …crwalters/papers/kline_walters.pdfEVALUATING PUBLIC PROGRAMS WITH CLOSE SUBSTITUTES: THE CASE OF HEAD START Patrick Kline and Christopher R

Rothstein Jesse lsquolsquoTeacher Quality in Educational Production Tracking Decayand Student Achievementrsquorsquo Quarterly Journal of Economics 125 (2010)175ndash214

mdashmdashmdash lsquolsquoTeacher Quality Policy When Supply Mattersrsquorsquo American EconomicReview 105 (2015) 100ndash130

Roy A D lsquolsquoSome Thoughts on the Distribution of Earningsrsquorsquo Oxford EconomicsPapers 3 (1951) 135ndash146

Saggio Raffaele lsquolsquoDiscrete Unobserved Heterogeneity in Discrete Choice PanelData Modelsrsquorsquo masterrsquos thesis Center for Monetary and Financial Studies2012

Schumacher Rachel Mark Greenberg and Janellen Duffy The Impact of TANFFunding on State Child Care Subsidy Programs (Washington DC Center forLaw and Social Policy 2001)

Stossel John lsquolsquoHead Start Has Little Effect by Grade Schoolrsquorsquo Fox BusinessTelevision March 7 2014

US Department of Health and Human Services (DHHS) Administration forChildren and Families lsquolsquoChild Care and Development Fund Fact Sheetrsquorsquo2012 available at httpwwwacfhhsgovsitesdefaultfilesoccccdf_fact-sheetpdf

mdashmdashmdash lsquolsquoHead Start Program Facts Fiscal Year 2013rsquorsquo 2013 available at httpeclkcohsacfhhsgovhslcmrfactsheetsdocshs-program-fact-sheet-2011-finalpdf

mdashmdashmdash lsquolsquoHead Start Servicesrsquorsquo 2014 available at httpwwwacfhhsgovpro-gramsohsabouthead-start

Walters Christopher lsquolsquoThe Demand for Effective Charter Schoolsrsquorsquo NBERWorking Paper 20640 2014

mdashmdashmdash lsquolsquoInputs in the Production of Early Childhood Human Capital Evidencefrom Head Startrsquorsquo American Economic Journal Applied Economics 7 (2015)76ndash102

QUARTERLY JOURNAL OF ECONOMICS1848

at University of C

alifornia Berkeley on O

ctober 28 2016httpqjeoxfordjournalsorg

Dow

nloaded from


Java Server Appliances - jbox.dk · Java Server Appliances Datalogforeningen ... • A server appliance for Java programs ... • File system (open, close, read,

Scaling Up to Close the Opportunity Divide for · economy. Practitioners have developed promising training programs to bridge this divide, but these programs have been small relative

Why do so many Electronics Programs close due to low enrollments?

Chuck Close, Up Close

Econometrics Laboratory, UC Berkeley - Identification of Causal …crwalters/papers/multiple_Z.pdf · 2019-02-11 · Identi cation of Causal E ects with Multiple Instruments: Problems

close to SAP, close to Alfresco, close to you! · ... close to SAP, close to Alfresco, close to you! BUSINESS OBJECTS PURCHASE ORDERS INVOICES CONTRACTS EMPLOYEES PRINT LIST INCOMING

Do Parents Value School Effectiveness?crwalters/papers/apsw_nyc.pdfDo Parents Value School Effectiveness?† By Atila Abdulkadiro glu, Parag A. Pathak,ˇ Jonathan Schellenberg, and

2012 PROFITstar Budget Manager Installation Guideprofitstarhelp.jackhenry.com/dev/BudgetManager2012...3 1. Welcome to the Budget Manager Installation Wizard – Close all Windows programs

Statement on Allegations of Academic Fraud by Jay …crwalters/other/greene_memo.pdfStatement on Allegations of Academic Fraud by Jay P. Greene Atila Abdulkadiroglu, Duke Parag Pathak,

Evaluating Public Programs with Close Substitutes: …pkline/papers/kline_walters_12...Evaluating Public Programs with Close Substitutes: The Case of Head Start Patrick Kline UC Berkeley/NBER

close(sql) close(sql) - IBPhoenix - Your Premier Source of ... · close(sql) close(sql) NAME close −close cursor SYNTAX close cursor-name DESCRIPTION The close statement terminates

주께로 가까이 주께로 가오니 Close to Thee, close to Thee, Close to Thee, close to Thee

Leveraging Lotteries for School Value-Added: Testing and ...crwalters/papers/VAM_appx.pdf · school admission lottery records, and information from the centralized BPS student assignment

Evaluating Public Programs with Close Substitutes: The ...pkline/papers/Slides_KW.pdf · Evaluating Public Programs with Close Substitutes: The Case of Head Start ... This Paper Revisit

A hands-on museum for children! SCHOOL PROGRAMS · Choose from over a dozen school programs to educate, ... 51-100 children $30, ... students will get an up-close look at eggs of

OnlineAppendixto“TheDemandforEffectiveCharterSchools”crwalters/papers/JMP_appx.pdf · problemathand. TheoremA1: Supposecharterapplication,lotteryoffer,andattendancerulesaregivenbyB

How close is close enough?

Operational Safety Excellence - osorm.com · Operational Safety Excellence ... Feel leadership commitment and support. ... – Recommend actions and programs needed to close the

CFHT [2013A - 2016B] Large Programs · CFHT [2013A - 2016B] Large Programs MaTYSSE: Magnetic Topologies of Young Stars & the Survival of close-in massive Exoplanets p 1/13 Proprietary

Evaluating Public Programs with Close Substitutes: …pkline/papers/Headstart_QJE.pdfEvaluating Public Programs with Close Substitutes: The Case of Head Start Patrick Kline UC Berkeley/NBER

LEAD FEARLESSLY - rhsmith.umd.edu › files › Images › Programs › ... · analyze big data and think critically to solve business challenges. We maintain close relationships

SFMA CALENDAR YEAR 2020 CLOSING SCHEDULES for … · EOM Close June 20 (Mo12) Weekly Report Run 18 19 20 Run Warrant Expiration Programs 21 CAFR Full Report Run 22 DAFJ620-Close Dates

Closing Out a Federal Aid Construction Project Missouri Local Programs How to Close-Out a Federal Aid Construction Project

The Impact of State Budget Cuts on U.S. Postsecondary ...crwalters/papers/deming_walters.pdf · David J. Deming Harvard University and NBER Christopher R. Walters UC Berkeley and

Installing Flash Player - Training and Developmenttraining.fullerton.edu/documents/installingflashplayer.pdfAdobe Flash Player, please close the following programs that will be listed,

Engaging Staff and Faculty through Green Office Certification Programs Bowen Close - Pomona College Kira Stoll - UC Berkeley

1 CLOSE UP WASHINGTON, DC CIVIC EDUCATION PROGRAMS

futurePRNT 7.3 Software Manual...(10) At this point, all drivers and programs have been installed. Click “Finish” to close the wizard. (11) Click "Exit" to close the following

Helping to Close the Homework Gap - Kajeet...Helping to Close the Homework Gap Funding Student Mobile ... BYOD, library/media center checkout programs, as well as school bus-enabled

Through the Looking Glass: Heckits, LATE, and …crwalters/papers/looking_glass.pdf · Through the Looking Glass: Heckits, LATE, and Numerical Equivalence Patrick Kline UC Berkeley

Spring Programs 2017 - The Bridge Center · Equine Programs April 24—June 10, 2017 Registration begins March 1, 2017 Register Online at Registration will close 5 days before the

International Information Programs - USEmbassy.gov · security concerns caused the Peace Corps to close some programs in some Central Asian countries and reassign them to areas such

Who Benefits from KIPP?crwalters/papers/KIPP.pdf · INTRODUCTION The question of whether charter schools boost academic achievement, and what types of students benefit most from charter

Truck Platoon Testing Allowed Under Assembly Bill … › - › media › dot-media › programs › ...traveling together, confers benefits when under close coordination. These closely

Close to close deal