http://aerj.aera.netJournal

American Educational Research

http://aer.sagepub.com/content/early/2013/05/20/0002831213486334The online version of this article can be found at:

 DOI: 10.3102/0002831213486334

published online 21 May 2013Am Educ Res JCarolyn J. Heinrich and Hiren Nisar

OutcomesThe Efficacy of Private Sector Providers in Improving Public Educational

  

 Published on behalf of

  American Educational Research Association

and

http://www.sagepublications.com

can be found at:American Educational Research JournalAdditional services and information for    

  http://aerj.aera.net/alertsEmail Alerts:

 

http://aerj.aera.net/subscriptionsSubscriptions:  

http://www.aera.net/reprintsReprints:  

http://www.aera.net/permissionsPermissions:  

What is This? 

- May 21, 2013OnlineFirst Version of Record >>

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

The Efficacy of Private Sector Providers inImproving Public Educational Outcomes

Carolyn J. HeinrichUniversity of Texas at Austin

Hiren NisarAbt Associates

School districts required under No Child Left Behind (NCLB) to provide sup-plemental educational services (SES) to students in schools that are not mak-ing adequate yearly progress rely heavily on the private sector to offer choicein services. If the market does not drive out ineffective providers, studentsmay not gain through SES participation. We estimate SES provider effectson students’ math and reading achievement in an urban school districtthat accounts for a significant share of participating students. We expectthis research to inform education policy on the viability of policy interven-tions employing a private market model to improve public sector outcomes,including the reauthorization of Title I and district tutoring interventionsunder NCLB and after federal waivers from NCLB.

KEYWORDS: educational choice, effectiveness, supplemental educationalservices

Introduction

The U.S. No Child Left Behind (NCLB) Act was designed to increase theachievement of economically disadvantaged students by introducing greaterchoice, flexibility, and accountability in public education. Key legislative

CAROLYN J. HEINRICH is the Sid Richardson Professor of Public Affairs and affiliatedProfessor of Economics and Director of the Center for Health and Social Policy atthe Lyndon B. Johnson School of Public Affairs, The University of Texas at Austin,2315 Red River St., PO Box Y, Austin, TX 78713-8925; e-mail: cheinrich@austin.utex-as.edu. Her research focuses on education, labor and social welfare policy, publicmanagement and performance management, and econometric methods for socialprogram evaluation.

HIREN NISAR is a senior analyst/PhD economist at Abt Associates Inc. Much of hisresearch addresses improving student achievement in K–12 education; the impactof scholarships in science, technology, engineering, and mathematics; and evalua-tions of job training programs.

American Educational Research Journal

Month XXXX, Vol. XX, No. X, pp. 1–39

DOI: 10.3102/0002831213486334

� 2013 AERA. http://aerj.aera.net

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

provisions include offering educational choice to those in persistently low-performing schools; empowering parents with more information about thequality of their children’s schools; holding states, districts, and schoolsaccountable for student achievement by identifying and imposing require-ments on schools in need of improvement; and targeting federal funds oneffective practices for improving teacher and school quality.1 As in thedecentralization of government functions in other policy areas (public assis-tance, publicly funded training, child welfare, etc.), the expectation is thatinvolving the private sector in public services delivery will bring abouta more efficient, innovative, competitive, results-oriented, and responsivepublic sector that better meets diverse public preferences, values, and needs(Frederickson & Smith, 2003; Rivlin, 1992).

We study the NCLB provision that requires schools that are not makingadequate yearly progress for 3 years to offer children in low-income familiesthe opportunity to receive extra academic assistance (or tutoring), known assupplemental educational services (SES). The primary responsibility for im-plementing SES lies with school districts, which largely rely on the privatesector to offer eligible students a range of choices for SES. NCLB obligatesschool districts to set aside 20% of their Title I funding for SES and laysout criteria and guidelines for state and local educational agencies in approv-ing SES providers and arranging for their services.2 Importantly, it requiresstate and local educational agencies to measure provider effectiveness inincreasing student achievement and to use this information to withdrawapproval of ineffective providers (Heinrich, 2010).

NCLB also directs school districts to determine eligibility for SES usingthe same information used for making within-district Title I allocations(e.g., free school lunch eligibility), and schools then notify families of theirchildren’s eligibility. If more eligible students sign up than there are fundsavailable for serving them, districts establish additional eligibility criteria, fre-quently based on student special needs and academic performance.However, some eligible students do not follow through in registering forand attending SES, and others stop attending before their SES dollar alloca-tion is expended. Therefore, selection into treatment (or who gets tutored inSES) and for how long is likely influenced by student characteristics as wellas program type and administration (Heinrich, Meyer, & Whitten, 2010;Steinberg, 2011). This makes it enormously challenging for researchers,and even more so for school districts with limited resources, to identify ef-fects of SES providers on student achievement while controlling for the ef-fects of other classroom and school interventions.

Chicago Public Schools (CPS) accounts for a disproportionately largeshare of the students eligible for SES in the United States, and as we discussfurther in the following, CPS has consistently made an effort to evaluate SESeffects, including provider effectiveness. We use the most recently availablelongitudinal data from CPS—the 2007–2008, 2008–2009, 2009–2010, and

Heinrich, Nisar

2 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

2010–2011 school years—to estimate provider-specific effects and the effectsof different types of SES providers using alternative estimation techniquesand subsamples and under differing assumptions about student and pro-vider selection. Based on prior literature and our conceptual framework (dis-cussed in the following section), we hypothesize that there are positiveeffects of attending SES on the achievement of eligible students that willincrease with hours attended, provider characteristics will be correlatedwith SES effectiveness, and some specific providers will be more effectivethan others in delivering SES and increasing student achievement.

We employ four alternative approaches to estimate the effects of SES anddifferent type of SES providers on changes in student test scores while con-trolling for student selection into SES and/or into different provider types.The SES provider types that we examine are: district versus non-district pro-viders, online provision, on-site versus off-site providers, and for-profit ver-sus not-for-profit (nonprofit or public) provision.3 We use gains in test scoresas our outcome in school value-added, student fixed effects, school and stu-dent fixed effects, and propensity score matching models. We control forschool and student time invariant characteristics using these four strategies.Each of these modeling approaches makes somewhat different assumptionsabout selection into SES, and as we are estimating different types of effects(e.g., provider specific, provider type), the analytical samples differ to someextent as well. Thus, while we look for overall consistency in the results, weexpect some differences as well.

In general, we find that the CPS district provider delivers significantlymore hours of tutoring to students who register to receive SES, and there ap-pears to be strong linkage between hours of SES attended and increases instudent achievement. In addition, the CPS district provider is an on-site pro-vider, and we also find that SES delivered by on-site providers is effective inincreasing student achievement. Students attending SES with online pro-viders and for-profit providers gain less in reading and math as comparedto other off-line and not-for-profit providers, respectively. We also identifya number of providers that consistently have positive and statistically signif-icant effects on student achievement. Overall, while average effect sizes aresmall (approximately .09 in reading and .06 in math) relative to other educa-tional interventions, there is some limited evidence that the SES programcontributes to improving student achievement.

In the next section, we discuss our conceptual framework and somebackground information, followed by a review of the literature to date onSES effectiveness. We then describe the data and estimation approachesthat we employ in the analysis of SES effects and present some descriptivestatistics on the different types of SES providers and the characteristics ofCPS students they serve. We follow with a discussion of the results of theestimation of SES effects for different provider types and specific providersin CPS and conclude with policy implications.

Efficacy of Private Sector Providers

3 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

Conceptual Framework and Background

Consistent with broader public sector management trends and the clas-sic market paradigm, an important component of recent initiatives toimprove K–12 education effectiveness are efforts to introduce market-drivenmanagement and increase pressures on educational institutions to developnew strategies for improving student learning and educational outcomes.In the context of SES, measuring provider performance and disseminatinginformation on provider effectiveness should foster a more competitive mar-ket for services, contribute to more informed student choices, encourageinnovative approaches to service delivery as providers compete for marketshare, and squeeze out inefficient or ineffective providers through choice.

Indeed, a large number of diverse organizations with widely varyinghourly rates, service costs, tutor qualifications, tutoring session length,instructional strategies, and curricula have entered the market to competefor the opportunity to provide SES. These include national and local organ-izations, for-profit and nonprofit providers, online and off-line providers,those offering services on-site at the schools (and off-site), and in somecases, schools districts engaging directly in SES provision. Burch,Steinberg, and Donovan (2007) described the market for SES as ‘‘a verynew market where hundreds of firms are flocking to take advantage ofthe promise of sizeable revenues’’ (p. 121). NCLB explicitly discouragesstates from taking any actions that might limit the supply of providers andrange of choices available to parents.4

Private companies have long been involved in the delivery of K–12 pub-lic educational services, from textbooks and instructional supports to testingand evaluation, tutoring, and more. More recently, the increasing participa-tion of for-profit entities in direct services provision through vouchers forschool choice, charter schools, and mandatory out-of-school-time interven-tions has drawn criticism of the prospects for ‘‘profiteering’’ in public educa-tion, where for-profit firms are viewed as willing to compromise on qualityand to shortchange students to better their bottom line (Horn, 2011). Theprimary opposing view counters that private firms have considerable poten-tial to cultivate critically needed innovation in educational practice; that is,they are more likely to have the capability and incentives to rapidly expandsuccessful practices and approaches, attract the required financial andhuman capital, and more cost-effectively deliver educational services.

In the context of this debate, Peterson (2003) recounts the compromisebetween Capitol Hill conservatives who supported vouchers as a key leverof accountability through choice and liberal politicians who opposed the en-croaching private sector role in K–12 public education that led to the crea-tion of SES under NCLB. SES allowed for the ‘‘back door’’ entry of privateproviders-for-profit, nonprofit, secular, and religious into public schoolsthat were failing to make adequate yearly progress while preserving an

Heinrich, Nisar

4 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

important role for school districts in arranging access for students to SES andin contracting with private providers. School districts identified as ‘‘in needof improvement’’ were, for the most part, prohibited from directly providingSES, on the premise that if they were not effective during the regular schoolday, they would be unlikely to do better after school. Private providers, alter-natively, might benefit from the fact that SES is voluntary rather than compul-sory, allowing them to potentially work with a more motivated group ofstudents enrolling in an afterschool academic program. In addition, theyare free to hire and fire teachers/tutors (unencumbered by the typical unionrules) and have broad leeway in program structure, focus, and curriculardesign.

School districts have assumed the major responsibility for disseminatinginformation on SES providers approved by state educational agencies andoperating in the district, although to date, this has largely consisted of pro-vider self-reported information on their attributes and effectiveness(Heinrich et al., 2010). SES providers also market their services directly to pa-rents and students. Some school district accountability and evaluation unitshave attempted to measure program and provider effectiveness with admin-istrative and student record data, as in Chicago Public Schools. They faceimportant challenges, however, in properly evaluating student- and pro-vider-level SES effects, given that participation in SES is voluntary among eli-gible students.

In general, school districts have few resources for monitoring SES pro-viders and little leverage for dismissing them,5 and they have criticized stateeducational agencies for their lack of responsiveness to reported problemswith providers, including fraud and ineffectiveness. Chicago PublicSchools has been particularly proactive in its efforts to disseminate availableinformation to parents and school principals about SES provider effective-ness and to develop district policies that support monitoring of providersthat use its school facilities for service delivery. Still, in the absence of accu-rate and fairly complete information on SES provider performance, statesand districts have little capability or leverage for disciplining the market(i.e., sanctioning or disqualifying ineffective providers), and the benefits ofchoice in a competitive market are unlikely to be realized if the purchasers(eligible students or their parents) have inadequate information for choosingproviders.

Review of Literature on SES Effectiveness

Previous research on out-of-school-time programs reports mixed find-ings on the effectiveness of these programs in improving student outcomes(Halpern, 2003; Little, 2007). Many after-school programs, particularly thosewith a greater focus on recreational than educational activities, have beenshown to have minimal effects on students’ academic progress (Hollister,

Efficacy of Private Sector Providers

5 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

2003). SES, however, was designed to explicitly address students’ educationalneeds, and the large literature on other after-school/tutoring programs con-firms their potential to increase student achievement with sufficient hours oftutoring (Dynarski et al., 2004; Halpern, 2003; Lauer et al., 2004; Little, 2007;Vandell et al., 2005). Yet to date, little is also known about what types or at-tributes of SES programs are effective and what policies at state or local levelscan maximize the potential benefits of SES for eligible students.

Early studies of SES effects on student achievement were primarilydescriptive and focused on the challenges of implementing the services inan evolving market (Burch et al., 2007), while more recent studies havesought to empirically estimate the effects of SES on student achievement.Chatterji, Kwon, and Sng (2006) estimate the effects of SES in one NewYork school and found small positive effects. Evaluations conducted byChicago Public Schools in 2003–2004, 2004–2005, 2006–2007 and Jones(2009) reported larger gains in reading and mathematics for students receiv-ing at least 40 hours of tutoring and for students in Grades 4 through 8 whowere not English language learners and who received at least 30 hours ofSES tutoring. A study by the Los Angeles Unified School District (Rickles &Barnhart, 2007) found fairly small program effects, attributed primarily toimproved performance by elementary students. Studies in Minneapolis(Heistad, 2005) and Milwaukee Public Schools (Heinrich et al., 2010), whereaverage SES hours attended are particularly low, did not find statistically sig-nificant, positive effects of SES participation. Zimmer, Gill, Razquin, Booker,and Lockwood (2007) reported average increases in math test score gains of.09 standard deviations for students attending any SES across the eight sitesin their study, and Springer, Pepper, and Ghosh-Dastidar (2009) likewiseestimated increases in test score gains of about .09 standard deviations inmathematics (and .076 standard deviations in reading). In an alternativespecification that accounted for those who registered but did not attendSES, however, Springer et al. did not find statistically significant effects onreading for students attending SES. Springer et al. caution that very few stud-ies rigorously adjust for student selection into SES, identifying only four stud-ies besides their own that did (Heistad, 2005; Heinrich et al., 2010; Zimmer,Christina, Hamilton, & Prine, 2006; Zimmer et al., 2007).

There are likewise few studies that rigorously examine the effects of spe-cific SES providers on students’ academic outcomes (Jones, 2009; Munoz,Potter, & Ross, 2008; Potter, Ross, Paek, Pribesh, & Nunnery, 2006; Ross,Potter, Paek, & McKay, 2008; Zimmer et al., 2007). This is especially prob-lematic given that it was the explicit intent of NCLB to hold providersaccountable by giving students and parents the necessary information onprovider performance to exercise choice and realize the benefits of a com-petitive market for services.

Jones (2009) used multilevel modeling to explore SES provider effectson students who attended SES, controlling for student- and school-level

Heinrich, Nisar

6 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

characteristics.6 He reported moderate effects of attending SES and positiveeffects of several individual providers in Chicago. Studies by Zimmer et al.(2007) and Socias, deSousa, and Le Floch (2009) used a difference-in-differ-ences strategy with value-added models to estimate SES provider effectsacross multiple districts after the introduction of district providers in 2006–2007. The Zimmer et al. study found that participation in SES had positiveeffect on students’ achievement in reading and math, with students partici-pating for multiple years realizing larger gains. Although the Zimmer et al.study did not estimate the effects of specific providers, they did estimatethe effects of a district provider and reported mixed results. Socias et al.found that the district SES provider had no effect on student achievementin Anchorage and Hillsborough.

The Center for Research on Education Policy (CREP) conducted several,multiyear studies in Kentucky, Tennessee, and Louisiana on provider-spe-cific effects using a matched treatment-comparison strategy. Students attend-ing SES were matched on observable characteristics to schoolmates whowere eligible for SES but did not participate. They found mixed results forthe overall effect of attending SES across different states. Specifically,Munoz et al. (2008) analyzed student-level achievement for those who at-tended SES in Kentucky and found no significant effects for any individualprovider or for all providers combined. Similarly, Ross et al. (2008) foundthree providers were significantly worse than the comparison group inmath (suggesting the potential for selection bias that is not adequatelyadjusted through matching) but no effects for any providers in reading forthe 2007–2008 school year in Tennessee. Finally, Potter et al. (2006) foundmost students in Louisiana who were served by SES providers did no betteror worse than their counterparts who were not served.

The gap in knowledge that we aim to fill is to identify not only effectiveproviders but also their attributes and approaches in delivering SES that con-tribute to their success. This should help policymakers better direct the re-sources that school districts are required to set aside for publicly fundedtutoring. Although we realize that these results are based primarily on a sin-gle school district and have limited generalizability, CPS has one of the larg-est numbers of students eligible for and receiving SES, accounting for 10% ofall SES recipients in the nation’s public schools in 2008–2009. It is also one ofa small number of school districts identified as in need of improvement thatsuccessfully petitioned the federal government to directly provide SES. Wemore fully explore the implications of direct service provision by districtsin the discussion of our findings and the concluding section.

Data and Methods

We obtained school record data for all students eligible for SES in CPSfor the school years 2007–2008, 2008–2009, 2009–2010, and 2010–2011.

Efficacy of Private Sector Providers

7 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

The longitudinal database that we constructed includes student test scores,demographics, and information on their registration for and participationin the SES programs. These data allow us to construct measures of students’SES attendance with specific providers, including the number of hours of SESattended and total expenditures from provider invoices. The district also pro-vided information on the SES providers, including whether they were online,off-site, or on-site; district or non-district; for-profit or nonprofit, which al-lows us to explore the types of organizations and methods of service deliv-ery that may contribute to improving student outcomes.

To construct the key outcome measures of student achievement gains(or changes) in student test scores, we use data from standardized tests(Illinois Standardized Achievement Tests; ISAT). For each grade and year,we construct z-scores using the district mean and standard deviation, sothat the test scores are comparable across grades and years. Table 1 showsthe number of CPS students who are eligible, registered, and attended SESfor the different grades and years as well as the percentage of studentswith missing scores. Across the three panels and school years, the distribu-tion of characteristics for those who are eligible, register for, and attend SESlook similar to the subset of those with gain scores. Therefore, the missingdata (other than Grade 3)7 will be treated as random.

Characteristics of Eligible Students and SES providers

Table 2 shows the characteristics of students who are eligible and regis-tered for SES and who attended SES for the 3 school years for which we esti-mate SES effects. These measures, along with the student’s grade year andschool attended, serve as the core set of control variables intended toaccount for selection into SES in the estimation of SES effects. Our choiceof these variables reflects specific criteria used by CPS in prioritizing eligiblestudents for SES, as well as other characteristics we expect might be associ-ated with the likelihood that eligible students will register for and attend SES.Per the law, the primary criterion for most school districts is free lunch eli-gibility, but as the number of students eligible for SES has expanded, districtshave specified additional criteria for registrations, such as grade level, pastperformance in school, and student special needs. For example, in the2009–2010 school year, CPS established the following hierarchy of criteriafor prioritizing registrations among eligible students: (a) free/reduced luncheligible; (b) all students in Grades 1 through 3 and in high school andEnglish language learners (ELL) and students with disabilities (SWD) inGrades 4 through 8; and then (c) students in Grades 4 through 8 by readingstanine. The new emphasis on registering SWD and ELL students in 2009–2010 is reflected in the larger percentage of SWD and ELL students that reg-istered that year (vs. 2008–2009), even though the percentage eligible wereapproximately the same from year to year. We also control for student

Heinrich, Nisar

8 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

9

Tab

le 1

Nu

mb

er o

f S

tud

ents

Elig

ible

, Reg

iste

red

for

Su

pp

lem

enta

l Ed

uca

tio

nal

Ser

vice

s (S

ES

) an

d W

ho

Att

end

ed S

ES

, W

ith

an

d W

ith

ou

t G

ain

Sco

res,

for

the

2008

–20

09, 2

009

–201

0, a

nd

201

0–20

11 S

cho

ol Y

ears

Dis

tric

t M

ean A

dju

sted

Gai

n S

core

s

A

ll St

uden

tsSt

uden

ts W

ith G

ain S

core

sSt

uden

ts W

ith M

issi

ng

Gai

n S

core

s (%

)

Gra

de

Elig

ible

Fr

equen

cyReg

iste

red

Freq

uen

cyA

tten

ded

SES

Freq

uen

cyElig

ible

Fr

equen

cyReg

iste

red

Freq

uen

cyA

tten

ded

SES

Freq

uen

cyElig

ible

Fr

equen

cyReg

iste

red

Freq

uen

cyA

tten

ded

SES

Freq

uen

cy

Yea

r 20

08–2

009

313

,363

7,25

26,

530

1,49

787

075

7N

/AN

/AN

/A4

11,8

235,

849

5,27

911

,336

5,60

55,

070

44

45

11,5

815,

308

4,75

511

,149

5,12

64,

602

43

36

13,0

885,

567

4,92

112

,629

5,37

94,

753

43

37

12,6

954,

441

3,81

812

,263

4,29

23,

697

33

38

12,6

984,

350

3,74

212

,297

4,21

93,

635

33

3To

tal

75,2

4832

,767

29,0

4561

,171

25,4

9122

,514

44

3

Yea

r 20

09–2

010

316

,739

8,03

07,

652

N/A

N/A

N/A

N/A

N/A

N/A

414

,380

3,05

22,

839

13,1

422,

876

2,68

49

65

513

,912

2,72

62,

515

12,7

382,

571

2,38

18

65

614

,182

2,68

72,

448

13,0

782,

536

2,31

48

65

714

,074

1,80

41,

618

12,9

911,

715

1,54

38

55

814

,255

1,73

71,

532

13,1

781,

626

1,43

58

66

Tota

l87

,542

20,0

3618

,604

65,1

2711

,324

10,3

578

65

Yea

r 20

10–2

011

318

,607

7,53

74,

748

1,49

373

759

6N

/AN

/AN

/A4

17,1

813,

073

2,42

316

,475

2,96

12,

384

44

25

17,2

562,

523

1,99

416

,541

2,45

31,

965

43

16

16,9

052,

337

1,79

016

,242

2,25

91,

754

43

27

15,7

801,

437

1,06

214

,992

1,35

21,

015

56

48

16,2

011,

442

1,03

315

,346

1,37

199

65

54

Tota

l10

1,93

018

,349

13,0

5081

,089

11,1

338,

710

44

2

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

absences in the prior school year, as we find, not surprisingly, that studentswho are less likely to attend regular school are also significantly less likely toattend SES after school.8 In addition, students who attended SES in a priorschool year are significantly more likely to attend SES again.

If after controlling for the student characteristics shown in Table 2, otherfactors influencing participation in SES are random, then our estimates of theeffects of SES should approximate the true effects. It is not possible to verify,however, that there are no unobserved, selective differences between eligi-ble students who participate in SES and those who do not (and serve as com-parison group members in our analysis). A well-executed randomassignment experiment would be needed to assume statistical equivalencebetween students participating in SES and eligible nonparticipants. Forexample, if there were consistently greater demand for SES than funding,school districts might be persuaded to allocate opportunities to participatein SES by randomly assigning some fraction of the eligible students toreceive services and others to a control group of students who would notbe invited to participate. This would facilitate a causal analysis of the effectsof SES on student outcomes. In the absence of random assignment, we canonly estimate potential effects or associations that are suggestive of a possiblecausal interpretation of the findings, which we undertake employing rigor-ous econometric methods that adjust for selection.9

Table 3 shows the characteristics of CPS students by SES provider type inthe 2008–2009, 2009–2010, and 2010–2011 school years. A few of the notabledifferences across provider types include a higher proportion of ELL studentsserved by the district provider, a lower proportion of ELL students served byonline providers, and a substantially larger proportion of students with dis-abilities receiving services from off-site providers. Students with disabilitieswere also significantly more likely to attend SES with online providers in2009–2010. These differences in student characteristics across different ser-vice provider types were also confirmed in the first stage propensity scorematching model (discussed in the following section) that predicted SESattendance with different provider types.10

Estimation Strategies

In our estimation of the average effects of SES and of attending SES withdifferent types of providers on changes in student test scores, we adjust forstudent selection in registering for SES and/or their enrollment with particu-lar types of providers (see again Table 3). In addition, we estimate individ-ual, provider-specific effects for SES providers serving at least 30 students ina given school year (and conditional on having data for at least 30 students).SES providers serving fewer than 30 students are combined in a small-pro-vider measure, allowing us to estimate the average effect of smaller

Heinrich, Nisar

10 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

11

Tab

le 2

Stu

den

t C

har

acte

rist

ics

of T

ho

se W

ho

Are

Elig

ible

, Reg

iste

red

, an

d A

tten

ded

Su

pp

lem

enta

l Ed

uca

tio

nal

Ser

vice

s (S

ES

; per

cen

tag

e u

nle

ss

oth

erw

ise

ind

icat

ed)

Yea

r 20

08–2

009

All

Elig

ible

St

uden

tsElig

ible

Stu

den

ts

With

Gai

n S

core

sAll

Reg

iste

red

Studen

tsReg

iste

red S

tuden

ts

With

Gai

n S

core

sAll

Atten

ded

St

uden

tsAtten

ded

Stu

den

ts

With

Gai

n S

core

s

Studen

t ch

arac

teristics

75,2

48

61,1

71

34,8

38

25,4

92

30,3

06

22,5

15

Asi

an1

11

01

0Bla

ck53

5459

6158

59H

ispan

ic44

4439

3840

39W

hite

22

11

11

Oth

er r

ace

00

00

00

Fem

ale

4949

5050

5050

Engl

ish lan

guag

e le

arner

1210

1411

1511

Free

/red

uce

d lunch

100

100

100

100

100

100

Spec

ial ed

uca

tion

1414

1515

1515

Atten

ded

SES

last

yea

r26

2838

4039

41Abse

nt la

st y

ear

65

55

55

Ret

ained

this

yea

r4

24

14

1Rea

d g

ain (

dis

t) z

-sco

re0.

000.

000.

030.

030.

030.

03M

ath g

ain (

dis

t) z

-sco

re0.

000.

000.

030.

030.

040.

03

Yea

r 20

09–2

010

All

Elig

ible

St

uden

tsElig

ible

Stu

den

ts

With

Gai

n S

core

sAll

Reg

iste

red

Studen

tsReg

iste

red S

tuden

ts

With

Gai

n S

core

sAll

Atten

ded

St

uden

tsAtten

ded

Stu

den

ts

With

Gai

n S

core

s

Studen

t ch

arac

teristics

87,5

42

65,4

14

20,0

36

11,3

93

18,6

04

10,4

24

Asi

an2

11

11

1Bla

ck49

4849

4948

49H

ispan

ic47

4846

4747

47W

hite

22

22

22

Oth

er r

ace

00

21

21

Fem

ale

4949

4745

4745

Engl

ish lan

guag

e le

arner

1210

2018

2018

(continued)

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

12

Free

/red

uce

d lunch

100

100

9898

9898

Spec

ial ed

uca

tion

1314

2229

2230

Atten

ded

SES

last

yea

r42

4258

5859

60Abse

nt la

st y

ear

44

44

44

Ret

ained

this

yea

r2

14

14

1Rea

d g

ain (

dis

t) z

-sco

re–0

.01

–0.0

10.

050.

050.

060.

06M

ath g

ain (

dis

t) z

-sco

re–0

.02

–0.0

20.

020.

020.

030.

03

Yea

r 20

10–2

011

All

Elig

ible

St

uden

tsElig

ible

Stu

den

ts

With

Gai

n S

core

sAll

Reg

iste

red

Studen

tsReg

iste

red S

tuden

ts

With

Gai

n S

core

sAll

Atten

ded

St

uden

tsAtten

ded

Stu

den

ts

With

Gai

n S

core

s

Studen

t ch

arac

teristics

101,

930

81,0

89

18,3

49

11,1

33

13,0

50

8,71

0

Asi

an2

21

11

1Bla

ck42

4345

4743

44H

ispan

ic53

5251

4953

52W

hite

22

22

22

Oth

er r

ace

11

11

11

Fem

ale

4949

4848

4847

Engl

ish lan

guag

e le

arner

1613

2621

2621

Free

/red

uce

d lunch

100

100

100

100

100

100

Spec

ial ed

uca

tion

1213

1621

1821

Atten

ded

SES

last

yea

r8

109

1511

16Abse

nt la

st y

ear

55

55

44

Ret

ained

this

yea

r2

35

85

8Rea

d g

ain (

dis

t) z

-sco

re0.

010.

010.

090.

090.

100.

10M

ath g

ain (

dis

t) z

-sco

re0.

010.

010.

090.

090.

100.

10

Tab

le 2

(C

on

tin

ued

)

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

13

Tab

le 3

Ch

arac

teri

stic

s o

f C

hic

ago

Pu

blic

Sch

oo

ls S

tud

ents

Ser

ved

by

Diff

eren

t Typ

es o

f S

up

ple

men

tal E

du

cati

on

al S

ervi

ces

(SE

S)

Pro

vid

ers

in t

he

2008

–20

09, 2

009

–201

0, a

nd

201

0–20

11 S

cho

ol Y

ears

(p

erce

nta

ges

un

less

oth

erw

ise

ind

icat

ed)

Pro

vider

Typ

eD

istric

tN

on-D

istric

tO

nlin

eO

n-S

iteO

ff-S

iteFo

r-Pro

fit

Nonpro

fit

Nu

mber

of

stu

den

ts, ye

ar

20

08

–20

09

4,03

721

,455

4,16

619

,377

9120

,770

626

Asi

an0

01

00

00

Bla

ck53

6270

6166

6263

His

pan

ic46

3628

3734

3636

White

11

11

01

1O

ther

rac

e0

00

00

00

Fem

ale

5050

5150

4950

54Engl

ish lan

guag

e le

arner

s13

117

117

117

Free

lunch

elig

ible

100

100

100

100

100

100

100

Studen

ts w

ith d

isab

ilitie

s16

1515

1527

1516

Atten

ded

SES

last

yea

r41

3940

4042

3938

Abse

nt la

st y

ear

55

55

55

5Ret

ained

this

yea

r1

11

20

12

Rea

d g

ain (

dis

t) z

-sco

re0.

040

0.02

00.

000

0.03

00.

050

0.02

00.

020

Mat

h g

ain (

dis

t) z

-sco

re0.

060

0.02

00.

000

0.03

00.

000

0.02

00.

050

Nu

mber

of

stu

den

ts, ye

ar

20

09

–20

10

1,18

210

,142

1,10

59,

757

759,

886

255

Asi

an0

12

125

10

Bla

ck48

5053

5041

5057

His

pan

ic49

4641

4631

4638

White

12

22

32

1O

ther

rac

e1

12

10

14

Fem

ale

4445

4345

3745

39Engl

ish lan

guag

e le

arner

s20

1714

1717

1714

Free

lunch

elig

ible

9898

9898

100

9896

Studen

ts w

ith d

isab

ilitie

s34

2938

2847

2937

Atten

ded

SES

last

yea

r42

4446

4433

4443

Abse

nt la

st y

ear

44

44

44

5Ret

ained

this

yea

r1

10

10

12

Rea

d g

ain (

dis

t) z

-sco

re0.

100

0.05

00.

040

0.05

00.

320

0.05

00.

110

Mat

h g

ain (

dis

t) z

-sco

re0.

070

0.02

00.

000

0.02

0–0

.060

0.02

00.

060

(continued)

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

14

Nu

mber

of

stu

den

ts, ye

ar

20

10

–20

11

1,13

37,

331

1,06

97,

114

1,41

77,

262

69Asi

an1

11

11

10

Bla

ck39

4580

4638

4552

His

pan

ic57

5119

5157

5148

White

22

02

32

0O

ther

rac

e2

10

11

10

Fem

ale

4647

4848

4447

54Engl

ish lan

guag

e le

arner

s26

209

2120

2020

Free

lunch

elig

ible

100

100

100

100

100

100

100

Studen

ts w

ith d

isab

ilitie

s23

2318

2224

2313

Atten

ded

SES

last

yea

r14

1718

1715

1720

Abse

nt la

st y

ear

44

54

44

4Ret

ained

this

yea

r9

811

88

812

Rea

d g

ain (

dis

t) z

-sco

re0.

148

0.09

10.

073

0.09

50.

115

0.09

10.

085

Mat

h g

ain (

dis

t) z

-sco

re0.

140

0.08

60.

074

0.10

00.

072

0.08

70.

012

Tab

le 3

(C

on

tin

ued

)

Pro

vider

Typ

eD

istric

tN

on-D

istric

tO

nlin

eO

n-S

iteO

ff-S

iteFo

r-Pro

fit

Nonpro

fit

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

providers relative to larger ones. We employ the following four strategies toaddress possible selection bias.

Value-Added Model

One way that education studies deal with selection is using value-addedmodels. The formal value-added model we employ is specified in Equation1. The value-added strategy allows us to control for other classroom andschool interventions that are fixed over time while identifying provider char-acteristics. For example, if there is a reading intervention at a school andthose students also attend SES, failing to control for the intervention (schoolfixed effect) would bias the results. The outcome measure is the achieve-ment gain made by a given student, which accounts for the possibility thatstudents with similar characteristics might enter SES with different underly-ing achievement trajectories (as reflected in their prior test scores). We esti-mate the following equation,

Ajst � Ajst�15aSESjt1bXjt�11ps1mgt1Ejst; ð1Þ

where Ajst is the achievement of student j attending school s in year t; SESjt isan indicator function if the student j attended SES in year t; Xjt21 are studentcharacteristics that include student demographics, percentage absent in prioryear, retained in prior year, and attended SES in prior year; ps is school fixedeffect; mgt are grade by year fixed effects; and Ejst is the random error term.Identification in this specification comes from the average gain in studentachievement after controlling for student characteristics and school andgrade year effects.

Student Fixed Effects Model

The value-added model assumes that selection depends on observedstudent characteristics. Hence, controlling for them allows us to deal withself-selection. However, if selection is on some unobserved or unmeasuredcharacteristics of the students, then a value-added strategy could still lead tobiased results. The student fixed effects model controls for all time-invariantcharacteristics of a student, including those that are not observed or mea-sured. The following model of an educational production differs fromEquation 1 in that it includes student fixed effects (dj) instead of school fixedeffects,

Ajst5aSESjt1bXjt�11dj1mgt1Ejst: ð2Þ

When we take the first difference of Equation 2, we eliminate the studentfixed effect (dj), and the model estimates the average difference betweenthe gains made by students attending SES with the gains made by similar

Efficacy of Private Sector Providers

15 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

students in CPS who were likewise eligible for SES. This formulation impo-ses some restrictions (or assumptions) that are important to note. First, theimpact of students’ prior experience does not deteriorate over time. This im-plies, for example, that the effect of the quality of kindergarten has the sameimpact on student achievement no matter the grade. The second assumptionis that the unobserved effect of attending SES only affects the level but notthe rate of growth in student achievement. A concern with this restrictionis that if students with lower growth are more likely to choose to attendSES, then this type of selection may bias the estimates obtained from a gainsmodel.

In order to relax this restriction, the following equation is estimated,

Ajst � Ajst�15aSESjt1bXjt�11dj1mgt1Ejst: ð3Þ

This approach to estimating the fixed effects model controls for any unob-served differences between students that are constant across time. The esti-mation of this model requires a first difference of Equation 3 and thereforeneeds three or more observations for each student.11 As students self-selectinto the SES program, we deal with this selection by using the gain scoresmade by same student in the prior year. Identification of the average impactof SES in this model comes from students who participate in one or more butnot all years. If these students differ in systematic ways from all students whoattend SES, then the estimator gives a ‘‘local’’ effect (specific to students withthese characteristics) instead of an average effect. In estimating provider-specific effects, identification comes from students who transfer from oneSES provider to another over the period of observation. Therefore, it isimportant that we check the robustness of the model results using alternativeestimation strategies. Table C.1 in Appendix C (in the online journal) showsthe differences in characteristics between the students who are used foridentification and those who are not in this estimation approach.

School and Student Fixed Effects Model

The base model for this estimation strategy is the combination of the twoaforementioned methods. A school fixed effect (ps) is added to Equation 3,which gives:

Ajst � Ajst�15aSESjt1bXjt�11ps1dj1mgt1Ejst; ð4Þ

adding a school fixed effect controls for unmeasured, time-invariant schoolquality. For example, in CPS, school administrators have a role in choosingthe providers that deliver services on-site at their schools. If principals inviteproviders that they believe are best suited to their students and school envi-ronments, provider effects may be correlated with unobservable schoolcharacteristics that might affect student performance. The inclusion of school

Heinrich, Nisar

16 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

fixed effects facilitates controlling for time-invariant school characteristicssuch as average school test scores, neighborhood attributes, parentalinvolvement in the school, and peer composition, to the extent these areunchanging over time. The inclusion of student fixed effects effectively con-trols for student ability and other time-invariant student characteristics.

As discussed previously, identification of the average impact of SES inthis model comes from students who participate in some but not all years,or in the estimation of provider-specific effects, from students who transferfrom one SES provider to another, whereas the identification of the schooleffect comes from students who switch schools. This model is generally pre-ferred over the value added and the student fixed effects models, as it con-trols for both school and student fixed effects.12 On the other hand, if thestudents who switch are different across some time-variant, unobservedcharacteristics, the results from this strategy could still be biased.

Propensity Score Matching Model

The focus of our analysis using matching methods is the estimation ofthe differential effects of different types of providers for students participat-ing in SES. We employ propensity score matching (PSM), a two-step processin which the probability of participation in SES (with a particular type of pro-vider) is first estimated based on student characteristics (X), generating pre-dicted probabilities of participation (propensity scores). The matchingprocess is thereby reduced to a one-dimensional problem of comparing stu-dents who receive SES from a particular type of provider with students withsimilar propensity scores who participate with other providers. In otherwords, if SES participants and comparison group members have the samepropensity scores, the distribution of X across these groups will be the same:

Y0?D jX ) Y0?D jPðXÞ; ð5Þ

and students can be compared on the basis of their propensity scores alone,where D is the treatment of attending SES with a given type of provider.

In applying PSM, we invoke the conditional independence assumption,which implies that after controlling for observable characteristics (X), a stu-dent’s treatment status is unrelated to what his outcome would be in thecounterfactual state (Rosenbaum & Rubin, 1983). The validity of this assump-tion depends largely on the set of variables or student characteristics (X)available for the estimation. There may be some unmeasured factors thatinfluence participation with particular types of SES providers; what is impor-tant is that participation not be predictive of the outcome that would haveoccurred with another type of provider. In addition, because our outcomevariables are defined as the difference between a pre-program and post-program measure, we use a panel form of the matching estimator

Efficacy of Private Sector Providers

17 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

(difference-in-difference matching) that allows for time-invariant, unob-served differences between SES participants and comparison students with-out biasing estimates of program impacts. In estimating this model, we makethe assumption that conditional independence holds for the periods bothbefore (t) and after (t1) treatment:

E Y0t1 � Y0t jD1 5 1; Xð Þ5E Y0t1 � Y0tjD1 5 0;Xð Þ: ð6Þ

This model estimates the average difference between the gains made bystudents attending SES with a specific type of provider with the gains madeby ‘‘matched’’ students attending with other providers, without puttinga functional form on the gain equation (as in the case of student fixed ef-fects). The control variables used in the first-stage matching model are thesame as those included in the other modeling strategies (shown in Table2). The primary PSM matching technique we apply in the second stagemodel is radius matching, which specifies a ‘‘caliper ’’ or maximum propen-sity score distance (.01 in our analysis) by which a match can be made. Ituses not only the nearest neighbor within each caliper, but all comparisoncases within the caliper (based on the specified distance), and the commonsupport condition is imposed to exclude poor matches from the analysis. It isimportant to reiterate that the sample used in this analysis only includes stu-dents who attended SES; thus, the estimates produced are relative compar-isons between providers (or types of providers) that show their differentialeffects on student achievement.

After-matching balancing tests suggested that the matching generallyworked effectively; for each of the different estimations (by year and pro-vider type), the covariates were fully balanced after matching, with justtwo exceptions in 2009–2010. In the estimation of the effects of on-site pro-viders for students attending SES in 2009–2010, English language learnerswere less likely to attend SES with on-site providers (a small difference of.013 that was not reduced by matching), and in this same year, studentswith disabilities were significantly more likely to attend with off-site pro-viders (a larger difference of .165 that was likewise not reduced by match-ing). There were no after-matching balancing concerns (statisticallysignificant differences in covariate means) for 2008–2009 or 2010–2011.

Results of Analyses

Overall Effects of Attending SES

Tables 4 and 5 show the average effects of attending SES (using a dummyvariable for any SES attendance) and the number of hours of SES attended,respectively. Irrespective of the variable or estimation strategy used, we findpositive and statistically significant results of attending SES on math andreading achievement gains for CPS students. The effect size is very similar

Heinrich, Nisar

18 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

using the student fixed effects or school and student fixed effects strategiesin reading and math. Table 4 shows that the effect size for reading is approx-imately .07 to .09 standard deviations, and for math, it is about .05 to .06.These effect sizes are about one-third of the average annual reading andmath gains for elementary and middle school students (as reviewed byHill, Bloom, Black, & Lipsey, 2007).13

Table 5 reports the effects of the number of hours of SES received onstudent achievement in math and reading. We find that there is a positiveand statistically significant effect of an additional hour of SES on studentachievement. Appendix A (in the online journal) shows histograms of thenumber of hours of SES received by students who attended SES in these 3school years (2008–2009, 2009–2010, and 2010–2011). These histogramsshow distinct spikes in the distribution of SES hours attended, typically closeto 40 and 60 hours of SES attended, reflecting, in part, that the number ofhours attended is a function of the rate the providers charge and the maxi-mum dollars allocated per student by CPS. For the average number of hoursof SES received (approximately 40 hours), estimated effect sizes are compa-rable to those shown in Table 4. In Appendix B (in the online journal), wereport findings on SES effects at the 40- and 60-hour attendance thresholds(using PSM methods) only for those students who attended SES (and forbrevity, only for school year 2008–2009). In general, these results are consis-tent with prior studies (Lauer et al., 2004), which suggest effect sizes arelarger for programs offering 45 or more hours of tutoring.

In addition, we also estimated generalized propensity score models14

using data on the total number of hours of SES students attended over theyears 2008 to 2011 to assess the cumulative effects of tutoring on students’math and reading achievement (for those tutored). We are not able to attainbalance across student characteristics at all intervals (e.g., quartiles or dec-iles) of hours of SES attended (likely due to unmeasured selection into dif-ferent levels of SES attendance), and thus, we view these results asillustrative rather than causal. The graphical display of these results inAppendix B suggests a linear relationship between hours tutored and read-ing gains through about 55 hours of tutoring, with diminishing returns toadditional hours of SES setting in around 60 hours of tutoring.Alternatively, the results for students’ math achievement suggest a steady,positive relationship between hours tutored and math gains through 80-plus hours.

Heterogeneous Effects of SES Providers

A diverse range of providers come and go from the tutoring market. Ifthere are identifiable attributes of the more effective providers and this infor-mation is made available to key stakeholders in SES (e.g., state and localeducational agencies, students, and parents), it could have the potential of

Efficacy of Private Sector Providers

19 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

20

Tab

le 4

Th

e A

vera

ge

Eff

ect

of A

tten

din

g S

up

ple

men

tal E

du

cati

on

al S

ervi

ces

(SE

S; u

nd

er a

lter

nat

ive

esti

mat

ion

str

ateg

ies)

Rea

din

g2-

Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Coef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

Atten

ded

SES

0.04

30.

006

0.09

40.

009

0.07

50.

009

0.07

80.

013

0.07

80.

013

0.08

50.

024

0.08

70.

024

Num

ber

of

obse

rvat

ions

61,1

7163

,506

80,5

1020

5,18

720

5,18

712

4,67

712

4,67

7

Num

ber

of

schools

227

454

302

466

466

458

458

Num

ber

of

studen

ts61

,171

63,5

0680

,510

119,

970

119,

970

83,9

4583

,945

M

ath

2-Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Coef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

Atten

ded

SES

0.04

60.

005

0.05

30.

008

0.06

40.

009

0.05

70.

012

0.05

70.

012

0.05

40.

021

0.05

50.

021

Num

ber

of

obse

rvat

ions

61,4

6463

,773

80,6

1420

4,09

420

4,09

412

4,05

912

4,05

9

Num

ber

of

schools

227

455

302

466

466

458

458

Num

ber

of

studen

ts61

,464

63,7

7380

,614

119,

441

119,

441

83,5

7983

,579

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

21

Tab

le 5

Ave

rag

e E

ffec

ts o

f th

e N

um

ber

of

Ho

urs

of

Su

pp

lem

enta

l Ed

uca

tio

nal

Ser

vice

s (S

ES

) at

ten

ded

o

n S

tud

ent A

chie

vem

ent

in M

ath

an

d R

ead

ing

Rea

din

g2-

Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Hours

of SE

S Atten

ded

Coef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

Atten

ded

SES

0.00

110.

0001

30.

0026

0.00

020.

0016

0.00

020.

0020

0.00

030.

0028

0.00

050.

0022

0.00

050.

0022

0.00

05N

um

ber

of

obse

rvat

ions

61,1

7163

,506

80,5

1020

5,18

720

5,18

712

4,67

712

4,67

7

Num

ber

of

schools

227

454

302

466

466

458

458

Num

ber

of

studen

ts61

,171

63,5

0680

,510

119,

970

119,

970

83,9

4583

,945

M

ath

2-Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Coef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

Atten

ded

SES

0.00

130.

0001

0.00

130.

0002

0.00

150.

0002

0.00

160.

0003

0.00

120.

0005

0.00

180.

0005

0.00

180.

0005

Num

ber

of

obse

rvat

ions

61,4

6463

,773

80,6

1420

4,09

420

4,09

412

4,05

912

4,05

9

Num

ber

of

schools

227

455

302

466

466

458

458

Num

ber

of

studen

ts61

,464

63,7

7380

,614

119,

441

119,

441

83,5

7983

,579

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

increasing the effectiveness of SES (and tutoring services more generally)over time. We first discuss results that compare the CPS district providerwith other (non-district) SES providers. Under federal regulations, schooldistricts that have been identified for improvement are not eligible to pro-vide SES, but CPS was one of a small number of districts granted waiversby the U.S. Department of Education to offer SES. Coming policy changesmay allow an increasing number of school districts to engage in direct pro-vision of SES, as well as other flexibility for innovative approaches to serviceprovision.

There are several possible reasons we might expect to observe a differ-ent effect size for the district provider (A.I.M. High) compared to other SESproviders. First, CPS uses only regular school-day teachers as tutors in itsprogram, with the intent to provide continuity in learning from the regularschool day to after-school instruction, as well as to take advantage of teacherknowledge about student needs. That said, if the instruction is less likely tobe innovative or just more of the same from the school day, this featuremight not benefit students. In addition, we know from our analysis of infor-mation on SES provider rates in CPS and other large urban districts that CPSnot only charges a relatively low hourly rate as a district provider (approxi-mately $28 per hour in 2009–2010), but it also appears to have influencedrate setting among other (non-district) providers in the Chicago area. Weobserve the same non-district providers operating in other districts chargingas much as twice the rate they charged for an hour of their services in CPS.Because the hourly rate charged directly affects the number of hours of SESstudents can receive before reaching the district maximum per-student allo-cation, CPS students are attending more hours of SES (compared to studentsin other districts). In this context, it is possible that a district provider couldcontribute to higher hours of SES tutoring received and correspondingly, togreater program effectiveness.

The first analysis includes both SES-eligible students who attended andthose who did not attend SES in the sample, so that the estimated effects arefor district providers and other (non-district) providers relative to outcomesfor eligible students who did not receive SES. (Alternatively, when we restrictour sample to include only students who attended SES, our estimated effectsare differential effects between the district and other providers.) The value-added (with school fixed effects), the student fixed effects, and school withstudent fixed effects results (see Table 6) all show statistically significant ef-fects of CPS district-provided services on students’ math and readingachievement relative to students who do not receive SES. The coefficientsare the changes (measured in standard deviations from district average read-ing and math test scores) in an average student’s outcome that can be ex-pected if the student participates in SES. The estimated coefficients areconsistently larger for district versus non-district providers, suggesting thatattending SES with a district provider may generate a larger effect on student

Heinrich, Nisar

22 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

achievement than attending with a non-district provider. We explicitly testthis in additional analyses discussed in the following and presented inTable 11. Although there are a few differences in estimated effect sizes (es-timates from the fixed effects estimation approaches are slightly larger), theygenerally represent about one-third the average annual student gain scores.

Tables 7 and 8 present the provider-specific effect sizes for the districtprovider and other providers who have at least 30 students attending SESwith them; smaller providers (serving less than 30 students) are groupedinto a single ‘‘small provider’’ indicator. There is fairly strong agreementamong the estimates of value-added and both types of fixed effect modelsresults, with a handful of providers standing out as particularly effective inboth sets of results and/or across the 3 years (i.e., the district providerA.I.M. High, Newton Learning, Orion’s Mind, School Service Systems, andSES of Illinois). With few exceptions, the provider-specific effect sizesfrom the school and student effects models are very close to those of the stu-dent fixed effects models.

In Appendix B (in the online journal), we report an average effect size ofapproximately .06 that was statistically significant for SES providers delivering40 or more hours of tutoring to students. The CPS district provider was gettingsignificantly more hours of SES to the students it served (an average of 48hours, and nearly twice as many as other providers). Thus, it is not surprisingthat the effect size of the district provider is approximately twice the size of theaverage for all providers.15 Effect sizes for Newton Learning, School ServiceSystems, and SES of Illinois are similarly large. These providers have differenthourly rates, although because we view the hourly rate and number of hoursas part of the treatment (as defined or designed by a given provider), we donot include number of hours attended as a covariate.

Tables 9 and 10 report average effects of the district and other providertypes on CPS students’ reading and math achievement in 2008–2009, 2009–2010, and 2010–2011, estimated using value-added models, student fixed ef-fects models, and school and student fixed effects models. Table 11 showsthe differential effects of different types of providers for those studentswho attended SES using the three aforementioned estimation techniquesand propensity score matching (respectively). Even though the samples dif-fer to some extent (particularly for Table 11, where students attending withdifferent types of providers are compared), the results are fairly consistentacross specifications. On-site providers consistently have positive effectson student math and reading achievement (effect sizes of .042–.081), andstudents attending SES with the district provider generally outperformedother on-site providers (although the difference between district and non-district providers is not statistically significant, as verified by an F test,Prob . F = .152).16 The results also suggest that online providers are gener-ally less effective than other providers. The difference in the coefficients ofonline versus on-site are statistically significant at 5% significance level. The

Efficacy of Private Sector Providers

23 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

24

Tab

le 6

Th

e A

vera

ge

Eff

ects

of A

tten

din

g S

up

ple

men

tal E

du

cati

on

al S

ervi

ces

(SE

S) W

ith

Dis

tric

t P

rovi

der

Ver

sus

a N

on

-Dis

tric

t P

rovi

der

Rea

din

g2-

Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Coef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

District

0.04

50.

013

0.12

80.

019

0.09

10.

020

0.11

10.

027

0.10

60.

026

0.11

30.

039

0.10

70.

040

Non-district

0.04

20.

006

0.08

80.

009

0.07

20.

010

0.07

30.

014

0.07

40.

014

0.07

80.

025

0.08

00.

025

Num

ber

of

obse

rvat

ions

61,1

7163

,506

80,5

1020

5,18

720

5,18

712

4,67

712

4,67

7

Num

ber

of

schools

227

454

302

466

466

458

458

Num

ber

of

studen

ts61

,171

63,5

0680

,510

119,

970

119,

970

83,9

4583

,945

M

ath

2-Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Coef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

District

0.06

70.

011

0.06

50.

020

0.09

20.

022

0.12

30.

032

0.11

50.

031

0.11

40.

039

0.10

40.

039

Non-district

0.03

70.

005

0.04

70.

008

0.05

90.

009

0.04

80.

013

0.04

80.

013

0.03

70.

020

0.03

90.

021

Num

ber

of

obse

rvat

ions

61.4

6463

,773

80,6

1420

4,09

420

4,09

412

4,05

912

4,05

9

Num

ber

of

schools

227

455

302

466

466

458

458

Num

ber

of

studen

ts61

,464

63,7

7380

,614

119,

441

119,

441

83,5

7983

,579

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

25

Tab

le 7

Eff

ects

of A

tten

din

g S

up

ple

men

tal E

du

cati

on

al S

ervi

ces

(SE

S) W

ith

th

e D

istr

ict

Pro

vid

er a

nd

Oth

er P

rovi

der

s o

n R

ead

ing

Gai

ns

2-Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Rea

din

g G

ain

Coef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

Dis

tric

t0.

045

0.01

30.

129

0.01

90.

091

0.02

00.

110

0.02

70.

105

0.02

60.

112

0.04

00.

106

0.04

0Sm

all

pro

vid

ers

0.03

30.

038

0.17

40.

048

0.08

20.

075

0.12

00.

060

0.11

90.

060

0.13

50.

121

0.14

20.

124

A+ T

uto

ring

Serv

ice,

LTD

0.09

50.

044

0.15

30.

098

0.14

70.

101

0.16

00.

140

0.15

90.

142

Aca

dem

ic A

dva

nta

ge

0.10

70.

045

–0.0

470.

134

–0.0

450.

137

ASP

IRA

0.00

90.

006

–0.0

290.

069

–0.0

240.

067

–0.0

660.

073

–0.0

590.

070

Bab

bag

e N

et S

chool

0.04

30.

034

0.08

10.

082

–0.0

200.

073

0.02

50.

055

0.03

70.

056

0.08

70.

098

0.08

90.

098

Bla

ck S

tar

Pro

ject

0.08

90.

069

0.05

10.

122

0.01

80.

136

0.03

50.

135

0.11

10.

170

0.11

90.

170

Bra

in H

urr

ican

e0.

080

0.02

40.

040

0.02

20.

107

0.02

50.

101

0.05

50.

090

0.05

30.

120

0.07

90.

122

0.07

9Bra

infu

se O

ne-

to-O

ne

–0.0

050.

031

0.04

40.

060

0.25

60.

075

0.00

70.

069

0.01

20.

070

–0.0

370.

123

–0.0

270.

125

Brilli

ance

Aca

dem

y0.

061

0.04

7

0.

009

0.03

00.

054

0.07

20.

067

0.07

00.

162

0.11

30.

168

0.11

0Cam

bridge

Educa

tional

0.

050

0.02

80.

094

0.04

10.

016

0.03

20.

032

0.05

60.

023

0.05

70.

029

0.10

20.

022

0.10

2Ches

s Aca

dem

y0.

034

0.02

9–0

.030

0.02

60.

114

0.04

00.

063

0.04

00.

059

0.04

10.

059

0.07

80.

067

0.08

1Child

ren’s H

om

e+A

id

Soc.

0.00

70.

020

0.04

40.

099

0.03

80.

113

0.02

50.

097

0.03

50.

099

–0.0

860.

120

–0.0

780.

127

Clu

bZ! T

uto

ring

Serv

ice

0.04

80.

047

0.12

60.

083

0.02

60.

046

0.06

00.

077

0.05

70.

078

0.16

20.

112

0.15

90.

115

CSC

Jule

x Le

arnin

g0.

051

0.05

1

0.

153

0.20

50.

129

0.18

8Educa

te O

nlin

e 0.

022

0.01

60.

194

0.04

9

0.

055

0.04

80.

055

0.04

80.

072

0.05

20.

067

0.05

4Fa

ilure

Fre

e Rea

din

g –0

.029

0.06

2

0.

248

0.24

20.

225

0.23

8H

untin

gton (

on-s

ite)

0.04

90.

020

0.03

50.

047

0.12

70.

038

0.07

30.

055

0.07

40.

055

0.01

50.

082

0.01

40.

083

IEP (

on-s

ite)

0.10

20.

037

0.01

30.

059

0.03

10.

085

0.09

70.

112

0.09

60.

113

0.08

10.

150

0.09

00.

148

Imag

ine

Lear

nin

g

–0.0

660.

080

Lear

n it Sy

stem

s

0.23

20.

085

Lite

racy

for

All

–0.0

040.

033

0.08

20.

038

0.05

20.

076

0.11

50.

058

0.11

20.

058

0.08

50.

097

0.09

20.

097

(continued)

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

26

Mai

nst

ream

D

evel

opm

ent

Educa

tional

Gro

up

0.05

70.

059

0.03

30.

069

0.02

60.

111

0.03

70.

112

0.07

60.

145

0.08

20.

147

NESI

0.00

00.

063

0.01

70.

071

0.04

60.

053

0.02

70.

040

0.10

50.

114

0.08

40.

096

New

ton L

earn

ing

0.05

30.

013

0.11

50.

019

0.06

90.

022

0.12

60.

028

0.12

40.

027

0.12

20.

041

0.11

60.

040

One

to O

ne

0.16

40.

068

Orion’s M

ind

0.04

40.

011

0.06

90.

015

0.09

40.

023

0.07

20.

024

0.07

40.

025

0.08

00.

038

0.08

10.

039

Pla

tform

Lea

rnin

g–0

.012

0.02

4

–0

.023

0.05

8–0

.013

0.06

1–0

.045

0.08

3–0

.032

0.08

4Po

der

Ser

(on-s

ite)

0.09

30.

035

0.11

30.

088

0.05

90.

136

0.08

00.

134

0.14

50.

123

0.14

40.

119

Prince

ton R

evie

w0.

038

0.01

7

0.

024

0.05

40.

034

0.05

40.

008

0.07

70.

015

0.07

7Pro

gres

sive

Lea

rnin

g0.

045

0.01

80.

082

0.03

0–0

.022

0.03

40.

027

0.03

40.

029

0.03

40.

023

0.04

80.

024

0.04

8Rock

et L

earn

ing

Partner

s0.

025

0.02

90.

058

0.03

3–0

.025

0.03

80.

029

0.04

90.

035

0.05

00.

032

0.07

90.

036

0.08

1Sc

hool Se

rvic

e Sy

stem

s0.

059

0.01

80.

130

0.03

20.

074

0.03

30.

111

0.04

60.

109

0.04

70.

160

0.07

30.

170

0.07

6SE

S of Illin

ois

0.04

40.

025

0.12

80.

019

0.05

60.

023

0.09

50.

034

0.09

80.

036

0.13

10.

058

0.13

80.

057

SPC E

duca

tional

Ser

vice

s

–0

.053

0.04

8

Sm

art K

ids,

Inc

0.21

60.

188

Span

ish L

earn

ing

Cen

ter

0.13

50.

064

Sylv

an L

earn

ing

0.16

80.

081

The

Hom

ework

Mas

ter

Cen

ter

0.22

20.

086

Tuto

rial

Ser

vice

s0.

036

0.04

2

–0

.038

0.18

1–0

.042

0.18

40.

051

0.18

40.

049

0.18

7U

npar

alle

led S

olu

tions

0.06

50.

045

0.10

20.

037

0.06

10.

048

0.05

00.

081

0.05

80.

084

0.11

40.

134

0.11

90.

138

Num

ber

of obse

rvat

ions

61,1

7163

,506

80,5

1020

5,18

720

5,18

712

4,67

712

4,67

7N

um

ber

of sc

hools

227

454

302

466

466

458

458

Num

ber

of st

uden

ts61

,171

63,5

0680

,510

119,

970

119,

970

83,9

4583

,945

Tab

le 7

(C

on

tin

ued

)

2-Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Rea

din

g G

ain

Coef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

27

Tab

le 8

Eff

ects

of A

tten

din

g S

up

ple

men

tal E

du

cati

on

al S

ervi

ces

(SE

S) W

ith

th

e D

istr

ict

Pro

vid

er a

nd

Oth

er P

rovi

der

s o

n M

ath

Gai

ns

2-Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Mat

h G

ain

Coef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

Dis

tric

t0.

065

0.01

10.

066

0.02

00.

092

0.02

20.

121

0.03

20.

114

0.03

20.

112

0.04

00.

106

0.04

0Sm

all

pro

vid

ers

0.03

20.

036

0.00

00.

044

0.08

00.

099

–0.0

090.

051

–0.0

080.

051

0.13

50.

121

0.14

20.

124

A+ T

uto

ring

Serv

ice,

LTD

0.02

10.

058

–0.0

230.

179

–0.0

280.

181

0.16

00.

140

0.15

90.

142

Aca

dem

ic A

dva

nta

ge

0.

166

0.04

0–0

.002

0.13

6–0

.003

0.13

6

ASP

IRA

0.02

20.

097

0.00

50.

086

0.01

00.

087

–0.0

660.

073

–0.0

590.

070

Bab

bag

e N

et S

chool

0.06

30.

028

–0.0

320.

025

–0.0

050.

083

0.01

40.

052

0.02

60.

053

0.08

70.

098

0.08

90.

098

Bla

ck S

tar

Pro

ject

0.04

90.

054

0.12

80.

111

–0.0

120.

116

–0.0

010.

118

0.11

10.

170

0.11

90.

170

Bra

in H

urr

ican

e0.

071

0.02

50.

061

0.03

30.

039

0.02

70.

101

0.04

20.

081

0.03

90.

120

0.07

90.

122

0.07

9Bra

infu

se O

ne-

to-O

ne

–0.0

220.

022

–0.0

320.

071

0.00

00.

051

–0.0

390.

069

–0.0

350.

070

–0.0

370.

123

–0.0

270.

125

Brilli

ance

Aca

dem

y0.

000

0.05

8

0.00

20.

044

–0.0

530.

087

–0.0

480.

087

0.16

20.

113

0.16

80.

110

Cam

bridge

Educa

tional

0.

090

0.03

10.

051

0.03

50.

063

0.03

10.

073

0.06

10.

060

0.05

90.

029

0.10

20.

022

0.10

2Ches

s Aca

dem

y0.

066

0.03

90.

065

0.02

50.

102

0.04

60.

092

0.07

90.

095

0.08

00.

059

0.07

80.

067

0.08

1Chi

ldre

n’s

Hom

e+Aid

Soc

.0.

173

0.02

10.

086

0.03

70.

024

0.14

60.

267

0.06

20.

285

0.06

4–0

.086

0.12

0–0

.078

0.12

7Clu

bZ! T

uto

ring

Serv

ice

–0.0

170.

039

0.04

70.

071

0.08

30.

015

0.01

70.

070

0.01

30.

066

0.16

20.

112

0.15

90.

115

CSC

Jule

x Le

arnin

g–0

.029

0.06

0

0.15

30.

205

0.12

90.

188

Educa

te O

nlin

e 0.

022

0.01

30.

041

0.04

4

–0

.005

0.04

2–0

.008

0.04

30.

072

0.05

20.

067

0.05

4Fa

ilure

Fre

e Rea

din

g 0.

027

0.04

4

0.24

80.

242

0.22

50.

238

Huntin

gton (

on-s

ite)

0.01

60.

016

0.03

70.

036

0.03

50.

029

0.02

80.

042

0.03

40.

042

0.01

50.

082

0.01

40.

083

IEP (

on-s

ite)

0.03

70.

049

0.02

00.

038

0.13

20.

093

0.06

50.

078

0.06

50.

079

0.08

10.

150

0.09

00.

148

Imag

ine

Lear

nin

g

0.

051

0.06

7

Le

arn it Sy

stem

s

0.

029

0.08

9

Li

tera

cy for

All

–0.0

030.

035

0.09

20.

031

0.03

70.

043

0.03

40.

054

0.03

30.

055

0.08

50.

097

0.09

20.

097

Mai

nst

ream

Dev

elopm

ent

Educa

tional

Gro

up

–0.0

120.

040

0.03

30.

044

–0.0

350.

140

–0.0

200.

141

0.07

60.

145

0.08

20.

147

(continued)

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

28

NESI

0.07

10.

038

0.11

80.

024

0.00

30.

153

–0.0

240.

151

0.10

50.

114

0.08

40.

096

New

ton L

earn

ing

0.06

10.

016

0.03

10.

017

0.07

10.

023

0.08

10.

028

0.07

70.

027

0.12

20.

041

0.11

60.

040

One

to O

ne

0.15

70.

083

Orion’s M

ind

0.05

00.

010

0.03

70.

013

0.08

10.

017

0.04

10.

024

0.04

50.

024

0.08

00.

038

0.08

10.

039

Pla

tform

Lea

rnin

g–0

.018

0.03

1

–0

.007

0.08

50.

004

0.08

5–0

.045

0.08

3–0

.032

0.08

4Po

der

Ser

(on-s

ite)

0.00

70.

072

–0.0

100.

034

–0.0

430.

064

–0.0

330.

065

0.14

50.

123

0.14

40.

119

Prince

ton R

evie

w0.

050

0.01

7

0.

076

0.05

60.

090

0.05

60.

008

0.07

70.

015

0.07

7Pro

gres

sive

Lea

rnin

g0.

008

0.01

70.

046

0.02

3–0

.005

0.03

70.

001

0.03

60.

004

0.03

60.

023

0.04

80.

024

0.04

8Rock

et L

earn

ing

Partner

s0.

005

0.02

4–0

.013

0.03

4–0

.021

0.04

4–0

.067

0.04

4–0

.054

0.04

40.

032

0.07

90.

036

0.08

1Sc

hool Se

rvic

e Sy

stem

s0.

012

0.02

30.

106

0.02

20.

051

0.03

70.

046

0.03

50.

103

0.03

90.

160

0.07

30.

170

0.07

6SE

S of Illin

ois

0.04

60.

026

0.08

60.

023

0.05

30.

021

0.10

70.

038

0.04

20.

035

0.13

10.

058

0.13

80.

057

SPC E

duca

tional

Ser

vice

s

–0.0

150.

091

Smar

t K

ids,

Inc

–0

.132

0.01

3

Sp

anis

h L

earn

ing

Cen

ter

0.

079

0.03

1

Sy

lvan

Lea

rnin

g

0.15

80.

107

The

Hom

ework

Mas

ter

Cen

ter

0.

012

0.10

5

Tuto

rial

Ser

vice

s–0

.005

0.04

7

0.00

80.

136

0.00

30.

134

0.05

10.

184

0.04

90.

187

Unpar

alle

led S

olu

tions

0.07

20.

032

0.02

60.

041

0.09

60.

046

0.06

90.

061

0.06

90.

059

0.11

40.

134

0.11

90.

138

Num

ber

of obse

rvat

ions

61,4

6463

,773

80,6

1420

4,09

420

4,09

412

4,67

712

4,67

7N

um

ber

of sc

hools

227

455

302

466

466

458

458

Num

ber

of st

uden

ts61

,464

63,7

7380

,614

119,

441

119,

441

83,9

4583

,945

Tab

le 8

(C

on

tin

ued

)

2-Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Mat

h G

ain

Coef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

sample size for off-site providers was relatively small in 2008–2009 and2009–2010 (75 and 91, respectively), although more than 2,000 students at-tended SES with off-site providers in 2010–2011. In 2009–2010, off-site pro-viders served a larger percentage of SWD students, and thus, we suggestinterpreting these results with some caution.17

The effect sizes of attending SES with for-profit providers versus othernonprofit or district providers (see Table 10) suggest that for-profit providersare generally less effective than district/public providers in increasing stu-dent achievement, particularly for math. Students attending with for-profitproviders gain about .03 standard deviations less than the district providersin reading and about .07 standard deviations less in math. Table 3 shows thatonly about 2.5% of students attended SES with a nonprofit provider, andanother 10% to 16% attended with the district provider; clearly, the largestshare of students attend SES with for-profit providers.

The results in Table 11 show the estimated differential effects betweenthe district and other providers, as well as the effects of other provider types,from the analysis that only includes students who attended SES. As the anal-ysis is done for only those students who attend SES (controlling for theirselection into specific provider types in the PSM analyses), a smaller samplesize leads to larger standard errors than in the fixed effects strategies andfewer statistically significant results. The differential impacts can also be cal-culated from the previous tables, and the results are consistent across thesedifferent estimation strategies.

The first four rows of results in Table 11 present the average differentialeffect between the district provider and other SES providers serving CPS stu-dents in 2008–2009, 2009–2010, and 2010–2011, estimated using value-addedmodels, student fixed effects models, school and student fixed effects mod-els, and propensity score matching (respectively). The results suggest that onaverage, students attending SES with the district provider gain more on read-ing tests (approximately .02–.04 standard deviations more where statisticallysignificant) and more on math tests (approximately .02–.06 standard devia-tions) than students attending with non-district providers. We also restrictedthe sample to only students who attended SES with an on-site provider,either the district or another on-site SES provider, to determine if the districtprovider performance differed from that of other on-site providers. Thesefindings (in the fifth to eighth rows of results in Table 11) suggest that thedistrict provider generally outperformed other on-site providers, with stu-dents who attended SES on-site with the district provider realizing largergains of .04 to .06 standard deviations more (where statistically significant)than other on-site providers.

Results of analyses comparing providers that deliver SES instruction on-line with other (off-line) providers (see Table 11) suggest that online pro-viders are generally less effective than other providers, although thecoefficients (differential effect sizes) are statistically significant only for the

29

Efficacy of Private Sector Providers

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

30

Tab

le 9

Eff

ects

of A

tten

din

g S

up

ple

men

tal E

du

cati

on

al S

ervi

ces

(SE

S) W

ith

th

e D

istr

ict,

On

line,

On

-Sit

e, a

nd

Off

-Sit

e P

rovi

der

s

Rea

din

g2-

Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Pro

vider

Typ

eCoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

Dis

tric

t0.

044

0.01

30.

128

0.01

90.

091

0.02

00.

102

0.02

40.

096

0.02

40.

108

0.04

00.

102

0.04

0O

nli

ne

0.00

70.

011

0.03

30.

024

–0.0

090.

025

0.00

10.

023

0.00

10.

023

–0.0

060.

033

–0.0

070.

034

On

-sit

e 0.

042

0.00

60.

081

0.00

90.

066

0.01

10.

069

0.01

50.

070

0.01

50.

071

0.02

50.

074

0.02

5O

ff-s

ite

0.08

20.

073

0.35

00.

077

0.08

90.

020

0.13

40.

050

0.13

50.

051

0.42

70.

228

0.43

60.

236

Num

ber

of obse

rvat

ions

61,1

7163

,506

80,5

1020

5,18

720

5,18

712

4,67

712

4,67

7N

um

ber

of sc

hools

227

454

302

466

466

458

458

Num

ber

of st

uden

ts61

,171

63,5

0680

,510

119,

970

119,

970

83,9

4583

,945

Mat

h2-

Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Coef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

Dis

tric

t0.

064

0.01

10.

065

0.02

00.

092

0.02

20.

106

0.02

80.

098

0.02

80.

112

0.03

90.

102

0.03

9O

nli

ne

–0.0

100.

010

–0.0

010.

018

–0.0

270.

025

–0.0

230.

023

–0.0

230.

023

–0.0

360.

034

–0.0

340.

035

On

-sit

e 0.

038

0.00

50.

049

0.00

80.

064

0.01

00.

050

0.01

30.

050

0.01

30.

043

0.02

10.

045

0.02

1O

ff-s

ite

–0.0

010.

059

–0.0

640.

053

0.05

60.

020

0.02

50.

057

0.03

10.

058

0.03

20.

170

0.05

10.

179

Num

ber

of obse

rvat

ions

61.4

6463

,773

80,6

1420

4,09

420

4,09

412

4,05

912

4,05

9N

um

ber

of sc

hools

227

455

302

466

466

458

458

Num

ber

of st

uden

ts61

,464

63,7

7380

,614

119,

441

119,

441

83,5

7983

,579

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

31

Tab

le 1

0E

ffec

ts o

f Att

end

ing

Su

pp

lem

enta

l Ed

uca

tio

nal

Ser

vice

s (S

ES

) Wit

h t

he

Dis

tric

t, Fo

r-P

rofit

, an

d N

ot-

for-

Pro

fit P

rovi

der

s

Rea

din

g2-

Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Pro

vider

Typ

eCoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

Dis

tric

t0.

045

0.01

30.

129

0.01

90.

091

0.02

00.

151

0.03

40.

142

0.03

40.

114

0.03

90.

108

0.04

0For

-pro

fit

0.04

20.

006

0.08

90.

009

0.07

10.

010

0.12

00.

028

0.11

70.

029

0.08

00.

025

0.08

10.

025

Not

-for

-pro

fit

0.05

40.

025

0.09

90.

061

0.03

00.

059

0.05

60.

030

0.05

30.

030

0.05

60.

090

0.07

00.

091

Num

ber

of obse

rvat

ions

61,1

7163

,506

80,5

1020

5,18

720

5,18

712

4,67

712

4,67

7N

um

ber

of sc

hools

227

454

302

466

466

458

458

Num

ber

of st

uden

ts61

,171

63,5

0680

,510

119,

970

119,

970

83,9

4583

,945

Mat

h2-

Yea

r D

ata

Val

ue-

Added

Model

(W

ith S

chool Fi

xed E

ffec

ts)

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Studen

t Fi

xed

Effec

ts M

odel

School an

d S

tuden

t Fi

xed E

ffec

ts M

odel

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Coef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

Dis

tric

t0.

067

0.01

10.

065

0.02

00.

092

0.02

20.

162

0.03

50.

151

0.03

50.

115

0.03

90.

105

0.03

9For

-pro

fit

0.03

70.

005

0.04

80.

008

0.05

80.

009

0.09

90.

025

0.09

60.

026

0.03

90.

021

0.04

10.

021

Not

-for

-pro

fit

0.06

30.

031

0.03

60.

036

-0.0

710.

068

0.06

60.

027

0.06

30.

027

0.02

00.

061

0.03

40.

062

Num

ber

of obse

rvat

ions

61.4

6463

,773

80,6

1420

4,09

420

4,09

412

4,05

912

4,05

9N

um

ber

of sc

hools

227

455

302

466

466

458

458

Num

ber

of st

uden

ts61

,464

63,7

7380

,614

119,

441

119,

441

83,5

7983

,579

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

32

Tab

le 1

1D

iffer

ence

s in

Su

pp

lem

enta

l Ed

uca

tio

nal

Ser

vice

s (S

ES

) E

ffec

ts b

y P

rovi

der

Typ

es (

for

stu

den

ts w

ho

att

end

ed S

ES

)

Rea

din

gM

ath

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Yea

r 20

08–2

009

Yea

r 20

09–2

010

Yea

r 20

10–2

011

Coef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SECoef

fici

ent

SE

Dis

tric

t ve

rsus

non-d

istric

tVal

ue-

added

.020

.018

.027

.030

.040

.035

.039

.017

–.00

3.0

27.0

24.0

27St

uden

t fixe

d e

ffec

ts.0

33.0

89Res

ults

are

for

all 3

year

s.0

63.0

99Res

ults

are

for

all 3

year

sSc

hool an

d s

tuden

t fixe

d e

ffec

ts.0

08.0

94.0

62.1

02Pro

pen

sity

sco

re m

atch

ing

.029

.012

.065

.020

.042

.020

.041

.011

.049

.018

.048

.018

Dis

tric

t ve

rsus

oth

er o

n-s

ite p

rovi

der

sVal

ue-

added

.016

.019

.032

.032

.060

.037

.040

.018

–.01

4.0

26.0

26.0

33St

uden

t fixe

d e

ffec

ts.0

25.0

82Res

ults

are

for

all 3

year

s.0

70.1

01Res

ults

are

for

all 3

year

sSc

hool an

d s

tuden

t fixe

d e

ffec

ts.0

05.0

89.0

65.1

06Pro

pen

sity

sco

re m

atch

ing

.016

.011

.059

.019

.041

.019

.033

.010

.043

.017

.046

.021

Onlin

e ve

rsus

not onlin

eVal

ue-

added

–.02

8.0

13.0

25.0

29–.

014

.034

2–.

042

.012

–.00

5.0

20–.

021

.033

Studen

t fixe

d e

ffec

ts–.

087

.089

Res

ults

are

for

all 3

year

s–.

047

.081

Res

ults

are

for

all 3

year

sSc

hool an

d s

tuden

t fixe

d e

ffec

ts–.

087

.095

–.04

9.0

86Pro

pen

sity

sco

re m

atch

ing

–.02

7.0

11–.

009

.022

.012

.022

–.03

8.0

10–.

024

.019

–.04

0.0

21

For-

pro

fit ve

rsus

nonpro

fit

Val

ue-

added

–.02

2.0

15–.

028

.032

–.03

6.0

325

–.04

1.0

16.0

16.0

22–.

007

.025

Studen

t fixe

d e

ffec

ts–.

022

.089

Res

ults

are

for

all 3

year

s–.

053

.090

Res

ults

are

for

all 3

year

sSc

hool an

d s

tuden

t fixe

d e

ffec

ts.0

06.0

93–.

048

.093

Pro

pen

sity

sco

re m

atch

ing

–.00

9.0

11–.

067

.020

–.04

6.0

20–.

024

.011

–.05

5.0

17–.

038

.018

at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

2008–2009 school year and in two specifications. The estimated (negative)differential effects for 2008–2009 are highly comparable between thevalue-added and PSM models, suggesting students attending with onlineproviders gain approximately .03 to .04 less than students attending SESwith other providers. A final set of results comparing effect sizes of attendingSES with for-profit providers versus other providers (nonprofit or public)again suggests that for-profit providers are generally less effective than non-profit and district/public providers in increasing student achievement, partic-ularly for math.

Conclusion

Supplemental educational services (out-of-school tutoring) are a coreprovision of NCLB, in which school districts are mandated to pay for thecost of provision of after-school tutoring for low income and disadvantagedstudents who attend schools that are not making adequate yearly progressfor 3 years. A key feature of this mandate is its reliance on the private sectorto offer eligible students greater choice in a competitive market that is ex-pected to encourage innovative service approaches and squeeze out ineffec-tive providers. Identifying tutoring provider effects on student achievementis essential to generating the information necessary for students and parentsto make informed choices of tutoring providers, but efforts to estimate pro-vider effects are complicated by the fact that participation is voluntary. In thisarticle, we have drawn on nonexperimental methods to estimate the effectsof SES providers on student achievement in a large urban school district(Chicago Public Schools), which accounts for a significant share of studentsreceiving tutoring under NCLB.

The findings of our empirical analyses of the effects of SES providerswho served eligible CPS students in the 2008–2009, 2009–2010, and 2010–2011 school years suggest that there is a statistically significant effect ofattending SES on student achievement, particularly for those who receiveat least 40 hours of tutoring. These effect sizes represent about one-thirdof the annual gains made by students in these schools, and the gains fromtutoring generally increase with the number of hours of tutoring received.

Additionally, we find that the district provider is more effective thannon-district and other on-site SES providers in increasing the math and read-ing test scores of students who attend SES, although we recognize that thiseffect may not generalize beyond CPS. In an ongoing qualitative componentof our study that involves multiple large urban school districts (Heinrich etal., 2012), we have identified several distinctive features of the implementa-tion of SES in CPS that might explain the district’s greater effectiveness,including the significantly lower hourly rate charged that allows more hoursof tutoring for students, the use of regular school day teachers as tutors, andthe use of school-based SES coordinators in monitoring and coordinating the

Efficacy of Private Sector Providers

33 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

delivery of tutoring services. Alternatively, we find that students receiving tu-toring from online providers appear to gain less in reading and math thanstudents attending with other providers. We are currently further document-ing and analyzing the different types of digital/online providers (vs. nondi-gital providers), in an effort to identify empirically and qualitatively whatcontributes to these providers’ lower average effectiveness (e.g., do the on-line sessions involve interactions with a live tutor, or are they preloaded, self-directed sessions?).18 This is particularly important given that some urbanschool districts have seen substantial expansions of online providers’ marketshares of students attending SES.19

Students receiving tutoring from for-profit providers gain less than thoseattending with nonprofit providers or the district provider, particularly inmath. We also identified individual SES providers that were significantlymore effective in producing math and reading gains for CPS students.These are a mix of for-profit and nonprofit SES providers, although eachof them offers SES on-site (at the public schools that students attend).

Given that unmeasured differences in students who attend SES or attendwith particular types of providers could still introduce bias in these results,we are encouraged by the fact that the findings are fairly consistent acrossthe four different rigorous estimation methods that make different assump-tions. We also note that our findings on provider-specific effects are consis-tent with the most recent CPS evaluation of SES as well (Jones, 2009), whichidentified many of the same providers as being among the most effective inCPS. Finally, we believe that our research has identified some basic charac-teristics of more effective approaches to the organization and managementof SES programs that might be considered by other school districts seekingto improve tutoring outcomes, although we again caution that our findingsshould be viewed as associations (rather than causal effects).

By design, if NCLB or its successor initiatives (under recently grantedwaivers to states from NCLB provisions) are to achieve the broader goal ofreducing the academic achievement gap through after-school tutoring forstudents in underperforming schools, tutoring program administratorsneed adequate and independent (not self-reported) information on providereffectiveness to guide students’ and parents’ choices. We have presentedreadily adaptable estimation strategies that can be used by states and schooldistricts to generate information on tutoring provider effectiveness for stu-dents and parents that will help them to make better informed choices.Whether under NCLB or waivers from its provisions, many school districtsacross the country will continue to spend millions of Title I funds on tutoringfor economically and academically disadvantaged students, and these find-ings will help to inform those who are looking for guidance in improvingthese services and their impacts on student achievement.

Finally, the different levels of SES program administration—primarilydistrict and state—could improve their coordination in oversight and

Heinrich, Nisar

34 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

monitoring of tutoring provider performance to more fully realize the poten-tial of the competitive market in improving student outcomes. For example,with the evidence from this and related studies, states and districts mightintroduce performance-based contracts to exert more control over providerrates, minimum levels of tutoring delivered, and other parameters of servicedelivery that this research suggests could contribute to improved outcomes.Indeed, under newly granted state waivers from NCLB provisions, someschool districts are already establishing maximum hourly rates and/or otherrequirements that will ensure students are offered a minimum of 40 hours oftutoring (Heinrich et al., 2012). In this regard, we expect our study to morebroadly speak to the viability of education and other policy interventionsthat employ a private market model to improve public sector outcomes.

Acknowledgments

We would like to thank the funder of this research, the Institute for EducationalSciences, PR/Award number: R305A090301, Education Policy, Finance and SystemsResearch Program, Goal 3. In addition, we express our most sincere thanks to thestaff in the study district for their time and support for this project. We would alsolike to thank the staff at the Wisconsin Center for Education Research,UW-Madison for their help and especially Martina Chura, Curtis Jones, Hyun SikKim, Huiping Cheng, Patricia Burch, Annalee Good, and Molly Stewart. We alsothank anonymous referees for very helpful comments on an earlier version of thismanuscript.

Notes1See Title I, Section 1116(e) of the Elementary and Secondary Education Act (ESEA),

reauthorized by the No Child Left Behind Act of 2001.2Title I federal funding, which began in the 1965 Elementary and Secondary Act, was

created to allow all students an equal opportunity to receive the highest quality educationpossible. Through Title I, school districts can hire teachers to lower student-teacher ratios,provide tutoring for struggling students, create school computer labs, fund parent involve-ment activities, purchase instructional and professional development materials for teach-ers, hire teacher assistants, and more. The 20% Title I set-aside for supplementaleducational services (SES) and school transfers cannot be spent on administrative costsfor these activities, although the district may reallocate any unused set-aside funds to otherTitle I activities after all eligible students have had adequate time to opt to transfer toanother school or apply for SES.

3Online providers use computers/digital technology as the primary format/platformfor delivering SES. On-site providers offer SES on a public school campus (and typicallyserve students attending that school), while off-site implies that the services are deliveredat a site other than the school (e.g., home-based or other location in the community).

4The guidance states: ‘‘[A state educational agency] that desires to set program designparameters should ensure that such parameters do not result in the inability of a wide vari-ety of providers, including non-profits, for profits, [local educational agencies] and faith-based and community organizations, from being able to participate as eligible providers,thereby limiting parental choice’’ (U.S. Department of Education, 2005, p. 7).

5One of the few provisions available to districts for requesting removal of a provideris following a violation of a district policy in use of its buildings/space.

Efficacy of Private Sector Providers

35 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

6Our study is distinct from Jones (2009) in that we employ alternative approaches toadjusting for student selection into SES at multiple stages (registration, attendance, level ofattendance) and also selection into particular types of providers. In addition, the ChicagoPublic Schools (CPS) studies primarily control for race and gender (although they also usegain scores to account for prior learning trajectories) and do not report results on studentselection; we find other significant predictors of student selection into SES (e.g., studentabsences from school, prior SES attendance, English language learners, students with dis-abilities, etc.) that provide important information for school districts.

7Since the Illinois Standardized Achievement Tests (ISAT) is taken in Grades 3through 8 in Illinois, we are not able to include students in Grade 3, as there is no pretestinformation for these students (with the exception of students who were retained in Grade3 in 2008–2009). In the 2009–2010 school year, 6% of students who registered and at-tended SES are missing (in Grades 4–8) because they don’t have test scores in the prioryear (and our outcome measure is gains in achievement). The loss of data for 2008–2009 school year is lower at 4%.

8Steinberg (2011) finds fewer prior-year disciplinary infractions among SES partici-pants in CPS versus those who did not participate in years 2006–2007 and 2007–2008.However, as we do not have access to CPS disciplinary data, this is an omitted factor inour analysis.

9We also explored the possibility of using a regression-discontinuity design for esti-mating SES effects in CPS, given that CPS identified explicit criteria for prioritizing studentsfor SES when the number of eligible students exceeded available funding for services.However, after it was discovered in the 2009–2010 school year that the monies paid outin the prior school year exceeded the funds budgeted for SES, registered students weredenied services (i.e., told that they could not participate in SES) after being assigned toproviders and showing up for sessions. As there was no systematic process followed inretracting the offer of services or tracking the students who were ultimately refused serv-ices, we determined that we could not achieve a clean approach to identification of effectsthrough regression-discontinuity analyses.

10Summary statistics on the characteristics of student served by specific (individual)SES providers are available upon request from the authors.

11As SES providers serve students at multiple grade levels, it is reasonable to poolinformation across grade levels.

12We also check the results by restricting the analysis to those students who do notchange schools and run the student fixed effects estimation using Equation 3, and weobtain similar results.

13Hill, Bloom, Black, and Lipsey (2007) find an average annual reading gain for fifthto sixth graders of about .32 standard deviations and of .23 to .26 standard deviations forsixth through eighth graders.

14See Hirano and Imbens (2004) for more details on their extension of propensityscore matching methods to cases where the treatment is continuous. In generalized pro-pensity score matching, a ‘‘dose-response function’’ is estimated, where in this example,the ‘‘dose’’ is the number of hours a student is tutored, and the ‘‘response’’ is the impactthat a given level of tutoring has on their reading and math gains.

15To test this hypothesis, we interacted the district indicator with the number of hoursof SES the students received, and the difference between district and non-district providerdisappeared.

16In each of the estimations reported, the standard errors of the coefficients are larg-est for the school and student fixed effects models.

17CPS prioritized students with disabilities in 2009–2010, with the result that eventhough the proportion of SWD among eligible students is the same in 2008–2009 and2009–2010, the proportion attending SES is nearly double in 2009–2010. We speculatethat this compositional change in a small subsample might explain the differences inthe magnitude of off-site effects (as seen in Table 9) across the school years.

18This has turned out to be a much larger undertaking than originally anticipatedbecause of the extent of variation among digital providers in practice and discrepanciesin materials used by providers to advertise services or seek state approval for SESprovision.

Heinrich, Nisar

36 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

19In our ongoing multisite study, we have observed online tutoring companies witha student ‘‘market share’’ as high as 88% in one urban school district and a single digitalprovider delivering tutoring to more than 14,000 students in another large urban schooldistrict.

References

Burch, P., Steinberg, M., & Donovan, J. (2007). Supplemental educational servicesand NCLB: Policy assumptions, market practices, emerging issues. EducationalEvaluation and Policy Analysis, 29(2), 115–133.

Chatterji, M., Kwon, Y. A., & Sng, C. (2006). Gathering evidence on an after-schoolsupplemental instruction program: Design challenges and early findings in lightof NCLB. Education Policy Analysis Archives, 14(12). Retrieved fromhttp://epaa.asu.edu/ojs/article/view/83

Dynarski, M., James-Burdumy, S., Moore, M., Rosenberg, L., Deke, J., & Mansfield, W.(2004). When schools stay open late: The national evaluation of the 21st CenturyCommunity Learning Centers Program: New findings. Washington, DC: U.S.Department of Education.

Frederickson, G. H., & Smith, K. B. (2003). The public administration theory primer.Boulder, CO: Westview Press.

Halpern, R. (2003). The challenge of systems-building in the after-school field: Lessonsfrom experience. Wellesley, MA: National Institute on Out-of-School Time.

Heinrich, C. J. (2010). Third-party governance under No Child Left Behind:Accountability and performance management challenges in supplemental edu-cational services provision. Journal of Public Administration Research andTheory, 20(1), 59–80.

Heinrich, C. J., Burch, P., Good, A., Acosta, R., Cheng, H., Dillender, M., . . . Stewart,M. S. (2012). Improving the implementation and effectiveness of out-of-school-time tutoring: A longitudinal multisite, mixed-method investigation. WorkingPaper, University of Texas at Austin, Austin, TX.

Heinrich, C. J., Meyer, R. H., & Whitten, G. (2010). Supplemental education servicesunder No Child Left Behind: Who signs up, and what do they gain? EducationalEvaluation and Policy Analysis, 32, 273–298.

Heistad, D. (2005). Analysis of 2005 supplemental education services in Minneapolispublic schools: An application of matched sample statistical design. Minneapolis,MN: Minneapolis Public Schools.

Hill, C. J, Bloom, H. S., Black, A. R., & Lipsey, M. W. (2007). Empirical benchmarks forinterpreting effect sizes in research (MDRC Working Papers on ResearchMethodology). New York, NY: MDRC. Retrieved from www.mdrc.org/publica-tions/459/full.pdf

Hirano, K., & Imbens, G. W. (2004). The propensity score with continuous treat-ments. In A. Gelman & X.-L. Meng (Eds.), Applied Bayesian modeling andcausal inference from incomplete-data perspectives (pp. 73–84). West Sussex,UK: Wiley InterScience.

Hollister, R. (2003). The growth in after-school programs and their impact.Washington, DC: The Brookings Institution. Retrieved from http://www.broo-kings.edu/papers/2003/0225poverty_hollister.aspx

Horn, M. B. (2011). Beyond good and evil: Understanding the role of for-profits ineducation through the theories of disruptive innovation Private Enterprise inAmerican Education, American Enterprise Institute for Public Policy, Special

Efficacy of Private Sector Providers

37 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

Report 1). Retrieved from http://www.aei.org/files/2011/04/27/Enterprise-Issue-1.pdf

Jones, C. (2009). The 2009 Supplemental Educational Services program: Year 4 sum-mative evaluation. Chicago, IL: Chicago Public Schools Office of ExtendedLearning Opportunities and Office Research, Evaluation, and Accountability.

Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin- Glenn, M.(2004). The effectiveness of out-of-school-time strategies in assisting low-achiev-ing students in reading and mathematics: A research synthesis. Washington, DC:U.S. Department of Education, Institute of Education Sciences. Retrieved fromwww.mcrel.org/topics/products/151/

Little, P. M. D. (2007). The quality of school-age child care in after-school settings(Research-to-Policy Connections No. 7). Cambridge, MA: Harvard FamilyResearch Project.

Munoz, M. A., Potter, A., & Ross, S. M. (2008). Supplemental educational services asa consequence of the NCLB legislation: Evaluating its impact on student achieve-ment in a large urban district. Journal of Education for Students Placed at Risk,13, 1–25.

Peterson, P. E. (2003). The future of school choice. Stanford, CA: Hoover InstitutionPress.

Potter, A., Ross, S. M., Paek, J., Pribesh, S. L., & Nunnery, J. A. (2006). Supplementaleducational services in the State of Louisiana: 2004–2005. Memphis, TN: TheUniversity of Memphis, Center for Research in Educational Policy.

Rickles, J. H., & Barnhart, M. K. (2007). The impact of supplemental educational serv-ices participation on student achievement: 2005-06. Report of the Los AngelesUnified School District Program Evaluation and Research Branch, Planning,Assessment and Research Division Publication No. 352, Los Angeles, CA.

Rivlin, A. M. (1992). Reviving the American dream: The economy, the states, and thefederal government. Washington, DC: The Brookings Institution.

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score inobservational studies for causal effects. Biometrika, 70, 41–55.

Ross, S. M., Potter, A., Paek, J., & McKay, D. (2008). Implementation and outcomes ofsupplemental educational services: The TN state-wide study. Journal ofEducation for Students Placed at Risk, 13, 26–58.

Socias, M., deSousa, J-M., & Le Floch, K. C. (2009). Supplemental educational servicesand student achievement in waiver districts: Anchorage and Hillsborough(Report to U.S. Department of Education). Jessup, MD: Ed Pubs.

Springer, M.G., Pepper, M. J., & Ghosh-Dastidar, B. (2009). Supplemental educationalservices and student test score gains: Evidence from a large, urban school dis-trict. Working paper, Vanderbilt University, Nashville, TN.

Steinberg, M. (2011). Educational choice & student participation: The case of the sup-plemental educational services provision in Chicago public schools. EducationalEvaluation and Policy Analysis, 33(2), 159–182.

U.S. Department of Education. (2005, August 22). No Child Left Behind: Supplementaleducational services non-regulatory guidance (Final Guidance). Washington,DC: Author.

Vandell, D. L., Reisner, E. R., Brown, B. B., Dadisman, K., Pierce, K. M., Lee, D., &Pechman, E. M. (2005). The study of promising after-school programs:Examination of intermediate outcomes in year 2. Madison, WI: WisconsinCenter for Education Research.

Zimmer, R., Christina, R., Hamilton, L. S., & Prine, D. W. (2006). Evaluation of twoout-of-school program in Pittsburgh Public Schools: No Child Left Behind’s

Heinrich, Nisar

38 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from

supplemental educational services and State of Pennsylvania’s educationalassistance program (RAND Working Paper). Santa Monica, CA: RAND.

Zimmer, R., Gill, B., Razquin, P., Booker, K., & Lockwood, J. R. (2007). State and localimplementation of the No Child Left Behind Act: Volume I Title I school choice,supplemental educational services, and student achievement (Report to the USDepartment of Education, Office of Planning, Evaluation, and PolicyDevelopment). Santa Monica, CA: RAND.

Manuscript received October 23, 2011Final revision received December 31, 2012

Accepted February 28, 2013

Efficacy of Private Sector Providers

39 at University of Texas Libraries on May 31, 2013http://aerj.aera.netDownloaded from