ttr hoxby charters

8/14/2019 TTR Hoxby Charters

1/29

Summary of Review

How New York Citys Charter Schools Affect Achievement estimates the effects onstudent achievement of attending a New York City charter school rather than a traditionalpublic school and investigates the characteristics of charter schools associated with themost positive effects on achievement. Because the report relies on an inappropriate set of statistical models to analyze the data, however, the results presented appear to overstatethe cumulative effect of attending a charter school. In addition, the report does not pro-vide enough technical discussion and detailed description to enable a reader to assess thevalidity of some aspects of the reports methodology and results. Policymakers, educators,and parents, therefore, should not rely on these estimates until the authors provide moretechnical detail and the analysis has undergone rigorous peer review.

DOCUMENT REVIEWED : How New York Citys Charter Schools AffectAchievement.

AUTHOR : Caroline M. Hoxby, Sonali Murarka & Jenny Kang

PUBLISHER /THINK TANK : New York City Charter Schools Evaluation Project

DOCUMENT RELEASE DATE : September 2009

REVIEW DATE : November 12, 2009

REVIEWER : Sean F. Reardon

E-MAIL ADDRESS : [email protected]

PHONE NUMBER : (650) 736-8517SUGGESTED CITATION :Reardon, S.F. (2009) Review of How New York Citys Charter Schools Affect Achievement.

Boulder and Tempe: Education and the Public Interest Center & Education Policy Re-search Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-How-New-York-City-Charter


2/29

http://epicpolicy.org/thinktank/review-How-New-York-City-Charter

Executive Summary

The September 2009 report by CarolineHoxby, Sonali Murarka, and Jenny Kang,

How New York Citys Charter Schools Affect

Achievement , has the potential to add usefullyto the growing body of evidence regardingthe effectiveness of charter schools. The re-port analyzes data from New York City inorder to estimate the effects on studentachievement of attending a New York Citycharter school rather than a traditional publicschool. In addition, the report investigates thecharacteristics of charter schools in NewYork that are associated with the most posi-tive effects on achievement. Because of flaws

in the reports statistical analysis, however, itappears to overestimate the effect of charterschools in New York City.

The key strength of the report is that it relieson the fact that the majority of students inNew York City charter schools were admit-ted via lottery. The availability of rando-mized lotteries that determine admission tomost of New York Citys charter schoolsprovides the researchers with the opportunity to obtain highly-credible estimates of the ef-fect of attending a charter school rather than atraditional public school in New York City.Each lottery can be thought of as a small,randomized controlled experiment, with eachschool, year, and grade in which a lottery isconducted for admission serving as one suchexperiment. In principle, this allows re-searchers to estimate the effect of being ad-mitted to a specific charter school, in a spe-cific grade and year, on student achievementin subsequent years. Nonetheless, the mereavailability of lotteries does not guarantee that the estimates provide credible approxi-mations of how charter schools affect studentachievement. In order to take advantage of the opportunity presented by the many lotte-ries, the researchers must use appropriate sta-tistical models to analyze the data. As de-

scribed below, the statistical models used inmost of the analyses are not appropriate.

Several issues in the reports analysis indi-cate the need for caution in accepting someof the reports conclusions. In particular:

The report relies on an inappropriate setof statistical models to analyze the data.One feature in particular of the modelsusedthe inclusion of students testscores from the prior yearwill likelylead to mis-estimation of the charterschool effects. Because these test scores

are measured after the lotteries takeplace, and so could be affected bywhether students attend a charter schoolor not, they destroy the benefits of ran-domization. This flaw in the analysis af-fects the estimated effects of charterschooling on student achievement ingrades 4-12. The estimated effects of charter schools on achievement by thirdgrade are based on a different statisticalmodel that does not share this flaw, how-

ever, and so are more credible. The report includes claims regarding thecumulative effects of attending a NewYork City charter school from kinder-garten through eighth grade that arebased on an inappropriate extrapolation.

The report does not include adequatelydetailed information in some areas to al-low a reader to fully assess its methods,results, or generalizability.

The report uses a criterion for statistical

significance that is weaker than thatconventionally used in social science re-search;

The report describes the variation incharter school effects across schools in away that may distort the true distributionof effects by omitting many ineffectivecharter schools from the distribution.


3/29

http://epicpolicy.org/thinktank/review-How-New-York-City-Charter

As a result of the flaws in the report's statis-tical analysis, it likely overstates the effectsof New York City charter schools on stu-dents' cumulative achievement, though it isnot possiblegiven the information missing

from the reportto precisely quantify theextent of overestimation. It may be that NewYork City's charter schools do indeed have

positive effects on student achievement, butthose effects are likely smaller than the re-port claims. Policymakers, educators, andparents should not rely on the report's con-clusions regarding charter school effects in

grades 4-12 until these issues have been ful-ly investigated and the analysis has under-gone rigorous peer review.


4/29

http://epicpolicy.org/thinktank/review-How-New-York-City-Charter Page 1 of 26

Review

I. I NTRODUCTION

The question of whether charter schools are

more effective than traditional publicschools at improving students academicachievement is of considerable interest topolicymakers, educators, and parents. Some1.5 million students in the U.S. attendroughly 4,500 charter schools, a number theObama administration has been pushingstates to increase. 1 Although charter school-ing is a high-profile topic in educational pol-icy, high-quality, systematic evidence re-garding the effects of charter schooling is

relatively rare. In the past few years, howev-er, researchers have produced a growingnumber of studies and reports that attempt toprovide evidence regarding the effectivenessof charter schools relative to the local tradi-tional public schools that charter school stu-dents would have attended had charterschools been unavailable. 2

Prominent among these recent reports is aSeptember 2009 report by Caroline Hoxby,Sonali Murarka, and Jenny Kang, How NewYork Citys Charter Schools Affect Achieve-ment ,3 which describes enrollment patternsand average achievement impacts of charterschools in New York City. 4 This report is thesecond in a series of reports by Hoxby andher colleagues analyzing data from NewYork City to answer three questions: (1)What kinds of students enroll in New York City charter schools? (2) What are the effectson student achievement of attending a NewYork City charter school rather than a tradi-tional public school? (3) What features of charter schools are associated with morepositive effects of charter schools (that is,what kind of charter schools have the mostpositive effects on achievement)?

The Hoxby, Murarka, and Kang report is

written for a general audience and so containslittle of the technical detail regarding data,sample sizes and attrition, and analytic me-

thods that would typically be contained in ascholarly, peer-reviewed publication. Some,but not all, of the relevant supporting tech-nical documentation, however, is containedin an earlier technical report by Hoxby andMurarka, Charter Schools in New York City:Who Enrolls and How They Affect Their Stu-dents Achievement .5 Both the report and thetechnical report will be referred to throughoutthis review. In addition, I have had severalconversations with Caroline Hoxby over the

last few weeks, in which she has graciouslyanswered a number of questions to clarifyaspects of the reports data and methodology.Nonetheless, this review will focus primarilyon material included in the written reports;additional information provided by Hoxby isdescribed in the endnotes.

Outline of This Review

Section II of this review summarizes themain findings and conclusions of the report.Section III assesses at some length the ana-lyses regarding the effects of charter schoolson student achievement (Chapter IV of thereport), as these constitute the heart of thereport. In particular, Section III calls atten-tion to three specific weaknesses in the re-port. Section IV discusses a set of additionalconcerns regarding the report, concludingthat some of these concerns are minor, whileothers require more detailed information toassess. Section V briefly discusses the ana-lyses regarding the variation in charterschool effects (Chapter V in the report). Thefinal section of this review (Section VI) ex-plores the usefulness of the report for guid-ing policy and practice. A technical appen-dix contains extensive details regarding thestatistical issues discussed in the review.


5/29


II. F INDINGS AND C ONCLUSIONSOF THE R EPORT

The report contains three main sets of find-ings, corresponding to the three questions it

poses.

What kinds of students enroll in New York City charter schools? The report finds thatstudents who enroll in charter schools in NewYork are disproportionately non-HispanicBlack and poor, relative to all students inNew York Citys public schools (see TablesIIa and IIc in the report). Charter school stu-dents have average prior test scores similar tothose of the average student in New York

City schools (see Table IIb in the report). Thereport notes that these comparisons of priortest scores apply only to the roughly 20% of charter school students for whom test scoresare available prior to their enrollment in char-ter schoolsthat is, students who enroll incharter schools in grades 4 and higher afterattending another New York City school,since the test is initially administered to stu-dents in grade 3. These findings are discussedonly briefly in this review. Instead greaterattention is given to the next two sets of find-ings described below.

What are the effects on student achievement of attending a New York City charter schoolrather than a traditional public school? Interms of policy implications, the most im-portant findings of the report are the esti-mates of the effects of attending a charterschool rather than a traditional public schoolin New York City. These are summarized inTable IVc of the main report. That table in-dicates that by third grade the average stu-dent enrolled in a charter school in earlyelementary school gains 0.14 and 0.13 stan-dard deviations on the math and Englishachievement tests, respectively, relative tohow well she or he would have scored if enrolled in a traditional public school. The

typical charter school student will have beenin a charter school three to four years by theend of third grade, so these estimates implythat charter schools increase studentachievement by roughly 0.04 standard dev-

iations per year in grades K-3. Likewise, thereport finds that the average charter schoolstudent gains 0.12 and 0.09 standard devia-tions in math and English each year ingrades 4 through 8, relative to what she orhe would have gained each year in a tradi-tional public school. These results are statis-tically significant at the p


6/29


effect of charter schools on social studies testscores through fifth grade, Table IVe reportsan estimated charter school effect on socialstudies test scores of 0.17 standard deviationsper year from sixth through eighth grade.

While these estimated effects are large, noneare statistically significant by conventionalsocial science standards; each has a p-valueof roughly 0.15; the report describes the es-timated effects as marginally statisticallysignificant (see Table IVe).

The report finds that charter schools signifi-cantly ( p


7/29


patterns can then be attributed to the type of school they attend.

The availability of randomized lotteries thatdetermine admission to most of New York

Citys charter schools therefore provides re-searchers with the opportunity to obtain high-ly credible estimates of the effect of attendinga charter school in New York. Each school,year, and grade in which a lottery is con-ducted for admission serves as a small ran-domized controlled experiment that, in prin-ciple, allows researchers to estimate the ef-fect of being admitted to a specific charterschool, in a specific grade and year, on stu-dent achievement in subsequent years. By

using a statistical model to average the ef-fects of being admitted to charter schools ineach of the hundreds of lotteries, one can ob-tain an estimate of the average effect of at-tending a charter school among the popula-tion of students who attend charter schools.Only a few other studies of charter schoolshave relied on lotteries; 7 among those, thisstudy has by far the largest sample of schoolsand students (more detail below), and so hasthe potential to provide strong evidence re-garding the effects of oversubscribed charterschools, at least within New York City.

Statistical Analysis of Lottery Data

Although the availability of lotteries usedfor admission to most New York City char-ter schools provides the opportunity to ob-tain unbiased estimates of charter school ef-fects, the lotteries do not guarantee that theestimated effects are credible and unbiased.Several flaws in the statistical models usedin the report call into question the reportsestimates of the effects of charter schools.

First, the statistical models used to esti-mate the effects of charter schools mustnot introduce bias into the estimates. If the study consisted of a single charter

school lottery in which all lotteried-instudents enrolled in a charter school andall lotteried-out students enrolled in atraditional public school, a statisticalmodel would not be needed; we could

simply compare the later test scores of the charter and public school students.Because this study includes studentswho participated in hundreds of lotteriesin different years and grades, and whowere observed for different numbers of years and in varying grades, however, astatistical model is required to estimatethe average effects of charter schooling.As a result, it is important to assess theappropriateness of the statistical models

used for these purposes. As describedbelow, the statistical models used to es-timate the effects of charter schooling ingrades 4-12 are inappropriate.

Second, because there are relatively fewstudents who have been in a lottery formany years, estimates of cumulative ef-fects of charter schooling over manyyears must be based on an appropriateextrapolation. The report claims that the

annual charter school effects are suffi-ciently large that a student who attendedcharter schools from kindergartenthrough eighth grade would close almostall of the Scarsdale-Harlem gap As de-scribed below, the report relies on an in-appropriate extrapolation to estimate thecumulative effect of attending a charterschool for many years.

Third, the reader should have access to

sufficiently detailed information to un-derstand what set of students and schoolsare used to estimate the effects. For ex-ample, the report should include ade-quately detailed information to allow areader to determine the extent to whichthe estimated effects are based primarilyon data from a few charter schools or


8/29


from all charter schools. Although somesuch information is included in the tech-nical report, much is absent. The reportdoes not include adequately detailed in-formation in some areas to allow a read-

er to fully assess its methods or generali-zability.

Appropriateness of the Statistical Models

Bias due to inclusion of prior test scores inthe statistical model . Page III-6 of the reportdescribes the statistical model used in thereport to estimate the effects of charterschooling. One feature of this model is thatit relies on multiple observations of each

student, one for each post-lottery year inwhich a student has a test score in the data-base. So a student who participated in a lot-tery to enroll in fourth grade in a charterschool in 2004-05 will have (typically) fourobservations in the data: a fourth grade ob-servation from 2004-05; a fifth grade obser-vation from 2005-06; a sixth grade observa-tion from 2006-07; and a seventh grade ob-servation from 2007-08. The regressionmodel then predicts a students score in agiven year as a function of whether he or shewas enrolled in a charter school, controllingfor his or her prior years test score (whichstatisticians call a lagged score). Becausethis student participated in a lottery to enrollin a charter in school in fourth grade, thestudents prior years test scores in all butthe first observation will have been meas-ured after the lottery took place.

There are two primary problems with thisstatistical model. First, because the prioryears test score is measured after randomi-zation (except for the first year a student isin a charter school), the model destroys therandomization that is the strength of thestudys design. As discussed below, this willlikely lead the models to overstate the ef-fects of charter schools. A second problem

resulting from the inclusion of test scoresmeasured after randomization in the statis-tical model is that these test scores aremeasured with error (i.e., test scores do notperfectly measure students academic

achievement). This also will lead to themodels to overstate the effects of charterschools. Both issues are discussed in moredetail below.

Controlling for a test score measured after randomization destroys the studys claim tovalidity based on the lotteries random as-signment . To see this, note that the statisticalmodel that controls for the prior years testscores implicitly compares charter school

students to traditional public school studentswho had the same test score the prior year.While that may sound reasonable, it assumesthat charter and traditional public school stu-dents with the same prior test scores canstand as valid counterfactuals for one anoth-er. That is, it assumes that students in charterschools, had they instead attended a tradi-tional public school in a given year, wouldhave learned the same amount in that year asthose students in traditional public schoolswho started the year at the same achievementlevel. This would be a valid assumption if students were randomly assigned, each year ,to attend a charter school or a traditional pub-lic school. If this were the case, students as-signed to charter and traditional publicschools who had the same prior years testscore would be, on average, the same as oneanother, so a comparison of their subsequenttest scores would provide a valid estimate of the effect of attending a charter school in thatyear. But because enrollment to a charterschool is not randomly assigned every year,students who are in a charter school cannotbe assumed to be the same, in every way (in-cluding how much they would learn in a giv-en year in a given school), as students in tra-ditional public schools who start the yearwith the same level of achievement.


9/29


The statistical model used in the report thusrelies on an unverifiable assumption. Al-though the existence of the lotteries providesan opportunity to obtain unbiased estimates of charter school effects without relying on such

strong assumption, the report does not rely onthe randomization, but instead relies on thisunverifiable assumption about the similarity of students with the same prior years test scores.If there were no lotteries, this might be defens-ible (or might at least be the best one coulddo), but given the existence of the lotteries, amuch more straightforward and defensibleanalysis is possible. The estimated effectsfrom the models containing lagged scoresshould not be treated as unbiased estimates.

Moreover, a relatively straightforward statis-tical analysis indicates that the bias due toincluding the lagged test scores in the modelwill likely tend to exaggerate the true effects(if the true effects are positive, the laggedscore model will yield estimates that are toolarge; if the true effects are negative, thelagged score model will yield estimates thatare too negative). See the technical appendixfor more detailed discussion of this.

Finally, it is important to note that the estimatedcumulative effects of charter schools in gradesK-3 shown in the report are not subject to thistype of bias because the models estimatingthem do not contain any prior test scores. Theestimated annual effects of charter schools ingrades 4-8, on the Regents Examinations, andon graduating with a Regents Diploma, howev-er, are all subject to this type of bias, becausethey each rely on models that include prior testscores measured after the lotteries.

Measurement error in lagged test score willlead the estimates to overstate the charter school effect . A secondary problem resultingfrom the inclusion of test scores measuredafter randomization in the statistical model isthat these test scores are measured with error.

It is a well-established fact that statisticalmodels that condition on a variable measuredwith error will yield biased estimates. 8 In themodels used here, the bias resulting frommeasurement-error will tend to exaggerate

the effects of charter schools (if the true ef-fects or charter schools are positive, the esti-mated effects will appear be larger than thetrue effects; if the true effects are negative,the estimated effects will appear as morenegative than the true effects; for technicaldetail on this point, see the Appendix to thisreview). As noted above, the models used toestimate the effects of charter schooling bythird grade are not subject to this type of bias.

The inclusion of lagged scores measured witherror in the statistical models may account, inpart, for the results showing that charterschool effects are larger in grades 4-8 than ingrades K-3. Returning to Table IVc, for ex-ample, the reported annual effects of charterschools in the early elementary grades areroughly 0.04 standard deviations per year(0.14 s.d. over 3-4 years), while the estimatedannual effects in grades 4-8 are two to threetimes times larger (0.09-0.12 standard devia-tions per year). These larger figures are likelyinflated by measurement-error-induced bias of the type described above (in addition to biasdue to the fact that charter school enrollmentis not random, conditional on students priorscores). Without knowing the reliability of thetests and the average number of years elapsedbetween the lottery and the observations usedin the models, it is impossible to say exactlyhow large the measurement-error-induced biasmay be, but it clearly biases the estimatesaway from zero, making average charterschool effects appear larger than they are.

Computation of the Cumulative Effectsof Charter Schooling

Do charter schools significantly close theScarsdale-Harlem gap? The report claims


10/29


that the effects of attending a charter schoolfrom kindergarten through eighth grade aresufficiently large that they would close thegap between the average student in Harlemand the average student in Scarsdale by 66%

in English and 86% in Math. There are threereasons to doubt this claim. First, as de-scribed above, several aspects of the dataand the statistical models could lead the es-timated effects to be biased. If the true an-nual effects are smaller than the reportedeffects, then the true cumulative effects mustlikewise be smaller.

Second, the calculation of the cumulativeeffects of charter schooling in grades 4

through 8 is erroneous. Recall that one set of models attempts to estimate the annual ef-fects of charter schooling in each grade fromgrade 4 to 8. Even if the models did this cor-rectly (and they do not, because of the inclu-sion of the lagged test scores in the modelsand because of the presence of measurementerror in the test scores), one cannot simplyadd the annual impacts estimated from thesemodels in order to obtain the cumulativeimpact over a number of years. The reasonfor this is that achievement gains do notpersist perfectly from year to year. Studentswho have a good year one year dont alwayshave quite as good a year the next. Existingdata from New York City indicates thatabout 76%-80% of a students achievementgains (relative to his or her grade peers) inone year persist to the next year. 9 Thismeans that annual achievement effects mustbe discounted before summing them up tocompute a cumulative effect. The study doesnot appropriately discount the estimated ef-fects. This error, in conjunction with themeasurement error bias described above,likely results in the cumulative five-year ef-fect of charter schooling from grades 4through 8 being overestimated by as much50% (see the technical appendix to this re-view for details).

Third, roughly two-thirds of the students inthe study participated in lotteries to enter acharter school in the later years of 2004-05or 2005-06 (see technical report, Table 5),meaning that most of the charter school stu-

dents in the study have been in charter schoolsfor only three or four years. Moreover, it doesnot appear that any students in the study couldhave participated in lotteries and been in acharter school for nine years from kindergar-ten through eighth grade, because the first lot-teries in the study are from 2000-01. Even thenumber of students in the study who havebeen enrolled in charter schools for seven oreight years is likely relatively small, and mostsuch students will have attended one of only a

few charter schools in existence by 2001-02.This means that the only information about thelong-term cumulative effects of attending acharter schools comes from a relatively smallsubset of students who enrolled in one of thevery few charter schools operating before2002.

Even if the effects of these few early-established charter schools were as strong as isclaimed, it is far from obvious that such char-ter schools are typical of all the charter schoolsin operation today. Claims regarding the cu-mulative effects of attending a charter schoolfor nine years are therefore based on an un-warranted extrapolation of the available da-ta.10 The report would be much more informa-tive if it simply reported the cumulative effectsof attending a charter school for a given num-ber of years for the subset of students who at-tended for that number of years. This wouldallow a reader to see clearly how the charterschool effects accumulate over multiple years,without relying on unwarranted extrapolation.

Inclusion of Sufficiently DetailedTechnical Information

Data and sample . The report is based onanalysis of test score data and charter school


11/29


lottery data for approximately 30,000 stu-dents 11 who applied to a New York Citycharter school in the years 2000-01 through2005-06 and whose admission was deter-mined by participation in one of 725 charter

school lotteries. According to the report,roughly 94% of charter school students wereadmitted through randomized lotteries (p.vii). Of the 47 charter schools operating inNew York City as of 2005-06, 43 are in-cluded in the report (more detail below). Thereport relies on standardized test score data(New York State tests and Regents Examina-tions) and Regents Diploma data that wereavailable for the years 2000-02 through2007-08. Because the large sample of stu-

dents and charter schools, the large numberof charter school lotteries, and the availabilityof test score data spanning grades 3-12 overeight years, the report aims to provide un-biased estimates of the effects of charterschools that are generalizable to most charterschools and their students in New York City.

The report contains very little in the way of detailed descriptive information about thedata used in the estimation. There is no de-tailed information about (1) how many stu-dents participated in lotteries in each yearand grade; (2) what proportion of lotteryparticipants were lotteried-in and lotteried-out in each year and grade; and (3) whatproportion of lotteried-in and -out studentsare observed in the data, how many yearsthey are observed, and in what years andwhat grades they are observed. It is thus un-clear where the bulk of the information thatdrives the estimates comes from. Moreover,the lack of such information makes it diffi-cult to assess the extent to which schoolsand students the reports estimates apply orthe extent to which there may be bias in theestimated charter school effects.

To which schools and students do the reports estimates apply? The report states that

the estimates are representative of NewYork Citys charter school students ; themore students a school has enrolled, themore influence it will have on the results of the study (p. IV-2). More precisely, how-

ever, the estimates are representative of NewYork Citys charter school students whowere admitted via lottery and who were inthird grade or higher by 2007-08 ; the morestudents in test-taking grades that a schoolhas admitted via lottery, the more influenceit will have on the results of the study.

There were 47 charter schools operating inNew York City as of fall 2005; data from 43of them are included in this study (two char-

ter schools declined to participate; oneclosed in 2005-06; and one serves a specialpopulation of students). The report implies,but never states clearly, that each of theseschools was oversubscribed in at least onegrade and year and so admitted some stu-dents by lottery. However, some schoolsmay have a very small number of studentsadmitted via lottery and who were in test-taking grades by 2007-08.

If the proportion of students in a school ad-mitted via lottery is correlated with the effec-tiveness of the school, then the effect esti-mates will be biased in the direction of the ef-fects of the schools with the largest shares of students admitted via lottery. If better schoolsare more likely to be oversubscribed, andtherefore likely to have higher proportions of their students admitted by lottery, they will bedisproportionately overrepresented in the char-ter school effect estimates. Because the reportincludes no descriptive data regarding the pro-portion of students admitted via lottery in eachschool, we have no information that allows anassessment of whether the estimates providedhere generalize to the population of New York City charter school students. It would be help-ful for readers and users of this study if thestudys authors provided estimates of the as-


12/29


sociation between charter schools estimatedeffects and the proportion of their studentswho were admitted via lottery. This would beinformative not only regarding the generaliza-bility of the results, but also regarding the ex-

tent to which parental choices to apply to char-ter schools are related to school quality(another important policy question).

IV. A DDITIONAL C ONCERNS AND ISSUES

In addition to the issues described above,several issues are not adequately discussedin the report. These are briefly discussed be-low. Some appear to pose little threat to the

validity of the report; in other cases, there isinsufficient evidence in the report to assessthese issues.

Potential bias due to differential matchingrates . Depending on the year of lottery, be-tween 9% and 21% of charter school appli-cants in the study who participated in lotte-ries were not able to be matched to the NewYork City Department of Education data(Table 5, technical report), meaning that testscores were unavailable for these students.The technical report argues (see p. 12) thatthe primary reason some applicants were notable to be matched is that they were notenrolled in a New York City public schoolprior to applying to a charter school, and thennever attended one (because they subsequent-ly attended a private school, a school outsideNew York City, or were home-schooled).

The fact that outcome data are not observedfor all students who participated in lotteriesmay or may not lead to bias in the estimatedeffects of charter schools. For example, if themore academically able of the lotteried-outstudents disproportionately chose to go to pri-vate schools, then the study would be compar-ing charter school students to a less academi-cally able comparison group of students in

traditional public schools. This would makethe charter school students later test scoresappear better, relative to the comparisongroup, even if charter schools had no effect onstudent achievement.

The report does not indicate whether thematching rates differ between lotteried-inand lotteriedout students, which wouldhave been helpful in determining the extentto which missing data of this sort may biasthe estimates. The report would be moreuseful and informative if it provided moredetailed information on the extent to whichstudent records could not be matched toachievement data and how this non-

matching varies by lottery status, cohort,grade of lottery, and student demographiccharacteristics. Because the report containsvery little detail about the extent of non-matching, it is unclear to what extent thismay bias the estimated results. The technicalappendix to this review contains a brief dis-cussion of the extent to which differentialmatching may result in biased estimates.

Reliance on balanced lotteries . The reportrelies for its effect estimates only on thesubset of lotteries that appear to be ba-lancedthat is, those where the characte-ristics of lotteried-in and lotteried-out stu-dents appear to be similar. These include94% of students in lotteries. Under the pre-sumption of random assignment within eachlottery, standard practice would be to in-clude all lotteries in the analysis rather thana subset of the lotteries. If the lotteries areindeed random (as they appear to be), 12 thenestimates based on all lotteries will be un-biased estimates of the average effect of charter schools on all students who apply tocharter schools and participate in lotteries.Despite this, the main report includes onlyestimates based on balanced lotteries, andnotes that results based on all lotteries aresimilar (p. III-5) to those based only on


13/29


balanced lotteries. Because there is no evi-dence suggesting the lotteries (even the un-balanced ones) were not conducted via ran-dom assignment, it is unclear why the esti-mates from the balanced lotteries are pre-

ferred. At a minimum, the report should in-clude a presentation of results based on ba-lanced and unbalanced lotteries, as did thetechnical report (albeit using data from2005-06). 13

Description of charter school student popu-lation . The report describes how studentswho apply to and enroll in charter schoolscompare with students in New York Citypublic schools as a whole, which is useful.

Equally useful would be a description of how these students compare with the stu-dents in the subset of traditional publicschools that the charter school studentswould have attended if they were lotteried-out. That is, we would like to know if char-ter schools attract students who are dispro-portionately high- or low-achieving relativeto the public schools they come from. Thiswould directly answer the concern of somewho argue that charter schools cream thehighest achieving students from traditionalpublic schools. 14

The technical report contains some informa-tion relevant to this question. It shows thatstudents who applied to charter schools hadhigher average prior test scores than the av-erage student who attends the traditionalpublic schools from which the charter schoolstudents come (see Table 9 of the technicalreport). That is, within a given traditionalpublic school, the students who apply tocharter schools tend to be higher achieving(by 0.15-0.30 standard deviations) thanthose who do not. However, Hoxby told methat she has redone the analyses reported inTable 9 of the technical report, and thatthese new analyses show that charter schoolstudents do not have higher average prior

achievement than their peers at the tradition-al public schools from which they come.Because these new analyses are not includedin the report, however, a reader cannot as-sess their validity or implications.

Lottery oversubscription rates . The statistic-al models (described on page III-6) includewhat statisticians call lottery fixed effects.The inclusion of the fixed effects ensuresthat charter school students are implicitlycompared only to traditional public schoolstudents who participated in the same lot-tery. One consequence of using a fixed-effects model, however, is that the modelimplicitly weights students more in the esti-

mation if they participated in a charter schoollottery that was highly oversubscribed than if they participated in a lottery that was lessoversubscribed (see Appendix for a moretechnical discussion). This means that if indi-vidual charter schools effects are related tothe extent to which they are oversubscribed,then more effective charter schools may besystematically over- or under-weighted in theestimation, leading to bias in the results.There is no information of lottery oversub-scription rates in the text, though Hoxby re-ported to me that all NYC charter schoolshave approximately half of the students inthe lotteried-in group and half in the lotte-ried-out group. 15 This implies no correla-tion between lottery oversubscription ratesand charter school effects, and so impliesthat there is no resulting bias in the estimatesdue to this source. This should be clearlydocumented in the text.

The reported finding that all lotteries areroughly equally oversubscribed is somewhatsurprising, given that the lotteries includeboth new schools and more mature schoolsand span a range of years and grades. Thismerits discussion in the text. In particular, itraises an additional set of important ques-tions. Market competition theory would sug-


14/29


gest that if parents have information abouteither the quality of their local traditionalpublic schools or the available charterschools, they will be more likely to apply tothe charter schools that are highly effective

relative to the local traditional public schools.This would lead, all else being equal, to moreapplicants to the most effective charterschools (or more applicants among familieswhose alternative traditional public school isineffective). In fact, part of the rationale forcharter schools (and for private schoolvouchers and other forms of school choice) isthat competition among schools will forceschools to improve or to lose enrollment. If almost all charter schools are not only over-

subscribed (according to the report, 43 of 47charter schools in New York City rely on lot-teries for admission), but are all equally over-subscribed, it suggests parents may not beable to differentiate among charter schools interms of quality (though it does suggest thatthere is excess demand for charter schools inNew York City). Although this study is notdesigned to address the reasons for this, theauthors (or others) are encouraged to considerit in future analyses. At a minimum, thisstudy should report detailed data on lotteryoversubscription rates that would inform fu-ture studies.

Compliance-effect correlation bias . The re-port relies on instrumental variables (IV)models to provide estimates of the averageeffect of charter schooling. One feature of IV models of the sort used in the report isthat they implicitly weight students by howlong they remain in a charter school. If stu-dents for whom charter schools have largerpositive effects are more likely to stay in char-ter schools if lotteried-in than are students forwhom charter schools have smaller effects,then those students for whom charter schoolsare more effective will tend to get moreweight in the estimates, meaning the estimatedeffects will be overstated. The New York

City report states that very few (8%) lotte-ried-in students who enroll in a charterschool for at least one year ever return to thetraditional public schools. Given these lowrates of attrition from charter schools, the

threat to internal validity posed by using theIV model appears minimal. Nonetheless, thelow attrition rate relative to that reported inother studies also suggests the uniqueness of the New York City charter school context.

Existing research from other studies pro-vides some evidence both of high attritionrates in charter schools and that continuedcharter school enrollment may indeed becorrelated with the magnitude of charter

school effects. An analysis of attrition fromKIPP charter schools in the San FranciscoBay area, for example, found that studentswho left the KIPP schools prior to eighthgrade had experienced smaller gains in theKIPP schools in fifth grade than their coun-terparts who stayed in the KIPP schools. 16 Astudy of Boston charter schools found highrates of attrition from middle and highschool charter schools, though it did not ex-amine whether those who left charterschools had different patterns of achieve-ment gains in charter schools than those whostayed. 17 The low attrition rate in New York City relative to that reported in other studiessuggests the possible uniqueness of the NewYork City charter school context.

Are the estimated effects of charter schoolingstatistically significant? As described above,the report estimates the effects of charterschooling on four sets of outcomes: (1) mathand English tests in third through eighthgrade; (2) science and social studies tests infourth through eighth grade; (3) Regents Ex-aminations in high school; and (4) graduationwith a Regents diploma by age 20. Of these,the reported effects on the math and Englishtests in third through eighth grade and on theRegents Examinations are generally statisti-


15/29


cally significant at the conventional p


16/29


interpreted as causal relationships. For exam-ple, the existence of a correlation between theuse of performance-based teacher pay and theeffectiveness of charter schools does not imp-ly that if a school adopts performance-based

pay its students will learn more.

In addition to the policies of charter schoolsthemselves, there are two potentially impor-tant reasons for variance in achievement ef-fects among charters schools, neither of which is explored in the report. First, differ-ent charter schools enroll different popula-tions of students. If New York City charterschools as a whole are more effective forsome types of students than others, then char-

ter schools enrolling more of those types of students will appear more effective than thoseenrolling fewer such students, even if the twoschools are otherwise identical. For instance,New York City charters may, relative to tra-ditional public schools in the City, take moreadvantage of involved parents. If this is thecase, then charters that enroll more studentswith such parents might be expected to showbetter outcomes than those that do not.

Second, it is worth noting that each charterschool (each lottery, actually) has a some-what different counterfactual. What is de-scribed as the effect of a given charterschool is more accurately described as theeffect of attending that charter school ratherthan the traditional public school that a stu-dent would have attended had she or he notbeen lotteried-in to the charter school. Be-cause the traditional public schools attendedby lotteried-out students differ across lotte-ries (students applying to a charter school inthe Bronx are unlikely to attend a traditionalpublic school in Brooklyn if lotteried-out,and vice versa), the comparison traditionalpublic schools differ across charter schools.As a result, the variation in the charterschool effects reported in the study may re-sult as much from variation in the quality of

the traditional public schools that are alter-natives to each charter school as from varia-tion in the quality of charter schools them-selves. A given charter school of moderatequality may appear very effective if com-

pared with a low-quality traditional publicschool that the students would otherwisehave attended. But that same charter schoolwould appear ineffective if the students al-ternative school happened to be a high-quality traditional public school. This doesnot in any way undermine the lottery-baseddesign of the study; it is meant only to pointout a potential reason for variation amongcharters that was not discussed in the report.

None of the above necessarily disqualifiesthe descriptive results reported regarding theassociation between charter school policiesand their effectiveness, but it should makeclear that we cannot unambiguouslyattribute variation in charter school effec-tiveness to charter school policies alone.Nor, as the report notes, can we be sure thatadopting policies such as a longer schoolday, performance pay for teachers, or aca-demic-based mission statements would re-sult in improved student achievement.

VI. U SEFULNESS OF THE R EPORTFOR G UIDANCE OF POLICYAND P RACTICE

The data used in the New York City charterschools report have the potential to be veryvaluable for informing policymakers andeducators regarding the effectiveness of oversubscribed charter schools in New York City. The data include information on morethan 40 charter schools, 725 distinct lotte-ries, and thousands of students observedover multiple years and grades.

Nonetheless, the analyses in the study con-tain several possible sources of bias. Themost significant of these stem from the in-


17/29


clusion of students prior years test scoresin the statistical models estimating the ef-fects of charter schooling for all gradesabove third grade. This both destroys therandomization that is the potential strength

of the studys design and introduces mea-surement-error bias in the estimates. In addi-tion, the estimated cumulative effects of charter schools are based on an incorrectcomputation and on an extrapolation beyondthe limits of the data. As a result of theseflaws, the report considerably overstates theeffects of New York City charter schools onstudents cumulative achievement.

The likely bias in the estimates could be eas-

ily eliminated by using a more appropriatestatistical model, one that does not conditionon lagged test scores. Estimates from suchmodels would be more defensible, moretransparent, and would avoid relying on un-tenable assumptions and invalid extrapola-tion. Given the prominence of charterschools in current education policy discus-sions, it is important that policymakers, par-ents, educators, and scholars have access tothe most accurate possible evidence regard-ing the effects of charter schools.

That said, it is worth noting that the esti-mates of the effects of charter schooling inkindergarten through third grade do not suf-fer from the same statistical flaws as the es-timates of effects in fourth grade and higher.These estimates indicate that students whoattend charter schools in the early elementa-ry grades have higher achievement levels(0.13-0.14 standard deviations higher bythird grade) than they would have had if they had attended a traditional public school.

This is important evidence. In addition, de-spite the sources of bias in the estimates forthe higher grades, there may still be positiveaverage effects of charter schooling ingrades 4-8. A more appropriate statistical

analysis would provide a clear answer.

In addition, the report contains very little inthe way of detailed information regardinglottery participation and oversubscriptionrates. As a result it is unclear how generaliz-able the results are across charter schools,grades, and time. Moreover, it is unclearwhether there may be additional sources of bias, because the report does not always in-clude enough information for the reader to

assess the methods and results. The reportwould be far more useful to scholars seekingto understand if and how charter schools af-fect student achievement, for example, if itpresented a detailed accounting of applica-tion, enrollment, retention, and attrition pat-terns among charter school students.

As a result of the potential sources of biasand the lack of detailed information in thereports to assess the extent of such bias, it isnot possible for this reviewer or other read-ers to determine the degree to which the es-timated charter school effects in grades 4and above are valid. Policymakers and edu-cators should therefore not rely on these es-timates until the bias issues have been fullyinvestigated and the analysis has undergonerigorous peer review. Given the quality of the data, however, a revised version of theanalysis could provide a more definitive an-swer regarding the effectiveness of NewYork Citys oversubscribed charter schools.


18/29


Technical Appendix

A. Problems With the Statistical Model Used in the Report

A stylized example. Before discussing the statistical model, it is useful to simplify matters byconsidering a single lottery, in which students are randomly assigned to a treatment or controlgroup. Suppose all students comply with their treatment assignment for the duration of the timewe observe them (that is, if lotteried-in, students attend a charter school for the duration of thestudy; if lotteried-out, they attend a traditional public school for the duration of the study). Sup-pose also that we observe a test score for each student prior to randomization and at the end of each year after randomization. We standardize all test scores to have a mean 0 and a standarddeviation 1 in the control group (standardizing based on the control group will make all esti-mated effects expressible in terms of effect sizes relative to the control group distribution; it alsomake the derivations below a bit simpler).

We are interested in knowing the cumulative effect of the treatment after years. That is, wewant to know how much higher a students score would be in year if he attended a charterschool from year 1 to instead of a traditional public school. A straightforward way to do thiswould be to fit either the model

or the model

Either would yield an unbiased estimate of the average cumulative effect of attending a charterschool for years, measured in standard deviation units. If we had access to test scores meas-

ured prior to randomization ( ), the second model would be preferable because it would likelyyield more precise estimates because of the inclusion of pre-randomization test scores.

What model does the report fit? The report uses a model of the first type above in estimating theaverage cumulative effects of charter schooling by third grade. To estimate the annual effect of charter schooling in grades 4 and above, however, the report relies on a model of this form (seepage III-6):

This model says that a students observed test score in year depends linearly on his or her score

in the prior year plus an effect of attending a charter school, plus some random, i.i.d, mean-zero error. Because both and are standardized, , and This model is fit using the stacked data; that is, if student test scores are observedfor years following randomization, each student will have observations in the data. As isstandard in models like this, the standard errors are corrected for the fact that there are multipleobservations per student. Note that the model used in the report is actually more complex thanthisit also includes lottery fixed effects, grade fixed effects, year fixed effects, and student co-variates, and is fit using a lottery assignment as an instrument for enrollment in a charter school


19/29


in each year. Nonetheless, this additional complexity does not affect the issues described below.

Problem 1: Controlling for test scores measured after randomization destroys the benefits of ran-domization. In the model described above and used in the report, OLS will yield an unbiased es-timate of if treatment assignment is independent of the error, conditional on (that is, if

). Because is measured after randomization (except for the very first observa-tion, when ), however, the randomization does not ensure that ) for all . Thus, including lagged test scores measured after randomization destroys the primarybenefit of randomizationthe guarantee that treatment status is uncorrelated with all characteris-tics of students, observed or unobserved.

It is possible, under some mild assumptions, to determine the likely direction and magnitude of the bias in models that include lagged scores measured after randomization. Suppose each stu-dents potential outcomes in years 1 and 2 are defined by

where is the students score prior to randomization; and are the gains that a studentwould make in years and , respectively, if he or she attended a traditional public school; and are dummy variables indicating whether a student attended a charter school in years 1and 2, respectively; and and are the effects of attending a charter school in years 1 and 2,respectively. Note that the above implicitly assumes that the gain a student would make in year 2is independent of whether or not she or he attended a charter or traditional public school in year1. Suppose, moreover, that test scores of the control group are standardized at each wave.

In order to estimate the effect of charter schooling in year 2 on the students who attended charter

schools, we must estimate what the test scores of the charter school students would have been inyear 2 had they attended a traditional public school in year 2. It is tempting to use the observedyear 2 test scores of public school students who had similar test scores as the charter school stu-dents in year 1. However, this assumes that the gains that charter school students would havemade in year 2 had they gone to a traditional public school ( ) are the same as the gains madeby traditional public school students who had similar scores in year 1. Had we randomized stu-dents to charter or traditional public schools at the end of year 1, we could be sure that the out-comes of traditional public school students in year 2 could stand as a valid counterfactual for thecharter school students test scores in year 2. However, it we randomized students earlier thanthe end of year 1, it is not clear that this would be a valid counterfactual. To assess whether thisis valid, consider the following.

Suppose the effect of charter schooling in year 1 is positive. Then it will be true that

That is, among students with the same scores at time 1, the charter school students will have hadlower average scores at time 0 than the traditional public school students. If there is a non-zerocorrelation between students time 0 scores and the gains they would make in year 2 in a tradi-


20/29


tional public school (that is, if ), then traditional public school students scoresat time 2 cannot stand as valid counterfactuals for charter school students who had the samescores as them at time 1. Moreover, the direction of bias will depend on the sign of .If ,

which implies that the students in charter schools would have had higher scores, had they been intraditional public schools, than the traditional public school students who had the same scores attime 2. As a result, the estimates based on comparing charter and traditional public school stu-dents with the same time 1 scores will be biased upwards. If, on the other hand,

, the same logic implies that the estimated charter school effects in year 2 will be biased down-ward. Because the test scores are standardized within each wave, it is straightforward to write thecovariance of and as

In general, the correlation between student test scores at time 0 and time 2 will be lower than thecorrelation between scores at time 0 and time 1. 19 Thus, . This implies that, if the true effect of charter schools is positive, the likely direction of bias due to conditioning onlagged standardized scores is upward (because charter school students will, on average, havelower initial scores than public school students with the same scores at time 1; and so the factthat implies they would have gained more, on average, in a traditional publicschool in year 2 than did the students who were in a traditional public school). If is the cu-mulative effect by time

, then the magnitude of the bias will be

That is, the annual effects will be overstated by an amount that is proportional to the cumulativeeffect of charter schooling over the average number of years students have been in charterschools when their lagged scores are measured.

Problem 2: Measurement error in test scores will lead to biased estimates of the charter schooleffect. Let us suppose the statistical model is correct. That is, suppose treatment assignment isignorable conditional on the lagged score. In that case, OLS will yield an unbiased estimate of

so long as is measured without error. But test scores do not measure achievement withouterror. Therefore, it is important to consider if and how measurement error in may affect theestimate of .

To understand the impact of measurement error on the estimate of , it is useful to begin by con-sidering a model estimated on a single years observation for each student. Suppose lagged testscores are measured without error and the true data generating model is given by


21/29


Now, suppose the lagged test score includes independent random measurement error with va-riance . Because the observed test scores are standardized, this implies that the true score hasvariance

within treatment groups. That is,

Now define the reliability of the lagged score to be

Now if we fit the model to the observed data using OLS that is, if we fit

then OLS will yield

and

where is the difference in average test scores at time between charter school studentsand traditional public school students. The bias will depend on (the observed correlation be-tween students scores at times and ), the reliability of the observed scores, and . If the reliability is close to 1, , so the bias will be small. Likewise, if the difference in priorscores is small, the bias will be small.

Because the observed test scores are standardized, is the correlation between scores at time


22/29


and . This will be relatively high. Data from a study of the test scores of New York Citypublic school students show that the grade-to-grade correlations of student math scores rangefrom 0.76 to 0.80 across grades 3 to 8, with an estimated average of 0.79. 20 For the sake of illu-stration, let us assume . The reliability must be . Boyd et al. estimate thereliability of the New York state tests to be 0.831. 21 Then the bias will be

We know that, at the end of third grade, (TABLE IVc). This suggests that an esti-mate of the annual effect of charter schooling on achievement during fourth grade will be biasedupwards by . This is one-quarter to one-fifth of the estimated annualeffects on reading and math. Because the model is fit on the stacked data, the amount of bias willdepend on the magnitude of when pooled over all observations used in the estimation. Be-cause information regarding is not available in the report, it is not clear how large the mea-surement error bias will be. Nonetheless, the bias will clearly inflate the magnitude of the true

effects.

Problem 3: The cumulative effect of charter schooling is not equal to the sum of the estimatedannual effects. In order to compute the estimated cumulative effect of charter schooling over anumber of years, one cannot simply multiply the estimated annual effect by the number of years.This is because students achievement gains do not persist perfectly from year to year. To seethis, consider the following. Suppose the model used in the report represents the true data gene-rating process. That is, suppose the data are generated by the following structural model:

where . Note that, because each of the test scores are stan-dardized, .Then, substituting into the equation for , we get

Likewise, substituting this into the equation for , we get

and so on. The th equation will be


23/29


Thus, the cumulative effect of attending a charter school for years is equal to, where is the annual effect. Unless , . Because is the correla-tion of test scores in adjacent years, it will be less than 1 unless all students have identical growth

rates in test scores and there is no measurement error in the test scores. Any estimate of the cu-

mulative effect of attending a charter school that is based on simply summing the estimated an-nual effects with therefore overestimate the true cumulative effect.

By how much will using overstate ? As noted above, data from a study of the test scoresof New York City public school students show that the grade-to-grade correlations of studentmath scores range from 0.76 to 0.80 between across grades 3 to 8. For the sake of illustration, letus take . If the estimated value of the annual effect of charter schooling is 0.12 in math,the cumulative effect over the five years from grades 4-8 will be

Compare this to the value we would get if we simply sum the annual effects. If we simply mul-tiply the estimated annual effect (0.12) by 5 years, we get an estimated cumulative effect of 0.60;an estimate that is 50% larger than the correct cumulative effect implied by the structural modeland the correlations between annual test scores.

Note that the derivation above assumes that test scores contain no measurement error. If there is

measurement error in the test scores (as there certainly is), then the observed is due to a com-bination of measurement error and the extent of persistence in true achievement gains over time.The more measurement error there is, the less upward bias due to the incorrect accumulation of gains over time there will be (because if is primarily due to measurement error, then the corre-lation of true scores will be higher than 0.8), but the more upward bias due there will be due tomeasurement error, and vice versa. In other words, the combination of measurement error biasand the accumulation error will result in upward bias of the estimates, though the exact amountwill depend on the extent of measurement error in the test scores. Both sorts of bias could bereadily eliminated by estimating cumulative effects using a model that does not control for post-randomization test scores.

B.

Potential Bias Due to Non-Matching of Students to Department of Education RecordsIn an experiment, the estimated treatment effect is given by

where is the average outcome among students assigned to the treatment condition (charterschools in this case) and is the average outcome among those assigned to the control condi-


24/29


tion. Let and indicate the proportions of students assigned to the treatment and controlconditions, respectively, whose outcomes are observed (some students outcomes are not ob-served because they could not be matched to the Department of Education records, for example).Then we can write

where , , , and are the average outcomes among students in the treatmentand control groups who are observed and missing, respectively. Because we can only base ourestimates of the charter school effect on the cases for whom we observe outcomes, we will have

So the bias will be given by

The report does not describe what proportions of lotteried-in and -out students are matched, sowe do not know if is larger, smaller, or equal to . If students who are lotteried-out are morelikely to subsequently enroll in a private school or a school outside the NYC public school sys-tem than are lotteried-in students, then lotteried-out students will have lower matching rates thanlotteried-in students, so . Moreover, we cannot observe or , by definition,but the authors may be able to provide information on some characteristics of those who werenot matched to DOE data (because some demographic information is contained on the lotteryapplication forms and thus is available for students even if they are not matched). An analysis of these factors could help to assess whether non-matching may produce upward or downward biasin the estimates.

C. Weighting Due to Inclusion of Lottery Fixed Effects

In a model that includes lottery fixed effects, as do the models used in the report, the estimatedeffect will be

where is the number of students participating in lottery , is the proportion of students inthe lottery who are lotteried-in in lottery , and is the average effect of charter schoolingamong students participating in lottery . That is, the estimand from the fixed effects model is aweighted average of the lottery-specific effects, where the weights are proportional to

. The advantage of this weighting is that it produces a statistically efficient estimatelotterieswith the most information are weighted most, yielding precise estimates. The disadvantage of this weighting is that if the lottery-specific effects are correlated with the weights, the estimates


25/29


26/29


Notes and References

1 Maxwell, Lesli A. (2009, June 17). Obama Team's Advocacy Boosts Charter Momentum, Education Week , 28:Issue 35, pp. 1,24-25; Maxwell, Lesli A. (2009, November 2). Charter Schools Steadily Growing, Ed-week.org , Retrieved November 3, 2009, fromhttp://blogs.edweek.org/edweek/District_Dossier/2009/11/charters_continue_their_march.html .

2 For recent studies, see, for example

Abdulkadiroglu, A., Angrist J., Cohodes S., Dynarski, S., Fullerton, J., Kane, T., & Pathak, P. (2009). Informing thedebate: Comparing Bostons charter, pilot and traditional schools . Boston, MA: The Boston Foundation

Woodworth, K.R., David, J.L., Guha, R., Wang, H., & Lopez-Torkos, A. (2008). San Francisco Bay Area KIPPschools: A study of early implementation and achievement. Final report. Menlo Park, CA: SRIInternational

Center for Research on Education Outcomes (CREDO) (2009, June). Multiple Choice: Charter School Performancein 16 States . Palo Alto: CREDO, Stanford University

Hoxby, C. M. & Rockoff, J. E. (2004). The impact of charter schools on student achievement . Unpublished manu-script, Harvard University and Columbia University

Dobbie, W., & Fryer, R. G. (2009). Are High-Quality Schools Enough to Close the Achievement Gap? Evidence from a Bold Social Experiment in Harlem . Unpublished manuscript, Harvard University.

For reviews of the charter school effects literature, see

Betts, J. R. & Tang, Y. E. (2009, April). Charter School Achievement: What We Know , 5th Edition. Washington,DC: National Alliance for Public Charter Schools

Betts, J. R. & and Tang, Y. E. (2008). Value added and experimental studies of the effect of charter schools on stu-dent achievement . Seattle, WA: National Charter School Research Project, Center on Reinventing Public Education, University of Washington, Bothell

Carnoy, M., Jacobsen, R., Mishel, L., & Rothstein,R. (2006). Worth the Price? Weighing the Evidence on CharterSchool Achievement. Education Finance and Policy, Winter 2006, Vol. 1, No. 1, Pages 151-161;

Hill, P.T., Angel, L., & Christensen, J.. Charter School Achievement Studies. Education Finance and Policy, Win-

ter 2006, Vol. 1, No. 1, Pages 139-150Miron, G., Evergreen, S., & Urschel, J. (2008). The impact of school choice reforms on student achievement .

Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit RetrievedNovember 10, 2009, from http://epicpolicy.org/files/CHOICE-10-Miron-FINAL-withapp22.pdf .

3 Hoxby, C. M., Murarka, S. & Kang, J. (2009, September).How New York Citys Charter Schools AffectAchievement. Second report in series. Cambridge, MA: New York City Charter Schools EvaluationProject, September 2009. Retrieved October 1, 2009, fromhttp://www.nber.org/~schools/charterschoolseval/ .

4 More specifically, the report provides estimated achievement impacts among students admitted to New York Citycharter schools via lottery during the years 2000-01 to 2005-06. Because almost all charter schools in NewYork City were oversubscribed, the report covers almost all charter schools in New York City during thisperiod. More detail is provided on this point later in this review.

5 Hoxby, C. M., Murarka, S.. (2009, April). Charter Schools in New York City: Who Enrolls and How They AffectTheir Students Achievement. National Bureau of Economic Research Working Paper 14852, April 2009.Cambridge, MA: National Bureau of Economic Research. Retrieved October 1, 2009, fromhttp://www.nber.org/~schools/charterschoolseval/ .

6 Note also that these estimates are based on only two charter schools: those that enrolled students in grades 9-12 in2005-2006 (only students in these schools could have both attended a charter high school and be observedat age 20 in 2007-08). See Table IVa in main report and footnote 24 in the technical report.


27/29


7 Abdulkadiroglu, A., Angrist J., Cohodes S., Dynarski, S., Fullerton, J., Kane, T., & Pathak, P. (2009). Informingthe debate: Comparing Bostons charter, pilot and traditional schools . Boston, MA: The Boston Founda-tion

Hoxby, C. M. & Rockoff, J. E. (2004). The impact of charter schools on student achievement . Unpublished manu-script, Harvard University and Columbia University

Dobbie, W., & Fryer, R. G. (2009). Are High-Quality Schools Enough to Close the Achievement Gap? Evidence from a Bold Social Experiment in Harlem . Unpublished manuscript, Harvard University.

8 Greene, W. H. (2003). Econometric Analysis . Fifth Edition. Upper Saddle River, NJ: Prentice-Hall.9 Boyd, D., Grossman, P. Lankford, H., Loeb, S., & Wyckoff, J. (2008). Measuring Effect Sizes: the Effect of

Measurement Error. Paper prepared for the National Conference on Value-Added Modeling University of Wisconsin-Madison April 22-24, 2008. Retrieved October 20, 2009, fromhttp://www.teacherpolicyresearch.org/portals/1/pdfs/Measuring%20Effect%20Sizes%20%20the%20Effect%20of%20Measurement%20Error%20Boyd%20et%20al%20%2026Jun2008.pdf

10 In order to check whether this extrapolation is valid (or at least to check whether the small subset of students whohave been in charter schools for many years have benefited as much as is claimed), the report includes twofigures (Figures IVa and IVb) that compare the regression-based cumulative gain estimates with the raw

data for the subset of students who attended charter schools from third through eighth grade (or perhapsfrom kindergarten through first grade; the report includes contradictory statements regarding what studentsare included in the raw data lines in Figures IVa and IVb; see p. IV-8 and footnote 7, page A-1). It is un-clear what is meant by the statement that the figures use raw data from students who attended charterschool students. It may mean that the figure reports average annual gains for the subset of lotteried-in char-ter school students who are observed for the entire period from third through eighth grade. However, if thisis the case, the comparison is problematic. The relevant check for the extrapolation would be to estimatethe effects of charter schooling using only the subset of students who are observed from kindergartenthrough 8 th grade (if there are any such students who participated in lotteries; the report does not appear toinclude students who participated in lotteries earlier than 2000-01) and to compare this estimate to the es-timate based on the full sample of students. Figures IVa and IVb do not provide such a comparison. Allthey show is what the test scores of a subset of charter school students were from grades 3 to 8; this is notthe same as reporting what the effect of attending a charter school is from grades 3 to 8 or grades K to 8.

Accordingly, Figures IVa and IVb do not provide the necessary information to determine whether theextrapolation is valid even for the small subset of students who have attended charter schools for manyyears.

11 Neither the main report nor the technical report includes clear and detailed information regarding sample sizes, butTable 5 in the technical report implies that roughly 33,000 students applied to charter schools and partici-pated in lotteries from 2000-01 through 2005-06, of whom roughly 85-90% were matched to New York City Department of Education data.

12 Because each charter school controls its own lotteries (that is, the charter school lotteries are not run by an exter-nal agency, unlike the New York City public school choice program), it is impossible to verify that the lot-teries were random. Nonetheless, the report provides sufficient evidence to suggest that the lotteries were infact random (and no evidence that suggests they were not). First, Table 10 in the technical report shows nostatistically significant differences in demographic characteristics or prior test scores between those lotte-ried-in and those lotteried-out, when averaged over all lotteries in the data (see columns 1 and 4, Table 10,technical report). Second, the researchers examined each individual lottery to see if the lotteried-in and lot-teried-out students were similar in each lottery. They report that the vast majority (94%) of lotteries appearbalanced (if all lotteries were truly random, 95% would appear balanced according to the criterion used inthe report; sampling variation will result in some lotteries yielding unbalanced samples). Thus, there isnothing in the reported data to suggest that the lotteries were not conducted through actual randomization.Moreover, the unbalanced lotteries as a group appear to be collectively balanced. Table 10 in the tech-nical report shows that the overall balance on observable characteristics is just as good, if not better, whenall lotteries are included as when only the balanced lotteries are included. This implies that the 6% of lotte-


28/29


ries that are individually unbalanced are not systematically unbalanced in any one direction. They are justas likely to be unbalanced in favor of high-achieving as low-achieving students, for example.

13 Although the September 2009 report does not include estimates based on data from all lotteries, the technical re-port includes estimates based on balanced lotteries alone as well as estimates based on all lotteries (albeitusing 2005-06 data). Based on those earlier data, the technical report shows that the estimated effects aresmaller (40% smaller in math, 15% smaller in English) when all lotteries are included (see columns 5 and 8of Tables 12 and 13, technical report).

14 Evidence on the extent to which charter schools attract disproportionately more or less academically skilled stu-dents varies, and likely depends on the specific charter school and context. For some discussion of theseissues and related evidence, see

Carnoy, M., Jacobsen, R., Mishel, L. & Rothstein, R. (2005). The charter school dust-up: Examining the evidenceon enrollment and achievement . New York and Washington DC: Teachers College Press, and the Econom-ic Policy Institute.

Hoxby, C. M. & Murarka, S. (2006) Comprehensive Yet Simple: Floridas Tapestry of School Choice Programs.In Paul Peterson (Ed.), Reforming Education in Florida . Stanford: Hoover Institution Press.

Woodworth, K.R., David, J.L., Guha, R., Wang, H., & Lopez-Torkos, A. (2008). San Francisco Bay Area KIPP

schools: A study of early implementation and achievement. Final report. Menlo Park, CA: SRI Internation-al

Center for Research on Education Outcomes (CREDO) (2009, June). Multiple Choice: Charter School Performancein 16 States . Palo Alto: CREDO, Stanford University.

15 Caroline M. Hoxby, personal communication, 8 October, 2009 and 28 October, 2009.16 Woodworth, K.R., David, J.L., Guha, R., Wang, H., & Lopez-Torkos, A. (2008). San Francisco Bay Area KIPP

schools: A study of early implementation and achievement. Final report. Menlo Park, CA: SRI Internation-al

17 Abdulkadiroglu, A., Angrist J., Cohodes S., Dynarski, S., Fullerton, J., Kane, T., & Pathak, P. (2009). Informingthe debate: Comparing Bostons charter, pilot and traditional schools . Boston, MA: The Boston Founda-tion

Skinner, K.J. (2009). Charter school success or selective out-migration of low achievers. Boston: MassachusettsTeachers Association. Retreived October 25, 2009, fromhttp://www.massteacher.org/news/headlines/charterschools0909.pdf .

18 According to the report, the figures exclude charter schools whose estimated effects are so imprecise that an effectof 0.10 standard deviations per year would be statistically insignificant at the level unless the ef-fects are large enough that they are nonetheless statistically significant at the level (see notes forFigure IVg, p. IV-21 and footnote 8, p. A-1). Hoxby, however, informed me that the text is incorrect: thefigures exclude charter schools whose estimated effects are so imprecise that an effect of 0.10 standard dev-iations per year would be statistically insignificant at the level, regardless of the size of the esti-mated effect.

19 Boyd, D., Grossman, P. Lankford, H., Loeb, S., & Wyckoff, J. (2008). Measuring Effect Sizes: the Effect of Measurement Error. Paper prepared for the National Conference on Value-Added Modeling University of Wisconsin-Madison April 22-24, 2008. Retrieved October 20, 2009, fromhttp://www.teacherpolicyresearch.org/portals/1/pdfs/Measuring%20Effect%20Sizes%20%20the%20Effect%20of%20Measurement%20Error%20Boyd%20et%20al%20%2026Jun2008.pdf . See Table 1.

20 Boyd, D., Grossman, P. Lankford, H., Loeb, S., & Wyckoff, J. (2008). Measuring Effect Sizes: the Effect of Measurement Error. Paper prepared for the National Conference on Value-Added Modeling University of Wisconsin-Madison April 22-24, 2008. Retrieved October 20, 2009, fromhttp://www.teacherpolicyresearch.org/portals/1/pdfs/Measuring%20Effect%20Sizes%20%20the%20Effect%20of%20Measurement%20Error%20Boyd%20et%20al%20%2026Jun2008.pdf . See Tables 1 and 2.


29/29

21 Boyd, D., Grossman, P. Lankford, H., Loeb, S., & Wyckoff, J. (2008). Measuring Effect Sizes: the Effect of Measurement Error. Paper prepared for the National Conference on Value-Added Modeling University of Wisconsin-Madison April 22-24, 2008. Retrieved October 20, 2009, fromhttp://www.teacherpolicyresearch.org/portals/1/pdfs/Measuring%20Effect%20Sizes%20%20the%20Effect%20of%20Measurement%20Error%20Boyd%20et%20al%20%2026Jun2008.pdf . See p.17.

The Think Tank Review Project is made possible by fundingfrom the Great Lakes Center for Education Research and Practice.

ttr hoxby charters

Documents