kipp: effectiveness and innovation in publicly-funded, privately-operated schools october 4, 2012...
TRANSCRIPT
![Page 1: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/1.jpg)
KIPP: Effectiveness and Innovation in
Publicly-Funded, Privately-Operated SchoolsOctober 4, 2012
Presentation to the APPAM/INVALSI Improving Education Conference
Christina Clark Tuttle Philip Gleason Brian Gill Ira Nichols-Barrer
![Page 2: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/2.jpg)
Network of 125 charter schools serving over 39,000 disadvantaged students in 20 states and D.C.
Model involves high expectations and more time in school to prepare students for college
Early KIPP model served grades 5-8 (age 10-14)– Represent a majority of schools in operation (n=70)– First elementary and high schools opened in 2004
Mixed effects for charter schools generally
Positive pattern of findings for KIPP specifically
Background
2
![Page 3: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/3.jpg)
What are the impacts of KIPP middle schools on student achievement and other student outcomes?
Do KIPP schools engage in selective entry or exit?
Does the performance of KIPP students suggest they are on a path toward college?
Research Questions
3
![Page 4: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/4.jpg)
Quasi-experimental analysis of “all” 70 KIPP middle schools– De-identified data from states on student selection, test
scores, and attainment
Experimental (or lottery-based) analysis in 13 schools (1,000+ students)– School records– Parent and student surveys– Study-administered test
Validation of observational methods using experimental results
Evaluation Design
4
![Page 5: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/5.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
22 KIPP middle schools– 20 still operating– 2 “closed” by KIPP
Opened by SY2005-06– Allows for more than one cohort to be analyzed across
multiple years after KIPP entry
Located in jurisdictions with available data– Three consecutive years of longitudinally-linked student-
level data, typically through 2007-08– For both traditional public and charter schools– Between 3 and 8 cohorts per school
Pilot QED Sample Selection
5
![Page 6: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/6.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
Treatment group comprised students entering KIPP in 5th or 6th grade (n=5,993)
Defined three comparison groups:– District: all students within the district– Feeder: students in ES also attended by KIPP students at
baseline (and their MS)– Matched comparison: propensity-score matched comparison
group using baseline characteristics
Analyses– Student characteristics– Attrition and replacement– Impacts on achievement
Analytic Approach
6
![Page 7: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/7.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
Demographic Characteristics
7
Difference from KIPP is statistically significant at the 5% level
![Page 8: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/8.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
Baseline Achievement
8
Difference from KIPP is statistically significant at the 5% level
![Page 9: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/9.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
Attrition Rates, by Grade
9
Difference from KIPP is statistically significant at the 5% level
![Page 10: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/10.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
Average Baseline Achievement in Math, Stayers vs. Transfers
10
Difference from stayers is statistically significant at the 1% level
At both KIPP and district schools, early leavers are lower-achieving than students who stay
![Page 11: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/11.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
Incidence of Late Arrivals
11
KIPP schools replace more students than they lose in grade 6, but fewer in grades 7 and 8
District comparison schools replace more students than they lose in both grades 7 and 8
Replacement Ratio: Ratio of New Arrivals to Prior Attrition
Proportion of total
enrollmentGrade 5-6 Transition
Grade 6-7 Transition
Grade 7-8 Transition
KIPP 1.18 0.78 0.60 0.15
Feeder NA 1.32 1.35 0.14
![Page 12: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/12.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
Average Baseline Achievement in Math, On-Time vs. Late Arrivals
12
Difference from on-time arrivals is statistically significant at the 1% level
At KIPP schools, late arrivals are higher-achieving than on-time arrivals; at district schools, they are lower-achieving
![Page 13: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/13.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
Baseline reading and math scores, by grade
13
By 8th grade, KIPP classrooms comprise students higher-achieving at baseline
Baseline reading Baseline math
KIPP Feeder District KIPP Feeder District
5 -0.10 -0.09 0.03** -0.10 -0.09 0.04**
6 -0.03 -0.08** 0.05** -0.01 -0.07** 0.05**
7 0.06 -0.07** 0.05 0.07 -0.07** 0.05
8 0.07 -0.07** 0.05 0.13 -0.05** 0.06*
All -0.09 -0.09 0.03** -0.08 -0.09 0.03**
*Difference from KIPP is statistically significant at the 0.05 level**Difference from KIPP is statistically significant at the 0.01 level
![Page 14: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/14.jpg)
Model specification:
Retain students who leave KIPP in the treatment group
“Freeze” scores for grade repeaters (more common in KIPP: 11% vs. 2% of 5th graders)
Estimating Impacts
14
4
1nittitnitit TnXy
![Page 15: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/15.jpg)
Year 1 Year 2 Year 3 Year 4
Reading0.09**(0.011)
0.16**(0.013)
0.24**(0.018)
0.16**(0.027)
Math0.26**(0.011)
0.35**(0.014)
0.42**(0.020)
0.25**(0.027)
Number of KIPP schools
22 22 22 17
N (math) 11,242 8,019 5,439 2,576
N (reading) 11,218 8,041 5,447 2,570
Estimated Impact of Potential Exposure to KIPP
15
*Difference is statistically significant at the 0.05 level**Difference is statistically significant at the 0.01 level
![Page 16: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/16.jpg)
Percentage of KIPP Schools with Positive and Negative Impacts in Reading, by Years after KIPP Entry
16
![Page 17: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/17.jpg)
Percentage of KIPP Schools with Positive and Negative Impacts in Math, by Years after KIPP Entry
17
![Page 18: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/18.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
KIPP students are:– More likely to be a racial minority, eligible for FRPL– Less likely to be limited English proficiency or
special education – Lower-achieving at baseline than the district overall
but equivalent to other students at the same ES
Rates of attrition are similar in KIPP and district schools
Conclusions
18
![Page 19: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/19.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
Late arrivals present a mixed picture– Proportion of late arrivals relative to enrollment is similar
at KIPP and comparison schools– KIPP schools are less likely to replace in later grades– KIPP late arrivals are higher-achieving
Patterns of attrition and late arrivals mean later grades at KIPP comprise higher-performing students, but “peer effects” can explain no more than about a quarter of cumulative impacts
Estimated impacts on reading and math scores are positive, statistically significant, and of substantial magnitude
Conclusions
19
![Page 20: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/20.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
Please contact:– Christina Clark Tuttle
View reports online at:– Impacts:
http://www.mathematica-mpr.com/publications/PDFs/
education/KIPP_fnlrpt.pdf– Selection and Attrition:
http://www.mathematica-mpr.com/publications/PDFs/
education/KIPP_middle_schools_wp.pdf
For More Information
20
![Page 21: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/21.jpg)
Supplemental Slides
21
![Page 22: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/22.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
Location of KIPP Schools in Sample
22
KIPP state in study
Other KIPP state (as of 2005)
Recent KIPP state (as of 2012)
![Page 23: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/23.jpg)
Mathematica® is a registered trademark of Mathematica Policy Research.
KIPP Feeder District
Black 0.37 0.44** 0.45**
male 0.38 0.42 0.45**
Hispanic 0.24 0.29** 0.30**
male 0.27 0.30 0.30
FRPL 0.34 0.34 0.37**
Overall 0.34 0.34 0.36**
Average Cumulative Attrition by Subgroup
23
*Difference is statistically significant at the 0.05 level**Difference is statistically significant at the 0.01 level
![Page 24: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/24.jpg)
Size of Impacts in Reading after Three Years
24
KIPP Schools
![Page 25: KIPP: Effectiveness and Innovation in Publicly-Funded, Privately-Operated Schools October 4, 2012 Presentation to the APPAM/INVALSI Improving Education](https://reader036.vdocuments.net/reader036/viewer/2022062407/56649ceb5503460f949b6fe9/html5/thumbnails/25.jpg)
Size of Impacts in Math after Three Years
25
KIPP Schools