Beyond Triage: A Randomized Experiment in Sustained Pre-College AdvisingERIC P. BETTINGERBRENT J . EVANS

WITH ANTHONY ANTONIO, JESSE FOSTER, E ILEEN HORNG, RIE K I J IMA

UC DAVIS – 15 MARCH 2015

PRELIMINARY. PLEASE DO NOT CITE OR TWEET WITHOUT AUTHORS’ PERMISSION

Motivation Non-financial barriers related to information deter students from preparing for and enrolling in higher education, especially among low-income and minority students (Avery & Kane, 2004)

Gauntlets◦ Financial aid (ACSFA, 2005)◦ Admission process (Klasik, 2012)

High school counselors are overloaded and cannot meet the demand in their schools.

Huge investment by college access programs to provide information and guide students through the process

◦ TRIO (> $800 million)◦ Countless small local programs

Models vary substantially

Literature Information and guidance matter at very specific stages in the process

◦ H&R Block Study (Bettinger, Long, Oreopoulos, & Sanbonmatsu, 2012)◦ Information and guidance related to financial aid improves aid receipt and college enrollment◦ 2-8 percentage point effect

◦ Summer Melt Studies (Castleman, Page, & Schooley, 2014; Castleman, Arnold, & Wartman, 2012)◦ Outreach to students in the summer between high school completion and college enrollment improves likelihood of enrolling◦ 3-14 percentage point effect

Holistic advising◦ Mathematica’s Upward Bound Study (Myers et al., 2004)

◦ Primarily academic supports to promote college enrollment, but also provide information◦ RCT found no effects on college enrollment although some evidence of shift from two-year to four-year

◦ College Possible (Avery, 2013)◦ Targeted ACT prep, information, and admission support◦ High-touch intervention-360 hours◦ Large effects (15 pp) on shifting from two-year to four-year college but no overall enrollment effects

◦ Dartmouth Outreach (Carrell and Sacerdote, 2013)

Need for evidence of different models

Different models of providing information and guidance◦ Targeted versus school wide◦ In school versus out of school◦ Near-peer

By investigating wide range of models, we can start to tease apart mechanisms

Need well identified, causal estimates of efficacy

Randomized controlled trial evidence should be privileged where possible

Advise TX Part of the College Advising Corps model (active in 15 states 350-400 high schools)

Goal is to help low-income, first-generation college students attend and succeed in college

Recent college graduates serve as full-time near-peer advisers to entire student body at a high school

Full school college advising intervention-concentrate on seniors but some time spent developing college plans for underclassmen

Encourage attendance; provide information; assist with selection of colleges, college applications, and FAFSA completion

Program pilot in 2010-2011 in 15 schools with rapid expansion to over 120 schools in 2011-2012 school year

Experimental Design School level randomization

Schools invited to apply◦ > 35% FRL◦ < 70% graduates attend college within one year (average is 45% in 2009)◦ < 55% students undertaking a “distinguished” college-prep curriculum

Schools were ranked on these 3 criteria and a qualitative “fit” component

84 schools automatically selected among 237 who applied

Next 111 schools eligible for random assignment (experimental sample)

36 randomly selected for treatment assignment, others assigned control status (no program)

Blocked on region: individual lotteries were held within each region (23)

Data School level data from Texas Education Agency (TEA)

Student level data from Texas Higher Education Coordinating Board (THECB)

2 years of enrollment outcome data◦ From 2011-2012 (2012) and 2012-2013 (2013) high school graduating classes◦ All in-state public postsecondary institutions◦ Our results are lower bounds of effect of program if the program had any effect on out of state or

private enrollments◦ Outcomes: 2-year and 4-year enrollment and enrollment in Fall after high school graduation

Preliminary results from student surveys conducted in treatment/control schools

Treatment-Control BalancePanel A: School Level

Variable All TX High Schools All Experiment High Schools

All Treatment High Schools

Raw Difference T-C T-C Difference with Lottery Controls

Mean Stdev. Mean Stdev. Mean Stdev. Difference P-value Difference P-value

White0.400 0.311 0.225 0.227 0.214 0.223 -0.017 0.718 -0.015 0.665

Black0.123 0.177 0.171 0.171 0.217 0.208 0.068 0.049 0.085 0.006

Hispanic0.440 0.313 0.568 0.273 0.526 0.276 -0.062 0.266 -0.083 0.033

Asian0.017 0.045 0.021 0.029 0.029 0.037 0.011 0.060 0.013 0.017

Other race0.021 0.041 0.015 0.016 0.014 0.011 -0.001 0.783 0.000 0.948

Low-income0.542 0.258 0.635 0.179 0.636 .0169 0.001 0.980 0.006 0.843

Grad Rate*0.804 0.217 0.800 0.100 0.802 0.096 0.003 0.899 0.014 0.470

Total Students728 891 1683 838 1848 956 243 0.154 194 0.210

Total Seniors159 209 370 187 418 220 71.8 0.058 62.2 0.066

N1785 111 36

Descriptive Statistics & Balance Relative to all TX high schools, experimental schools

◦ Are much larger (1683 to 728)◦ Have a higher percentage of low-income status (64% to 54%)◦ Have a higher share of minority students (77.5% to 60%)◦ Have same graduation rate (80%)

Treatment schools relative to control schools◦ Have more Black and less Hispanic students (8 percentage points difference)◦ Have slightly more Asian students (1 percentage point difference)◦ Same graduation rates and college enrollment rates◦ We control for race in all models to account for this imbalance

Compliance 5 of 36 treatment schools declined to participate or eventually left the program

9 of 75 control schools eventually received treatment◦ Although we had a randomly determined waitlist, it was not consistently adhered to by program staff◦ E.g. Austin loved the program and offered to pay for all control and treatment schools to participate

after the first year.

Treatment on the treated estimates are approximately 33% higher than Intent to treat estimates Treatment Received Control Received Total

Treatment Assigned 31 5 36

Control Assigned 9 66 75

Total 40 72 111

Lottery controlled regression of treatment received on treatment assignment

0.745(0.072)

Main ImpactsPanel B: Separate Treatment Years

Outcomes Treatment Year Model 1 Model 2 Model 3Enrolled in Higher Education Fall after HS

2012 0.021 *(0.0102)

0.0146(0.0092)

0.0146(0.0090)

2013 0.0133(0.0105)

0.0089(0.0094)

0.0089(0.0092)

Enrolled in 2 Year Fall after HS

2012 0.019(0.0118)

0.0208 +(0.0112)

0.0208 +(0.0110)

2013 0.0061(0.0122)

0.0084(0.0114)

0.0084(0.0112)

Enrolled in 4 Year Fall after HS

2012 0.0053(0.0082)

-0.0029(0.0083)

-0.0029(0.0083)

2013 0.0077(0.0082)

0.0011(0.0076)

0.0011(0.0076)

Controls X XSE Clustered at SchoolxYear Level

X

N 77823 77823 77823

Intent to treat estimates 1.2 – 1.5 pp impacts on enrolling in fall after high school completion (significant at 10% level)

◦ Impacts are observed in first year of treatment◦ enrollment effects on two-years become more significant if just examine first year

If we pool the data, small and insignificant effects (~ 1 pp) on overall enrollment◦ Concentrated at 2-years◦ Point estimates at 4-years close to 0

All second year impacts attenuated

Impacts on Subgroups Subgroup

Outcomes Coefficient Female White Black Hispanic Asian Low-Income

Enrolled in Higher Education Fall after HS

Treatment Main Effect

0.0167 *(0.0076)

0.0155 *(0.0073)

0.0147 *(0.0071)

-0.0028(0.0095)

0.0124 +(0.0064)

-0.01(0.0090)

Interaction Effect

-0.0100(0.0081)

-0.0197(0.0140)

-0.0157(0.0139)

0.0257 +(0.0131)

-0.0183(0.0270)

0.0345*(0.0120)

Enrolled in 2 Year Fall after HS

Treatment Main Effect

0.0116(0.0080)

0.0165 +(0.0087)

0.0215 *(0.0086)

-0.0047(0.0103)

0.0155 +(0.0079)

0.0041(0.0097)

Interaction Effect

0.0059(0.0083)

-0.0103(0.0127)

-0.0370 *(0.0163)

0.0340 **(0.0128)

-0.0264(0.0263)

0.0166(0.0104)

Enrolled in 4 Year Fall after HS

Treatment Main Effect

0.0072(0.0061)

0.0016(0.0058)

-0.0045(0.0059)

0.0015(0.0085)

-0.001(0.0055)

-0.0135(0.0085)

Interaction Effect

-0.0161 **(0.0060)

-0.0127(0.0147)

0.0191(0.0133)

-0.0041(0.0092)

0.0045(0.0290)

0.0203 *(0.0097)

N 77823 77823 77823 77823 77823 77823

Subgroup Analysis Impact on Low-income students

◦ More likely to enroll overall, split between two-years and four-years (2-3 pp)

Impact on Hispanic students◦ More likely to enroll overall, mostly at two-years (2-3 pp)

Impact on Black students◦ Less likely to enroll in a two-year (3 pp)◦ Some evidence of substitution into four-year, but four-year enrollments not significant

Impact on Female students◦ Less likely than men to enroll in four-year and start at a four-year in fall after high school (1.6 pp)

Survey Sample Surveys conducted in spring 2014

◦ All treatment schools (36 + additional randomized group from 2013)◦ 42 control schools

Treatment/Control Balance◦ Less representation among Hispanic students (as in the administrative data)

Surveys conducted in spring 2014◦ All treatment schools (36 + additional randomized group from 2013)◦ 42 control schools

Preliminary Survey Results Key Differences

◦ Degree expectations improve ◦ Plans to work full- or part-time after college or to enter the military decrease◦ Similarly likely to participate in “soft” college prep activities (college visits, website, test prep)◦ More applications submitted and greater likelihood of receiving help◦ More likely to attempt AP exams or to retake college entrance exams◦ More likely to complete FAFSA (self-reported)◦ More likely to have submitted a deposit◦ Less likely to have talked to a high school counselor or teacher about admissions

Discussion Unsurprising that treatment effects are small at the school level

Advisers likely spend time with inframarginal students who are already going to college as opposed to targeting support at students who need it most

Degree of complementarity is potentially important part of the story

Cost-benefit analysis

Scalability

Future work◦ Will examine persistence outcomes to observe whether advisers have effect on “fit”