1 on money and motivation: a quasi-experimental analysis of financial incentives for college...
Post on 21-Dec-2015
214 views
TRANSCRIPT
1
On Money and Motivation:A Quasi-Experimental Analysis
of Financial Incentives for College Achievement
Judith Scott-ClaytonTeachers College, Columbia University
Higher Education in Michigan: Looking Back and Looking Ahead University of Michigan, Ann Arbor
December 10, 2009
2
Background We now lose as many students between college entry
and completion, as we lose between HS graduation and college entry
Financial incentives for achievement are increasingly popular policy intervention 13+ states have programs similar to WV PROMISE scholarship Programs provide both money and motivation We know that aid improves entry, less known on persistence
Learning about mechanisms is critical for optimal policy design But we have only a limited understanding of if/how these
programs work Administrative data like WVHEPC database offers the
opportunity to address these knowledge gaps with rigorous research
3
Research Overview Do large financial incentives for student achievement improve
college students’ outcomes, and if so, what are the mechanisms? Alleviation of financial constraints? Incentives for increased student effort?
I examine these questions in the context of a $40 million dollar state-funded merit scholarship program in West Virginia (the PROMISE scholarship)* Program covers tuition/fees at WV college for eligible students, as long
as they maintain a minimum GPA & courseload I exploit discontinuities in the eligibility formula and timing of
implementation to identify causal effects, use detailed data to tease out mechanisms
Summary of results: Significant and robust positive effects on range of outcomes Incentives are key to results; scholarship with no strings attached
unlikely to produce same pattern
*My research follows the first two cohorts of recipients, entering in the fall of 2002 and 2003. The cost of the program has risen, and eligibility rules have changed since then.
4
West Virginia’s PROMISE Scholarship WV began Providing Real Opportunities to Maximize In-
state Student Excellence (PROMISE) in 2002 [context] To qualify initially: must be a WV high school graduate,
with 3.0+ high school GPA overall and within “core” courses, and score a 21+ on the ACT (or 1000+ SAT)* No income limitations Must enroll full-time in a WV public or private 2/4 year college
Covers tuition and fees at any WV public univ* To renew: must complete 30+ credits per year and
maintain 3.0 GPA (2.75 in 1st year) 30-credit requirement corresponds to “on-time” progression Of 13 similar state programs, only WV & SC have this req.
5
The WVHEPC Data Comprehensive, administrative data from WV state
higher ed commission Covers all 1st time freshmen entering WV public 2/4 yr colleges
between 2000-2003 (~13,000 students per entry cohort). Includes hsgpa, ACT/SAT scores, basic demographics (not
income); enrollment, transcript, financial aid and graduation records for five years following entry
Linked with admin employment records from WV state employment office (but still rather early for post-college emp.)
How did this data partnership come to pass? Persistence on both sides, plus luck Support from the top, time/expertise from data coordinator WV contacts provided more than just data: also provided critical
contextual information, and on-the-ground perspective
6
Research Methods Simple comparison of recipients and non-recipients
does not provide causal estimates of program’s impact Instead, two strategies based on discontinuities
Initial eligibility rules set up a regression discontinuity (RD) analysis for 2002, 2003 entry cohorts [figure]
Availability of data for pre-program entry cohorts sets up a cohort analysis (before-after) [figure]
Both strategies made possible by rich admin data
Concern: differential selection into the “treated” group An explicit goal of the program was to attract “best & brightest” But additional data sources suggest that switchers from out of
state are too small in number to drive overall impacts
7
Results RD graphs: key thresholds and graduation Cohort graphs: key thresholds and
graduation RD table: main results and falsification
check (RD before PROMISE) Cohort table: main results Mechanisms: [credit PDFs] [credit CDFs]
[GPA CDFs] [table]
8
Concluding Remarks Financial incentives can have significant positive effects
on end-of-college outcomes incl. graduation & TTD Scholarship of same value with no strings (or different
strings) attached is not likely to produce same effects Courseload requirements appear particularly important Does not imply that money is unimportant Knowing something about mechanisms makes results useful
beyond just WV context
Social cost-benefit analysis is positive, but upfront cost to state is high – future research needed to examine whether smaller scholarships can produce similar effects
9
The End
10
West Virginia Context [BACK]
WV public system accounts for about 85% of in-state first-time entrants Total of about 20 public, 20 private 2/4-yr colleges WVU/Marshall account for 43% of public entrants (66% of public PROMISE
recips) WVU is only ranked school (“Tier 3”)
Tuition is lower than national average WVU is most expensive public (currently $5,100/yr, compared to national avg. of
$6,200 for 4-yr public) In 2002 average public tuition in WV was ~$3,000
PROMISE represented major increase in financial aid Prior to PROMISE, average young WV entrant received $1,350 in Pell+HEGP
(mostly Pell), with 46% receiving one or both After PROMISE average entrant received $1,200 in PROMISE, with 38%
receiving a grant PROMISE receipt generally does not affect Pell/HEGP eligibility [back]
11
Figure 1a - RD EstimationCompares Outcomes Above/Below Cutoff
0
10
20
30
40
50
60
70
11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
ACT Score (or equivalent)
Hyp
oth
etic
al C
oll
ege
Ou
tco
me
12
Figure 1b - Cohort AnalysisCompares Outcomes Before/After PROMISE
0
10
20
30
40
50
60
70
11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
ACT Score (or equivalent)
Hyp
oth
etic
al C
oll
ege
Ou
tco
me
Before PROMISE After
13
.1
.2
.3
.4
.5
.6
16 17 18 19 20 21 22 23 24 25ACT (or equivalent)
Earned 120+ credits within 4 yrs
.3
.4
.5
.6
.7
16 17 18 19 20 21 22 23 24 25ACT (or equivalent)
Had 3.0+ cumulative GPA after 4 yrs
0
.1
.2
.3
.4
16 17 18 19 20 21 22 23 24 25ACT (or equivalent)
Earned a BA Within 4 Years
.2
.3
.4
.5
.6
16 17 18 19 20 21 22 23 24 25ACT (or equivalent)
Earned a BA Within 5 Years
Selected Outcomes by ACT Score (2)
14
Table 2bRD Robustness Checks
(1) Baseline: (6) FalsificationFuzzy RD, LLR Exercise: Sharp RD16<=ACT<=25 Before 2002
Outcome B (SE) B (SE)
Received PROMISE 1.000 *** (0.000) n/aValue of PROMISE in Year 1 $3,012 *** ($21) n/aTotal PROMISE received (over 4 years) $8,338 *** ($180) n/a
GPA, end of year 1 0.156 *** (0.051) -0.033 0 (0.036)Credits earned, end of year 1 2.095 *** (0.461) -0.359 0 (0.330)
Number of semesters enrolled (over 4 years) 0.037 0 (0.130) -0.130 0 (0.094)Total credits earned (over 4 years) 4.644 * (2.519) -1.027 0 (1.799)Cumulative GPA (over 4 years) [a] 0.099 ** (0.045) -0.008 0 (0.032)Typical weekly school-year earnings [b] -$2.12 0 ($7.04) -$6.06 0 ($4.97)
Earned 120 credits by end of Year 4 0.095 *** (0.026) -0.003 0 (0.019)Had 3.0+ cumulative GPA at end of Year 4 0.090 *** (0.027) 0.023 0 (0.020)
Earned BA within 4 Years 0.094 *** (0.022) 0.011 0 (0.015)Earned BA within 5 Years 0.045 * (0.028) 0.000 0 (0.020)
Sample size 8,567 7,826
15
Figure 8. Selected Outcomes By Entry Cohort (2)For Entrants Who Qualified for PROMISE
Completed 120+ Credits by Yr. 4
0.30
0.35
0.40
0.45
0.50
0.55
0.60
0.65
0.70
2000 2001 2002 2003
Entry Cohort (Fall)
Had 3.0+ GPA at End of Yr. 4
0.30
0.35
0.40
0.45
0.50
0.55
0.60
0.65
0.70
2000 2001 2002 2003
Entry Cohort (Fall)
Earned a BA within 4 Years
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
2000 2001 2002 2003
Entry Cohort (Fall)
Earned a BA within 5 Years
0.30
0.35
0.40
0.45
0.50
0.55
0.60
0.65
0.70
2000 2001 2002 2003
Entry Cohort (Fall)
16
NOTES: All regressions include indicator controls for gender, race/ethnicity, age, and a quadratic function of high school GPA and indictors for each ACT score.
Table 3Cohort Analysis Estimates of the Effect of PROMISE
(1) (2) Before-After with ControlsPre- After- (3) OLS (4) IV
Outcome Mean Before B (SE) B (SE)
Received PROMISE 0.000 0.852 0.859 *** (0.009) 1.000 *** (0.000)Value of PROMISE in Year 1 $0 $2,621 $2,643 *** ($131) $3,077 *** ($120)Total PROMISE received (over 4 years) $0 $8,598 $8,677 *** ($310) $10,101 *** ($255)
GPA, end of year 1 2.849 0.062 0.066 *** (0.005) 0.077 *** (0.006)Credits earned, end of year 1 26.239 1.529 1.572 *** (0.085) 1.830 *** (0.118)
Number of semesters enrolled (over 4 years) 6.731 0.118 0.126 ** (0.037) 0.146 ** (0.045)Total credits earned (over 4 years) 97.225 4.668 4.967 ** (0.940) 5.782 ** (1.136)Cumulative GPA (over 4 years) [a] 2.982 0.025 0.033 0 (0.015) 0.039 0 (0.018)Typical weekly school-year earnings [b] $85.51 -$7.40 -$8.20 ** ($1.76) -$9.55 ** ($2.10)
Earned 120 credits by end of Year 4 0.431 0.091 0.095 *** (0.015) 0.111 *** (0.018)Had 3.0+ cumulative GPA at end of Year 4 0.590 0.025 0.030 * (0.010) 0.035 * (0.012)
Earned BA within 4 Years 0.267 0.054 0.058 *** (0.004) 0.067 *** (0.005)Earned BA within 5 Years 0.509 0.029 0.031 * (0.010) 0.037 * (0.012)
Sample size 12,911 12,911
17
0.00
0.05
0.10
0.15
0.20
0.25
10 15 20 25 30 35 40Credits Completed in Fresh. Yr.
2000 2001 2002 2003
0.00
0.05
0.10
0.15
0.20
0.25
10 15 20 25 30 35 40Credits Completed in Soph. Yr.
2000 2001 2002 2003
0.00
0.05
0.10
0.15
0.20
0.25
10 15 20 25 30 35 40Credits Completed in Jr. Yr.
2000 2001 2002 2003
0.00
0.05
0.10
0.15
0.20
0.25
10 15 20 25 30 35 40Credits Completed in Sr. Yr.
2000 2001 2002 2003
SOURCE: Author's calculations using WVHEPC administrative data on 12,911 first-time freshmen age 19 and younger,
who met PROMISE eligibility requirements.
Dist. of Credits Completed Each Year, By Cohort
18
0.00
0.20
0.40
0.60
0.80
1.00
10 15 20 25 30 35 40Credits Completed in Fresh. Yr.
2000 2001 2002 2003
0.00
0.20
0.40
0.60
0.80
1.00
10 15 20 25 30 35 40Credits Completed in Soph. Yr.
2000 2001 2002 2003
0.00
0.20
0.40
0.60
0.80
1.00
10 15 20 25 30 35 40Credits Completed in Jr. Yr.
2000 2001 2002 2003
0.00
0.20
0.40
0.60
0.80
1.00
10 15 20 25 30 35 40Credits Completed in Sr. Yr.
2000 2001 2002 2003
SOURCE: Author's calculations using WVHEPC administrative data on 12,911 first-time freshmen age 19 and younger,
who met PROMISE eligibility requirements.
CDFs of Credits Completed Each Year, By Cohort
19
0.20
0.40
0.60
0.80
1.002
.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
3.0
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
Cumulative GPA, End of Fresh. Yr.
2000 2001 2002 2003
0.20
0.40
0.60
0.80
1.00
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
3.0
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
Cumulative GPA, End of Soph. Yr.
2000 2001 2002 2003
0.20
0.40
0.60
0.80
1.00
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
3.0
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
Cumulative GPA, End of Jr. Yr.
2000 2001 2002 2003
0.20
0.40
0.60
0.80
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
3.0
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
Annual GPA, Senior Yr.
2000 2001 2002 2003
SOURCE: Author's calculations using WVHEPC administrative data on 12,911 first-time freshmen age 19 and younger,
who met PROMISE eligibility requirements.
CDFs of GPA Each Year, By Cohort
20
Table 6
OLS and IV Cohort Analysis Estimates of the Effect of West Virginia's PROMISE Scholarshipon Selected Outcomes (Using "After" as Instrument for PROMISE Receipt in Each Year)
Pre- Basic OLS IV: Freshman Year IV: Soph. Year IV: Junior Year IV: Senior YearOutcome Mean B (SE) B (SE) B (SE) B (SE) B (SE)
Received PROMISE: Year 1 0.000 0.859 *** (0.009) 1.000 *** (0.000)Year 2 0.000 0.652 *** (0.005) 1.000 *** (0.000)Year 3 0.000 0.520 *** (0.001) 1.000 *** (0.000)Year 4 0.000 0.440 *** (0.009) 1.000 *** (0.000)
Completed at least 30 credits in: Year 1 0.409 0.210 *** (0.003) 0.245 *** (0.004)Year 2 0.383 0.139 *** (0.006) 0.213 *** (0.010)Year 3 0.360 0.109 *** (0.007) 0.209 *** (0.013)Year 4 0.296 0.035 * (0.012) 0.080 * (0.027)
Cumulative 2.75+ GPA, end of Year 1 (a) 0.678 0.051 *** (0.007) 0.059 *** (0.008)Cumulative 3.0+ GPA, end of Year 2 0.572 0.050 *** (0.006) 0.077 *** (0.010)Cumulative 3.0+ GPA, end of Year 3 0.578 0.044 ** (0.008) 0.084 ** (0.015)Cumulative 3.0+ GPA, end of Year 4 0.590 0.030 * (0.010) 0.067 * (0.023)
Annual 2.75+ GPA, Year 1 (b) 0.633 0.044 *** (0.004) 0.052 *** (0.005)Annual 3.0+ GPA, Year 2 0.537 0.049 * (0.018) 0.075 * (0.028)Annual 3.0+ GPA, Year 3 0.562 0.032 ** (0.008) 0.061 ** (0.016)Annual 3.0+ GPA, Year 4 (a) 0.599 -0.008 0 (0.009) -0.018 0 (0.021)
Sample size 12,911 12,911 12,911 12,911 12,911
SOURCE: Author's calculations using WVHEPC administrative data on first-time degree-seeking freshmen aged 19 and younger, enrolling in the fall semester of school years 2000-01 through 2003-04. Unless otherwise noted, the sample is restricted to West Virginia residents who met the high school GPA (3.0+) and ACT/SAT (21/1000+) score requirements for PROMISE eligibility.NOTES: Robust standard errors, clustered by cohort, are in parentheses. All regressions use the basic before-after specification and include controls for gender, race/ethnicity, age, high school GPA and ACT score (or equivalent). (a) I present both cumulative and annual GPAs. PROMISE renewal is contingent upon cumulative GPAs in the first three years; but because the cumulative GPA in Year 4 is mostly determined by behavior prior to Year 4, it does not reveal behavioral changes as clearly as the annual GPA measure. (b) In Year 1, the cumulative and annual GPA measures are not identical because of slight differences in how certain courses (such as transfer and/or remedial courses) are counted. For students not enrolled in a given year, annual GPA is first imputed as the semester GPA if the student enrolled for at least one semester, otherwise it is imputed as the cumulative GPA as of last enrollment (71 percent of the sample enrolled for at least part of year 4; 68 percent enrolled full-time for the full year; 62 percent enrolled full-time for all four years).
Table 9