![Page 1: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/1.jpg)
Opportunities and Challenges in a Multi-Site Regression Discontinuity Design
Stephen W. RaudenbushUniversity of Chicago
Presentation at the MultiLevel Theory and Research Conference
The Pennsylvania State University University Park, PA, May 17, 2015
The research reported here was supported by a grant from the WT Grant Foundation entitled “Learning from Variation In Program Effects: Methods, Tools, and Insights from Multi-site Trials.”
![Page 2: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/2.jpg)
Outline
Counter-factual account of causation
The “drug-trial paradigm” for causal inference
An alternative paradigm for social interventionsHeterogeneous agentsSocial interactions among participants
Curriular reform in chicago
Conventional RDDIncorporating Agents and Social InteractionsIdentification: School-specific IV
Conclusions
![Page 3: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/3.jpg)
Counter-factual account of Causation
In statistics (Neyman, Rubin, Rosenbaum)
In economics (Haavelmo, Roy, Heckman)
![Page 4: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/4.jpg)
Drug trial paradigm for causation
Y(1): Outcome if the patient receives Z = 1
(the “new drug”)
Y(0): Outcome if the patient receives Z = 0
(the “standard treatment”)
Y(1) – Y(0): Patient-specific causal effect
E (Y(1) – Y(0)) = : Average causal effect
![Page 5: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/5.jpg)
Stable Unit Treatment Value Assumption (Rubin, 1986)
• Each patient has two potential outcomes• Implies
– Only one “version” of each treatment– No “interference between units”
• Implies the doctor and the other patients have no effect on the potential outcomes
![Page 6: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/6.jpg)
Formally…
)();,...,,( 11211 zYdzzzY n
![Page 7: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/7.jpg)
Failure of SUTVA in Education
• Teachers enact instruction in classrooms– Multiple “versions of the treatment”
• Treatment assignment of one’s peers affects one’s own potential outcomes– EG Grade Retention
– Hong and Raudenbush, Educational Evaluation and Policy Analysis, 2005
– Hong and Raudenbush, Journal of the American Statistical Association, 2006
![Page 8: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/8.jpg)
Group-Randomized Trials
Potential outcome
Thus, each child has only two potential outcomes – if we have “intact classrooms”– if we have “no interference between classrooms”
controltoassignedisjiftY
treatmenttoassignedisjiftY
tzzzY
jj
jj
jnjjjj
);0,...,0,0(
);1,...,1,1(
);,...,,(
1
1
211
![Page 9: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/9.jpg)
Limitations of cluster randomized trial
Mechanisms operate within clusters
* Example: 4Rs
teachers vary in response
classroom interactions spill over
We may have interference between clusters
* Example: community policing
![Page 10: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/10.jpg)
Alternative Paradigm for social interventions
Treatment setting (Hong, 2004):
A unique local environment for each treatment composed of * a set of agents who may implement an intervention and* a set of participants who may receive it
Each participant possesses a single potential outcome within each possible treatment setting
Causal effects are comparisons between these potential outcomes
);,...,,( 21 jnjjij tzzzYj
![Page 11: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/11.jpg)
Example: Community Policing (Verbitsky-Shavitz and Raudenbush, 2012)
• Let Zj=1 if Neighborhood j gets community policing
• Let Zj=0 if not
• Under SUTVA
)0()1( jjj YY
![Page 12: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/12.jpg)
“All or none”
)0,0()1,1(
jjj YY
1
1
1
1
10
0
0
0
0
![Page 13: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/13.jpg)
Do it only in high-crime areas: effect on low-crime areas
)0,0(),0( ''''
jjjj YZY
1, HC
1, HC
0, LC
0, LC
1, HC
0, HC
0, HC
0, LC
0, HC
0, LC
![Page 14: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/14.jpg)
Results
Having community policing was especially good if your surrounding neighbors had it
Not having community policing was especially bad if your neighbors had it
*** So targetting only high crime areas may fail***
![Page 15: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/15.jpg)
Application: Double-dose AlgebraNomi and Raudenbush (2015)
Requires 9th-graders to take Double-dose Algebra if they scored below 50 percentile on 8th-grade math test
12,000 students in 60 Chicago high schools
![Page 16: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/16.jpg)
Double-dose Algebra enrollment rate by math percentile scores (city wide)
Enro
llmen
t Rat
es
ITBS percentile scores
![Page 17: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/17.jpg)
Conventional Mediation Model (T, M,Y model)
Cut off (T)Double-Dose Algebra (M)
Algebra Learning (Y)
• γ = average effect of cutoff on taking Double Dose• δ=average effect of taking Double Dose on Y for compliers (“CACE” effect)• Assume no direct effect of T on Y (exclusion restriction)• β= γδ (“ITT” effect)• So δ= β/ γ
Nomi, T., & Allensworth, E. (2009)
γ δ
![Page 18: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/18.jpg)
Conventional Model is Founded on SUTVA
CACE
BYY
E
MM
1)1)(Pr(|E(
Y"oneffectITT"E(B)
Yoncutofimpact)0()1(
Y[M(0)]-Y[M(1)]
M"oneffectITT")(
DDtakingoncutofimpact)0()1(
![Page 19: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/19.jpg)
Results of Conventional Analysis
• Large average impact of Cut on taking DD (ITT effect on M)
• Modest average impact of Cut on Y (ITT effect on Y)
• Modest CACE (Average Impact of M on compliers)
![Page 20: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/20.jpg)
ITT effect on Y
-1.5
-1-.
50
.5A
lgeb
ra S
core
s
-50 -40 -30 -20 -10 0 10 20 30 40
Math Percentile Scores
observed values
fitted values (Lowess)
![Page 21: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/21.jpg)
But the policy changed classroom composition!!
![Page 22: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/22.jpg)
Classroom average skill levels by math percentile scores
Pre-policy (2001-02 and 2002-03 cohorts)
Post-policy (2003-04 and 2004-05 cohorts)
![Page 23: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/23.jpg)
Implementation varied across schools in---
• Complying with the policy • Inducing classroom segregation
![Page 24: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/24.jpg)
Exclusion Restriction RevisedT-M-C-Y model
Cut off (T) Double-Dose Algebra (M)
Algebra score (Y)
Classroom Peer skill (C)
1
2
21
![Page 25: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/25.jpg)
Identification Problem
We have one equation, two unknowns:
Strategy is school-specific
21
21
/
CACE
ITT
)(
21
21
jjj
jjj
jjjjj
u
![Page 26: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/26.jpg)
A simple two-level model
At level 1
At level 2
jijijijjijjjij TXfTTYY .),().(.
jjjj u 211
![Page 27: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/27.jpg)
-.6
-.4
-.2
0.2
Eff
ects
of
the
cut-
scor
e on
cla
ssro
om p
eer
abil
ity
0 .2 .4 .6 .8 1
Effect of the cut-score on double-dose enrollment
![Page 28: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/28.jpg)
Derivation of assumptions using potential outcomes
CACECACECACE
EBECACE
CMYCMYB
CC
MM
21
21
21
)1|()1|(
)]0(),0([)]1(),1([
)0()1(
)0()1(
![Page 29: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/29.jpg)
Parameter Estimate SE
ITT impact on M 0.72 0.03
ITT impact on C -0.28 0.03
ITT impact on Y 0.07 0.03
CACE of M 0.20 0.05
CACE of C 0.22 0.09
![Page 30: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/30.jpg)
-.6
-.4
-.2
0.2
Eff
ects
of
the
cut-
scor
e on
cla
ssro
om p
eer
abil
ity
0 .2 .4 .6 .8 1
Effect of the cut-score on double-dose enrollment
![Page 31: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/31.jpg)
5. Conclusions on DDThe reform• Increased instructional time• Changed class composition
Median skill kids• Gained a lot if not tracked into low-skill classes• Gained little if they were
![Page 32: Opportunities and Challenges in a Multi-Site Regression Discontinuity Design Stephen W. Raudenbush University of Chicago Presentation at the MultiLevel](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649de95503460f94ae401b/html5/thumbnails/32.jpg)
Conclusions on Causal Inference
Conventional causal paradigm:* a single potential outcome per participant under each treatment
Alternative paradigm:* a single potential outcome per participant in each treatment setting
RDD as a means-tested program
Potentially large policy implications of causal paradigm