effect sizes for meta-analysis of single-subject designs s. natasha beretvas university of texas at...
TRANSCRIPT
![Page 1: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/1.jpg)
Effect Sizes for Meta-analysis of Single-Subject Designs
S. Natasha BeretvasUniversity of Texas at Austin
![Page 2: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/2.jpg)
Beretvas grant
Three studies: 1.a) Summarize practices used for meta-
analyzing SSD results 1.b) Summarize methods used to calculate
effect sizes (ESs) for SSD results 2. Simulation study evaluating
performance of selection of ESs 3. Conduct actual meta-analysis of school-
based interventions for children with autism spectrum disorders.
![Page 3: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/3.jpg)
Outline
Large-n designs’ data Large-n Effect Sizes Single-n designs’ data Single-n Effect Sizes (sample)
Problems 4-parameter model (AB designs)
Explanation Continuing research
![Page 4: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/4.jpg)
Large-n Studies’ Data
Most simply: consists of a randomly selected and assigned sample of participants in each of the Treatment and Control groups.
Each participant is measured once on the outcome.
Each participant provides an independently observed data point.
The standard deviation provides an estimate of the variability of these independent data points.
![Page 5: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/5.jpg)
Large-n Effect Sizes
Provides a practical measure of the size and direction of a treatment’s effect.
In large-n studies, the standardized mean difference is most typically used: Represents how different the two groups’
means are on the outcome of interest. The “standardized” part originates in the
difference being measured in standard deviations:
s
MMd CT
![Page 6: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/6.jpg)
Single-n Studies’ Data
0
1
2
3
4
5
6
7
8
9
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Time
Out
com
e
![Page 7: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/7.jpg)
Single-n Studies’ Data
Most simply: repeated measures on an individual over time in two phases (time series data):
Baseline: phase A = “control”
Treatment: phase B = “treatment”
Score at time point t is related to score at time (t – 1): not independent.
![Page 8: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/8.jpg)
Single-n Studies’ DataVisual Analysis:
Plots are evaluated for the presence of a treatment effect by simultaneously considering the following :
Sustainable level and/or trend changes
Baseline trends in expected direction
Overlapping data between phases
Variability changes within and across phases.
![Page 9: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/9.jpg)
Single-n Effect Sizes
Seems reasonable that a standardized difference between scores in phase A and B could be used as an effect size (ES):
It seems feasible that this effect size would be on the same metric as for large-n designs?!
No!!
s
MMd AB
![Page 10: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/10.jpg)
Problems with d for single-n designs
The standard deviation, s, for single-n designs describes different variability than for large-n designs.
If these were not problems, then it would also only make sense to use d when there is no trend in the data.
![Page 11: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/11.jpg)
Trend in A and B phases, tx effect
0
5
10
15
20
25
30
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Time
Ou
tco
me
A single number cannot summarize changes in level and slope
![Page 12: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/12.jpg)
Trend in B phase, tx effect
0
1
2
3
4
5
6
7
8
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Time
Ou
tco
me
![Page 13: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/13.jpg)
Trend in A and B phases, no tx effect
0
1
2
3
4
5
6
7
8
9
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Time
Ou
tco
me
What would d indicate about this pattern?
![Page 14: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/14.jpg)
Alternative single-n ESs
Percent Non-overlapping data (PND) is one of the most frequently used ES descriptors.
If treatment’s effect is anticipated to increase outcome then: Horizontal line drawn through highest point in
phase A through points in phase B
PND = % of phase B points above line
The higher the PND, the stronger the support for a treatment’s effect.
![Page 15: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/15.jpg)
PND
0
10
20
30
40
50
60
70
80
90
1 2 3 4 5 6 7 8 9 10 11
PND = 6/6 = 100%
Baseline Treatment
![Page 16: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/16.jpg)
PND
0
10
20
30
40
50
60
70
80
90
0 5 10 15 20
Baseline Treatment
PND = 11/13 = 84.6%
![Page 17: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/17.jpg)
PND
PND is simple to calculate and interpret and takes into consideration:
Baseline variability
Slope changes, but
![Page 18: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/18.jpg)
PND
0
1
2
3
4
5
6
7
8
9
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Time
Ou
tco
me
What would PND indicate about this pattern?
![Page 19: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/19.jpg)
Alternative single-n ESs
Assuming linear trends, it seems that two ESs should be used to describe change in level and trend.
Huitema and McKean (2000) suggested using a four-parameter regression model (extension of piecewise reg’n suggested by Gorman and Allison, 1996).
Appropriate parameterization of this model provides two coefficients that can be used to describe change in intercept and in slope from phase to phase:
![Page 20: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/20.jpg)
4-parameter model
The model:
where
Yt = outcome score at time t
Tt = time point
D = phase (A or B)
n1 = # time points in phase A
tttttt eDnTDTY )]1([ 13210
![Page 21: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/21.jpg)
4-parameter model – interpretation
Coefficients represent the following:
0 = baseline intercept (i.e. Y at time = 0)
1 = baseline linear trend (slope over time)
2 = difference in intercept predicted from treatment phase data from that predicted for time = n1+1 from baseline phase data
3 = difference in slope
Thus 2 and 3 provide estimates of a treatment’s effect on level and on slope, respectively.
![Page 22: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/22.jpg)
4-parameter model - interpretation
0
5
10
15
20
25
30
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Time
Ou
tco
me
![Page 23: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/23.jpg)
4-parameter model - interpretation
0
5
10
15
20
25
30
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Time
Ou
tco
me
![Page 24: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/24.jpg)
4-parameter model - interpretation
0
5
10
15
20
25
30
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Time
Ou
tco
me
![Page 25: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/25.jpg)
4-parameter model - interpretation
0
5
10
15
20
25
30
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Time
Ou
tco
me
2
![Page 26: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/26.jpg)
4-parameter model
Model can be estimated using OLS or autoregression (to correct SEs if residuals are autocorrelated).
The four-parameter model can be expanded for ABAB designs.
Multiple baseline designs can be thought of as multiple dependent, within-study AB designs. 2 and 3 can be calculated for each individual
and then summarized across individuals for a study.
![Page 27: Effect Sizes for Meta-analysis of Single-Subject Designs S. Natasha Beretvas University of Texas at Austin](https://reader036.vdocuments.net/reader036/viewer/2022062423/56649ea15503460f94ba412e/html5/thumbnails/27.jpg)
4-parameter model
How does estimation of these coefficients function for differing true coefficient values?
How does an omnibus test work? F-ratio testing addition of both predictors (with
coefficients 2 and 3)
How to standardize regression coefficients for meta-analytic synthesis? No procedure yet established for regular
regression. Comparison with long list of other SSD ESs.