applying sgp to the star assessments daniel bolt dept of educational psychology university of...
TRANSCRIPT
![Page 1: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/1.jpg)
Applying SGP to the STAR Assessments
Daniel BoltDept of Educational Psychology
University of Wisconsin, Madison
![Page 2: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/2.jpg)
Some Unique Features of STAR Assessments
• Same CAT assessment is administered within and across years (possible to generate look-up tables for the calculation of SGP)
• Multiple STAR administrations to a student are possible throughout the year, and can occur at different times and/or frequencies for different students
![Page 3: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/3.jpg)
Why SGP may be Useful with STAR Assessments
• Different STAR scores may be associated with different amounts of measurement error (e.g., extremely low or high STAR scores are sometimes of questionable validity)
• There often exists more/less variability in growth observed across students at different initial STAR score levels
![Page 4: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/4.jpg)
Some Practical Issues Related to Administration of STAR Assessments
• How frequently and at what intervals should STAR be administered to get reliable estimates of end-of-year scores? Does the answer depend on the initial (fall) score of a student?
• Is there practical value in the use of SGP for answering this question (in contrast to alternative approaches, such as ordinary least squares---OLS--- regression methods)?
![Page 5: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/5.jpg)
Goals of the Present Study
• Examine SGP as a methodology for quantifying growth and for studying the precision of end-of-year growth predictions using STAR Assessments
• Compare SGP against competing methodologies (OLS regression) in terms of their reported precision of end-of-year predictions
![Page 6: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/6.jpg)
Grade Math Reading Early Literacy
1 99283 100000 968772 100000 100000 286573 100000 100000 57584 100000 100000 554
5 100000 1000006 100000 1000007 86975 1000008 72599 1000009 20835 61919
10 13810 3745511 9796 2627812 5155 12763
Sample Sizes (Students with Fall, Winter & Spring Assessments)
![Page 7: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/7.jpg)
Evaluating the Precision ofSpring Score Predictions
• How well do winter assessments improve our predictions of end-of-year outcomes?• Are the winter assessments more/less useful depending on the fall score obtained by the student?• In answering these questions, we find it useful to examine changes in the confidence intervals for spring scores defined by the SGP percentile cuts
![Page 8: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/8.jpg)
Academic Year
ST
AR
Ma
th S
cale
d S
core
*
95 %ile
80 %ile
50 %ile
20 %ile
5 %ile
Fall Spring
500
600
700
800
900
1000
Example of SGP Percentile Cuts, One Covariate
![Page 9: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/9.jpg)
Academic Year
ST
AR
Ma
th S
cale
d S
core
**
95 %ile
80 %ile
50 %ile
20 %ile
5 %ile
Fall Winter Spring
50
06
00
70
08
00
90
01
00
0
Example of SGP Percentile Cuts, Two Covariates
![Page 10: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/10.jpg)
Comparison of SGP & OLS Intervals, Math Grade 1
According to OLS Standard Error of Prediction
According to SGP %ile cuts
![Page 11: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/11.jpg)
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 1
0 200 600
150
200
250
300
350
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 2
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 3
0 400 800 1200
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 4
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 5
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 6
STAR Math, Grades 1-6
OLSSGP
![Page 12: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/12.jpg)
0 400 800 1200
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 7
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 8
200 600 1000
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 9
0 400 800
200
300
400
500
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 10
200 600 1200
200
300
400
500
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 11
200 600 1000
200
300
400
500
600
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 12
STAR Math, Grades 7-12
OLSSGP
![Page 13: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/13.jpg)
STAR Reading, Grades 1-6
OLS SGP
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 1
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 2
0 400 800
200
400
600
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 3
0 400 800
200
250
300
350
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 4
0 400 800 1400
250
300
350
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 5
0 400 800 1400
200
250
300
350
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 6
![Page 14: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/14.jpg)
0 400 800 1400
100
200
300
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 7
200 600 1200
0100
200
300
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 8
0 400 800 1400
-100
0100
300
500
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 9
0 400 800 1400
-200
0100
300
500
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 10
200 600 1200
0100
300
500
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 11
200 600 1200
100
200
300
400
500
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 12
STAR Reading, Grades 7-12
OLS
SGP
![Page 15: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/15.jpg)
300 500 700 900
200
250
300
Fall Score
Siz
e o
f 80%
Conf. Inte
rval Grade K
300 500 700 900
100
200
300
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval Grade 1
300 500 700 900
100
200
300
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval Grade 2
400 600 800
0100
200
300
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval Grade 3
400 600 800
50
150
250
Fall Score
Siz
e o
f 80%
Conf. Inte
rval Grade 4
STAR Early Literacy, Grades K-4
OLSSGP
![Page 16: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/16.jpg)
Adding Winter Scores as Covariates with SGP
• Using 80% interval width curves as a baseline, we can further examine how much the intervals are reduced when adding a winter assessment
• The decline in the 80% interval can be used as an indicator of the added precision provided by the winter assessment
![Page 17: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/17.jpg)
STAR Math Example
![Page 18: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/18.jpg)
STAR Reading Example
![Page 19: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/19.jpg)
STAR Early Literacy Example
![Page 20: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/20.jpg)
Some Examples of STAR Score Patterns
Subject Fall Winter Spring Change in 80% Interval
MATH 501 32 531 (+432)625 91 790 (+321)
3 833 852 (-208)2 846 771 (-209)
READING 392 891 589 (+41)376 915 551 (+40)984 1346 1333 (-148)928 1345 1255 (-256)
LITERACY 835 315 832 (+297)777 312 840 (+282)328 858 823 (-242)306 829 851 (-256)
![Page 21: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/21.jpg)
0 200 400 600 800 1000
02
04
06
08
01
00
Fall Score
De
clin
e in
Siz
e o
f 80
% C
on
f. In
terv
al
Grade 5
STAR Math Example
OLS
SGP
![Page 22: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/22.jpg)
0 400 800
20
40
60
80
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 1
0 200 60020
40
60
80
100
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 2
0 400 800
20
60
100
140
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 3
0 400 800 1200
20
40
60
80
100
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 4
0 400 800
20
40
60
80
100
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 5
0 400 800
20
40
60
80
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 6
STAR Math, Grade 1-6
OLS SGP
![Page 23: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/23.jpg)
0 400 800 1200
20
40
60
80
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 7
0 400 80020
40
60
80
100
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 8
200 600 1000
20
40
60
80
100
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 9
0 400 800
50
100
150
200
250
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 10
200 600 1200
50
100
150
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 11
200 600 1000
050
100
150
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 12
STAR Math, Grades 7-12
OLS SGP
![Page 24: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/24.jpg)
STAR Reading Example
OLS
SGP
![Page 25: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/25.jpg)
0 400 800
30
40
50
60
70
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 1
0 400 80030
40
50
60
70
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 2
0 400 800
20
40
60
80
100
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 3
0 400 800
25
30
35
40
45
50
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 4
0 400 800 1400
30
35
40
45
50
55
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 5
0 400 800 1400
35
40
45
50
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 6
STAR Reading, Grades 1-6
OLS SGP
![Page 26: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/26.jpg)
0 400 800 1400
36
38
40
42
44
46
48
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 7
200 600 120020
25
30
35
40
45
50
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 8
0 400 800 1400
10
20
30
40
50
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 9
0 400 800 1400
-20
020
40
60
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 10
200 600 1200
10
20
30
40
50
60
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 11
200 600 1200
010
20
30
40
50
60
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 12
STAR Reading, Grades 7-12
OLS SGP
![Page 27: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/27.jpg)
300 400 500 600 700 800 900
02
04
06
08
01
00
12
0
Fall Score
De
clin
e in
Siz
e o
f 8
0%
Co
nf. In
terv
alGrade 1
STAR Early Literacy Example
OLS
SGP
![Page 28: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/28.jpg)
300 500 700 900
20
30
40
50
60
70
80
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade K
300 500 700 900
020
40
60
80
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 1
300 500 700 900
020
40
60
80
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 2
400 600 800
020
40
60
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 3
400 600 800
-10
010
20
30
40
50
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 4
STAR Early Literacy, Grades K-4
OLS SGP
![Page 29: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/29.jpg)
Evaluating Predicted Spring Scores in Terms of State Proficiency Thresholds
• The accuracy of SGP and OLS predictions can also be compared against the thresholds associated with state-specific proficiency categories
• By assuming normally distributed residuals (with constant variance) for OLS, SGP and OLS can each be used to define a probability that the spring score will exceed a predefined threshold
![Page 30: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/30.jpg)
Academic Year
ST
AR
Ma
th S
ca
led
Sco
re
**
95 %ile
80 %ile
50 %ile
20 %ile
5 %ile
Fall Winter Spring
50
06
00
70
08
00
90
01
00
0
ProficiencyThreshold
Example of SGP Percentile Cuts against State-Defined Proficiency Category
![Page 31: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/31.jpg)
MATH READING
GradeProficiency
Cuta Percentile OLS R2,b SGP R2,bProficiency
Cuta Percentile OLS R2,b SGP R2,b
3 617 48 .40 .41 445 48 .53 .66
4 709 61 .40 .40 531 52 .53 .68
5 758 60 .42 .42 592 49 .52 .69
6 810 65 .46 .46 647 42 .52 .70
7 823 74 .48 .49 825 51 .55 .74
8 859 68 .47 .48 933 53 .55 .73
aEstimated STAR cutscores for the Tennessee Comprehensive Assessment Program (TCAP)bEfron’s Pseudo R2
Comparing SGP and OLS on Accuracy of Proficiency Predictions
![Page 32: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/32.jpg)
Extending SGP to Accommodate Multiple Intermediate Assessments
• How can additional intermediate assessments be used in SGP to further improve predicted spring scores?•Challenge: Handling varying-time point assessment schedules•One possible solution: Linear interpolation to fixed node locations
![Page 33: Applying SGP to the STAR Assessments Daniel Bolt Dept of Educational Psychology University of Wisconsin, Madison](https://reader036.vdocuments.net/reader036/viewer/2022070400/56649ef25503460f94c0345f/html5/thumbnails/33.jpg)
Conclusions and Future Directions
• Our SGP analyses suggest substantial variability in the precision of spring score predictions for STAR Math, Reading and Early Literacy depending on fall scores
• There is clear value in incorporating winter assessments into SGP---the largest value occurs for students with extreme fall scores in STAR Math, intermediate fall scores in STAR Reading
• More experimentation needed to determine how best to make use of multiple intermediate assessments within SGP