1 measure up! benchmark assessment quality assurance process rcan september 10, 2010
TRANSCRIPT
![Page 1: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/1.jpg)
1
Measure Up!Benchmark Assessment
Quality Assurance Process
RCANSeptember 10, 2010
![Page 2: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/2.jpg)
2
Measure Up! ObjectiveTo monitor and improve Benchmark Assessments in order to attain the most accurate possible measurement of student achievement with respect to California Content Standards Tests.
![Page 3: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/3.jpg)
Measure Up! Components• Content—Structure & Course Guides• Predictability
– Correlation of Benchmark Exams to CST scores– Association of Benchmark Exams with CST Performance Levels
• Item Analysis– Difficulty– Discrimination– Representative of CST items
Evolving
![Page 4: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/4.jpg)
4
PSUSD Benchmark Structure
CST
~153 ID
60 to 75 Items
Blueprint Aligned
Benchmark#1
~45 ID
~20-35 Items
1st 45 ID Paced
Standards
Partial Match to
CST
Benchmark#2
~90 ID
~20-35 Items
2nd 45 ID Paced
Standards
Partial Match to
CST
Benchmark#3
~135 ID
~20-35 Items
3rd 45 ID Paced
Standards
Partial Match to
CST
![Page 5: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/5.jpg)
5
2008-2009
Algebra I(8th Grade)Aggregation of 3 Benchmarks
2008-209
Algebra I(8th Grade)
CST
Predictability
![Page 6: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/6.jpg)
6
Prof = 38
38
Algebra I BM 08-09 (8th Grade)
Alge
bra
I CST
200
9 (8
th G
rade
)
![Page 7: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/7.jpg)
7
N = 23
N = 119 N = 125
N = 51
N = 5
![Page 8: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/8.jpg)
8
Comparison of 2008-2009 Benchmark Exams with 2009 CST Performance Bands for Grade 8 Algebra I
using STAR Performance Band Cut Points
1 5 38
189
3
32 56
11
14
50
54
3
5
32
18
1
1-Far BelowBasic
2-Below Basic 3-Basic 4-Proficient 5-Advanced
Benchmark Exams Performance Levels
CS
T P
erfo
rman
ce L
evel
s
5-Advanced
4-Proficient
3-Basic
2-Below Basic
1-Far Below Basic
N = 9
N = 70
N = 123
N = 100
N = 21
![Page 9: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/9.jpg)
9
2008-2009
ELA 10th GradeAggregation of 3 Benchmarks
2008-209
ELA 10th Grade
CST
Predictability
![Page 10: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/10.jpg)
10ELA 10th Gd BM 08-09
ELA
10th
Gd
CST
2009
Prof = 53
53
![Page 11: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/11.jpg)
11
N = 26
N = 151
N = 326
N = 249
N = 36
![Page 12: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/12.jpg)
12
Comparison of 2008-2009 Benchmark Exams with2009 CST Performance Bands for Grade 10 English Language Arts
using STAR Performance Band Cut Points
18 28 21
5143
7
50
162
30
7
107
85
31
76
35
41-Far Below
Basic2-Below Basic 3-Basic 4-Proficient 5-Advanced
Benchmark Exams Performance Levels
CS
T P
erf
orm
an
ce
Lev
els
5-Advanced
4-Proficient
3-Basic
2-Below Basic
1-Far Below Basic
6 Basic6 Below Basic
7 Proficient
4 Far Below Basic
2 Basic6 Below Basic
2 Advanced
N = 26
N = 138
N = 364
N = 202
N = 58
![Page 13: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/13.jpg)
13
2008-2009
US History11th GradeAggregation of 3 Benchmarks
2008-209
US History11th Grade
CST
Predictability
![Page 14: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/14.jpg)
14
r = .46
US His 11th Gd BM 08-09
US
His
11th
Gd
CST
2009
Prof = 40
63
![Page 15: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/15.jpg)
15
N = 68
N = 116
N = 142
N = 72
N = 7
![Page 16: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/16.jpg)
16
Comparison of 2008-2009 Benchmark Exams with2009 CST Performance Bands for Grade 11 US History
using STAR Performance Band Cut Points
48
19 2410
25
24 18
12
25
2644
23
15
30
28
5
5
13
2
3
1-Far BelowBasic
2-Below Basic 3-Basic 4-Proficient 5-Advanced
Benchmark Exams Performance Levels
CS
T P
erfo
rman
ce L
evel
s
5-Advanced
4-Proficient
3-Basic
2-Below Basic
1-Far Below Basic
2 Below Basic2 Far Below Basic
2 Basic
N = 100
N = 84
N = 121
N = 86
N = 14
![Page 17: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/17.jpg)
17
2008-2009
Science8th GradeAggregation of 3 Benchmarks
2008-209
Science8th Grade
CST
Predictability
![Page 18: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/18.jpg)
18
r = .77
Science 8th Gd BM 08-09
Scie
nce
8th G
d CS
T 20
09
Prof = 39
44
![Page 19: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/19.jpg)
19
N = 84
N = 307
N = 494
N = 275
N = 24
![Page 20: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/20.jpg)
20
Comparison of 2008-2009 Benchmark Exams with2009 CST Performance Bands for Grade 8 Science
using STAR Performance Band Cut Points
6430 13
67
89
3515
31
110
77
20
6
59
154
111
10
40
154
91
1
5
1-Far BelowBasic
2-Below Basic 3-Basic 4-Proficient 5-Advanced
Benchmark Exams Performance Levels
CS
T P
erfo
rman
ce L
evel
s
5-Advanced
4-Proficient
3-Basic
2-Below Basic
1-Far Below Basic
1 Basic1 Below Basic
N = 168
N = 293N = 319
N = 301
N = 103
![Page 21: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/21.jpg)
21
2008-2009
Math6th GradeAggregation of 3 Benchmarks
2008-209
Math6th Grade
CST
Predictability
![Page 22: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/22.jpg)
22
r = .84
Math 6th Gd BM 08-09
Mat
h 6th
Gd
CST
2009
Prof = 43
48
![Page 23: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/23.jpg)
23
N = 164
N = 393
N = 460
N = 224
N = 31
![Page 24: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/24.jpg)
24
Comparison of 2008-2009 Benchmark Exams with2009 CST Performance Bands for Grade 6 Math
using STAR Performance Band Cut Points
41 57
42
202
89
116
246
43
9129
142
83
416 4 4
3
1-Far BelowBasic
2-Below Basic 3-Basic 4-Proficient 5-Advanced
Benchmark Exams Performance Levels
CS
T P
erfo
rman
ce L
evel
s
5-Advanced
4-Proficient
3-Basic
2-Below Basic
1-Far Below Basic
N = 98
N = 384
N = 472
N = 272
N = 45
2 Basic
8 Proficient5 Advanced
![Page 25: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/25.jpg)
25
2008-2009
ELA4th GradeAggregation of 3 Benchmarks
2008-209
ELA4th Grade
CST
Predictability
![Page 26: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/26.jpg)
26
r = .80
Prof = 44
102
![Page 27: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/27.jpg)
27
N = 88
N = 176
N = 334
N = 388
N = 93
![Page 28: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/28.jpg)
28
Comparison of 2008-2009 Benchmark Exams with 2009 CST Performance Bands for Grade 4 English Language Arts
using STAR Performance Band Cut Points
25 27 8
1842
54
13
39
167
62
8
106
167
31
23
103
169
1
7
7
1-Far BelowBasic
2-Below Basic 3-Basic 4-Proficient 5-Advanced
Benchmark Exams Performance Levels
CS
T P
erfo
rman
ce L
evel
s
5-Advanced
4-Proficient
3-Basic
2-Below Basic
1-Far Below Basic1 Proficient
1 Far Below Basic
N = 51
N = 115
N = 358N = 346
N = 209
![Page 29: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/29.jpg)
29
Item Level Analysis
![Page 30: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/30.jpg)
30
Item Difficulty
• The p value for any item– percentage of correct answers– usually in decimal form
• Ideally p value range is .30 to .80 for most items• For example
– p value of .28 = 28% of the test takers got the item right– p value of .75 = 75% of the test takers got the item right– P value of .95 = 95% of the test takers got the item right
![Page 31: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/31.jpg)
31
Item Difficulty MonitoringItem Number N Students Standard p Value
1 853 NS1.2 .37
2 853 NS1.2 .58
3 853 NS1.2 .77
4 853 MG2.1 .25
5 853 MG2.1 .35
6 853 MG2.1 .40
7 853 PS3.3 .43
8 853 PS3.3 .59
9 853 PS3.3 .75
10 853 PS3.3 .95
![Page 32: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/32.jpg)
32
Item Discrimination
• Is item “discriminating” appropriately between higher & lower scoring students
• Discrimination Index (DI) = difference between how upper half and lower half of students score on an item
• DI ranges between -1 and +1• We want items to discriminate positively
![Page 33: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/33.jpg)
33
Item Discrimination MonitoringItem
NumberN
Students Standard p Value DI Max DI
1 853 NS1.2 .37 .45 .742 853 NS1.2 .58 .53 .843 853 NS1.2 .77 .30 .464 853 MG2.1 .25 -.12 .505 853 MG2.1 .35 .10 .706 853 MG2.1 .40 .62 .807 853 PS3.3 .43 .40 .868 853 PS3.3 .59 .58 .829 853 PS3.3 .75 .41 .50
10 853 PS3.3 .95 .09 .10
![Page 34: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/34.jpg)
34
2008-2009
7th Grade Math3rd Benchmark
2009-2010
7th Grade Math3rd Benchmark
Representative of CST Items and our continual Revision Process
![Page 35: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/35.jpg)
35
7NS1.6
2008-2009 Item
p = .33DI = .34
![Page 36: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/36.jpg)
36
RTQ for 7NS1.6
CST = 1RTQ = 1
![Page 37: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/37.jpg)
37
7NS1.6
2009-2010 Item
p = .42DI = .31
![Page 38: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/38.jpg)
38
Additionally…
• Item #3 replaced (as #3) with item modeled after RTQ # 16 (7NS1.7*)—CST = 5, RTQ = 7
• Item #15 replaced (as #13) with item modeled after RTQ #47 (7AF1.5)
• Item #s 23 & 24 replaced (as #s 21 & 22) with items modeled after RTQ #s 89 & 88 (7MG3.4*)
![Page 39: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/39.jpg)
39
Measure Up! Next Steps
• Benchmark Exam Structure• Institutionalize System• CAHSEE• Writing Prompt• Discrimination• Distractor Shaping
![Page 40: 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010](https://reader036.vdocuments.net/reader036/viewer/2022062621/551c0d02550346b24f8b5222/html5/thumbnails/40.jpg)
40
Questions?