an exploration of answer-changing on a computer-based high-stakes achievement test
DESCRIPTION
An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test. Gail Tiemann Neal Kingston. Purpose. Gather baseline answer-changing patterns on a computer-based achievement test Add to the knowledge base in this area. Methods. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/1.jpg)
An Exploration of Answer-Changing on a Computer-
based High-Stakes Achievement Test
Gail Tiemann Neal Kingston
![Page 2: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/2.jpg)
Purpose
Gather baseline answer-changing patterns on a computer-based achievement test
Add to the knowledge base in this area
![Page 3: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/3.jpg)
Methods
Third grade summative test in mathematics, one state testing program
One, 2012 form 70 items, no time-limit Students can review and change items
as desired
![Page 4: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/4.jpg)
Methods
Assessment System Logso Linear path through the testo Time spent viewing an item oNumber of reviewsoNumber and pattern of changes
Unique benefit of CBT
![Page 5: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/5.jpg)
Sample and Test Characteristics
Students 5328
Raw Score Range 6 - 70Mean 56.19Standard deviation
8.19
Coefficient alpha 0.89
![Page 6: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/6.jpg)
Test Score Distribution
![Page 7: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/7.jpg)
Answer-ChangingItems
ChangedCount Percent Cumulative
Percent
0 2537 47.6 47.6
1 1419 26.6 74.2
2 652 12.2 86.5
3 337 6.3 92.8
4 155 2.9 95.7
… … … …
16 2 0.04 99.9
18 1 0.02 99.9
19 1 0.02 99.9
22 2 0.04 100
Items = 70 Students = 5328
![Page 8: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/8.jpg)
Wrong-to-Right & Right-to-Wrong
Final change in patternoWRoRWRoRWWWR
![Page 9: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/9.jpg)
Wrong-to-Right by Items Changed
ItemsChanged
Wrong-to-Right Count
0 1 2 3 4 5 6 7 8 9 10
1 504 915
2 92 252 308
3 18 75 131 113
4 6 19 42 60 28
5 5 16 8 27 22
6 4 8 10 14 11 11
7 1 1 3 3 6 7 6 6
8 3 1 3 2 5
9 3 1 1 1 4 2
10 1 1 2 1 1 3 1 0
Items = 70 Students = 5328
![Page 10: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/10.jpg)
Wrong-to-Right by Items Changed
ItemsChanged
Wrong-to-Right Count
0 1 2 3 4 5 6 7 8 9 10
1 504 915
2 92 252 308
3 18 75 131 113
4 6 19 42 60 28
5 5 16 8 27 22
6 4 8 10 14 11 11
7 1 1 3 3 6 7 6 6
8 3 1 3 2 5
9 3 1 1 1 4 2
10 1 1 2 1 1 3 1 0Items = 70 Students = 5328
![Page 11: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/11.jpg)
Wrong-to-Right by Items Changed
ItemsChange
d
Wrong-to-Right Count
… 11 12 13 14 15 16 17
…
14 1
15 1 1
16 1
18 1
19 1
22 … 1
Items = 70 Students = 5328
![Page 12: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/12.jpg)
Wrong-to-Right vs. Right-to-Wrong
RtW
WtR
1 2 3 4 5 6 7 8 9 10
1 252 75 19 5
2 131 42 16 3 4 1 1
3 60 8 10 3 3 1 1
4 27 14 6 1 3 2
5 11 7 4 3
6 6 3 1 1 1 1 1
7 2 1 1 1 1
8 4 3 1 1
9 1
10 2
![Page 13: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/13.jpg)
Wrong-to-Right vs. Right-to-Wrong
RtW
WtR
1 2 3 4 5 6 7 8 9 10
11 1 1
14 1 1
15 1
16 1
17 1
![Page 14: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/14.jpg)
Answer-Changing Patterns
Pattern Students Items
1x Total 2731 5641WR 2101 3733
RW 672 818
WW 869 1090
2x Total 240 286RWR* 126 137
WWR* 61 66
RWW 7 7
W1 R W1 18 18
W1 R W2 26 26
W1 W2 W1 23 23
W1 W2 W3 8 8
![Page 15: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/15.jpg)
Answer-Changing Patterns
Pattern Students Items
3x Total 25 26WRWR* 7
WWWR* 7
RWRW 4
WRWW 6
WWWW 2
4x Total 4 4RWWWR* 1
WRWWR* 1
WWWRW 2
![Page 16: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/16.jpg)
Scatterplots
![Page 17: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/17.jpg)
Scatterplots
![Page 18: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/18.jpg)
Elapsed Time
Wrong-to-Right< 60 seconds
![Page 19: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/19.jpg)
Wrong-to-Right < 10 seconds
WtR Items < 10 Seconds
Students Percent Cumulative Percent
1 456 82.8 82.8
2 75 13.6 96.4
3 9 1.6 98.0
4 4 0.7 98.7
5 4 0.7 99.5
6 1 0.2 99.6
7 1 0.2 99.8
8 1 0.2 100.0
![Page 20: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/20.jpg)
Flagging Rule Application
Students Flagged 6
WtR / Total Changes (Proportion)
0.64 – 1.00
Elapsed Screen Time Range 4 seconds to 236 seconds
Weekend Submissions No
Multiple Changes to Single Items
No
Mean Wrong-to-Right Count (Target Form Only)
+ 8 Standard Deviations
![Page 21: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/21.jpg)
Discussion
LimitationsoOne form, one subject, one gradeoNested dataoDistributions are not normal
![Page 22: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/22.jpg)
Discussion
Answer patterns and elapsed time add to answer-changing interpretations
![Page 23: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/23.jpg)
Future
Further exploration of wrong-to-right changes by elapsed screen time and position of change
Other sources of varianceo Item-factors (difficulty, position)o Student-factors (demographics)oNesting (buildings, districts,
classrooms)
![Page 24: An Exploration of Answer-Changing on a Computer-based High-Stakes Achievement Test](https://reader035.vdocuments.net/reader035/viewer/2022062723/56813b48550346895da42e95/html5/thumbnails/24.jpg)
Thank you