universal access to assessments. project overview four implementing states: new hampshire, vermont,...
TRANSCRIPT
Universal Access to Assessments
Project Overview
• Four Implementing States: New Hampshire, Vermont, Rhode Island, and Maine
• Eight Partner States: Connecticut, Maryland, Montana, Iowa, Florida, North Carolina, South Carolina, and Georgia
• Goal: Examine the feasibility, effect, and capacity to deliver state achievement tests using a computer-based test delivery system specifically designed to provide universal access to test content
Operational Test Plan
• Spring 2009: Grade 11 Science
• Spring 2010: Grade 4, 8, & 11 Science
• Fall 2010: Grade 11 Reading & Mathematics
Operational Test Components - 2009• Notify Schools of NimbleTools Option
- Email, Training Sessions, WebEX, Mailers, Website
• Orientation & Practice Tests
- Mailed & Web Download
• Registration & Tool Assignment
• Secure Client Installation Disk Mailed
• Test Administration Manual
• Call Center
Tool Assignment
Access Tool Number of Students
Percent of Students
Standard Read Aloud 608 87 Magnifier 64 09 Reverse Contrast 70 10 Color Overlay 122 17 Color Chooser 90 13 Answer Masking 226 32 Custom Masking 148 21 Auditory Calming 218 31 Break 406 58
Tool Assignment
Number of Tools
Percentage of Students
1 30.6 2 29.6 3 14.4 4 9.5
5+ 15.8
Characteristics of Read Aloud Students
Computer Human
IEP 477 (.79) 368 (.84)
Non-IEP 125 (.21) 68 (.16)
Total 602 436
Note: Six students were marked as NT and Human. These students were excluded from analyses.
Regression Analysis
B SE Beta Sig.
Math .24 .03 .29 <.001
Reading .32 .02 .39 <.001
Computer 2.1 .47 .12 <.001
Paper Read Aloud and Computer Read Aloud
Survey ResponsesOverall Very Positive Feedback
• 92% of test administrators agreed or strongly agreed that it was easy to prepare computers for testing.
• 96% of test administrators agreed or strongly agreed that they would like their students to use NimbleTools in the future.
• 92% of test administrators agreed or strongly agreed that NimbleTools helped students appropriately access the test content.
• “The read aloud accommodation really allowed each student to work independently. They didn't need to wait for me to read to them and could work at their own pace.”
• “Students who utilized the computer based accommodations seemed to be less stressed or exhausted than those who took the paper and pencil method.”
Suggestions for the Future
• “Allow open end responses to be typed.”
• “I had a student with a broken arm last minute – it would have helped to have been able to allow him to use NimbleTools.”
Operational Test Components - 2010• Notify Schools of NT Option - Low Vision RA
- Email, Training Sessions, Mailers, Website
• Teacher & Student Orientation & Practice Tests with Tactile booklets
- Mailed & Web Download
• Registration & Tool Assignment
• Secure Client Installation Disk Mailed/Web download
• Test Administration Manual
• Call Center
2010 Participation
Grade 4 Grade 8 Grade 11 Total
Number of Students 629 606 653 1888
Number of Schools 83 50 52 185
Low Vision Scripting
Tactile Overlays
QuickTime™ and a decompressor
are needed to see this picture.
Scripting Research• Should all elements in a table, graph, or diagram
be read to students in the order they are presented in the item or should these elements only be available to be read to students when clicked on?
• Should chemical equations be read to students using chemistry symbols (e.g. Na) or words (e.g. sodium)?
• When reading table elements to a student, should units of measure be included in the description?
Scripting Research
• Participating students were randomly assigned to one of four 21 item test forms.
• Each test form contained the same 21 items in the same order, but with variations on the scripting rules applied.
• After each set of items, students were asked two survey questions about their preference in the way the items were read aloud.
• Analyses will look at differences in performance and preference between scripting rules.
• 8 of 16 preferred automatic reading of all information in tables
• 6 of 16 preferred no automatic reading of information in tables
• 2 of 16 had no preference
• Preferred - facilitated item comprehension– “It didn’t say the stuff on the chart. I know I can read it, but
what if I didn’t know what Celsius was or how to pronounce it. It is helpful to read”
– “I like how it did the table – it went over it so the grams and stuff so you don’t have to figure it out”
• Not Preferred – redundancy was distracting– “It doesn’t need to read it to me because I can read it to
myself. If I had wanted it to read it to me I would click on it”– “I think it would quicker doing it this way (not read - #6) it
would use up more of the time than reading stuff we don’t need to read”
• 12 of 16 students preferred formulas voiced as words
• 2 of 16 students preferred formulas voiced as letters
• 2 students had no opinion
• Formulas as words facilitated comprehension of items and formulas– “It just confuses me – NA2 + CH3”– “I didn’t like how #15 was read. It was a lot easier
to me so I know what the heck they are talking about”
– “I like it and it explains the equation. If I was a real smart chemist I would know all these formulas, but I think reading the formulas is better”
• 2 of 16 students preferred no reading of units of measure
• 6 of 16 students preferred reading of units of measure
• 8 of 16 students had no opinion (the majority of these students did not recognize any differences in scripting)
• “Annoying” factor (from the mouths of babes!) – proficient computer users used the word “annoyed” frequently when sitting through tutorials or items that took a perceived long time for scripting
• Pacing – some students preferred faster pace than test was scripted, some preferred slower pace. With current technology, natural reading cannot be adjusted for pace
• Every student preferred Nimble Assessment to previously taken assessments