state of delaware delaware comprehensive assessment …...question the student was considering,...

State of Delaware

Delaware Comprehensive

Assessment System

2010–2011

Volume 5

Test Administration

American Institutes for Research

DCAS 2010–2011 Technical Report: Volume 5

Test Administration i American Institutes for Research


Test Administration American Institutes for Research

TABLE OF CONTENTS

INTRODUCTION ...........................................................................................................................1

1. TESTING PROCEDURES AND TESTING WINDOWS .........................................................2

1.1 Testing Accommodations ......................................................................................................2

2. ADMINISTRATOR TRAINING................................................................................................4

2.1 DCAS Online Administration ................................................................................................5

2.2 Paper-and-Pencil Administration ...........................................................................................7

3. TEST SECURITY PROCEDURES ..........................................................................................11

APPENDICES

A. Delaware Comprehensive Assessment System (DCAS) Test Administrator User Guide

B. Delaware Comprehensive Assessment System (DCAS) Online Test Administration Manual, 2010–

2011

C. Delaware Comprehensive Assessment System (DCAS) Paper/Pencil Test Administration Manual,

2010–2011

D. Guidelines for Inclusion of Students with Disabilities and English Language Learners, July 30, 2010

E. Protocol of Data Security


Test Administration i American Institutes for Research

LIST OF TABLES

Table 1: Summary of DCAS Tests in 2010–2011 .......................................................................2

Table 2: Test Windows by Subject Area .....................................................................................2

Table 3: Allowable Resources for the DCAS Online Tests .........................................................6

Table 4: Allowable Resources for the DCAS Paper/Pencil Testing ............................................8

Table 5: Key Features of the Plan for Detecting Testing Irregularities .....................................14


Test Administration ii American Institutes for Research


Test Administration 1 American Institutes for Research

INTRODUCTION

The State of Delaware implemented a new online assessment for operational use during the

2010–2011 school year. This new test, referred to as the Delaware Comprehensive Assessment

System (DCAS), replaced the paper-and-pencil test, the Delaware State Testing Program

(DSTP). In 2010–2011, students who were enrolled in various grades in all public schools were

required to take the online assessment. A paper-and-pencil version was available as an

accommodation for students with special needs and students designated as homebound.

Much of the usefulness and interpretability of test scores rests on the premise that a test be

administered according to the test developer’s instruction and that all examinees have the same

opportunity to demonstrate their competencies. This volume documents the test administration

procedures employed in DCAS online and paper-and-pencil tests. Specifically, the purpose of

this volume is to provide empirical evidence to support the following:

Testing procedures used in DCAS ensure the validity of assessment results.

Test Administrators, District Assessment Coordinators, and School Test Coordinators

follow the protocols outlined in test administration manuals.

All secure materials (e.g., test items, test materials, and student-level testing information)

are appropriately handled to protect the integrity, validity, and confidentiality of

assessment questions, prompts, and student results.



1. TESTING PROCEDURES AND TESTING WINDOWS

Students from all Delaware public schools are required to participate in the DCAS unless they

receive a special exemption or they qualify as recently arrived English language learners. See the

DCAS Guidelines for Inclusion of Students with Disabilities and English Language Learners in

Appendix D of this volume for more information on recently arrived English language learners.

All students have the opportunity to practice with sample items in practice tests prior to the

administration. Table 1 below summarizes the content areas, grade levels, and testing

opportunities for DCAS tests in 2010–2011.

Table 1: Summary of DCAS Tests in 2010–2011

Subject Grade Levels Tested Number of Testing Opportunities

Reading 2–10 3*

Mathematics 2–10 3*

Science 5, 8, 10 1

Social Studies 4, 7 1

*Only one testing opportunity for grade 2 in Reading and Mathematics

There were three testing windows for grades 3 through 10 in both Reading and Mathematics with

one opportunity per window. The DCAS was administered as an operational test during the three

windows shown in Table 2. The DCAS-2 window included EFT items for grades 6 and 10 in

Mathematics only. DCAS-3 in all grades and subjects included embedded field-test items.

Table 2: Test Windows by Subject Area

DCAS-1 Initial Blended (Mathematics and Reading Grades 3–10) Test Window 1 Oct. 11–Dec. 14

DCAS-2 Mid-Year Blended (Mathematics and Reading Grades 3–10) Test Window 2 Jan. 5–Apr. 11

DCAS-3 Final Blended (Mathematics and Reading Grades 2–10)/DCAS Final Summative (Science Grades 5, 8, 10) Test Window 3

Apr. 18–June 3

DCAS Final Summative (Social Studies Grades 4, 7) Test Window 3 May 23–June 3

1.1 TESTING ACCOMMODATIONS

All students participating in the DCAS were able to use the standard online testing features in the

Test Delivery System. These features include ability to zoom in on test questions, highlight

passages (or sections of passages) in the Reading tests, pause the test and return to the last

question the student was considering, cross out response options by using the strikethrough

function, and mark a question for review using the flag function. It should be noted that if a test

was paused for more than 30 minutes (45 minutes in the spring window), the student was not

allowed to return to marked questions.

Furthermore, additional features that are available to improve the accessibility of test questions

were provided with the approval of the Test Administrator (TA). For example, students who used

a color overlay during regular classroom time were allowed to change the screen color (choices

included gray, magenta, yellow, and blue). Students who expected to use this accommodation



were given the opportunity to try out the color schemes during the practice test in order to choose

a color scheme well in advance of the test. Also, print on request, like the color overlay feature,

was available to all students with TA approval.

Available accommodations within the testing system for students with documented IEPs or 504

Plans or students identified as English language learners include English text-to-speech, Spanish

text-to-speech, and default large print. These accommodations must be set by entering the

accommodations in the DCAS Accommodations Database at least 72 hours prior to testing time.

Braille forms and paper-and-pencil tests printed in English are also available. In 2010–2011,

districts submitted orders for Braille and paper-and-pencil booklets in advance of the test date.

A student must be provided with all accommodations entered in the DCAS Accommodations

Database. Although students may decide not to use an accommodation, they must be offered the

opportunity to use the accommodations they are entitled to. During the DCAS, students with

IEPs and 504 Plans may only use accommodations they regularly use in the classroom.



2. ADMINISTRATOR TRAINING

To be eligible for the DCAS, a student must be in the Delaware Department of Education’s

DELSIS system. All students are required to participate in all three test windows. DCAS is

untimed in each window. For the online tests, students may begin a session and complete it at

another time. However, all testing must take place during the designated testing window. If

students pause their testing session for 30 minutes (45 minutes in the spring) or more, they

cannot change their answers to previously administered items.

Key personnel involved with DCAS administration included the State Assessment Coordinator,

school administrators, and Test Administrators who proctored the test. Test Administrators must

have been certified by successfully completing an online Test Administrator training course. (For

information about the roles and responsibilities of testing staff, see the next section.)

A secure browser developed by AIR was required to access the online DCAS tests. The secure

browser provides a secure environment for student testing by disabling the hot keys, copy and

screenshot capabilities, and access to desktop functionalities, such as the Internet and email.

Other measures that protect the integrity and security of the online test are presented in “Test

Security Procedures” below.

Prior to the beginning of the 2010–2011 school year, statewide State Assessment Coordinator

training sessions were conducted. A “train the trainer” model was employed to train coordinators

on the use of the Test Delivery System (TDS), Online Reporting System (ORS), and Test

Information Distribution Engine (TIDE). Also, teachers were required to successfully complete

the online TA Certification Course before they could administer a test. The DDOE also offered

regular web conferences (“Elluminate Sessions”) to help coordinators familiarize themselves

with the online testing system.

Also available online are manuals and guides on test administrations. The DCAS Test

Administrator User Guide, included as Appendix A, is designed to familiarize Test

Administrators with the Test Delivery System and contains tips and screenshots throughout the

text. The guide provides enough how-to information to enable teachers to access and navigate

the Test Delivery System. (Policy information is not included in this guide but can be found in

the administration manuals.) The user guide provides the following information:

Steps to take prior to accessing the system and logging in

Navigating the TA interface application

The Student Interface, used by students for online testing

Training sites available for Test Administrators and students

Secure browsers and keyboard shortcut keys

The DCAS Online Test Administration Manual (Appendix B of this volume) and the DCAS

Paper-Pencil Test Administration Manual (Appendix C of this volume) provide information

about policies and procedures for the DCAS. These Test Administration Manuals, which are

updated annually before each school year, include test administration information, guidance, and

directions.



2.1 DCAS ONLINE ADMINISTRATION

The DCAS Online Test Administration Manual provides easy-to-follow instructions about

creating testing sessions, monitoring sessions, verifying student information, assigning test

accommodations, and starting and pausing tests. Certified personnel involved with the DCAS

administration play an important role in ensuring the validity of the assessment by maintaining

both standardized administration conditions and test security. Their roles and responsibilities are

summarized below.

Roles and Responsibilities in the Online Testing System

The State Assessment Coordinator, School Test Coordinator, and Test Administrator each have

specific roles and responsibilities in the Online Testing System. The Test Administrator User

Guide in Appendix A provides an overview of the DCAS testing system and user roles within the

system.

State Assessment Coordinators (SACs)

State Assessment Coordinators are responsible for coordinating testing at the district level. They

ensure that the School Test Coordinators in each school are appropriately trained and aware of

policies and procedures, and that they are trained to use the reporting system.

School Test Coordinators (STCs)

School Test Coordinators are ultimately accountable for ensuring that testing is conducted in

accordance with the test security and other policies and procedures established by the Delaware

Department of Education. STCs are primarily responsible for identifying and training Test

Administrators. They also create or approve testing schedules and procedures for the school.

STCs work with technology staff to ensure that the necessary secure browsers are installed and

any other technical issues are resolved. During the testing window, STCs need to monitor testing

progress, ensure that all students participate as appropriate, and handle testing problems as

necessary.

STCs can also serve as Test Administrators, but they must complete the online Test

Administrator certification course located at http://de.portal.airast.org and review all user guides

and manuals. An STC can be a principal, vice principal, technology coordinator, counselor, or

other staff member. However, it is recommended that the STC be a person with non-instructional

or limited instructional duties so that she or he can coordinate testing activity in the school.

Test Administrators (TAs)

Test Administrators administer the DCAS online tests. Personnel who will administer the DCAS

must complete the online Test Administrator certification course, which is located at

http://de.portal.airast.org. Valid Test Administrator certification was required for the 2010–2011

school year.

The DCAS online tests should be administered by Delaware-certified educators, which include

teachers, administrators, and guidance counselors. However, the tests can also be administered



by a paraprofessional, translator, or substitute teacher as long as he or she is closely supervised

by a Delaware-certified educator. In case of a severe shortage of staff, a test may be administered

by student teachers or school support staff as long as they are closely supervised by a Delaware-

certified educator.

TAs are responsible for reviewing necessary manuals and user guides to prepare the testing

environment and ensuring that students have the necessary equipment and materials, including

scratch paper, pencils, and rulers as appropriate. They are required to administer the DCAS

online tests following the “Directions for Administration” found in the DCAS Online Test

Administration Manual. Any deviation in test administration must be reported by TAs to the

School Test Coordinator, who reports it to the State Assessment Coordinator. The State

Assessment Coordinator then reports it to DDOE. Used scratch paper from the testing session

and any documents that contain personally identifiable student information must be securely

shredded and recycled immediately after each testing session.

TAs must also ensure that only resources that are allowed for specific tests are available and no

additional resources are being used during the DCAS. Table 3 below provides the resources that

are allowed during the DCAS online test.

Table 3: Allowable Resources for the DCAS Online Tests

Test Resources

Mathematics

Calculators (school provided hand-held calculators or online calculators)

Department-approved formula and conversion sheets (the formula sheet is also

provided online)

Science Department-approved periodic table (for applicable grades) (the periodic table is

provided in both online and paper format)

All DCAS Tests

Pen or pencil

Blank or grid scratch paper and a ruler for drawing on scratch paper only

Posters offering students encouragement or inspiration without any specific content

related to content standards

Online test keyboard navigation symbols (found in the Test Administrator User Guide)

Printed out English version of the help screens for students using the Spanish-English

side-by-side accommodation (all the help screens, instructions, and error messages will

be in Spanish only)

Headphones for students using the text-to-speech feature (in which the computer reads

aloud portions of the test)

As specified in the Test Administration Manual, the TA must provide two sheets of graph paper

per student for each mathematics test. Students in grades 2–5 use half-inch grid paper. Students

in grades 6–10 use quarter-inch grid paper. Both grids can be found in the Test Resources section

of the DCAS portal at http://de.portal.airast.org/tab_testadmin.html.

The School Test Coordinator and Test Administrators work together to determine the most

appropriate testing option(s) and testing environment and the average time needed to complete

each test. The appropriate protocols are established to maintain a quiet testing environment

throughout the testing session. The DCAS online tests are NOT timed, and students may begin a

computer session and complete it at another time. All testing must take place during the

http://de.portal.airast.org/tab_testadmin.html



designated testing window. Students who finish early are encouraged to use any remaining test

time to check answers. TAs also need to ensure that adequate time is available to start computers,

load secure browsers, and log in students.

2.2 PAPER-AND-PENCIL ADMINISTRATION

In DCAS 2010–2011, paper-and-pencil forms were administered only to students needing them

as an accommodation; certain students who were designated as homebound; and students

enrolled in schools managed by the Department of Services for Children, Youth and Their

Families (DSCYF). The Braille forms were administered to students who were visually impaired.

The paper-and-pencil and the Braille versions of the tests are typically offered along with the

computer-based administration during the same test window but with a shorter test

administration (a two-week window). The test windows for paper-pencil and Braille tests are

listed in the Paper-Pencil Test Administration Manual (Appendix C in this report).

The administration manual for the paper-and-pencil tests covered the instructions for handling

secure materials, test schedule and time allotment, testing room preparation, test distribution,

breaks, interruptions in testing sessions while maintaining test security, test collection, and

handling and returning secure material. The secure materials included test booklets, answer

sheets, scratch paper, and any reports or other documents that contain personally identifiable

student information.

Roles and Responsibilities in Paper-and-Pencil Testing

The roles and responsibilities of the key personnel involved in the paper-and-pencil testing are

similar to those involved in online testing with some differences due to the nature of the test

materials and handling procedures.

State Assessment Coordinators (SACs)

State Assessment Coordinators are responsible for coordinating testing at the district level. They

ensure that the School Test Coordinators in each school are appropriately trained and aware of

testing policies and procedures, and that they are trained to use the reporting system.

School Test Coordinators (STCs)

Similar to the online tests, School Test Coordinators are accountable for ensuring that testing is

conducted in accordance with the test security and other policies and procedures established by

the DDOE. STCs are primarily responsible for identifying and training Test Administrators.

They also create or approve testing schedules and procedures for the school. STCs monitor

testing progress, ensure that all students participate as appropriate, and handle testing problems

as necessary. The STC also handles testing materials at the school level, securing the materials

and distributing them to Test Administrators as needed.

STCs can also serve as Test Administrators. An STC can be a principal, vice principal,

technology coordinator, counselor, or other staff member. However, it is recommended that the

STC be a person with non-instructional or limited instructional duties so that she or he can

coordinate testing activity in the school.



Test Administrators (TAs)

Test Administrators administer the DCAS paper-and-pencil tests. The DCAS paper-and-pencil

tests should be administered by Delaware-certified educators, including teachers, administrators,

or guidance counselors. However, the tests can be administered by a paraprofessional, translator,

or substitute teacher as long as he or she is closely supervised by a Delaware-certified educator.

In case of a severe shortage of staff, a test may be administered by student teachers or school

support staff as long as they are closely supervised by a Delaware-certified educator.

TAs are responsible for reviewing the necessary manuals and materials (presented in the

appendix of the Test Administration Manual) to prepare the testing environment. TAs ensure that

students have the necessary equipment and materials, including scratch paper, pencils, and rulers

as appropriate. TAs must ensure that students receive the necessary accommodations (Special

Education/504, ELL). They are required to administer the DCAS paper-and-pencil version

following the “Directions for Administration” found in the Test Administration Manual. Any

deviation in test administration must be reported by TAs to the School Test Coordinator, who

reports it to the State Assessment Coordinator. The State Assessment Coordinator then reports it

to DDOE. Used scratch paper from the testing session and any documents that contain personally

identifiable student information must be securely shredded and recycled immediately after each

testing session.

TAs must also ensure that only resources that are allowed for specific tests are available and no

additional resources are being used during the DCAS. The Test Administration Manual provides

details on establishing appropriate testing conditions. Table 4 below provides the resources that

students are allowed to use during the DCAS paper-and-pencil test.

It should also be noted that TAs, for both online testing and the paper-pencil version, are

required to report any testing irregularities during the testing sessions. Testing irregularities are

unusual circumstances that impact a group of students who are testing and may potentially affect

student performance on the test or interpretation of those scores. Examples of testing

irregularities include major disruptions to a test, such as a fire drill, a schoolwide power outage,

or a natural disaster that impacts either test security or test validity. The What to Do When

section of the Test Administration Manual provides guidelines to address such an emergency

situation. That section also includes how to handle testing irregularities resulting from the

administration of an accommodation to a student or a group of students when the student or

students are not eligible for the accommodation or the failure to administer an accommodation to

which a student or group of students are entitled. School Test Coordinators are required to report

testing security incidents to the Delaware Department of Education.

Table 4: Allowable Resources for the DCAS Paper/Pencil Testing

Test Resources

Mathematics Calculators (supplied by the school to students for the Mathematics test only)

Department-approved formula and conversion sheets

Science Department-approved periodic table (for applicable grades)

All DCAS Tests Pencil and eraser

Blank or grid scratch paper and a ruler for drawing on scratch paper only



Test Resources

Posters offering students encouragement or inspiration without any specific content

related to content standards.

The School Test Coordinator and Test Administrators work together to determine the most

appropriate testing option(s) and testing environment and the average time needed to complete

each test. The appropriate protocols are established to maintain a quiet testing environment

throughout the testing session. The paper/pencil tests are NOT timed. However, once a student

begins a paper/pencil session, he or she should complete the test within the session whenever

possible. All testing must take place during the designated testing window. Students who finish

early are encouraged to use any remaining test time to check answers

Ordering, Receiving, and Collecting Paper-and-Pencil Testing Materials

In 2010–2011, paper-and-pencil materials were shipped to schools using two different ordering

methods. For the fall administration, paper books were sent to schools based on the number of

students at the school. The number sent was a percentage of the total test-takers. In order to

streamline this process, for the winter and spring administrations, an order form was made

available to the field. Districts requested materials by returning this form to the Help Desk.

Orders were then processed by a member of the project team.

In 2010–2011, answer documents for grades 3 and 4 contained easy-to-use response grids

(bubbles) that were larger than those in the answer sheets for the higher grades. Sample answer

documents are included in Appendix C of the Paper/Pencil Test Administration Manual. The

answer sheets were returned and scored. For 2011–2012, this process was streamlined. Students

answered the questions in their test books, and TAs entered student responses into the Data Entry

Interface. Once students’ responses are entered into the Data Entry Interface, students receive

their scores instantly. Potential error introduced by handling, returning, and scanning answer

sheets is minimized.

Handling Additional Test Administration Issues in Paper-and-Pencil Administration

This section summarizes the additional issues that must be handled by the School Test

Coordinators and Test Administrators to ensure that the paper-and-pencil tests are administered

in the same way as intended by the test developer. These issues include following the testing

protocols, securing the test materials, handling absences and makeup testing, reporting testing

security incidents, and returning answer documents and test booklets. Refer to the Paper/Pencil

Test Administration Manual in Appendix C for details.

School Test Coordinators are required to ensure that special education students (Section 504 and

ELL students) are provided with appropriate accommodations. Special protocols must be

followed for translating, signing, scribing, reading aloud, and other accommodations. TAs must

be familiar with these protocols if they are administering tests to students with these

accommodations.

The School Test Coordinator is also responsible for monitoring and ensuring the secure storage

of all test materials between testing sessions. When not in use, test booklets and answer sheets

must be stored in a locked, secure location in each school.



All students who can be appropriately tested must participate in the test. Thus, students who are

absent during their regular testing period or those who have not completed their tests should be

allowed to do so within the DCAS paper/pencil test window. STCs must keep a record of

absences for any student in grades 3–10 eligible for paper/pencil testing who was absent from the

regular and makeup testing. Such documentation helps each school verify the receipt of student

scores when they are released on the online reports system.

The STC is responsible for ensuring that schools follow the return procedures outlined in the

Paper/Pencil Test Administration Manual so that answer documents are received and scored

expeditiously and test booklets are returned. Furthermore, State Assessment Coordinators must

count and return ALL unused secure materials in the district overage shipment, using the same

timeframe and procedures as schools will to return test materials.



3. TEST SECURITY PROCEDURES

Maintaining a secure test environment is critical to ensure that scores represent what students

know and are able to do. Because the DCAS is administered online and with a paper-and-pencil

version as an accommodation, test security procedures must guard against item exposure,

cheating, or other security problems for all testing modes.

The test security procedures for DCAS involve the following:

Procedures to ensure security of printed test materials

Procedures to ensure security of online tests

Procedures to investigate test irregularities

See Appendix E for the policy on the test security protocols implemented for DCAS.

Security of Printed Materials

All test items, test materials, and student-level testing information are secure documents and

must be appropriately handled. Secure handling protects the integrity, validity, and

confidentiality of assessment questions, prompts, and student results. Any deviation in test

administration must be reported to ensure the validity of the assessment results. Mishandling of

test administration puts student information at risk and disadvantages the student. Failure to

honor security severely jeopardizes district and state accountability requirements and the

accuracy of student data.

Because the DCAS is administered online, printed test materials were shipped only for those

students requiring the paper-and-pencil accommodation. Printed manuals were shipped to each

school. Manuals and other information can also be found online at http://de.portal.airast.org.

School Test Coordinators and Test Administrators will be able to print additional materials from

this site. A separate manual, the Paper/Pencil Test Administration Manual (Appendix C of this

volume), contained more information.

The following test materials must be securely shredded and recycled immediately after each

testing session and may not be retained from one testing session to the next:

1. Scratch paper and all other paper handouts written on by students during testing

2. Any reports or other documents that contain personally identifiable student information,

including cards with student ID numbers used to help students log into the online system

It is considered a testing security violation for an individual to fail to follow state assessment

system administration procedures promulgated by the Department, and no individual shall:

Give any examinee access to secure test items or materials except in the regular course of

an authorized administration of the state assessment system;

Give unauthorized individuals or other persons access to secure test items or materials;



Copy, reproduce, use, or otherwise disclose in any manner inconsistent with test security

regulations and procedures any portion of secure test materials; Provide answers during

the administration of the test orally, in writing, or by any other means to any examinee;

Coach any examinee during testing by giving the examinee answers to secure test

questions, or otherwise directing or guiding a response, or by altering or interfering with

the examinee’s response in any way;

Fail to follow security regulations and procedures for the storage, distribution, collection,

and return of secure test materials or fail to account for all secure test materials before,

during, and after testing;

Fail to properly monitor test administration, including permitting inappropriate

collaboration between or among individuals; fail to remove or cover non-allowable

resources from the test site; or fail to destroy scratch paper used by the students during

testing;

Fail to prohibit students from accessing or using unauthorized electronic equipment (e.g.,

cell phones, PDAs, iPods, or electronic translators) during testing;

Fail to confirm proper identification of students being administered the testing or give

students the wrong student identification number during the log-in, causing students to

log in and test under another student’s records;

Fail to collect and destroy any materials with student identification number(s) and student

name(s) used to provide student(s) with this information;

E-mail, fax, or inappropriately reproduce any student identification number(s) associated

with student name(s) or other personally identifiable student data;

Produce unauthorized printed copies of test content from the computer website; fail to

properly destroy authorized printed copies; or allow printed copies to leave the test site;

Allow tests to be administered by unauthorized personnel or personnel who have not

received test administration certification;

Administer secure tests on dates other than those authorized by the Department;

Participate in, direct, aid, counsel, assist, encourage, or fail to report any of the acts

prohibited;

Refuse to disclose to the Department information regarding test security violations; or

Refuse to cooperate in the investigation of a suspected breach of test security, whether

such investigation is conducted by a school district or the Department. The investigation

shall include a review of mitigating circumstances, if applicable.

Data Reporting Violations

School districts and individuals shall not:

Fail to report test scores, numbers of students tested, or any other data element required

to be reported to the Department;



Report incorrect or otherwise inaccurate test scores, numbers of students tested, or any

other data element required to be reported to the Department;

Exclude a student from participation in the state assessment system except in accordance

with the regulations of the Department;

Refuse to disclose to the Department information concerning a violation of the foregoing

data reporting requirements; or

Refuse to cooperate in the investigation of a suspected data reporting violation, whether

such investigation is conducted by a school district or the Department. The investigation

shall include a review of mitigating circumstances, if applicable.

Security in Accessing the Online Tests

Secure Browsers

A secure Internet browser is required to access the DCAS tests. The secure browser provides a

secure environment for student testing by disabling the hot keys, copy and screenshot

capabilities, and access to the desktop (Internet, email, and other files or programs installed on

school machines). The secure browser will not display the IP address or other URL for the site.

Users cannot access other applications from within the secure browser, even if they know the

keystroke sequences. The “back” and “forward” browser options are not available, except as

allowed in the testing environment as testing navigation tools. Students will not be able to print

from the secure browsers. During testing, the desktop is locked down, and students must “Pause”

(to save the test for another session) or answer all items and complete a test in order to exit the

secure browser.

Because the browser locks down the computer, students cannot access the Internet or any other

applications on the computer during a testing session.

Investigating Test Irregularities

While not implemented during the 2010–2011 test administrations, the following protocols are

planned for upcoming administrations of the DCAS. Throughout the testing window, a set of

quality assurance/control reports will be routinely produced to ensure the quality of the

assessment system. The quality assurance/control reports will provide information on unexpected

item behavior, item exposure rates, and blueprint match rates. In addition, a cheating analysis

report will be produced to flag any unlikely patterns of behavior in a testing session. The quality

assurance/control reports will be generated every week in the first month of the testing window,

and monthly afterwards, except for the cheating analyses. The cheating analysis report will be

produced every two months.

The following indicators are produced to monitor the online environment:

Item fit: AIR computes classical and IRT-based measures of item fit to evaluate whether

the items are working as intended.

Item exposure control: The percentage of students responding to the same item is

produced to evaluate any unusually high or low rates of item exposure.



Cheating detection: The following are produced to flag unusual test instances:

– Large fluctuations in a student’s performance between test opportunities

– Page latency to assess whether students respond quickly to a given item (which

assumes some possible prior knowledge of the item)

– Inconsistent item fit to assess whether students have aberrant response patterns to

items

Table 5 below highlights the major features of the quality assurance plan to investigate any test

irregularities for the online environment.

Table 5: Key Features of the Plan for Detecting Testing Irregularities

QA Reports Purpose Rationale QA Report Frequency

Item Statistics

To confirm whether items work as expected

Early detection of errors (key errors for MC and scoring errors for CR)

Every week in 1st month and monthly afterwards

Blueprint Match Rates

To monitor unexpected low BP match rates

Early detection of unexpected BP match issue


Item Exposure Rates

To monitor unlikely high exposure rates of items or passages or unusually low item pool usage (high unused items/passages)

Early detection of any oversight in the BP specification


CR Item Position

To monitor whether CR items are positioned throughout the test, beginning, middle, and end of the test

Early detection of unexpected CR item positions


Cheating Analysis Monitor testing irregularities Early detection of testing irregularities

Every two months during the testing window

state of delaware delaware comprehensive assessment …...question the student was considering,...

Documents