caveon webinar series - mastering the us doe test security requirements january 2016
TRANSCRIPT
www.caveon.com 1
Caveon Webinar Series
Mastering the U.S. DOE Peer Review Requirements on Test Security
January 27, 2016
www.caveon.com 2
Objective
Today, we will overview the Test Security requirements of Peer Review, and share some readily available resources to help you meet them.
www.caveon.com 3
Today’s Presenters
Dr. John Fremer Dr. John Olson
Marc J. Weinstein, J.D.
www.caveon.com 4
Agenda for Today• Purpose of Peer Review• Background of ESSA• Our Background• The Directive• Critical Element 2.5• Four Requirements• Example Evidence
• Resources Available• Summary• Q&A
www.caveon.com 5
Purpose of the Peer Review Process• To support States by identifying expectations
that they can use as they develop, administer, and improve their assessment systems
• To help States prepare for the peer review of their assessment systems
• To guide the peer reviewers in the review of the State assessment systems
www.caveon.com 6
Key Dates for Peer Review
Due to ESSA being signed into law:• January date cancelled• Revised review dates • April 2016 • June 2016
www.caveon.com 7
From the U.S. DOE
“The field has learned a lot about the importance of having strong test security measures, before, during, and after the assessments .”
Ann Whalen, Acting Assistant Secretary for Elementary and
Secondary Education, letter to CCSSO, September 25, 2015
www.caveon.com 8
ESSA Act – December 2015Some background – from NCLB to ESSA.
www.caveon.com 9
From the U.S. Senate on ESSA
Senator HATCH. [regarding ESSA] I ask the chairman, is it your understanding that the references in section 1201 to activities and services that ensure and improve the ‘‘validity and reliability of state assessments’’ were intended to allow funds to be used for test security activities and services designed and utilized to prevent, detect, and respond to testing irregularities and incidents that threaten the validity of assessment results?
See, Congressional Record at S8469 (December 8, 2015) .
www.caveon.com 10
From the U.S. Senate on ESSASenator ALEXANDER. Mr. President, the Senator is correct. Student assessments must be designed and administered with a high degree of quality assurance. State assessment results can be used as the basis for critical decisions affecting the lives of students and the funding and operation of schools, and given the significant taxpayer investment for statewide assessments, we must provide States with the flexibility to use funds to preserve and maintain the integrity and validity of these important assessments.
See, Congressional Record at S8469 (December 8, 2015).
www.caveon.com 11
ESSA – December 2015
• The U.S. Department of Education will seek public comments on ESSA before issuing implementation regulations later in 2016.
• Accountability systems under the new law go into effect for school year 2017-18.
• Test security continues to be a U.S. DOE priority under the new law.
www.caveon.com 12
• Co-authors of CCSSO PublicationsCo-Editor of HoTS
Our Background
Olson & Fremer , 2013 Fremer & Olson, 2015 Wollack & Fremer, 2013
www.caveon.com 13
The Directive, DefinitionsCritical Element• Statement of a relative requirement• State must submit evidence meeting the
requirementExamples of Evidence• Generally illustrative• Variety of evidence
acceptable• Intended to facilitate
preparation of peer review submission
• Must address the critical element in whole or in part
www.caveon.com 14
Critical Element 2.5
Prevent test security irregularities and ensure the integrity of test results through:
• Prevention• Detection• Remediation• Investigation
www.caveon.com 15
Example Evidence, 2.5Per the U.S DOE:
• State Test Security Handbook
• Summary results or reports of internal or independent monitoring, audit, or evaluation of the State’s test security policies, procedures and practices
www.caveon.com 16
Resources to Meet Critical Element 2.5State Test Security Handbook
www.caveon.com 17
Part 1 - PreventionPrevention of any assessment irregularities including : • Implementing proper test preparation
guidelines and administration procedures
• Maintaining the security of test materials
• Providing annual training at the district and school levels
www.caveon.com 18
Example Evidence, Prevention
Per the U.S DOE, implemented & documented:
• Test administration manual (TAM)• Chain of custody procedures• Efforts to reduce the likelihood of
unauthorized communication• Security procedures for
accommodations providers• Protocol and consequences for violations• Internal communication procedures
www.caveon.com 19
Example Evidence, Prevention
Additional suggestions:
• State Test Security Audit or Evaluation Report
• State Test Security Handbook• Technical Report with appropriate sections• DFA and monitoring procedures• Training materials for DTCs, STCs, TAs and
Proctors
www.caveon.com 20
A List of Resources to Help*• TILSA Guidebook, Section 3• TILSA Lessons Learned Report, Section 2• Operational Best Practices for State
Assessments• Handbook of Test Security, Chapter 6 &7• Standards for Educational and
Psychological Testing by AERA/APA/NCME • Test and Data Integrity in the
Administration of Statewide Student Assessment Programs, NCME (2013)
*This is a partial list. You will be provided with a comprehensive list of linked resources after the session, which will address Parts 1-4.
www.caveon.com 21
Part 2 - Detection
Detection of Testing Irregularities
www.caveon.com 22
Example Evidence, DetectionPer the U.S DOE, implemented and documented:
• Incident reporting procedures• Data analysis procedures• Historical data of previous year’s
incidents
www.caveon.com 23
Example Evidence, DetectionAdditional Suggestions:• Those in Part 1, and:• Procedures for securing testing room
from cheating aids• Data Forensics analysis• Monitoring the proctoring process• Web & social media monitoring• Incident intake and tracking
mechanism• Training materials• Tip line
www.caveon.com 24
Part 3 - Remediation
Remediation following any test security incident involving any of the State’s assessments
www.caveon.com 25
Example Evidence, Remediation
Per the U.S DOE, policies & procedures in place, including:
• Clear definition of an incident• Levels of severity identified
• Security Incidence Response Plan• Communication strategies• Stakeholders identified • Media relations tactics
www.caveon.com 26
Example Evidence, Remediation
Additional Suggestions:• Those in Part 1-2, and:
• Escalation Path Flowchart• Documentation of procedures for
responding to testing breaches• Documentation of state's consequences
for confirmed violations
www.caveon.com 27
Part 4 - Investigation
Investigation of any alleged or factual test irregularities
www.caveon.com 28
Example Evidence, Investigation
Per the U.S DOE, investigation procedures including policies for:
• Securing evidence• Rules for investigating• Standards for districts• Privacy policies• Escalation path flowchart
www.caveon.com 29
Example Evidence, Investigation
Additional Suggestions:Those in Part 1-3, and:• Data forensics analysis indicating irregularities• Web patrol findings• Testing Irregularity or Incident Report from a
school or district• Log of tips received by state DOE • Licensure action/other sanctions imposed
following investigation• Risk analysis• Investigative Response Plan• Investigative Training
www.caveon.com 30
Key Points Summarized
• Detailed, documented policies and procedures are key.
• A State Test Security Handbook is an invaluable tool for this process.
• Keep careful records of past efforts and incidents.
• Training must extend beyond test administrators.
www.caveon.com 31
We can help!• Assistance with preparing Peer
Review documentation• Security Audit• Security Handbook• Planning and implementing data forensics analyses• Investigation Training and Process
development
www.caveon.com 32
Thank You! Follow Caveon on twitter @caveonCheck out our blog www.caveon.com/blogLinkedIn Group “Caveon Test Security”
Dr. John [email protected]
Dr. John [email protected]
Marc Weinstein, [email protected]