caveon webinar series: lessons learned from eatp and csdptf november 2013
DESCRIPTION
Presented by Dr. John Fremer, Dennis Maynes and Steve Addicott, Caveon Test Security Two important industry conferences have been held in the last couple of months, the European Association of Test Publishers (E-ATP) Conference and the Conference on Statistical Detection of Potential Test Fraud (CSDPTF). Caveon was at both of these events and wants to share some important information with you. Join Dr. John Fremer, President of Caveon Consulting Services, Steve Addicott, Vice President, Sales and Marketing, and Dennis Maynes, Chief Scientist, Caveon Data Forensics, who attended both conferences and presented sessions. They will explore key takeaways and lessons learned on security. Stay updated on the latest and greatest industry security trends.TRANSCRIPT
Lessons Learned fromThe European Association of Test Publishers Conference and
The Conference on Statistical Detection of Potential Test Fraud
November 20, 2013
Caveon Webinar Series:
John Fremer, President Caveon Consulting ServicesSteve Addicott, Vice President, Client Services
Dennis Maynes, Chief Scientist, Caveon Data Forensics
Agenda for Today
John Fremer, Observations from E-ATP Handbook of Test Security CCSSO TILSA Test Security Guidebook The Emergence of Social Media
Steve Addicott, Observations from E-ATP and CSDPTF ATP/CCSSO OBP ITC Guidelines Electronic Badging ATP Security Committee CSDPTF Observations
Dennis Maynes, Observations from CSDPTF Progress, Research and Direction Techniques that work, and don’t work Challenges in this field
Lessons Learned fromThe European Association of Test
Publishers Conference
John Fremer
5th Annual E-ATP conference: Growing Talent In Europe: Gaining Advantage Through Assessment
Brings together European and other international test publishers and related organizations
Held in beautiful St. Julian, Malta
European Association of Test Publishers (E-ATP) Annual Conference
1. What security vulnerabilities exist for all genres of testing?
2. The critical importance of security planning
3. Practical and proven strategies for preventing and detecting cheating
4. How security breaches have been dealt with in specific programs
5. What lessons have we learned from past instances of testing misbehavior?
HANDBOOK OF TEST SECURITY, FIVE TAKEAWAYS
In many high stakes testing programs:1. Cheating detection statistical analyses
will be performed routinely2. Computer-based testing will increasingly
become the norm3. Technology developments will be critically
important to test security4. The internationalization of the program
will increase5. “Test Security Expert” will become a
recognized and valued position
HANDBOOK OF TEST SECURITY,FIVE PREDICTIONS
Review the Table of Contents “Skim read“ the chapters Earmark chapters and sections for staff to
read or for you to return to Read sections as you deal with the issues
addressed
ADVICE FOR THE READER
To get the Handbook,http://www.psypress.com/books/details/9780415816540/ use discount code HYJ82 for a 20% discount.
TILSA GUIDEBOOK
Guidebook: Preventing, Detecting, and Following up on Testing Irregularities
Center for Chief State School Officers
Designed for State Assessment Directors
Communicate Zero Tolerance1. Clear 2. Consistent3. Frequently emphasized
messages Develop a Security Plan
Comprehensive Up to date
PREVENTION(Sample)
Use Multiple Data Forensics Methods1. Unusual Gains2. Similarity3. Erasures (Answer Changing)4. Person Fit Analyses
DETECTION(Sample)
Areas Needing Attention Level of evidence warranting an
investigation Roles for state, district, and other
staff Time lines Model investigations kit
TEST SECURITY INVESTIGATIONS (Sample #1)
Model Investigations Kit Clear definition of
responsibilities Procedures for evaluating
evidence Planning and conducting
interviews Developing a report with
recommendations
TEST SECURITY INVESTIGATIONS (cont.)(Sample #2)
To get a copy of the Guidebook,
Download a PDF for free or purchase your own copy for only $3.59.
http://www.ccsso.org/Resources/Publications
TILSA GUIDEBOOK
SOCIAL MEDIA
Importance of Social Media -- several presentations
Sessions suggest using “Glocal” approach
Social Media continue to gain users in testing internationally
Ignite Session: Are You One Post Away From a Social Media Crisis?
Speaker: Matthew Poyiadgi, Managing Director, Pearson VUE
3 P’s to deal with a social media crisis: Police Proactive Stance Proportional Response
Customers expect a 1-day response from you. 20% get it.
Ignite Session: Are You One Post Away From a Social Media Crisis? (cont)
Summary Choose your communication channels wisely Write your guidelines – make them practical Choose and train those representing your
agency Be proactive – put your guidelines in action Be proportionate with resources and
responses Track what you do – watch for patterns
Be personable on social media! You’re dealing with people!
Lessons Learned fromThe European ATP Conference &
The Conference on Statistical Detection of Potential Test Fraud
Steve Addicott
ATP/CCSSO Operational Best Practices for Statewide Large Scale Assessment Programs
International Test Commission Guidelines for the Security of Examinations, Tests and Other Assessments
Important Test Security Resources
Published in Spring 2010: Enhance the quality, accuracy,
and timeliness of student test data derived from large scale assessments
Strengthen public confidence in the accuracy and quality of testing data and their uses
ATP/CCSSO Operational Best Practices
2013: Revised OBP for CBT and Online Testing Just published September 2013!
May not be precisely applicable to “…(test programs) used on an international basis”
Provides a solid framework from which others might seek to define a set of practices tailored to their testing programs
ATP and the CCSSO encourage others to use this document
ATP/CCSSO Operational Best Practices
To get a copy of the OBP,
Purchase a copy for $29.99 at
http://www.ccsso.org/resources/publications
ATP/CCSSO Operational Best Practices
International Test Commission Guidelines for the Security of Examinations, Tests and Other Assessments
International Test Commission (ITC) Security Guidelines
With increasing security problems, the guidelines are:
intended to share key elements of security best practices to promote better security, and
defend the value of scores produced through the assessment process
ITC Guidelines: Purpose
All high-stakes assessments, tests and exams
All stakeholders in the assessment process (publishers, users, students, test developers, etc.)
The entire assessment process from development to administration to results processing and scoring
Across all areas of testing (e.g., education, employment, certification/licensure government, clinical psychology, etc.)
ITC Guidelines: Scope
The ITC Guidelines provide answers to these questions:
Threats. What are the dangers?
Test Fraud. How can we deal effectively with increasing security problems?
Terminology. How can we best communicate about test security
ITC Guidelines: Value
Electronic Badging
What is electronic badging?
Big industry players involved
Compelling value
Badging + accreditation + assessment = ???
Stay tuned
ATP Security Committee
Picking up from last February’s conference
Annual Survey results released: https://www.createspace.com/4418924
Hands-on demonstrations of new tech innovations used by test thieves and pirates
Harder and harder for the “good guys” to keep up
About the Conference
The Conference on Statistical Detection of Potential Test Fraud (CSDPTF)
2nd Annual Conference this year
Presents various methods of cheating detection and offers an important forum to examine a variety of methods
Huge thanks to University of Wisconsin at Madison’s Dr. James Wollack and team!
Conference on Statistical Detection of Potential Test Fraud
Last year’s event was groundbreaking
This year, growing momentum… Premier academic & industry organizations
supporting/sponsoring Attendance steadily increasing Publishing of selected papers Plans for future years already underway! Rumors of a scholarly journal – stay tuned!!
Conference on Statistical Detection of Potential Test Fraud
Compelling dynamics behind the conference: Growth in testing Stakes growing ever-higher Focus on trustworthy test results How could we NOT focus on statistical detection?
Conference on Statistical Detection of Potential Test Fraud
Challenge: Current research on the use of statistics. What about the policies and procedures- the legal
defensibility- wrapping their use?
Lessons Learned fromThe Conference on Statistical Detection
of Potential Test Fraud
Dennis Maynes
There were three general sessions on Friday with four or five presentations and a discussant.
About ten poster presentations were on display Friday evening.
There were three concurrent sessions on Saturday with five presentations in the first session, four in the second, and a symposium/workshop in the third.
The conference was attended by over 120 researchers!
Summary of CSDPTF
Similarity and Person Misfit Pre-knowledge and “Braindumps” Unusual Gains Robustness and Sensitivity Issues Test Tampering Computer-Based Testing Policy Test Fraud Detection Challenge Security Planning
Topics Covered at CSDPTF
CSDPTF: 2012 versus 2013
The conference this year was more varied in terms of topics and research than last year. Attendees were from all over the world. The conference has expanded in its reach, focus, and attractiveness.
For this field to progress, the following are needed:
A set of data sets that have known characteristics of security breaches.
Commonly accepted ways to compare the efficacy of methods.
Consensus on the utility and implementation of statistical methods.
Consensus on how to make inferences concerning potential test fraud.
Guidance to practitioners on how to use the statistics.
Progress
The current state of research is that of individual or organization-based efforts with low cross-fertilization between those efforts.
Research
Greg Cizek presented the keynote speech. He discussed where this field has been and where it is going.
Direction
Test Tampering aka “Erasure Analysis”
Similarity aka “Answer Copying”
Techniques that work well and/or are gaining greater acceptance
Detection of item compromise in all its forms such as braindumps, rogue review courses, and disclosure of live exam content by teachers and/or instructors
Analysis of gain scores for detecting potential test compromise
Analysis of response times to detect potential test compromise
Techniques that do not work well
1. Using statistics to invalidate test scores because the scores are not valid. The problem is that the public and measurement professionals have focused on the behavior not the scores.
2. Conducting investigations when the evidence that misbehavior occurred is circumstantial. There are no missing persons, broken doors, or misappropriated funds.
Challenges Practitioners in This Field Face
3. Demonstrating the science behind the techniques that are being used:
Reproducible results Sound reasoning Quantitative measures Defensible assumptions Measurement of error
Challenges Practitioners in This Field Face
Questions?
Caveon Online
Caveon Security Insights Blog http://www.caveon.com/blog/
twitter Follow @Caveon
LinkedIn “Caveon Test Security” Group
Please contribute! Dr. John’s Test Security Tip of the Day Caveon Security Minute
Facebook “Like” us!
Dr. John Fremer
President, Consulting Services
@TestSecurityGuy
Steve Addicott
Vice President, Client Services
@SdAddicott
Dennis MaynesChief Scientist
[email protected]@DennisMaynes
Thank you for your attendance!
At Caveon, we fundamentally believe in quality testing and trustworthy test results.