Upcoming Caveon Events

Conference for the Statistical Detection of Potential Test Fraud, University of Kansas, May 23-24 – Register Now!

National Conference on Student Assessment in Minneapolis, June 26-29

International Testing Commission Conference in Amsterdam, July 3-5

Joint Statistical Meetings in San Diego, July 28 – August 2

National Collegiate Testing Association in Minneapolis, August 1-4

Visit www.caveon.com/events for more info

Caveon Online

Caveon Security Insights Blog http://www.caveon.com/blog/

Twitter @Caveon

LinkedIn Caveon Company Page Caveon Test Security Group

Please contribute!

Facebook Will you be our “friend?” “Like” us!

Background

The following is an encore presentation of Dr. Fremer’s Plenary Session at April’s National Association of Test Directors Conference

We are pleased to share his Certificate of Recognition for “Outstanding Contributions to Educational Assessment”

National Association of Test Directors Certificate of

Recognition• 40 Years Improving Educ.

Assess.• Revision of the SAT• Book “Measuring Up”• Writing about Computer

Assessment• As NCME President - Advocated

Inclusion for States, Districts, and Schools

• Helping Promote Testing Integrity

“How to Avoid Becoming Another Atlanta Public Schools”

John Fremer President, Caveon Consulting Services

May 16, 2012

Caveon Webinar Series

Presentation Overview

The Atlanta Story Lessons External

Assistance

Caveon Role in APS

Contract with Atlanta Educational Foundation Not APS Not state of Georgia

Security Audit Limited Data Forensics

Atlanta Story – Short Form

Persistent Accusations of Cheating in APS Schools

Consistent Denials Reports are wrong Newspaper is prejudiced Governor out to get

Superintendent Governor’s Office of Student

Achievement and Atlanta Journal Constitution Stayed on Case

What Did the Data Show?

Unmistakable Evidence of Improprieties Impossible to miss Extreme cases very pronounced –

one over 10 to 52nd power! Governor Lost All Patience

Georgia Bureau of Investigation 61 agents Subpoena powers “Deals” offered

Outcome for APS

Superintendent Out – A National and Highly Regarded Educator

All Deputy Superintendents Out Many Principals and Teachers

Confess Those Pleading Fifth

Amendment Declared Guilty Names of all Judged Guilty

Published APS Situation Known

Nationwide

Questions?

Testing Management Lessons from Atlanta Don’t Make Political Leaders

Mad at You! Perform Cheating Analyses

Routinely Evaluate the Adequacy of Your

Security Procedures Follow Up Thoroughly on All

Suspicions Take Appropriate and Public

Action

Media Lessons from Atlanta

Expect Media Attention Get Out Front with Your Own

Story Cultivate Local Media

Most do not understand test results

Looking for interesting story – a no problem administration is not a good story

Learn Who You Can Trust Don’t Expect a Cheating Story

to Go Away – You Cannot Control It

Any Questions?

Possible External Assistance

Test Security Audit Data Forensics Web Patrol (Monitoring

Internet) Security Investigations

Test Security Audit: The Importance of Knowing

Validates your Security Practices Identifies Vulnerabilities Recommends and Prioritizes

Actions to Correct Shows Your Seriousness about

Security

Test Security Audit: What is it? Systematic Review by

Experienced Professionals Objective Third Party

Evaluation Over 250 Standards Immediate Feedback Practical and Specific

Test Security Audit Topics

1. Security Plan

2. Roles & Responsibilities

3. Budget & Funding

4. Legal Precautions & Agreements

5. Test & Item Design

6. Test & Item Development & Maintenance

7. Test Publication

8. Test Administration

9. Test Scores & Results

10. Physical security

11. Information Security

12. Web & Media Monitoring

13. Security Awareness & Training

14. Security Incidence Response Plan

15. Conducting Security Investigations [New]

Psychometric Analyses - Detecting Cheating

Data Forensics Looking at Your Test Data Has a Test been Compromised? When and Where was it Compromised? “Traditional” Procedures No Longer Good Enough

Data Forensics Benefits

Use Data to make Informed Decisions

Train from Known Threats Detect and Deter Cheaters Stay Ahead of Cheating

Methods Evaluate Test Admin Security

Procedures Evidence of Strong Action –

Bringing in Independent Experts

Success Story - State Assessment

72% reduction – 2 years

Test admin monitoring

Conditional status Districts follow

through “Students confess” Coordinator training

Reduction in "Irregularities"

0.0%

1.0%

2.0%

3.0%

4.0%

5.0%

6.0%

Rep

ort 1

Rep

ort 2

Rep

ort 3

Rep

ort 4

Rep

ort 5

Rep

ort 6

Rep

ort 7

Rep

ort 8

Rep

ort 9

Web Patrol

Are “Secure” Items on the Internet?

Occurs in High Stakes Testing Long Testing Windows Tend to

Lead to “Leakage” Involves Searching Tools,

Expert Sleuthing, and Quick Follow-Up

Very Common with Certification Programs

Conducting Investigations

Evaluating Cases Warranting Follow-Up

Using all Available Data Interviewing Teachers, Test

Coordinators, Principals First Response is Always –

“There is no problem” Careers are at Stake – Need

Very Solid Evidence

Continue The Conversation

- Follow us on twitter @caveon for updates and events

- Check out our blog…www.caveon.com/blog/

- LinkedIn Group – Join our Caveon Test Security group to ask questions and join discussions about test security topics

- Slides of this and past Webinars are available at www.caveon.com/resources/webinars

Please Join Us Next Time!

“Tales from an Examnivore: Utilizing test development resources to support your testing program”

Featuring Jamie MulkeyVice President, Test Development ServicesCaveon

Save The Date - June 20th

Thank you!

Dr. John Fremer, Ph.D.PresidentCaveon Consulting Servicesjohn.fremer@caveon.com