confronting data manipulation

Confronting Data Manipulation on Your

Campus

Dr. Will Miller

Director of Institutional Research and Effectiveness

Flagler College

Context

• We rely on others for data• IPEDS

• Outcomes reports

• Closing the Loop

• Course evaluations

• Technology has eliminated the paper trail

Data Manipulation

• Comes in a variety of flavors

• Different than data error!• Survey responses

• Course evaluations

• Course outcomes

• Closing the Loop reports

• Or more troubling forms

Flagler College

• Founded in 1968

• Baccalaureate only

• Approximately 2700 students

• U.S. News Top Regional

• The unicorn of admissions• Increase applicant pool

• Increase cohort size

• Decrease merit scholarships

• Increase class profile

But…

• Unicorns aren’t real

The Swift Fall of the Unicorn

• Background of the offender

• New Director of IRE and New Director of College Writing in Summer 2013

• During a random audit of student placements from fall semester, we notice a vast majority of students were under-placed in writing courses according to information in Student Management System

• Initially blame advisors

• But, test scores in the system do not match hardcopy test scores in student files• The two cases that sealed his fate

Internal Process

• Notification of VPAA (February 7)

• More analysis (Weekend of February 8-9)• Audit of data entry changes shows one user affiliated with all changes on weekends in

September for numerous years (beginning in 2004)

• Update of VPAA (February 9 around 2am)

• Notification of President (February 10 at 8am)

• More analysis

• Final notification of President (February 12 at 8am)

• Trip with VPEM (February 13)

• Removal (February 14)

• Measure Impact (Weekend of February 15-16)

• Notify Cabinet on February 17 at 8am

Sampling of Findings

Sampling of Findings

Contributing Factors

• Relationships

• Whistle blower

• Access to databases

• Test score entry

• Policies on paper v. policies in practice

• Silos

The Public Process

• External investigators

• External data auditors

• Crisis communication consultants

• Openness and transparency• Stakeholders

• Campus community

Sampling of Findings

Sampling of Findings

Reporting and Response

• Open and transparent• Easier to tell all at the beginning

• No sugar coating

• Responses from community

• Responses from stakeholders• Accrediting bodies

• Government

• Publications

Impact

• Faculty

• Students

• Staff

• Institution as a whole

And the Other Shoe Falls

• Students aren’t impacted…if they take math and English first semester• Which some students don’t

• 223 students end up misplaced in math, English, or in a few cases both due to test score fraud

• A whole new process• Pull transcripts, courses taken, time of registration v. time of data

manipulation, determine remediation while consulting with faculty, presenting remediation, and in some cases scrub transcripts

• Interests DoE

Lessons Learned

Any press does not necessarily make for good press

Be open and transparent

Be selective in choosing investigators

Do not worry about cost…you’ll be remembered by how you respond

Lessons Learned

Do not sugarcoat

Policies exist to be followed

Do not hold press conferences

Some people don’t need access to some things

Lessons Learned

Do not blindly trust

Lessons Learned

It can be a positive impetus for change

• We’re actually a unicorn• Decreased applications, increased yield, made the class (and then

some), and increased profile

• Re-established relationships between enrollment management and faculty

• Personnel changes

Questions