bbworld 2012 marcus merrin & james m. riceexamination challenges in a caribbean medical school...

Examination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in Examinations

Marcus MerrinDirector of Educational Technology.James RiceAssociate Dean of Research.

American University of AntiguaCollege of Medicine

2

Background: Who we are and what we do.• Medical School, accredited in New York and

California. Also CAAM accredited• Provide medical education for candidates

expecting to practice in the US• 2000 + Medical Students, 1000 in Antigua, the rest

on clinical rotations in the US

3

Our Students

• Many from non-traditional backgrounds for medical school.

• Students can be admitted without MCAT scores

• About 60% privately funded by self / parents

• 40% in receipt of student loans

4

Our Challenges

• University policy mandates regular in-house exams in all subjects. NBME external exams are administered at the end of every two semesters.

• Some of our students are not well equipped to deal with the rigours of medical education upon entry.

• Many students are under immense financial and social pressure to succeed.

• Motivation to cheat on exams is high for a significant segment of our student population.

• While Antigua enjoys good internet connectivity with mainland North America, brief outages are common and bandwidth is expensive.

5

What we try to achieve

• Maximize our students’ chances of performing well on examinations

• Provide a fair exam for all. Approximately 250 – 300 students are taking exams at the same time.

• Appropriate accommodations for students with disabilities

• Prevent students getting illicit assistance

• Preserve integrity and secrecy of exam materials

6

What we try to achieve

• Provide exams that have formative value

• Expose and accustom students to the style of exams used by NBME

• Deny exams to students who do not have an 80% attendance record in lectures / small group sessions or labs

7

Our Solutions to our Challenges

• All departments prepare and upload exams using the Respondus 4.0 exam authoring software (campus-wide license)

• Some faculty reuse questions with well-established performance data (à la NBME)

• Some faculty prefer to write new questions for each exam.

• Exams are set to be available only through the Respondus LockDown Browser

• Exams are password protected

8

Our Solutions to our Challenges (setup)

• Exams always set for one-at-a-time presentation

• Adaptive release is used to exclude students whose attendance is inadequate.

• Separate versions of the exams are created for students requiring testing accommodations. Adaptive release is used to ensure these versions are only visible to qualified students

• Because of connectivity issues, we are forced to permit students to resume an exam in progress – this presents additional security challenges.

9

Our Solutions to our Challenges (exam administration)

• The exam center provides hard-wired exam rooms with university supplied laptops with Respondus LockDown Browser installed

• The exam center provides randomized seating assignments for each exam

• Students are examined in rooms of approximately 100 students with 3 proctors per room.

• Cameras are available in rooms to provide evidence that a particular student was present (not always used, but a useful threat)

10

Our Solutions to our Challenges (administration)

• Before the students are admitted, the proctors ensure all previous logins have been reset

• At the start of the exam, The exam link is made available and the password projected on LCD overhead screens.

• GSM blockers have been installed to prevent students messaging the exam password to external accomplices

• As soon as all students have started the exam, the password is changed.

11

Our Solutions to our Challenges (administration)

• Students needing bathroom breaks must be chaperoned

• Bathrooms are searched before exam sessions

• At the conclusion of the exam, Students are not permitted to leave the room.

• The exam link is closed, and feedback options are activated for the exam.

• After a 15 minute review period, the review options are closed, and the students are permitted to leave the room.

12

13

We’re going to talk about research & statistics?

14

Examination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in Examinations

• So, the experimental questions we looked at were:• Do our medical students report stress, and “decision

fatigue”-type problems?

• Are there any data to suggest “decision fatigue” in our medical students?

15

Examination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in Examinations

• Do our medical students report stress, and “decision fatigue” problems?

• Our students? –265 AUA medical students in their first semester of medical school

• Students also completed a survey after one exam examining their test preparation, amount of sleep, and test-taking strategies

16

Examination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in Examinations

Let’s look at some sleep survey dataHow many hours did you sleep last night (before the exam)?• 42.5% of students got 5 hours or less sleep• 18% got 4 hours of sleep OR LESS

What is the usual number of hours you sleep per night?• 22% of students reported 5 hours or less sleep• 4% of students reported 4 hours of sleep or less

17

Examination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in Examinations

Let’s look at some nutrition survey data:• My breakfast today (morning of the exam) consisted of the following items (check all that apply):• 25% of the students reported consuming a beverage without caffeine• 46% of the students reported consuming a beverage with caffeine• 63% of the students consumed a carbohydrate food item• 30% of the students consumed a protein food item)• 12% of the students consumed a vegetable/fruit item

18

Examination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in Examinations

• Are there any data to suggest “decision fatigue” in our medical students?

• Our students? –265 AUA medical students in their first semester of medical school

• The data we collected were from three internal exams in Human Structure and Function I, a course that attempts to integrate both Anatomy and Physiology

• All Exams were given on Blackboard

19

Examination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in Examinations

Are there any data to suggest “decision fatigue” in our medical students?

• But first, a brief digression: Let’s talk about “Decision Fatigue”

• A term coined by Dr. Roy Baumeister, a Social Psychologist at Florida State University who is interested in decision-making and the amount of energy expended in making decisions and other types of mental activity.

20

Examination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in ExaminationsSome methodological concerns/issues:• One assumption is that students essentially complete exams in an ordinal manner. So they answer Item #1 first, Item #2 second, etc. We know this is not entirely true, and students do go back and answer difficult items ;• GPA may mitigate against decision fatigue—that is, if you are “smart” enough, you may be able to work through any fatigue and still get the item correct• Overall item difficulty may also be a covariate. In-other-words, the ordinal location of an item may not matter; the item may be missed because it is a difficult item; and• It was important to confirm that Blackboard indeed randomizes item presentation on the exams•  

21

Examination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in Examinations

Are there any data to suggest “decision fatigue” in our medical students?

• So, the experimental question was:

 • After controlling for GPA and Overall Item Difficulty,

does the Order of item presentation affect whether the item was missed or not?

22

Examination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in Examinations

Are there any data to suggest “decision fatigue” in our medical students?

The data from four quizzes were reviewed for this analysis:

• Mini-exam #2 70 mult choice questions

• Mini-Exam #3 80 mult choice questions• Mini-Exam #4 (Part 1) 60 mult

choice questions• Mini-Exam #4 (Part 2) 60 mult choice

questions

23

Examination Challenges in a Caribbean Medical School and the Study of "Decision

Fatigue" in Examinations

What kind of questions are being asked of

our medical students?

Why don’t you try a couple and find out?

Welcome to Human Structure & Function I

Mini-Exam #2!!

24

Examination Challenges in a Caribbean Medical School and the Study of "Decision

Fatigue" in Examinations

A 46- year old male patient had surgery to repair the thoracic diaphragm skeletal muscle. The patient did not survive the operation. If the patient had recovered from the surgery, the muscle fibers removed during surgery would have been replaced by proliferation of which of the following cells?

• a) Fibroblast

• b) Monocyte

• c) Satellite

• d) Multiplication of mature muscle fibers

25

Examination Challenges in a Caribbean Medical School and the Study of "Decision

Fatigue" in Examinations A 46- year old male patient had surgery to repair the

thoracic diaphragm skeletal muscle. The patient did not survive the operation. If the patient had recovered from the surgery, the muscle fibers removed during surgery would have been replaced by proliferation of which of the following cells?

• a) Fibroblast

• b) Monocyte

• c) Satellite

• d) Multiplication of mature muscle fibers

Item Difficulty was 13.06%

26

Examination Challenges in a Caribbean Medical School and the Study of "Decision

Fatigue" in ExaminationsA 500 Daltons protein was injected into the venous

systemic system of a 50 year old male patient. This protein will not be detected or present in which of the following tissues or organs?

• a) Brain

• b) Kidney

• c) Spleen

• d) Adrenal gland

• e) Liver

27

Examination Challenges in a Caribbean Medical School and the Study of "Decision

Fatigue" in ExaminationsA 500 Daltons protein was injected into the venous

systemic system of a 50 year old male patient. This protein will not be detected or present in which of the following tissues or organs?

• a) Brain

• b) Kidney

• c) Spleen

• d) Adrenal gland

• e) Liver

Item Difficulty is 62.69%

28

Examination Challenges in a Caribbean Medical School and the Study of "Decision

Fatigue" in ExaminationsAnalysis of the data

• Logistic regression

• What is it? It is a statistical test used when the dependent (criterion) variable is binary (yes/no , correct/incorrect), and there are one or more continuous predictor variables or covariates. It is NOT the same as multiple regression and does not produce a standard solution, but a probabilistic solution.

29

Examination Challenges in a Caribbean Medical School and the Study of "Decision

Fatigue" in ExaminationsAnalysis of the data

So, what did we find?

• Mini-exam #2—18,760 lines of data-- both covariates were significant—“Order” was not a significant predictor

• Mini-exam #3—20,960 lines of data—both covariates AND “Order” were statistically significant—p < .01

• Mini-Exam #4—31,800 lines of data—both covariates were significant—“Order” was not a significant predictor

30

Examination Challenges in a Caribbean Medical School and the Study of "Decision

Fatigue" in ExaminationsSo, what are the implications from this study?

• Medical students are stressed. Many students sleep too little, have marginal nutrition, and are stressed and distracted during exams;

• There is weak evidence with longer exams that “decision fatigue”, defined as getting questions incorrect because of the order the item is presented, may in fact exist. This effect was found (in one of three exams analyzed) even after student GPA and overall item difficulty were factored out of the model.

• Since the purpose of examinations is to give medical students the opportunity to demonstrate mastery over their Basic Science coursework, AUA Administration may wish to consider more innovative strategies to provide more rest breaks during exams in order to minimize the possibility of decision fatigue so that a more accurate assessment of student performance can be obtained.

31

And we hope you don’t think this was!!

Thank you!!

32

Contributing Colleagues

Dr Roslyn Haley

• Chairperson, Educational Enhancement Department, AUA College of Medicine

Cecelia Downs, M.A.

• Faculty, Educational Enhancement Department, AUA College of Medicine

33

We value your feedback!Please fill out a session evaluation.