introduction to dundee's mmi process
TRANSCRIPT
1
The Role of Multiple Mini Interviews (MMI) in selection
Dr Jon Dowell, Admissions Convenor, Dundee Medical School
OverviewRationale behind MMI for selectionA brief historyWhat do MMIs assessHow do MMIs workPerformance in UK contextFeedback from applicants and assessorsThe future?
Primus
Secundus
Tertious
Alexander Monros1720 - 1846
Medicine has ‘previous’
"I dislike him & his lectures so much that I cannot speak with decency about them. He is so dirty in person & actions."
RationalePersonal qualities considered ‘important’ but not reliably
assessed by interview and without much predictive validity. (Goho + Blackman 2006. cognitive r 0.06, clinical r 0.17)
Introduced for medicine at McMaster, Canada (Reiter, Eva et al).Piloted 2002 – 12 x 8 min. Generalisability since +/- 0.7 Predicted OSCE performance ß 0.44 (interview ns)Clerkship ratings ß 0.57 (interview ns)MCC Part 1 (selected components) ß 0.3-0.4 (interview ns)
Increasing detail emergingA cost efficiency comparison between the multiple mini-interview and traditional
admissions interviews. Set up costs, running equates
Rosenfeld JM, Reiter HI, Trinh K, Eva KW. Adv Health Sci Educ Theory Pract. 2008 Mar;13(1):43-58.
The multiple mini-interview: how long is long enough? 10 – 8 – 5 minsDodson M, et al. Med Educ. 2009 Feb;43(2):168-74
Factors affecting the utility of the multiple mini-interview in selecting candidates for graduate-entry medical school. 8Q reliability 0.7
Roberts et al. Med Educ. 2008 Apr;42(4):396-404 applicant 22% variance
Should candidate scores be adjusted for interviewer stringency or leniency in the multiple mini-interview? Rasch Modeling
Roberts C, Rothnie I, Zoanetti N, Crossley J. Med Educ. 2010 Jul;44(7):690-8
Increasingly widespread
Countries• Canada,• Australia• UK• US
Disciplines• UG Medicine• UG Veterinary• UG Dentistry
MMI in Dundee 7 minute x 10 stations
Reliability– 2008/9 n 450 0.7– 2009/10 n 500 0.69– 2010/11 n 600 0.69/.67 (0.88)
– 2011/12 Dental school collaboration
MMI in St Georges 5 minute stations x 8 Stations
Muriel Shannon +Aileen O’Brien
Reliability– 2009/10 n 1078 0.69– 2010/11 n 1333
MMI in Belfast 5 minute x 9 stations
Keith Steele Reliability
– 2010/11 n 500 0.56
MMIs typically aim to assess
– Interpersonal and communication skills (inc empathy)– Teamwork and leadership– Preparation and motivation– Critical thinking, problem solving– Ethical/moral reasoning– ‘Integrity’
How?
PracticalitiesCapacity (n = 600)
20 applicants per ‘Run’ per sessionTwo teams in parallel (80/day)8 days total
Staffing (2 academic leaders + 2 admin)For each run
5 faculty (interview)5 student or Simulated Patients (interactive)2 ‘actors’ (employed junior students)3 ‘role players’ (students or SP)
Preparation = big jobStations - Drafted and piloted with students• Assessor instructions• Candidate instructions• Resources (script, shapes, materials, UCAS forms etc)• Actor training
Training at start of every session 30 min general 20 min station specific
Station/Assessor/Candidate Information
Domain Scores
Overall Judgement
Red Flag & Narrative
Performance of Dundee MMI
Score Distribution
Station Reliability
Overall Alpha 0.69Item-Total Correl
Alpha if Item
Deleted1 0.34 0.672 0.31 0.673 0.17 0.704 0.37 0.665 0.43 0.656 0.47 0.647 0.34 0.678 0.28 0.679 0.38 0.6610 0.39 0.66
Domain Reliability
Number of Stations
Number of Scores
Cronbach’s alpha
Communication 9 12 .767Critical Thinking 6 6 .453Moral Reasoning 4 4 .170Prep +Motivation 3 4 .515Teamwork 2 3 .395Integrity 1 1 -
Gender difference - YesInteractive One - one
Females performed significantly better than males. 2009 Female average 109 /150 vs 105 p = < .01 Effect size: 0.14
Applicant Group difference?
2009 – 2010 2010 - 2011
Apologies but we are intending to eventually publish from this data
so have removed it from circulated slides.
Assessors Survey 2008-9
116 /201 (58%) assessors completed online survey.• 91% thought the process fair.• 88% thought applicant stress moderate or less. • All stations rated highly relevant, except for the
‘Question of Ethics’ where opinions divided.
Candidate Survey 2008-9
• 324 /433 (75%) completed online survey.• 94% felt MMI was ‘fair’.• 90% felt it is a ‘valid way to assess candidates’.• 71% preferred MMI to traditional interview.• ‘Student Counselor’ (role play) most enjoyable
We are happybut where next?
Hawk-Dove Effect
1 2 3 4 5 6 7 8 9 10
STATIONS
A
E
B
D
C
P. H. Harasym, Ph.D.
Dundee 2011 Rasch vertical ruler
MFRM creates a common scale for candidate ability, examiner stringency, item difficulty and ......Comparing raw and ‘fair’ scores for 350 offers 34 (9.7%) different candidates selected.
Conclusions and Next StepsMMI appear very promising:
• Set up ‘costs’ considerable• Running costs comparable (if students used)• Student involvement positive• Applicants are positive• Predictive validity emerging and encouraging.
Outstanding issues:• Rotating content.• Application of IRT (DIFF, Fair scores etc)• Oh yes – and the bloopers are great.
Thank you
Discussion?