annual meeting 2008 creating a better tomorrow gsa transition forum: is the mspe dead? aamc annual...
TRANSCRIPT
Annual Meeting 2008
Creating a Better Tomorrow
GSA Transition Forum:
Is the MSPE Dead?
AAMC Annual MeetingSan Antonio, TexasNovember 1, 2008
Creating a Better Tomorrow2008 Annual Meeting
Chair
Molly Osborne, M.D., Ph.D.
Associate Dean for Student Affairs
Oregon Health & Science University
Chair-Elect, AAMC Group on Student Affairs
Creating a Better Tomorrow2008 Annual Meeting
Speakers
Richard P. Wheeler, M.D.
Executive Associate Dean for Academic Affairs, College of Medicine
University of Arkansas for Medical Sciences
Lanny Garth Close, M.D.
Professor and Chair, Department of Otolaryngology
Columbia University College of Physicians & Surgeons
Creating a Better Tomorrow2008 Annual Meeting
Speakers
Neil H. Parker, M.D.
Senior Associate Dean, Student Affairs & GME
University of California Los Angeles Geffen SOM
Angela Nuzzarello, M.D., MHPE
Associate Dean, Student Programs & Professional Development
Northwestern University Feinberg School of Medicine
Creating a Better Tomorrow2008 Annual Meeting
Speakers
Maria C. Savoia, M.D.
Vice Dean for Medical Education
Professor of Medicine
University of California San Diego School of Medicine
Creating a Better Tomorrow2008 Annual Meeting
Question # 1
Which of the choices below is your primary affiliation?
1. Student Affairs (GSA)
2. Undergraduate Medical Education (GEA)
3. Graduate Medical Education (Program Directors)
4. Student (OSR)
5. Resident (ORR)
6. Other
Creating a Better Tomorrow2008 Annual Meeting
Question # 2
If you are involved with GSA what part are you primarily involved with?
1. MAS
2. COSR
3. COSA
4. COSFA
5. OSR
6. All
Creating a Better Tomorrow2008 Annual Meeting
Question # 3
How long have you worked in this position?
1. < 1 year
2. 1-5 years
3. 5-10 years
4. >10 years
Creating a Better Tomorrow2008 Annual Meeting
Question # 4
Are you the sender, recipient or subject of the MSPE?
1. Sender
2. Recipient
3. Subject
4. None of the Above
Creating a Better Tomorrow2008 Annual Meeting
Richard P. Wheeler, M.D.
Executive Associate Dean for Academic Affairs, College of Medicine
University of Arkansas for Medical Sciences
Creating a Better Tomorrow2008 Annual Meeting
History of the Dean’s Letter
AKA – M S P E
AKA – MISSPEEEEEEEE
AKA – “It’s not time yet to do those darn things again this year, is it?”
Creating a Better Tomorrow2008 Annual Meeting
National Library of Medicine
4th or 5th Century AD
Creating a Better Tomorrow2008 Annual Meeting
National Library of Medicine
4th or 5th Century AD
Creating a Better Tomorrow2008 Annual Meeting
National Library of Medicine
4th or 5th Century AD
Creating a Better Tomorrow2008 Annual Meeting
This was too damn hard. Have the Associate Dean do it from now on!!
Creating a Better Tomorrow2008 Annual Meeting
National Library of Medicine
4th or 5th Century AD
Creating a Better Tomorrow2008 Annual Meeting
National Library of Medicine
4th or 5th Century AD
Creating a Better Tomorrow2008 Annual Meeting
National Library of Medicine
4th or 5th Century AD
Creating a Better Tomorrow2008 Annual Meeting
Henry M. Sondheimer, M.D.November 1, 300
Creating a Better Tomorrow2008 Annual Meeting
No rules or standardization.
Personal letters written to individual program directors.
Many times a real letter of recommendation rather than
reference.
Creating a Better Tomorrow2008 Annual Meeting
Creating a Better Tomorrow2008 Annual Meeting
UAMS Dean’s Letters = 1 to 2 pages
Creating a Better Tomorrow2008 Annual Meeting
1989
Creating a Better Tomorrow2008 Annual Meeting
UAMS Dean’s Letters = 2 to 3 pages
Creating a Better Tomorrow2008 Annual Meeting
November 1 release date approved through a complex
process.
Intended to slow down the process - and it worked. Few interviews before November.
Creating a Better Tomorrow2008 Annual Meeting
ERASSeptember 1 release established by ERAS Committee but not
through complex approval process like the November 1 release date
1995
Creating a Better Tomorrow2008 Annual Meeting
Creating a Better Tomorrow2008 Annual Meeting
Camel
Creating a Better Tomorrow2008 Annual Meeting
2002
Creating a Better Tomorrow2008 Annual Meeting
UAMS Dean’s Letters = 8 to 9 pages
Creating a Better Tomorrow2008 Annual Meeting
TODAY
Creating a Better Tomorrow2008 Annual Meeting
OR
Creating a Better Tomorrow2008 Annual Meeting
Lanny Garth Close, M.D.
Professor and Chair, Department of Otolaryngology
Columbia University College of Physicians & Surgeons
Creating a Better Tomorrow2008 Annual Meeting
THE PROGRAM DIRECTOR’S POINT OF
VIEW
The Medical Student Performance Evaluation
Creating a Better Tomorrow2008 Annual Meeting
MSPE
2004 Survey of Oto/HNS PD’s
Rank Order of ERAS Candidate Information
• USMLE step 1 score
• Performance as sub-I
• LOR from fellow Chairs/PD’s
• AOA status
• Candidate’s Personal statement
• Dean’s Letter (MSPE)
Creating a Better Tomorrow2008 Annual Meeting
MSPE
The PD’s dream MSPE………
“ Josephine Smith’s class rank at the completion of the third year of medical school is **/***.”
Creating a Better Tomorrow2008 Annual Meeting
MSPE
The PD’s nightmare MSPE………..
“Although the Dean’s Letter Advisory Committee
of the AAMC recommends that the MSPE
contain comparative performance statements,
………..’s evaluation system was not designed to
provide information comparing one student to
another. For that reason, we encourage review
of this evaluation in it’s entirety.”
Creating a Better Tomorrow2008 Annual Meeting
MSPE
Why is the non-revealing “stealth” MSPE a problem for the PD in a highly competitive specialty for which large numbers of highly qualified candidates are applying?
450 applicants
4 positions
40 interview slots
One PD
Creating a Better Tomorrow2008 Annual Meeting
MSPE
What does the PD want from the MSPE
Earlier release
Clearly stated, uniform performance scoring
Clearly stated and illustrated performance comparisons for
• All subjects (rotations) for all 3 years
• An overall ranking of performance at the end of the third year (at least by quartiles)
• As few adjectives and adverbs as possible
Creating a Better Tomorrow2008 Annual Meeting
MSPE
Where does the PD turn when the MSPE does not provide a meaningful comparison of performance for the candidate?
Other schools
USMLE scores
AOA status (when available)
LOR from colleagues
College transcript
Creating a Better Tomorrow2008 Annual Meeting
MSPEExample of scored/illustrated
performance by course
Nu
mb
er
of
Stu
den
ts
Creating a Better Tomorrow2008 Annual Meeting
MSPE
Example of scored/illustrated overall
performance at completion of third year
%
Stu
den
ts
Creating a Better Tomorrow2008 Annual Meeting
MSPE
A Review of 2007 MSPEs from
62 U.S. Medical Schools
Creating a Better Tomorrow2008 Annual Meeting
MSPE
SCORING CRITERIA
3 CATAGORIES
MS performance scored
MS performance compared
MS performance comparison illustrated
Creating a Better Tomorrow2008 Annual Meeting
MSPE
MS Performance Scored
0- not done
1- poorly/incompletely done
2- average (acceptable)
3- well done
4- uniformly done (excellent)
Creating a Better Tomorrow2008 Annual Meeting
MSPE
MS Performance Compared
0- not done
1- poorly/incompletely done
2- average (acceptable)
3- well done
4- uniformly done (excellent)
Creating a Better Tomorrow2008 Annual Meeting
MSPE
MS Performance Comparison Illustrated
0- not done
1- poorly/incompletely done
2- average (acceptable)
3- well done
4- uniformly done (excellent)
Creating a Better Tomorrow2008 Annual Meeting
MSPE
Overall Score for Institution
0- No attempt to follow AAMC guidelines
6- Acceptable presentation of MS performance
12- Outstanding (Gold Standard)
Creating a Better Tomorrow2008 Annual Meeting
MSPE
SCORES
# o
f In
stit
uti
on
s
Creating a Better Tomorrow2008 Annual Meeting
MSPE Study Results
20/62 (32%) U.S. Medical Schools failed to present MS Performance in acceptable format
42/62 (68%) U.S. Medical Schools present MS Performance in acceptable format
6/62 (10%) U.S. Medical Schools present MS Performance in outstanding format
Creating a Better Tomorrow2008 Annual Meeting
Timing of MSPE
Present date is November 1st
• Applications reviewed from mid-September until mid-October
• First batch of interview invitations go out 1st week of October
• Interview slots filled by 2nd to 3rd week of October
• MSPE used by PD to help rank candidates after interviews
Creating a Better Tomorrow2008 Annual Meeting
Timing of MSPE
Proposed date is October 1st
• Applications reviewed mid-September until mid-October
• Interview slots filled by 2nd or 3rd week of October
• MSPE used by PD to select resident applicants for interview
Creating a Better Tomorrow2008 Annual Meeting
Student Affairs Dean Prospective on the MSPE
Neil H. Parker, M.D.
Senior Associate Dean, Student Affairs & GME
University of California Los Angeles Geffen SOM
Creating a Better Tomorrow2008 Annual Meeting
Student Affairs Dean Prospective on the MSPE
Neil H. Parker, M.D.
Senior Associate Dean, Student Affairs & GME
University of California Los Angeles Geffen SOM
Creating a Better Tomorrow2008 Annual Meeting
MSPE – One Dean’s PerspectiveAccurate assessment of student’s performance
Perspective over three plus years
Preclinical coursework minor component and not very useful
Clinical coursework major component and should be very useful
Summary provides perspective on student, potential and overall assessment
Predictive of success as a houseofficer
Creating a Better Tomorrow2008 Annual Meeting
MSPE – Unique CharacteristicsProvides insight into what makes this student different, special, unique….
Documents the Road Travelled
Different from Student’s Activity and Background sections of ERAS as seen through the eyes of the Student Affairs Dean and Educator
Non-cognitive variables important for choosing residents - Programs three to seven years in length
Creating a Better Tomorrow2008 Annual Meeting
MSPE – Education DemographicsImportant for years of school
Leaves of absence important but very variable in how schools represent leaves and reasons
Academic difficulty and disciplinary issues
Predictive of future problems in licensure, etc
Creating a Better Tomorrow2008 Annual Meeting
MSPE – Years One and TwoToo short to be meaningful
All students about the same
Little information from course chairs
Can be left out without / not very different from transcript grades
Trumped by USMLE Step 1 score
Creating a Better Tomorrow2008 Annual Meeting
MSPE – Clinical Years
Not same as clinical transcript
Critical information on student’s performance in required clerkships
Best predictor of abilities, attitudes and future potential and problems
This is the meat of the MSPE
Essentially same as evaluations of residents by faculty in programs
Needs faculty and resident development
Creating a Better Tomorrow2008 Annual Meeting
MSPE – SummaryShort and puts residency applicant into perspective
Not specialty oriented
Not a recommendation
Would be better if was recommendation and uniform as to prediction of success as a resident
Creating a Better Tomorrow2008 Annual Meeting
MSPE – Graphs & AppendixIn Pass Fail Schools, totally useless
What does it predict
Adds multiple pages that make MSPE seem long
Appendix E – Does anyone read in selection or only when there is a problem in residency
Creating a Better Tomorrow2008 Annual Meeting
MSPE – One Dean’s PerspectiveWritten by Student Affairs Dean
169 Students
Follow AAMC MSPE recommendations
Use Automated System for Processing
Senior reviews for accuracy
1 ½ - 2 hours per student
Creating a Better Tomorrow2008 Annual Meeting
MSPE – Production Time / CostAt UCLA, production of MSPE reduced by 50% through automation
Students complete web-based survey on their activities, background and other unique characteristics – 2+ hours
Individual meeting with students 20-30 minutes, SAD
Production of Unique characteristics and Summary – SAD ( Student Affair Dean – and is sad) 1 – 1 1/2 hours
Electronic formulation of clerkships information from evaluation system – HAL
Cost $300-400 per student x 169 students = $ 45,000 +
Creating a Better Tomorrow2008 Annual Meeting
TimingStudents complete questionnaire on Activities, Background and Characteristics - August
Letters written between August and October
Students review letter for accuracy between September and mid October
Uniform release November 1
This allows for all the Third Year electives and first Sub I’s of fourth year to be received and place in the MSPE
Creating a Better Tomorrow2008 Annual Meeting
MSPEGeffen School of Medicine
Pass/Fail all Years
Nu
mb
er
of
Stu
den
ts
0
20
40
60
80
100
120
140
160
180
PASS FAIL
Creating a Better Tomorrow2008 Annual Meeting
MSPE - ProblemsPD’s and faculty increasingly busy with clincial & research productivity and ACGME requirements
Extremely useful but not appreciated by PD’s
Decisions on Interviews often decided before MSPE released
Undervalued
Clerkship evaluations require 6-8 weeks to collect
Senior electives evaluations difficult to collect and late
Creating a Better Tomorrow2008 Annual Meeting
MSPE – Perspective Cont.MSPE really only 2-3 Pages
5 -6 pages of graphs, Appendices, etc
Written comments 2-3 pages
Has the MSPE gone the way of the Physical Exam and the Medical Write-up?
Should it be reduced to Consumer Report style?
Is complete, valuable information overruled by need for bottom lines and byte size information?
Creating a Better Tomorrow2008 Annual Meeting
MSPE – Thoughts from the Weird sideConsider component release – everything short of Senior electives by October 1st
Hold ERAS process till two weeks after release of MSPE – lite
No interviews scheduled till after MSPE lite
Senior Elective evaluations supplement Dec 1st
Transcripts folded into MSPE’s
Creating a Better Tomorrow2008 Annual Meeting
MSPE - From the Far SideDon’t change, they will come around
Go back to letter of recommendation
Clerkship evaluation development
Do away with MSPE
Just send unabridged clerkship evaluations
Develop electronic MSPE that PD’s can sort for the parts they want
Creating a Better Tomorrow2008 Annual Meeting
Are we dealing with MillennialsNot the Students but PD’s – younger and have tenure of 3 years on average
Want everything electronic
Information must be in Byte size
Want it their way and don’t except No as an answer
Multitaskers who don’t have time to read 8 pages
Just give me the bottom line
Creating a Better Tomorrow2008 Annual Meeting
Angela Nuzzarello, M.D., MHPE
Associate Dean, Student Programs & Professional Development
Northwestern University Feinberg School of Medicine*
*As of January 1, 2009 – Oakland University William Beaumont School of Medicine
Creating a Better Tomorrow2008 Annual Meeting
What are we duplicating?
Unique Characteristics
• Leadership, research, community service, already in personal statement, CV, and LoRs
Academic History
• Grades already on transcript
Summary
• If summary statements are included, information has already been presented in LoRs or unique characteristics
Creating a Better Tomorrow2008 Annual Meeting
What information is unique to the MSPE?
Explanation of any extension of time
Recipient of any adverse action
Assessment of professional performance
Clerkship narratives
Comparative performance (ranking, or quartiles, etc)
Grade percentages for each clerkship
Creating a Better Tomorrow2008 Annual Meeting
The Problem with Creating MSPEs
Information from multiple sources required a significant amount of “cutting and pasting”
Students previewed MSPE as PDF and suggested changes through email or in person
Changes to document were required every time a new grade was finalized
Setting up comparative charts and tables was time consuming
Creating each student’s graphical record was also time consuming.
Creating a Better Tomorrow2008 Annual Meeting
Solutions
Web-based system allows students easy access to MSPE document
Can make suggestions for changes through web interface which decreases number of emails
Any information that is stored in a database can be put in to the MSPE template
New information (i.e. grades) continually updated
Creating a Better Tomorrow2008 Annual Meeting
Unique Features of Web-based MSPE
Email notification to students when the draft is ready; when student comments have been addressed
Forum-style system for collaboration on various sections of MSPE letter
Real-time clerkship narrative and grade inclusion
Dynamic chart and table generation
MS Word document generation
Allows tracking of progress of individual MSPEs
Creating a Better Tomorrow2008 Annual Meeting
Medical Student Performance Evaluation for
John Smith November 1, 2008
Identifying Information [Comment] # of Comments: 0
John is a fourth-year student at Northwestern University Feinberg School of Medicine in Chicago, Illinois.
Unique Characteristics EDIT [Comment] # of Comments: 0
Over the course of his medical training, John has participated in a variety of activities and gained experiences that will add to his strengths as a physician. Although all of his pursuits have been listed elsewhere on his ERAS application, several pursuits have been particularly meaningful to his development. John has done……..
Academic History EDIT [Comment] # of Comments: 0
Date of Expected Graduation from Medical School: May 2009
Date of Initial Matriculation in Medical School: July 2005
Please explain any extensions, leave(s) of absence, gap(s), or break(s) in the student’s educational program.
Not applicable
For transfer students:
Date of Initial Matriculation in Prior Medical School: Not applicable
Date of Transfer from Prior Medical School: Not applicable
For dual/joint/combined degree students:
Date of Initial Matriculation in other degree program: Not applicable
Date of Expected graduation from other degree program: Not applicable
Type of other degree program: Not Applicable
Was this student required to repeat or otherwise remediate any coursework during his/her medical education?
No
Was this student the recipient of any adverse action(s) by the medical school or its parent institution?
No
Professional Performance [Comment] # of Comments: 0
We have assessed all students’ capabilities to meet standards for: accountability, self-improvement and adaptability, appropriate relationship with patients, relationship with healthcare team, initiative, professional demeanor, and behavior under stress. Unless otherwise noted, all students have met the stated objectives for professionalism at Feinberg.
Academic Progress EDIT [Comment] # of Comments: 0
John earned grades of Pass in all preclerkship courses and recorded a score of 239 on Step I of the USMLE exam.
Preclinical/Basic Science Curriculum [Comment] # of Comments: 0
(Details are provided in Appendix E.)
Core Clinical Clerkships EDIT [Comment] # of Comments: 0
Following are the unedited narrative evaluations of John’s performance on the successive clerkships and electives. Appendix B summarizes this information graphically. NEUROLOGY: PASS 7/2/2007 - 7/27/2007
John did a very good job during his neurology rotation. This was his first clinical clerkship. He demonstrated good history and examination skills. His histories were thorough and he was able to pay attention to detail. He also demonstrated appropriate techniques in the neurological examination. His oral presentations were well organized and clear. He appeared to have a good fund of knowledge based on his clinical interactions and discussion; he scored below the mean on his written shelf examination. He was motivated and eager to learn. He was hard working, organized and responsible in his work. He had great interpersonal skills and interacted very well with his team members and patients. For his overall performance on this rotation, he is awarded the grade of PASS.
PRIMARY CARE: HONORS 7/30/2007 - 8/24/2007 John is a bright medical student who did an outstanding job in all aspects of the Primary Care Clerkship. He has excellent interactions with patients and staff alike. His history and physical exam taking skills are on par with a 4th year medical student. His oral case presentations are concise and well organized. He has a solid knowledge base and very good clinical judgment. In Weekly Report he was an active participant with frequent insightful comments. His Learner Centered Learning Goal presentation on depression screening was outstanding. The preceptor wrote, "Not
APPENDICES
Appendix A: Comparative Performance in Preclinical/Basic Science Coursework [Comment] # of Comments: 0
During the first and second years of the curriculum, the only grades assigned to students are Pass or Fail. Students must receive a grade of “Pass”, in each preclinical course before beginning the clerkships.
Appendix B: Graphic Representations of Comparative Performance in Core Clinical Clerkships [Comment] # of Comments: 0
Grades in Required M3 Clerkships/Courses
John’s grade for each clerkship is indicated inside the bracket beside the clerkship name.
Table 1- Grade Percentages for Each Required M3 Clerkships/Courses
IDM3 MED NEURO OBGYN PEDIA PRIMCARE PSYCH SURG
H 25 36 34 22 23 31 24 31
HP 16 27 30 34 29 26 40 18
P 59 37 36 43 46 43 36 48
F 0 0 0 1 2 0 1 2
Creating a Better Tomorrow2008 Annual Meeting
Comments from students
“I think it worked well. The ability to send e-comments was very helpful and response time was quick.”
“I think it was great that we could view our MSPEs and any comments I had were immediately addressed.”
“I feel that the bi-directional feedback and electronic format made this quite efficient.”
“I liked that everything was web based.”
Creating a Better Tomorrow2008 Annual Meeting
Maria C. Savoia, M.D.
Vice Dean for Medical Education
Professor of Medicine
University of California San Diego School of Medicine
Creating a Better Tomorrow2008 Annual Meeting
““BOTH SIDES NOWBOTH SIDES NOW””
A VIEW ABOVE THE FRAYA VIEW ABOVE THE FRAY&&
A REVIEW OF THE RELEVANT A REVIEW OF THE RELEVANT LITERATURELITERATURE
Creating a Better Tomorrow2008 Annual Meeting
Do MSPEs Accurately Reflect Do MSPEs Accurately Reflect Medical School Performance?Medical School Performance?
Are they based on accurate data?
Are data accurately reflected in letters?
Creating a Better Tomorrow2008 Annual Meeting
Based On Accurate Data?Based On Accurate Data?
Evaluation often done by the least experienced
› Experience important in preceptor’s ability to assess cognitive levels Taylor C., Lipsky MS Fam Med 1990 Jul-Aug;
(2294)296-8
Creating a Better Tomorrow2008 Annual Meeting
Based On Accurate Data?Based On Accurate Data?
For data to be reliable, either many observations or many observers
For data to be valid, they have to measure what they say they measure
› “Knowledge and skill” in a Psychiatry clerkship correlated with personality characteristics and not with NBME Subject exam or OSCE scores Chibnall JT, BlaskiewiczRJ. Acad Psychiatry 2008
May-Jun;32(3); 199-205
Creating a Better Tomorrow2008 Annual Meeting
Based On Accurate Data?Based On Accurate Data?
“Objective” data often maligned
Although many recent studies have not demonstrated gender or ethnic biases… it is better to be good looking and thin & to work closely with the person who evaluates you
› Supervisors grade higher than control graders McKinstry BH, et. al. BMC Medical Education 2004,
4;28
Creating a Better Tomorrow2008 Annual Meeting
Based On Accurate Data?Based On Accurate Data?The great group of “Very Good”
› In Pass/Fail curriculum, how do you distinguish among those who excel &those who scrape by?
› P/F grading puts students at a disadvantage in applying to General Surgery programs Dietrick JA, et. al., Am J Surg 1991 Jul;162(1);63-6
› In a survey of 110 program directors across specialties, only 8 favored a P/F system Provan JL, Cuttress L, Can Med Assoc J Oct.1, 1995;
153 (7)
Creating a Better Tomorrow2008 Annual Meeting
1992 55%
2001 65%
2005 75%
“Adequate MSPEs”
Are Data Accurately Reflected?Are Data Accurately Reflected?
Hunt DD, et. al., Acad Med 2001 Jul;76(7): 727-33
Shea JA, et. al., Acad Med 2008 Mar; 83(3):284-91
Creating a Better Tomorrow2008 Annual Meeting
Are Data Accurately Reflected?Are Data Accurately Reflected?
We live in a “fantasy land” where all students “will make a fine physician,” & are “above average” “with excellent interpersonal skills.”
Dean’s Letter for Adolph Hitler: “A natural leader…good communication skills…assisted in the development of technological advances…likes to find solutions to problems”
Friedman RB, NEJM 1983 308;651-3
Creating a Better Tomorrow2008 Annual Meeting
Are Data Accurately Reflected?Are Data Accurately Reflected?
Negative information found on transcript not mentioned in Dean’s Letter 34% of time.
› Conclusion: “Some deans suppress negative information in their letters and potentially obfuscate the residency selection process.” Edmond M, et. al. 1999 Acad Med 74(9);1033-5.
Creating a Better Tomorrow2008 Annual Meeting
Are Data Accurately Reflected?Are Data Accurately Reflected?
Excellent: of superior merit, exceptionally good
56/75 schools used “Excellent” as a category.
For >50% of schools, a student described as excellent might be in bottom half of class. For 2 schools, all in “Excellent” category were in the bottom half of the class
Naidich JB, et. al., Acad Rad 2007 14; 1121-6.
Creating a Better Tomorrow2008 Annual Meeting
Are Data Accurately Reflected?Are Data Accurately Reflected?
Professionalism comments evaluated in 293 MSPEs for 2005 graduates.
70% mentioned professionalism, with fewer than 1% of comments that could be considered negative.
› Conclusion: “Most professionalism comments in MSPE’s are generic and somewhat bland…” Shea JA, et. al., Acad Med 2008 Oct. 83(10
Suppl);S1-4
Creating a Better Tomorrow2008 Annual Meeting
What Factors Accurately Predict What Factors Accurately Predict Residency Performance?Residency Performance?
Univ. Rochester looked at 2 graduating classes, divided into:• “Outstanding” (upper quartile),• “Excellent” (second quartile),• “Very Good” (lower 2 quartiles)• “Good” (lowest few percentiles)
Found “Dean’s Letter rankings were a significant predictor of later performance.”
Lurie SJ, et. al., Teach Learn Med 2007 19(3);251-6
Creating a Better Tomorrow2008 Annual Meeting
What Factors Accurately Predict What Factors Accurately Predict Residency Performance?Residency Performance?
At the Uniform Services University, appearing before Standings & Promotions Committee (for any reason) was predictor of below average performance during internship.
Durning SJ, et. al., Teach Learn Med 2008 20(3);267-72
Creating a Better Tomorrow2008 Annual Meeting
What Factors Accurately Predict What Factors Accurately Predict Residency Performance?Residency Performance?
At the Medical University of South Carolina, GPA (preclinical + clinical) + interpersonal score on Step 2CS best predicted internship performance but only accounted for 30% of variance.
Taylor ML, et. al., Acad Med 2005 80(5); 496-501
Creating a Better Tomorrow2008 Annual Meeting
What Factors Accurately Predict What Factors Accurately Predict Residency Performance?Residency Performance?
Otolaryngology: “excellent academic performance in medical school”
Calhoun KH, et. al., Otolaryngol Head Neck Surg 1997 116(6 pt1); 647-51
Emergency Medicine: medical school attended, MSPE and “distinctive factors”
Hayden SR, et. al., Acad Emerg Med 2005 12(3); 206-10
Creating a Better Tomorrow2008 Annual Meeting
What Factors Accurately Predict What Factors Accurately Predict Residency Performance?Residency Performance?
Radiology: “Noncognitive factors were as important as cognitive factors,” “Objective measures, such as NBME scores, failed to adequately predict residents’ performance.”
Wood PS, et. al., Invest Radiol 1990 25(7);855-9
Radiology: Success on ABR exam predicted by grades and NBME scores. No good predictors of clinical performance.
Boyse TD, et. al., Acad Radiol. 2002 9(4);437-45
Creating a Better Tomorrow2008 Annual Meeting
What Factors Accurately Predict What Factors Accurately Predict Residency Performance?Residency Performance?
PM&R: 205 residents at the University of Washington. Clinical residency performance predicted by clerkship honors grades (p=.0001). Probation was predicted by failing a basic science course (p=.0001)
Amos DE, Massagli TL, Acad Med 1996 71(6);678
Creating a Better Tomorrow2008 Annual Meeting
What Factors Accurately Predict What Factors Accurately Predict Residency Performance?Residency Performance?
Orthopedics: “Academic score,” using only objective measures, made screening process more objective but didn’t correlate with outcomes of the training program.
Dirschl DR, et. al., Clin Orthop Relat Res 2006 449;44-9
Surgery: “Dean’s letter seems to give the most accurate picture of applicant’s suitability for residency training.”
Naylor RA, et. al., Arch Surg 2008 143(7); 647-52
Creating a Better Tomorrow2008 Annual Meeting
PD’s: Ignore at your own risk!With an edge of annoyance, Hunt asked (Barrows) about Swango. “What kind of guy did you send us?”
Barrows said that Hunt should have seen plenty of warning flags in Swango’s dean’s letter. “Well,” Hunt retorted, “I don’t read dean’s letters.”
….Soon after, Hunt called back: he’d found the letter
“Oh, my God,” Hunt said. “You’re right. You did tell me.”
Stewart, JB, Blind Eye , 1999, pps. 59-60
Creating a Better Tomorrow2008 Annual Meeting
What Factors Accurately Predict What Factors Accurately Predict Residency Performance?Residency Performance?
Surgery: Only Step 2 scores (and not Step 1, overall GPA, AOA status, or 3rd year clerkship grade) predicted internship performance for 87 graduates from Wash U.
Andriole DA, et. al., Am J Surg 2004 188;161-4
Anesthesia: Best predictor of academic performance number of years spent in another specialty. Step 1 score inversely correlated with in service exam score
Warrick SS, Crumrine RS, J Med Ed 1986 61(7); 591-5
Creating a Better Tomorrow2008 Annual Meeting
What Factors Accurately Predict What Factors Accurately Predict Residency Performance?Residency Performance?Anesthesia: “Selection committee score,” which correlates with Step 1 score, doesn’t correlate with resident performance.
Metro DG, et. al., Anesth Analg 2005 100;502-5
Pediatrics: In the case of 69 residents at UVA, medical school grades, performance on standardized exams, interviews and match list rankings did not predict clinical performance during residency.
Borowitz SM, et. al., Arch Pediaatr Adolesc Med 2000 154;256-60
Creating a Better Tomorrow2008 Annual Meeting
In Conclusion:In Conclusion:
We need better measures of performance during medical school.
Multiple types of evaluation may provide a more accurate picture of performance.
It’s about the Match. The best program may not be the best program for a particular student.
Honesty is the only policy.
Creating a Better Tomorrow2008 Annual Meeting
In Conclusion:In Conclusion:
More categories of grading may help students in the Match
The MSPE is as accurate a measure as any in predicting residency performance.
NBME Step 2 but not NBME Step 1 scores have occasionally been found to predict clinical performance
Creating a Better Tomorrow2008 Annual Meeting
In Conclusion:In Conclusion:
Creating a Better Tomorrow2008 Annual Meeting
Question # 5
Do you favor continuing the MSPE in some form?
1. Yes
2. No
3. Unsure
Creating a Better Tomorrow2008 Annual Meeting
Question # 6
If we keep the MSPE, do you favor changing the current format of the MSPE?
1. Yes
2. No
Creating a Better Tomorrow2008 Annual Meeting
Question # 7
Would you remove the unique characteristics section of the MSPE?
1. Yes
2. No
Creating a Better Tomorrow2008 Annual Meeting
Question # 8
Would you remove the clerkship narratives from the MSPE?
1. Yes
2. No
Creating a Better Tomorrow2008 Annual Meeting
Question # 9
Do you favor changing the MSPE release date?
1. Yes
2. No
Creating a Better Tomorrow2008 Annual Meeting
Question # 10
If you favor changing the MSPE release date, which would be the optimal date?
1. September 1st
2. September 15th
3. October 1st
4. October 15th
5. November 1st
6. January 11th
Creating a Better Tomorrow2008 Annual Meeting
Question # 11
Would you favor a single date for MSPE release and the start of ERAS upload?
1. Yes
2. No
Creating a Better Tomorrow2008 Annual Meeting
Question # 12
Would you favor a uniform release date of
Oct 1st for transcripts?
1. Yes
2. No
Creating a Better Tomorrow2008 Annual Meeting
Question # 13
Do you think the GSA steering committee should propose a change in the MSPE, develop a survey on the MSPE, or should we a new advisory group be convened?
1. Propose an MSPE change
2. Develop a survey
3. Convene a new advisory group
Creating a Better Tomorrow2008 Annual Meeting
Thank you for coming!!