validation of the manitoba practice assessment program cme congress: june 1, 2012
DESCRIPTION
Validation of the Manitoba Practice Assessment Program CME Congress: June 1, 2012. MPAP Program Team. Jose Francois MD, Associate Dean Marilyn Singer MD, Director Jillian Horton MD, Assistant Director Brenda Stutsky PhD, Program Advisor Debbie Lemkey, Program Assistant - PowerPoint PPT PresentationTRANSCRIPT
Validation of the Manitoba Practice Assessment ProgramCME Congress: June 1, 2012
MPAP Program TeamJose Francois MD, Associate DeanMarilyn Singer MD, DirectorJillian Horton MD, Assistant DirectorBrenda Stutsky PhD, Program AdvisorDebbie Lemkey, Program AssistantRobert Renaud PhD, Dept. Ed. Admin., Foundations & Psychology
Conflict of Interest: Nothing to disclose: Pilot study self-funded
Background/Purpose• MPAP developed based on need from the College of
Physicians and Surgeons of Manitoba (CPSM)• Involves the assessment of practicing physicians on the
CPSM conditional register
• The purpose of the pilot study was to test selected MPAP processes and examine the preliminary reliability and validity of the practice assessment tools
Method• Mixed methods design• 6 Stages:
1. Training of assessors2. Selection of family physicians and 1st year family
medicine residents3. Self-assessment process and 360 degree surveys4. Onsite assessment5. Collation of data and feedback to the physician
candidates6. Final debriefing session
Results: Assessor Online Training (N = 8)Length of time to complete the self-directed online learning activities:
– Mean 7.13 hrs. (Range of 4-9 hrs.)
Enjoyed online format:– Yes 75%– It was okay 25%– No 0%
After completing the online activities, assessors believed they had a good understanding of the various assessment strategies/processes:
– Yes 100%– No 0%
Results: Assessor Face-to-Face Training (N = 12)I am ready to begin to assess physicians:
– Yes, and I do not require additional training: 50%– Yes, but I will need to shadow/buddy an experienced
assessor or participate in a pilot/mock assessment: 50%– No, I will need additional guidance and training: 0%
I recommend that the Assessor Workshop be modified:– Yes: 0%– No: 100%
Results: Comparison of Mean Scores between Family Physicians and Residents & Correlation to PAR
Note: Higher scores reflect more positive ratings
CanMEDS Roles Overall Onsite Visit Scores (range 1-2)
360 Degree Survey Scores Self-Assessment Scores
(range 1-4) Physician Colleagues (range 1-4)
Interprofessional Colleagues (range 1-4)
Patients (range 1-4)
FP (5) R (1) FP (5) R (4) FP (5) R (3) FP (5) R (1) FP (5) R (4) Medical Expert 1.99 2.00 3.39 2.71 3.60 3.23 3.70 3.36 2.74 2.65 Communicator 2.00 2.00 3.39 2.62 3.55 3.38 3.60 3.21 2.97 2.83 Collaborator 2.00 2.00 3.45 2.68 3.47 3.31 3.64 3.75 2.69 2.78 Manager 2.00 -- 3.31 2.28 3.52 3.22 3.20 2.94 2.47 2.38 Health Advocate 2.00 2.00 2.95 2.74 3.53 3.21 3.50 3.20 2.44 2.61 Scholar 2.00 -- 3.31 2.95 3.60 2.80 3.44 3.00 2.64 2.19 Professional 2.00 2.00 3.23 3.02 3.60 3.39 3.72 3.58 2.68 2.97 Correlation between Mean total MPAP scores and Mean total Alberta scores
0.96 0.72 0.61
Results: Debriefing Session1. Overall, what did you like about the MPAP Process?
– Interprofessional collaboration, training process, usability and variety of tools
2. Overall, what could improve?– Office scheduling, patient selection, access to EMR/charts, use of
EMR for data, space, length of time between training and onsite assessment
3. How did you feel during the assessment process?– Assessors: uncomfortable, anxious → confident over time– Candidates: nervous, pre-occupied with other roles, “like a resident
again”4. If you were to provide guidance to physicians
undergoing the assessment process, what would you tell them?– Relax, process allows your strengths to shine, encouraging
comments - want them to succeed
Conclusions• Blended learning was effective• Provided valuable training experience for
assessors in a low-stakes environment• Limited generalizability given small sample
size• Validity and reliability and of tools is promising
– Continue to work on validity and reliability • Need only minor changes to the processCopy of Presentation (Follow MPAP links):
www.umanitoba.ca/cpd