The Campaign for McMaster University The Campaign for McMaster University

§  THE MINI MILESTONES ASSESSMENT

(MINI-MAS)

A DIRECT OBSERVATIONAL TOOL TO ASSESS CLINICAL MILESTONES IN THE ERA OF COMPETENCY-BASED

EDUCATION

MOYEZ B. LADHANI MD, FAAP, FRCPC

Introduction: CBME

the ‘tea-steeping model’, whereby medical educators “…put the student (tea) in medical school (hot water) for a fixed period of

time and, voila! After a historically determined interval of time, we assume a competent practitioner, like a good cup of tea, will

result”

Hodges, B. D. (2010). A tea-steeping or i-doc model for medical education?. Academic Medicine, 85(9 Suppl), S34-44.

The Campaign for McMaster University The Campaign for McMaster University

Introduction: CBME

Competency-based education is an approach to preparing physicians for

practice that is fundamentally oriented to graduate outcome abilities and

organized around competencies derived from an analysis of societal and patient needs. It deemphasizes time-based training and promises

greater accountability, flexibility, and learner centeredness

Frank, J. R., Mungroo, R., Ahmad, Y., Wang, M., De Rossi, S., & Horsley, T. (2010). Toward a definition of competency-based education in medicine: A systematic review of published definitions. Medical Teacher, 32(8), 631-637.

The Campaign for McMaster University The Campaign for McMaster University

The International Conference on Residency Education | La Conférence internationale sur la formation des résidents

What the beaver must do

§  Statement of learning outcome

§  Communication with staff and students

§  Educational strategies

§  Learning opportunities

§  Course content

§  Student progression

§  Assessment

§  Educational environment

§  Student selection §  Harden, R. M. (2007). Outcome-based education--the ostrich, the peacock and the beaver.

Medical Teacher, 29(7), 666-671.

4

Introduction: Assessment  

KNOWS

KNOWS HOW

SHOWS HOW

DOES

MCQs

Standardized Patients

Faculty Observation

Impact on Patient

Clinical vignettes

Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65(9 Suppl), S63-7.

Introduction: WBA

“…the assessment of trainees and physicians across the

continuum of day-to-day competencies and practice in

authentic, clinical environments…It enables the evaluation of

performance in context” Kogan, J. R., & Holmboe, E. (2013). Realizing the promise and importance of performance-based assessment. Teaching & Learning in Medicine, 25(Suppl 1), S68-74.

The Campaign for McMaster University The Campaign for McMaster University

Introduction: WBA

The In-Training Evaluation Report (ITER):

q  Does not discriminate (Gray, 1996; Holmboe & Hawkins, 1998 )

q  Completed retrospectively (Turnbull et al., 1998).

q  Often faculty who have not observed are completing the form (Epstein, 2007)

q  Halo effect (Wilkinson & Wade, 2007)

q  Raters fail to use the entire scale (Gray, 1996)

The Campaign for McMaster University The Campaign for McMaster University

Introduction: WBA

mini-CEX

•  The scale used in the mini-CEX is designed for linear gradations of performance.

•  The scores do not give the evaluators a point of reference to

help align a trainee to a category or score

•  Faculty assessors resort to norm-referencing.

Crossley, J., & Jolly, B. (2012). Making sense of work‐based assessment: Ask the right questions, in the right way, about the right things, of the right people. Medical Education, 46(1), 28-37.

The Campaign for McMaster University The Campaign for McMaster University

Introduction: WBA

mini-CEX

•  The raters do not use the full nine-point scale.

•  The distribution is right shifted towards the higher end of the

scale.

•  The use of the lower end of the scale is infrequent raising

concerns about identifying weaknesses.

•  Individual competencies tended to be highly correlated. Hawkins, R. E., Margolis, M. J., Durning, S. J., & Norcini, J. J. (2010). Constructing a validity argument for the mini-clinical evaluation exercise: A review of the research. Academic Medicine, 85(9), 1453-1461.

The Campaign for McMaster University The Campaign for McMaster University

Introduction: Designing a Better Tool •  WBA assessment tools should have anchors measuring the

trainees’ level of progression and development

•  Assessors make more reliable judgments of performances they can see clearly in a particular context or activity.

•  The tool should focus on the competence relevant to the activity, and avoid having multiple competencies to assess at the same time

Crossley, J., & Jolly, B. (2012). Making sense of work‐based assessment: Ask the right questions, in the right way, about the right things, of the right people. Medical Education, 46(1), 28-37. Crossley, J., Davies, H., Humphris, G., & Jolly, B. (2002). Generalisability: A key to unlock professional assessment. Medical Education, 36(10), 972-978.

The Campaign for McMaster University The Campaign for McMaster University

Dreyfus and Dreyfus

Figure 2 General Curve of skills Acquisition. (ten Cate et al., 2010)

The Campaign for McMaster University The Campaign for McMaster University

Introduction: Faculty Development

The Campaign for McMaster University The Campaign for McMaster University

Purpose

The purpose of this study is to: 1.  Implement a competency-based curriculum into the McMaster

University, pediatric residency program. 2.  Develop a tool, the Mini Milestones Assessment (Mini-MAS)

to assess six medical competencies and progression through milestones using the Dreyfus Developmental Model.

3.  Test the psychometric theories to assess the reliability,

validity, acceptability and feasibility of the Mini-MAS tool.

The Campaign for McMaster University The Campaign for McMaster University

Research Question

Is the Mini-MAS a valid, reliable, acceptable and feasible tool for

the assessment of milestones in history taking, physical exam

skills, clinical reasoning, communication and collaboration for

PGY 1 and PGY 4 pediatric residents at McMaster Children’s

Hospital?

The Campaign for McMaster University The Campaign for McMaster University

Methods

•  Implement CBME curriculum.

Clinical Exposure-Total 22 weeks: q  CTU-4 weeks q  Community Brampton-4 weeks q  Community St. Joseph’s Healthcare-4 weeks q  Float call at McMaster-6 weeks

The Campaign for McMaster University The Campaign for McMaster University

Methods

•  12 PGY 1 residents at McMaster Children’s Hospital were

required to complete 40 observations

•  10 history taking, 10 physical exam

•  5 clinical reasoning, 5 communication with families,

5 communication with staff and 5 collaboration

•  during the 2013-2014 academic year.

•  9 PGY 4 residents were also observed for the same

competencies over the same time period. This group was

required to complete 15-20 encounters.

The Campaign for McMaster University The Campaign for McMaster University

Methods:

•  Following the study period, a survey was completed by the

residents and faculty to assess acceptability and feasibility of

the Mini-MAS tool.

•  Kane’s validity framework which is divided into four

components (scoring, generalization, extrapolation and

decision) was used to evaluate the Mini-MAS tool.

The Campaign for McMaster University The Campaign for McMaster University

Procedures

§  Learners

§  Assessors

§  Other Data

The Campaign for McMaster University The Campaign for McMaster University

Results

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

12 PGY 1

474 observations

39 removed

435 (mean 36)

9 PGY 4

1 lost book 1 LOA

7 PGY 4

96 forms (mean of 16)

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

Discussion

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

Scoring

•  Learners met requirements 90% and 100% completion rates

•  76% by faculty 24% by residents

•  Residents assessors more lenient but not significant

•  Trend for observations occurring in later half of year

•  Faculty engagement 3.9 PGY 1 and 4.6 PGY 4

The Campaign for McMaster University The Campaign for McMaster University

Scoring

•  Faculty completed the forms in a timely manner 4.8, 5.3, 5.3

•  Faculty provided valuable feedback 5.3, 5.0, 5.6

•  Faculty felt appropriately trained, though wanted more training

on providing feedback

•  Scale was used appropriately 2-5 for PGY 1 and 3-5 fro PGY 4

The Campaign for McMaster University The Campaign for McMaster University

Scoring

•  Learners started with different skill levels

•  Scores also help determine where a PGY 1 resident should be

•  Individual competencies did not correlate-positive finding

•  Faculty and residents found descriptors were long and

sometime vague.

The Campaign for McMaster University The Campaign for McMaster University

Generalization

•  435 encounters for PGY 1 (mean=36) and 96 encounters for PGY 4 ( mean =16)

•  45 assessors including the 8 senior residents

•  Wide variety of clinical cases

•  CTU, ER, SRC, Community Office

The Campaign for McMaster University The Campaign for McMaster University

Generalization

•  The G coefficient overall was 0.8 for the PGY 1 group

•  The variance analysis showed the majority of the variance

was from the trainee as would be expected

•  For the PGY 4 group, the G coefficient was 0.5

The Campaign for McMaster University The Campaign for McMaster University

Generalization

•  A D-study conducted showed increasing the number of

observations to 10-12 could increase the G-coefficient to

acceptable levels for history taking, communication with

families, communication with health care professionals and

clinical reasoning

The Campaign for McMaster University The Campaign for McMaster University

Extrapolation

§  Involved observation of what really happens in clinical

practice across a variety of settings.

§  Progression of skills for PGY 1 residents through the year

§  PGY 4 scores increased through the year but not significant

§  The significant difference between PGY 1 and PGY 4

residents overall and in all the competencies

The Campaign for McMaster University The Campaign for McMaster University

Extrapolation

§  This study did not look at correlation with exit high stakes

exams nor how residents do in practice

§  There were comparisons done with concurrent assessment

tools used in the program, the mini-CEX, MCQ, SAQ and

OSCE exams

The Campaign for McMaster University The Campaign for McMaster University

The Campaign for McMaster University The Campaign for McMaster University

Decision

§  Progression of scores through the academic year and that

there is a difference between levels of training makes these

scores defendable.

§  Residents and faculty both reported that the implementation

of the tool improved the frequency of observation,4.0, 4.9, 5.1

and valuable feedback was provided 5.3, 5.0, 5.6

The Campaign for McMaster University The Campaign for McMaster University

Decision

§  The PGY 4 residents further reported the assessment

process influenced their education 5.1

§  Did not look at the effect remediation and improvement in

scores

The Campaign for McMaster University The Campaign for McMaster University

Acceptability and Feasibility

§  High completion rate

§  Residents and faculty satisfied with tool 3.8, 4.6, 5.3

§  Faculty satisfaction with tool 7.7/9

§  One lost book

The Campaign for McMaster University The Campaign for McMaster University

Conclusion

§  We successfully implemented a CBME pilot program in our

residency

§  The Mini-MAS added as a formative assessment mode to a

multi-modal assessment program will benefit the trainee,

informing them on where they stand compared to their level of

training, what competencies they can improve on and how

they can do that.

The Campaign for McMaster University The Campaign for McMaster University

Conclusion

§  Work-based assessment tool is one that assess trainees

across the continuum of competencies in clinical

environments enabling the evaluation of performance in

context

§  Kogan, J. R., & Holmboe, E. (2013). Realizing the promise and importance of performance-based

assessment. Teaching & Learning in Medicine, 25(Suppl 1), S68-74.

The Campaign for McMaster University The Campaign for McMaster University

Future Direction

•  Having scheduled assessments weekly.

•  More assessment by senior residents or fellows to improve

acceptability.

•  Simplifying and shortening the anchors.

•  Continued faculty training with an emphasis on effective

feedback.

The Campaign for McMaster University The Campaign for McMaster University

Future Direction

•  Expanding the assessment of trainees to all levels of training

and all rotations.

•  Consider different competencies to assess for different levels

of training.

•  Further studies to assess concurrent validity.

The Campaign for McMaster University The Campaign for McMaster University

Thank you

§  Dr. Kelly Dore

§  Dr. Meghan McConnell

§  Dr. Karen McAssey

§  Dr. Jonathan Sherbino

§  Sharyn Kreuger

§  Pediatric Residents

§ My Family

§ My Online Classmates

DISCUSSION/QUESTIONS