nancy piro, phd – no conflicts of interest to disclose kim walker, phd - no conflicts of interest...

Nancy Piro, PhD – No conflicts of interest to disclose

Kim Walker, PhD - No conflicts of interest to disclose

SES:096Voice of the Director: Your experience on what works and doesn’t work with CCC meetings

Nancy Piro, PhDEducation Specialist/Program ManagerStanford University Medical Center

Kim Walker, PhDInstructional DesignerSchool of Medicine, Stanford University

Confidential – For Discussion Purposes Only

Session Overview

Introduction and background on multi-institution survey

Report on survey results from Program Directors at multiple institutions on the CCC process and outcomes

− Emergent themes on directors’ experiences and their roles in the CCC

Quantitative outcomes

Qualitative outcomes

− What directors tell us was ineffective in the CCC process

− Best practices for coordinators supporting a successful CCC

Action planning for implementation of best practices


Objectives & Session Focus

This session will focus on understanding and discussing the results from a multi-institution survey on CCC practices from the Program Directors’ perspective.

At the conclusion, participants will be able to:

− Review their own practices in light of the survey outcomes

− Identify best practices that best suit their program or institution needs

− Develop and take home an action plan enabled by these best practices to better facilitate the work of their CCCs


Background


Milestones


Clinical Competency Committee

How the CCC does its work is decided by the Program Director


TEMPLATE

Confidential – For Discussion Purposes Only 9

Survey Idea – from Inspiration to Implementation


Survey – Fun Facts

What:

− A 14 Question* web-based survey with both quantitative and open ended questions was developed to assess the program director and coordinator experience so far with respect to CCCs.

When and To Whom:

− Between November 12, 2014 and January 15, 2015, the survey link was distributed to a wide sample of directors and coordinators (both within Stanford and across the country to the 15 largest institutions, and to PC networks in OB/GYN, Anesthesia, and Surgery)

Analytic Methods Used – Both Qualitative and Quantitative Analyses were used.

Response: 135 Program Directors and 325 Program Coordinators responded to our survey by January 15, 2015

* Questions detailed on the following slide


Survey Questions

1. What's your program?

2. Are you a voting member of your CCC?

3. During your CCC meeting, how much time (on the average) per resident was spent on the milestone assessments?

4. What actions has your program taken to better align your training program with your milestones (e.g., changed curriculum, added didactics, linked current evals to milestones, used spreadsheets to aggregate evaluation data, added new evaluation tools)?

5. What data was used in your CCC meeting for trainee assessment?

6. What data was more valuable in your overall milestone assessments?

7. Please describe any aspects of the milestone evaluation process that you thought were particularly beneficial.


Survey Questions (2)

8. Please describe any aspects of the CCC process that were particularly cumbersome.

9. To what extent do milestone evaluations provide for a more thorough semi-annual evaluation?

10.To what extent do milestone evaluations increase our documentation of trainee strengths?

11. Milestone evaluations increase our documentation of trainee weaknesses/areas for improvement.

12.The CCC process for trainee milestone evaluations is worth the time/effort.

13. If not worth the time and effort, why not?

14.Lastly, what did we miss? Are there any other areas of concern or suggestions?


How to Apply to Your Institution

CURRENT PRACTICE (Starting State)

POSSIBLE PRACTICE(End State)

CCC Membership and Size

Faculty Pre-work

Alignment of Milestones

Evaluation Tools

CCC Process

Challenges


Q1. Distribution of Programs Surveyed – 60 programs responded

Anesthesiology

Cardiovascular disease

Emergency medicine

Internal medicine

Neonatal-perinatal medicine

Nephrology

Obstetrics and gynecology

Orthopaedic surgery

Otolaryngology

Pathology-anatomic and clinical

Pediatric cardiology

Pediatric infectious diseases

Pediatric nephrology

Pediatrics

Physical medicine and rehabilitation

Urology

Other (1 response)

Other (2 responses)

0.00% 10.00% 20.00% 30.00% 40.00%

3.31%

3.31%

3.31%

4.96%

2.48%

3.31%

3.31%

3.31%

2.48%

2.48%

3.31%

3.31%

2.48%

3.31%

2.48%

2.48%

20.75%

29.70%


Q2. PDs - voting member of your Clinical Competency Committee (CCC)?

Yes No0%

25%

50%

75%

100%

83.97% 16.03%


PC Survey: Who is on your program's CCC?

Program Director

Associate Program Direc-tor

Chief Resident(s)

Rotation Director

Program Faculty Member

Program Director from other program

Attending who does sign out

Nurse

Case Manager

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

88.93%

69.74%

7.01%

25.83%

89.30%

7.38%

21.03%

8.86%

2.21%


Q3. How much time (on the average) per resident was spent on the milestone assessments?

1-10 minutes

11-20 minutes

21-30 minutes

31-40 minutes

41-60 minutes

>60 minutes

0.00% 10.00% 20.00% 30.00% 40.00% 50.00%

23.08%

41.54%

18.46%

9.23%

5.38%

2.31%


Q4. Alignment of milestones with training: Emergent Themes

18

New forms of evaluation (n=82) New curriculum/didactics (n=26)



New direct observation tools

“Started surgical skill evaluations”

“Created a computer based tool to provide instant feedback and document milestone progress.”

“…ramped up eval tools over time (mini CEX, tech skills assessment)…”



20

Dashboard Spreadsheets

“Spread sheets, color coded resident measures such as conference attendance”

“created spreadsheet to make sure our evaluation tools capture all the milestones”


Q5. Data used in CCC meetings for trainee assessment

Aggregate Direct Milestone Evalua-

tions

Aggregate Rotation Evaluations

Staff or patient (360) Evaluations

Technical Skills Assessments/Mini-

CEX's

Scholarly Activities

Case Volumes

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

65.38%

93.85%

76.92%

44.62%

63.08%

36.92%


Aggregate milestone evaluation scores

Board scores

Conference Attendance

Continuity clinic chart review

Curriculum Vitae

Didactic attendance

Duty Hour Summary

Duty Hour reporting compliance

Evaluation completion rate

In service exam scores

Learning modules completion

Medical records outstanding OP reports

Medical student evaluations

Mock oral results

Narrative evaluation comments

Peer Evaluations

Presentations

Previous evaluation

Procedure logs

Research projects

Self assessment scores

Spider diagrams with comparison data

Continuity clinic attendance

Data included in the CCC Review: Overall Emergent Themes


Q6. What data was more valuable in your overall milestone assessments?

Quantitative ag-gregate evaluation

scores

Qualitative comments and direct observa-tions by CCC

members

Qualitative and quantitative data

were equally valuable

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

12.40%

46.51%

41.09%


Q7. Beneficial aspects of the milestone evaluation process

24

“Group discussion to clarify our own thinking and understanding of the

Milestones and their meaning”

Committee-based review and discussion


“…recognition of deficits in our teaching”





26

“Multiple opinions strengthen the rigor of the process.”



“…provides a much more comprehensive look at an

individual resident's performance.”





“

“common language”

“more objective evidence”

“more detailed evaluation”

More objective and focused reviews


Identification of trainee strengths and areas for improvement




Identification of gaps in education (curriculum)


Q8. Cumbersome aspects: Emergent Themes

Milestone interpretation

"what are they really getting at

here?"

“contrived”

“nebulous”

“uncalibrated”



Time Consuming



33

Acquisition of sufficient data

Another 50 milestone evals due tomorrow?!

I can’t take it anymore!!!


Q9. Milestone evaluations provide for a more thorough semi-annual evaluation.

Very Strongly Agree

Strong Agree

Agree

Disagree

Strongly Disagree

Very Strongly Disagree

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

6.87%

19.08%

41.22%

16.79%

9.92%

6.11%


Q10. Milestone evaluations increase our documentation of trainee strengths.

Very Strongly Agree

Strong Agree

Agree

Disagree

Strongly Disagree


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

6.11%

21.37%

38.93%

20.61%

7.63%

5.34%


Q11. Milestone evaluations increase our documentation of trainee weaknesses/areas for improvement

Very Strongly Agree

Strong Agree

Agree

Disagree

Strongly Disagree


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

5.34%

24.43%

40.46%

18.32%

9.16%

2.29%


Q12. The CCC process for trainee milestone evaluations is worth the time/effort.

Very Strongly Agree

Strong Agree

Agree

Disagree

Strongly Disagree


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

6.92%

16.92%

40.00%

21.54%

6.15%

8.46%


Q12. Why not worth the time: Emergent Themes

“already doing it in other ways”

“CCC does not tell me more about my residents than I already know.”



Cumbersome and Esoteric Milestones



No prior research, testing for validity

40



PDs need educational mission support.


Other thoughts from Coordinators: Emergent themes

Approaches to CCC faculty review process

Recognizing time-consuming nature of work and need for support

“Food for thought” for ACGME from program coordinators


Overall Results Summary

Overall Improved Evaluation and Documentation

− Semi-annual evaluations improved

− Documentation of trainee strengths and weaknesses

Time Consuming

− Need leadership support

− Administrative Resources

− Ample planning time

CCC Membership Size

− Share the burden!

− Value of diverse perspectives

Faculty Roles

− Responsibilities

Pairing faculty with trainees

Assigning faculty to become “milestone” experts


Lessons Learned / Best Practices

It’s your turn and your voice that counts!!!

How might you change your practice in light of these findings?


Questions?

45


Contact Information:

Nancy Piro, Program Manager/Education Specialist

− [email protected]

Kim Walker, Instructional Designer

− [email protected]

mailto:[email protected]

mailto:[email protected]

nancy piro, phd – no conflicts of interest to disclose kim walker, phd - no conflicts of interest...

Documents

program director slide

following slide slide

discussion purposes

ccc practices

survey outcomes

background slide

implementation slide

cccs slide