clinical skills verification rater training module 4 strategies for clinical skills assessment:...

Clinical Skills Verification Rater TrainingMODULE 4

Strategies for Clinical Skills Assessment:Models and Best Practices

Michael Jibson, M.D., Ph.D.David Kaye , M.D.

Sandra Sexson , M.D.

Module 4 Pre-Test

1. The strength of a Clinical Skills Verification Examination stems from its ability to serve as a forum for all of the following except:

A. A high stakes examination

B. direct observation of clinical skills

C. provision of immediate feedback on performance

D. repeated administrations during residency

2. Which of the following models for CSV exam is the least standardized?

A. Workshop Model

B. Mock Board Model

C. Clinical Evaluation Model

Module 4 Pre-Test

Module 4 Pre-Test

3. Which of the following models for the CSV exam makes the greatest use of resident peer feedback?

A. Workshop Model

B. Mock Board Model

C. Clinical Evaluation Model

Module 4 Pre-Test

4. Within a program, consistency of faculty expectations and rating of residents on the CSV exam can be maximized by:

A. All raters using the same evaluation model

B. Limiting assessment to the three competency areas outlined by the ABPN

C. Using standardized patients

D. Monitoring inter-rater reliability

Clinical Skill Verification (CSV)

Strengths of Clinical Skills Verification• Direct observation of clinical skill• Multiple observations and observers• Defined areas for evaluation• Specific anchors for performance• Immediate feedback• Ample opportunity for remediation

Evaluation Model

We recommend that the evaluation:

• Be done early in residency and repeated frequently

• Include immediate feedback

• Avoid the “high-stakes exam” model

Evaluation Model

Mock Board Model

• Use the CSV for the current practice of “Mock Board” exams

• Prearranged patients

• Scheduled interviews

Mock Board Model

Advantages• Familiar format• Highly standardized

Disadvantages• Retains high-stakes exam format• Narrow patient selection• Difficult to schedule

Evaluation Model

Workshop Model

• Use the CSV form for a workshop or class on interviewing

• Involve peer resident observers

• Videotape the encounter

Workshop Model

Advantages• Multiple residents

involved• Highly

standardized• Excellent

feedback (if videotaped)

Disadvantages• Few opportunities

for each resident to interview

• Narrow patient selection

Evaluation Model

Clinical Model

• Use the CSV form for any initial clinical encounter with supervising faculty

• Inpatient admission

• Emergency room evaluation

• Outpatient assessment

Clinical Model

Advantages• Broad range of

patients• Many

opportunities to interview

• Easy to schedule

Disadvantages• Moderately

standardized• Faculty buy-in

essential

Evaluation Model

We recommend that the evaluation:

• Occur during regularly scheduled clinical rotations

• Involve the standard interview used in that clinical setting

• Involve randomly selected patients

• Be done as part of the routines of clinical care

Additional Issues

Programs may want to consider:

• Using the exam to test additional competencies for their own purposes

• Requiring residents to pass the examination before advancing through training

Additional Competencies

CSV requirements may be combined with other clinical competencies:

• Case formulation

• Diagnostic assessment

• Treatment planning

Faculty Training

We recommend that programs:

• Offer training to faculty in how to conduct and score the examination

• Make an effort to standardize scoring on the examination

• Monitor inter-rater reliability internally

• Consider credentialing faculty as evaluators

Post-Test Module 4

1. The strength of a Clinical Skills Verification Examination stems from its ability to serve as a forum for all of the following except:

A. A high stakes examination

B. direct observation of clinical skills

C. provision of immediate feedback on performance

D. repeated administrations during residency

Post-Test Module 4

1. The strength of a Clinical Skills Verification Examination stems from its ability to serve as a forum for all of the following except:

A. A high stakes examination

The in –residency CSV exam allows programs to administer repeatedly and avoid the high-stakes aspect.

Post-Test Module 4

2. Which of the following models for CSV exam is the least standardized?

A. Workshop Model

B. Mock Board Model

C. Clinical Evaluation Model

Post-Test Module 4

2. Which of the following models for CSV exam is the least standardized?

C. Clinical Model

The clinical model utilizes patients seen on regular clinical services without pre-selection of patients

Post-Test Module 4

3. Which of the following models for the CSV exam makes the greatest use of resident peer feedback?

A. Workshop Model

B. Mock Board Model

C. Clinical Evaluation Model

Post-Test Module 4

3. Which of the following models for the CSV exam makes the greatest use of resident peer feedback?

A. Workshop Model

Post-Test Module 4

4. Within a program, consistency of faculty expectations and rating of residents on the CSV exam can be maximized by:

A. All raters using the same evaluation model

B. Limiting assessment to the three competency areas outlined by the ABPN

C. Using standardized patients

D. Monitoring inter-rater reliability

Post-Test Module 4

4. Within a program, consistency of faculty expectations and rating of residents on the CSV exam can be maximized by:

D. Monitoring inter-rater reliability

Along with faculty training on how to conduct and score the examination, tracking inter-rater reliability among faculty after observing the same performance and giving feedback to faculty regarding their ratings compared to others is a very effective way of standardizing ratings.