Use of a Novel, Web-based Educational Platform Facilitates

Intra-operative Training in a Surgical Oncology Fellowship

Program

Paul B. Roach, Kevin K. Roggin, Eugene Selkov Jr., James Dignam,

Mitchell C. Posner, Jonathan C. Silverstein

The University of ChicagoDepartment of Surgery

Disclosure Statement

• The principal and contributing authors have no conflicts of interest to report.

• The software program presented has an “open source” copyright, and will be distributed freely to any interested surgical program directors.

Background

• No widely available, comprehensive, easy-to-use platform for evaluating surgical resident/fellow operative performance.

• Current educational reforms depend on one being developed and integrated into training programs.

• Examples exist from within Naval Aviation, and Business Management.

Hypothesis

• Can a system be created for achieving continuous operative performance assessments?

• Can it be integrated into our daily routine / educational curriculum?

“The Surgical Training and Assessment Tool” or “STAT”

1. A web-based software system to de- construct the complex training events of an operative case into distinct elements;

2. Enable methodical, thoughtful, subjective assessments of these distinct elements;

3. Re-construct data into usable formats to impact / improve the surgical training process.

“STAT” Login

4 Sections:

Two structured: General Capabilities Specific Capabilities

Two unstructured: Comments Box Overall Assessment

Results

Results

Training Objectives

Distribution Times

0 20 40 60 80 100

Nine-36

64-92

119-146

174-202

229-256

284-312

339-366

394-422

449-476

504-532

Sec

on

ds

Incidence

Seconds

Median = 39 seconds

Mean = 60 seconds

Time per case entry

Data: SYSTEM USAGESystem Usage: Submissions / Cases

0

50

100

150

200

1 3 5 7 9 11 13 15 17

Month

To

tals Submissions

Cases

18 Month Totals: 1318 submissions832 cases

Submissions: ATTENDING/TRAINEE

Submissions: Attending/Trainee

0

20

40

60

80

100

120

140

160

Month

To

tals Attendings

Trainees

18 Month Totals: 607 Attending 710 Trainee

Overall Assessment Distribution

0

20

40

60

80

100

120

140

160

180

F D- D D+ C- C C+ B- B B+ A- A A+C

ou

nt

Attending

AttendingsTraineesOverall Assessment Distribution

0

50

100

150

200

250

300

F D- D D+ C- C C+ B- B B+ A- A A+

Cou

nt

Resident

7.6 = B- 8.71 = B

Discriminatory Capacity

Do the “General Capabilities” correlate with the “Overall Score”?

• Each of the general capabilities is significantly and strongly correlated with the overall score.

Knowledge 0.6036

Skill 0.7638

Independence 0.6935

Linear correlation with overall score:

Do the general capabilities predict the overall score?

• In a multiple regression model, the general capabilities significantly predict the overall score.

Predictor Coef. (slope) P

Knowledge 0.312 <0.001

Skill 1.255 <0.001

Independence 0.546 <0.001

Model R2 = .62

Multiple regression model for overall score:

How well do attending ratings of a given trainee correlate with one another?

0 1 2 3 4

r1

r2

r3

r4

Average Rank

1.4

2.6

2.8

3.2

How well do the self-ratings of the trainees compare with the ratings their attendings

give them?

Trainee “A”Attending

Trainee

Trainee- “B”

• Abundant trainee performance data• Real-time, transparent, longitudinal

record of trainee performance• Inexpensive & time-efficient• Influences the educational process:

– Reflection & self-awareness– Feedback – Formative & summative

Summary

Conclusion1. STAT is PRACTICAL:

FeasibleComprehensiveRelevant

2. STAT appears to be RELIABLE:Internal consistencyInter-rater reliability

3. STAT is EASILY INTEGRATED into the daily work- flow and educational curriculum.

Voluntary, self-sustaining use

• Surgical Oncology Faculty

• Pritzker School of Medicine

• General Surgery Faculty & Residents

• Bob McBride – Education

• CAPT JJ Lee – Naval Aviation

• Steve Roach – Managerial Science

Acknowledgements

Future Directions1. STAT Validation Study:

4 military training programs + Hadassah University

2. STAT Expansion into other specialtiesOrthopedic, Thoracic, Plastic, Obstetric-

Gynecologic Surgery

3. HIPEC study.Training ComponentReporting Component

3. Society of Surgical Oncology:Possible tool for fellowship training

“STAT” General Capabilities

Criteria Checklist- General

Criteria Checklist

Criteria Checklist- Case Specific

Unstructured Elements