Evaluating Resident Candidates: Does Closed File Review Yield Better Information?

Shahnaz Chowdhry MS MD1, Linnea S. Hauge PhD1, Steve Stroessner PhD2, and Norman L. Wool MD1

Department of General Surgery, Rush University Medical Center, Chicago, IL1

and Department of Psychology, Barnard College, New York, NY2

Background

Bias created by open file interviewsRobin, Bombeck, Pollak, Nyhus (1991)Miles, Shaw, Risucci (2001)

Closed file interviews designed to reduce bias

Purpose of the Study

To determine the efficacy of enhanced interview and rating process

Research Questions

Is there a difference between faculty ratings of applicant characteristics in open and closed file conditions?

Do ratings of applicants, in these conditions, vary by gender and known status?

Study Participants

Large, urban academic general surgery program

70 residency candidates (interviewees)44 Male26 Female

17 surgery faculty members (interviewers)12 Male5 Female

Closed File Rating Process

Each applicant was interviewed by 2 faculty members (140 total interviews)

Closed file interviewNamePersonal statementUndergraduate institutionMedical school

Faculty completed evaluation instrument

Open File Rating Process

Faculty reviewed applicant filesERAS applicationTranscriptUSMLE ScoresMSPELetters of Recommendation

Faculty completed the same evaluation instrument

Evaluation Instrument Motivation for learning Motivation to do research Ability to work effectively with

others Commitment to general surgery Likelihood to succeed in surgery Overall fit in our program

Self-confidence Communication Intelligence Maturity Compassion Interpersonal

Skills Motivation for

Surgery

Miles WS, Shaw V, and Risucci D. The Role of blinded interview in the assessment of surgical residency candidates. The American Journal of Surgery. 182 (2001) 143-162.

Data Analysis

Factor analysis to yield rating components MANOVA on:

Interview type (closed vs. open)Known status (known vs. unknown)Gender (male vs. female)Ratings

Results

No significant differences betweenRatings between open vs. closed file

(F=.34, p=.56)

Open vs. closed file ratings by gender (F=.04, p=.84)

Results: Interaction Effects

Significant interaction effects between:Open vs. closed file ratings by known status

(F=4.7, p=.04)Open vs. closed file ratings by trait (F=1.8,

p=.04)Open vs. closed file ratings by trait, gender

and known status (F=1.98, p=.02)Trait by gender (F=3.1, p=.001)

Results: Closed vs. Open

By trait (F=1.8, p=.04)

Closed OpenIntelligence 4.19 4.33Maturity 4.16 4.29Commitment to general surgery 4.32 4.41Qualifications to succeed 4.20 4.32

Results: Closed vs. Open

By known status (F=4.7, p=.04)

Known UnknownMaturity (open) 4.02 4.43Motivation for general surgery (closed) 4.26 4.59Working with others (closed) 4.91 4.36Qualifications to succeed (open) 4.04 4.41Likelihood to succeed (open) 4.00 4.45Fit to program (open) 4.00 4.33

Results: Closed vs. Open Closed

Open

Known females Communication 4.4 4.0 Interpersonal skills 4.4 3.7 Motivation for research 3.6 3.1

Known males Qualifications to succeed 4.4 4.0

Results: Closed vs. Open

Maturity Ratings

Closed Open

Known Male 3.8 4.0

Unknown Male 4.2 4.3

Known Female 4.6 4.4

Unknown Female 4.2 4.6

Results: Trait by GenderSelf-confidence 4.3 4.4

Communication 4.3 4.3

Intelligence 4.4 4.3

Maturity 4.1 4.4

Compassion 4.3 4.4

Interpersonal skills 4.3 4.2

Motivation for surgery 4.5 4.6

Motivation for learning 4.6 4.6

Motivation for research 4.2 3.6

Ability to work effectively with others 4.5 4.4

Commitment to general surgery 4.4 4.6

Qualifications to succeed in surgery 4.2 4.3

Likelihood to succeed in surgery 4.2 4.3

Overall fit for our program 4.2 4.2

M F

Factor Analysis Results

Social Traits Communication = 0.87 Maturity = 0.85 Self-confidence = 0.77 Interpersonal skills =0.76

General Surgery Commitment to general surgery = 0.871 Motivation to do general surgery = 0.85

Research Motivation to do research = 0.89

Conclusions

No significant change in closed and open file evaluations except:Differences in ratings of known vs. unknown

applicantsDifferences in ratings of some traits by the

gender of the applicant Constructs evaluated are social traits,

motivation for general surgery, and research