emergency medicine standardized letter of recommendation: predictors of guaranteed match

648 SLOR Girzadas et al. • SLOR AND GUARANTEED MATCH

Emergency Medicine Standardized Letter ofRecommendation: Predictors of Guaranteed Match

DANIEL V. GIRZADAS JR., MD, ROBERT C. HARWOOD, MD, MPH,STEVE N. DELIS, MD, KATHLEEN STEVISON, MD, GEORGE KENG, MD,

NANCY CIPPARRONE, MA, ANDREA CARLSON, MD, GEORGE D. TSONIS, MD

Abstract. Objective: The Council of EmergencyMedicine Residency Directors (CORD) standardizedletter of recommendation (SLOR) has become a com-mon, reliable, and useful tool in the evaluation ofemergency medicine (EM) applicants. A ‘‘guaranteedmatch’’ (GM) is the SLOR’s bottom-line superlativeresponse. It is also the SLOR’s least common super-lative response. Because candidates receiving a GMare a select group, the authors thought it would beuseful to identify SLOR information that predicts aGM recommendation. Methods: This was a secondaryanalysis of a database of all EM SLORs submitted toa single EM residency during the 1998–1999 appli-cation cycle to one EM residency program. Responseto GM and 16 data points in the background/qualifi-cation sections were analyzed by chi-square, univar-iate analysis, and logistic regression. Results: Fourhundred eleven SLORs were analyzed. Qualificationinformation was more predictive than background in-formation for applicants receiving a GM. The highestunivariate odds ratios for background informationwere ‘‘staff author’’ (OR = 1.7, 1.0–2.8), ‘‘extended

contact’’ (OR = 2.2, 1.0–4.5), ‘‘clinical contact outsidethe ED’’ (OR = 3.0, 1.5–5.9), and ‘‘honors on EM ro-tation’’ (OR = 5.4, 3.0–9.8). The highest univariateodds ratios for qualification information were ‘‘out-standing differential diagnosis ability’’ (OR = 10.1,5.8–17.4), ‘‘outstanding work ethic’’ (OR = 13.1, 5.2–33.3), and ‘‘outstanding global assessment’’ (OR = 58,24.2–139). Logistic regression analysis demonstrated‘‘outstanding global assessment’’ (p < 0.000; r = 0.92)and ‘‘outstanding work ethic’’ (p = 0.028; r = 0.71) tobe statistically predictive of GM. Conclusions: Therewere both background and qualification data pointspredictive of a ‘‘guaranteed match.’’ Qualification in-formation had a greater predictive value than back-ground information. Medical student applicants, let-ter writers, and letter evaluators may find thisinformation useful when dealing with SLORs. Key

words: letters of recommendation; emergency medi-cine; resident; applicant; postgraduate education; rec-ommendation. ACADEMIC EMERGENCY MEDI-CINE 2001; 8:648–653

RESIDENCY selection is an annual event con-suming considerable time and financial re-

sources.1–3 Letters of recommendation (LORs) arean important factor in resident selection.4–9 Aca-demic physicians have identified several factors as-sociated with the best, most highly rated LORs.The best LORs give a superlative appraisal andare most supportive of a candidate.7–10

The best LORs were written by authors who: 1)practice in the applicant’s desired specialty, 2)know the applicant very well, 3) are comfortablecomparing the applicant with other students, and4) mentioned an attempt to recruit the applicantto one’s own program.7,9,10

From the Department of Emergency Medicine, Christ Hospitaland Medical Center (DVG, RCH, SND, KS, GK, NC), OakLawn, IL; and the University of Illinois at Chicago MedicalSchool (AC, GDT), Chicago, IL.Received August 3, 2000; revisions received October 11, 2000,and December 11, 2000; accepted January 9, 2001.Address for correspondence and reprints: Daniel V. GirzadasJr., MD, Department of Emergency Medicine, ChristHospital and Medical Center, 4440 West 95th Street, OakLawn, IL 60453. Fax: 708-346-1028; e-mail: [email protected]

Other factors that added to the perceivedstrength of a LOR were higher word count, person-alization of the LOR, describing the applicant asfunctioning at the level of an intern,7 the reader’sfamiliarity with the author,8,9 and an evaluation ofan applicant’s personal characteristics in compar-ison with other applicants.9 Academic rank of theauthor of a LOR was shown to modestly contributeto the strength of a LOR.7,9

Factors associated with less-positive LORswere absence of detail, vagueness, brevity, notingthe applicant was in the top half of his or her classor above average, hesitancy on the part of the au-thor, and the comment ‘‘If I can provide any addi-tional information, please call me.’’7,10

In 1996, the Council of Emergency MedicineResidency Directors (CORD) created a standard-ized letter of recommendation (SLOR) for appli-cants to emergency medicine (EM) residencies. TheSLOR provides background data and qualificationdata on applicants.11 Qualification data evaluateand compare an individual applicant in relation tothe overall applicant pool. The SLOR limits adjec-tive inflation and ambiguous terms that are fre-

ACADEMIC EMERGENCY MEDICINE • June 2001, Volume 8, Number 6 649

Figure 1. Univariate odds ratios for standardized letter of recommendation (SLOR) background information.

quent frustrations of a narrative letter of recom-mendations (NLOR).11–14 The SLOR has beenshown to have superior interrater reliability thatis not dependent on the level of experience of theinterpreter. It also takes less time to interpret com-pared with a NLOR.15

‘‘Guaranteed match’’ (GM) is the SLOR’s bot-tom-line superlative response. Also, GM is theleast-common superlative response given to appli-cants on the SLOR and as such is the most sup-portive of a candidate.16 A response of GM ad-dresses the ultimate question a program directorasks of every applicant—‘‘How should we rank thisapplicant?’’ It identifies a select group of applicantswho EM faculty believe are most capable of match-ing and excelling in our specialty. It has beenshown that applicants with better National Resi-dent Matching Program (NRMP) rank are per-ceived by supervisors and coworkers as strongerresidents than those with lower NRMP ranks.3

There is above-average competition for positions inEM residencies.17,18 Emergency medicine programdirectors expect this competition to increase.19 Ap-plicants, letter writers, and advisors need to knowwhat factors predict a GM response. The objectiveof this study was to determine which specific SLORbackground data and qualification data predict aGM recommendation.

METHODS

Study Design. We conducted a secondary anal-ysis of a database of all EM SLORs submitted to asingle EM residency.16 The original analysis of thedatabase determined the frequency of each possi-ble response and identified the least-common su-perlative responses in the entire cohort of SLORs

received in one application cycle.16 The presentstudy evaluated the subset of SLORs that con-tained a final bottom-line match recommendation.This subset comprised 411 SLORs from the origi-nal database of 432 SLORs.

Twenty-one SLORs from the original databasewere excluded from this analysis because theylacked a bottom-line match recommendation. Toensure that they were similar to the study group,these 21 excluded SLORs were analyzed for theirresponse frequencies to the 16 data points andcompared with the 411 SLORs in the study. Theonly response that did not match the study groupwas the author’s gender. The excluded SLORs had38% women authors compared with 14% in thestudy group (p = 0.019).

Within the subgroup of 411 SLORs, an entirelynew analysis of the data was performed. We ana-lyzed the relationship between qualification infor-mation and background information vis-a-vis GMrecommendation. Our institution’s investigationalreview board approved this study.

Study Protocol. Every LOR received by a singleEM residency program for the 1998–99 applicationcycle was reviewed. This included 555 applicationsand 1,948 LORs. All 1,948 letters were searched toseparate SLORs for further analysis. Each SLORwas analyzed for the responses to 16 data points(Figs. 1 and 2). Data from each SLOR were enteredon a standardized data abstraction sheet. If a par-ticular data point was omitted from a SLOR, thatinformation was searched for in the applicant’stranscripts, dean’s letters, and other letters of rec-ommendation. If a SLOR author’s gender was notspecified by his or her name, the institution wascontacted for the information. This method of


Figure 2. Univariate odds ratios for standardized letter of recommendation (SLOR) qualification information.

building a SLOR database has been described pre-viously.16

Of 555 applications, 432 SLORs were identifiedand reviewed. Four hundred eleven (95%) con-tained a final bottom-line match recommendation.These 411 SLORs constituted the database for thisstudy.

Data Analysis. The relationship between GMand each data point was collapsed into a dichoto-mous variable as guaranteed match vs all others(very likely, likely, possible, and unlikely to match).All background information was analyzed usingPearson’s chi-square test. All covariates with a p-value # 0.2 in the univariate analysis were putinto a logistic regression model to assess the as-sociation between GM and each independent var-iable. Odds ratios with 95% confidence intervals(95% CIs) were calculated from univariate analysisfor all variables with respect to GM.

RESULTS

Of the 411 SLORs that contained a bottom-linematch recommendation, 23% were a GM. Qualifi-cation data were more predictive than backgrounddata (Table 1). The highest univariate odds ratiosfor background information were ‘‘staff author’’(OR = 1.7, 1.0–2.8), ‘‘extended contact’’ (OR = 2.2,1.0–4.5), ‘‘clinical contact outside the ED’’ (OR =3.0, 1.5–5.9), and ‘‘honors on EM rotation’’ (OR =5.4, 3.0–9.8). The highest univariate odds ratiosfor qualification information were ‘‘outstanding dif-ferential diagnosis ability’’ (OR = 10.1, 5.8–17.4),‘‘outstanding work ethic’’ (OR = 13.1, 5.2–33.3),and ‘‘outstanding global assessment’’ (OR = 58,

24.2–139). Logistic regression analysis demon-strated ‘‘outstanding global assessment’’ (p < 0.000;r = 0.92) and ‘‘outstanding work ethic’’ (p = 0.028;r = 0.71) to be statistically predictive of GM. Hos-mer-Lemeshow goodness of fit was very good (p =0.780).

DISCUSSION

Resident selection is an inexact science. Criteriaused by EM residency selection committees are im-precise indicators of the actual performance of acandidate in an EM residency.2 For example, a can-didate’s medical school academic performance cor-relates only moderately with that candidate’s sub-sequent performance in residency.20,21 To reducethis type of uncertainty, those involved in the res-ident selection process seek multiple evaluations ofa candidate’s academic, clinical, and interpersonalskills.22

Paperwork evaluations such as LORs, EMgrades, clinical grades, board scores, and AlphaOmega Alpha status are an important first step inthe overall selection process. Paperwork evalua-tions are used as a screening tool to select whichapplicants obtain an interview. A February 1998survey of EM program directors found that EM ro-tation grade, clinical grades, and LORs were con-sidered the ‘‘most important’’ factors in resident se-lection.8 Earlier work also found that EM rotationgrade and clinical grades were ‘‘near critical.’’19

The SLOR contains two of these ‘‘most important’’factors, a LOR and an EM rotation grade. It ad-ditionally contains a match recommendation. Re-ceiving a GM is actually less common than receiv-ing an honors grade on an EM rotation (23% vs


TABLE 1. Univariate Results

Variable OR (95% CI)

Background informationAuthor’s position (staff vs all

others) 1.7 (1.0, 2.8)Know indirectly 0.56 (0.33, 0.91)Clinical contact outside ED 3.0 (1.5, 5.9)Extended contact 2.2 (1.0, 4.5)Grade (honors vs all others) 5.4 (3.0, 9.8)

Qualification informationCommitment to EM (outstand-

ing vs all others) 5.4 (3.1, 9.3)Work ethic (outstanding vs all

others) 13.1 (5.2, 33.3)Ability for differential diagno-

sis (outstanding vs all oth-ers) 10.1 (5.8, 17.4)

Personality (superior vs allothers) 5.3 (2.9, 9.5)

Global assessment (outstand-ing vs all others) 58.0 (24.2, 139)

55%).16 This combination of LOR, EM rotationgrade, and match ranking makes the SLOR a pow-erful screening tool.

The SLOR’s utility is evident to the EM com-munity, and it is becoming a common factor in theEM residency application process. Despite havingits use restricted to only EM academic faculty, it isthe most common format of LORs submitted byemergency physicians. It accounted for 22% of allLORs submitted to a single EM residency.23 TheSLOR has met with a high degree of satisfactionwith EM program directors.11 Compared withNLORs, the SLOR is easier to complete, read, andincorporate into a ranking scheme. The SLOR al-lows readers to better discriminate differences be-tween candidates.11

This study’s findings may be helpful to appli-cants looking to improve their chances of matchingin an EM residency. Extended contact with EMfaculty is an important factor for an applicant.Applicants who spent more than ten hours in theemergency department (ED) under the direct ob-servation of the recommending physician had atwo times better chance of a GM recommendationthan those having less contact. Students, whowork with a recommending physician in a settingoutside the ED, such as research, poison centers/toxicology, out-of-hospital care, and mass gather-ings, also increase their chances of a GM recom-mendation. This finding is in agreement with otherwork.7

Residency applicants frequently seek out rec-ommendations from senior physicians in the de-partment such as the chair or residency director,as opposed to staff physicians. However, this studyshows that having a staff EM physician write aSLOR for an applicant gives the applicant a 1.7times better odds of receiving a GM recommenda-tion. Further analysis of our data shows that thisadvantage is not independent of time spent withan applicant.

The staff physician may have more opportunityfor direct clinical observation of the applicant andthus feel more comfortable giving a GM recom-mendation. Alternatively, more senior physiciansmay have a better perspective on the entire resi-dency pool and be more selective in who receives aGM recommendation.

Some physicians believe selecting GM is a vio-lation of NRMP rule 6.3.24 A careful reading byCORD legal counsel indicates that confidentialityis breached only if the actual NRMP rank list isdivulged. A GM on a SLOR is only a rough esti-mate of a candidate’s eventual rank with wide con-fidence intervals. This estimate also precedes theactual match by several months (Sam Keim, Pres-ident, CORD, personal communication, September2000). If an applicant has spent more time with a

staff physician than with one of the departmentheads, the applicant appears to be better servedseeking a SLOR from the staff physician.

A related finding of our study was that a SLORwritten by a physician who knew an applicant onlyindirectly actually decreased the chance of the ap-plicant’s receiving a GM. An example of this situ-ation would be a chair or program director writinga SLOR based on evaluations from other faculty.This strengthens the concept that close clinicalcontact with the applicant is crucial, and possiblyoverrides any benefit obtained by the author’s ac-ademic rank.7,9

Emergency medicine rotation grade was againfound to be important. Of all SLOR backgroundinformation, EM rotation grade was the strongestpredictor of a GM recommendation. For residencyselection, importance of rotation grade has beenshown for EM and surgery.7,8,19

Although specific background information datacould improve a candidate’s chance of a GM rec-ommendation, qualification information had a rel-atively much greater impact on an applicant’s finalrecommendation. Outstanding ‘‘work ethic’’ andoutstanding ‘‘ability to develop and justify an ap-propriate differential and cohesive treatment plan’’were five times more likely to predict a GM rec-ommendation than any background information.An outstanding ‘‘global assessment’’ ranking wasthe most predictive factor on the SLOR of a GMrecommendation (odds ratio = 58). A SLOR authormarking this box went on to rank the applicant asa GM 50% of the time.

Applicants to EM residencies, their advisors,and letter writers should know that the probabilityof a GM recommendation is highest for applicants


who have had extended contact with the letterwriter, including work outside the ED. Staff emer-gency physicians are more likely to give a GM thanchairs and residency directors. Most importantly,applicants need to show an outstanding workethic, hone their differential diagnosis and treat-ment plan skills, and obtain as high a grade aspossible on their EM rotation. Applicants who ex-cel in these areas are most likely to receive a guar-anteed match.

LIMITATIONS AND FUTURE QUESTIONS

The study included only one EM residency pro-gram. However, our program captured 38% of allU.S. applicants to EM. Also, this group of appli-cants came from 70% of the United States’ allo-pathic medical schools.25 A sample of this size anddiversity gives our data the potential to be appliedto the general universe of EM residency programs.An important future question examining the con-cept of GM will be to test the external validity ofour findings with a prospective, multicenter, lon-gitudinal study.

It is possible that a GM recommendation mayvary between programs. A stronger EM residencyprogram may be less likely to rank an applicant asa GM than a weaker program. This, however, isnot different from interpreting dean’s letter rec-ommendations from two medical schools. Astronger medical school may have a candidaterated as ‘‘high recommendation,’’ whereas if thesame applicant attended a weaker medical school,he or she would be rated as ‘‘highest recommen-dation.’’ There are no validated rankings of medicalschools or EM residencies to help discern the rel-ative values of specific recommendations. Thus, onthis question individual programs are each left totheir own interpretations.

We did not assess the interrater reliability be-tween researchers abstracting data from SLORs.The process, however, is straightforward andrarely required judgment on the part of the datagatherer. We simply took the responses noted oneach SLOR and entered them onto a standardizeddata abstraction form. If a question arose abouthow the data should be entered on the form, it wasbrought to the whole research group for a decision.

The confidence intervals we report for the qual-ification data are wide. Due to the subjectivity ofthese measures, qualification data are a less-pre-cise measurement than background data. This in-troduces more standard variation.

The following question was not asked or an-swered by this study—Is it more advantageous fora medical student applicant to obtain a great LORfrom a staff physician or an average LOR from adepartment chair, residency director, etc.? For sur-gery residents, a LOR author’s academic rank

wasn’t considered ‘‘very important’’ or ‘‘important,’’but only ‘‘somewhat important.’’7

The structure of the SLOR presents possibili-ties for research that were not previously availablewith NLORs. The relationship between U.S. Med-ical Licensing Examination scores and LORs canbe examined. Author bias toward the physicalcharacteristics of applicants such as age and gen-der can be explored. There may be other factors,such as a personal interview, not directly capturedby the SLOR that may affect an applicant’s matchrecommendation. The most difficult questions arewhether SLORs predict actual resident perfor-mance and whether they predict performance bet-ter than NLORs and other evaluative tools.12

CONCLUSIONS

Background data points (staff author, extendedcontact, clinical contact outside the ED, and honorson EM rotation) and qualification data points (out-standing differential diagnosis ability, outstandingwork ethic, and outstanding global assessment)were predictive of a GM. Qualification informationhad a greater predictive value than background in-formation. Medical student applicants, advisors,and letter writers should be aware that the prob-ability of a GM recommendation is increased incertain situations. Guaranteed match probabilitygoes up when the applicant spends extended timewith a staff EM physician, works with the recom-mending physician in a medical realm outside theED, receives honors in his or her ED rotation, hasan outstanding work ethic, and has an outstandingability to develop and justify a differential diag-nosis and cohesive treatment plan. These are theapplicant traits that increase the likelihood of aGM in EM.

References

1. Aghababian R, Tandberg D, Iserson K, et al. Selection ofemergency medicine residents. Ann Emerg Med. 1993; 22:1753–61.2. Balentine J, Gaeta T, Spevack T. Evaluating applicants toemergency medicine residency programs. J Emerg Med. 1999;17:131–4.3. Sklar DP, Tandberg D. The relationship between NationalResident Match Program bank and perceived performance inan emergency medicine residency. Am J Emerg Med. 1996; 14:170–2.4. Frankville D, Benumof J. Relative importance of factorsused to select residents: a national survey [abstract]. Anesthe-siology. 1991; 75:A876.5. Leichner P, Eusebio-Torres E, Harper D. The validity of ref-erence letters in predicting resident performance. J Med Educ.1981; 56:1019–21.6. Binder LS. Babies and bathwater: standardized vs narra-tive data (or both) in applicant evaluation [commentary]. AcadEmerg Med. 1998; 5:1045–8.7. Greenburg AG, Doyle J, McClure DK. Letters of recommen-dation for surgical residencies: what they say and what theymean. J Surg Res. 1994; 2:192–8.8. Crane JT, Ferraro CM. Selection criteria for emergencymedicine residency applicants. Acad Emerg Med. 2000; 7:54–60.


9. Wagoner NE, Suriano JR, Stoner JA. Factors used by pro-gram directors to select residents. J. Med Educ. 1986; 61:10–21.10. Johnson M, Elam C, Edwards J, et al. Medical school ad-mission committee members’ evaluations of and impressionsfrom recommendation letters. Acad Med. 1998; 73(10, OctRIME suppl):S41-S43.11. Keim SM, Rein JA, Chisholm C, Dyne PL. A standardizedletter of recommendation for residency application. AcadEmerg Med. 1999; 6:1141–6.12. Schaider JJ, Rydman RJ, Greene CS. Predictive value ofletters of recommendation vs questionnaires for emergencymedicine resident performance. Acad Emerg Med. 1997; 4:801–5.13. Friedman RB. Fantasyland [letter]. N Engl J Med. 1983;308:651–3.14. Garmel GM. Letters of recommendation: what does goodactually mean?. [letter]. Acad Emerg Med. 1997; 4:833–4.15. Girzadas DV Jr, Harwood RC, Dearie J, Garrett S. A com-parison of standardized and narrative letters of recommenda-tion. Acad Emerg Med. 1998; 5:1101–4.16. Harwood RC, Girzadas DV Jr, Carlson A, et al. Character-istics of the emergency medicine standardized letter of rec-ommendation. Acad Emerg Med. 2000; 7:409–10.

17. Binder L. 1997 NRMP match in emergency medicine.SAEM Newslett. 1997; 9(4):8.18. Blanchard J. Board scores and resident performance: isthere a link? Ann Emerg Med. 2000; 36:64–7.19. Wagoner NE, Suriano R. Program directors’ responses toa survey on variables to select residents in a time of change.Acad Med. 1999; 74:51–8.20. Markert RJ. The relationship of academic measures inmedical school to performance after graduation. Acad Med.1993; 68(2 suppl):S31-S34.21. Calhoun KH, Hokanson JA, Bailey BJ. Predictors of resi-dency performance: a follow-up study. Otolaryngol Head NeckSurg. 1997; 116:647–51.22. Pilon S, Tandberg D. Neural network and linear regressionmodels in residency selection. Am J Emerg Med. 1997; 15:361–4.23. Tsonis G, Harwood RC, Girzadas DV Jr. Standardized let-ter of recommendation for residency application [letter]. AcadEmerg Med. 2000; 7:963.24. Policies of the NRMP. Main Match Guide. http://nrmp.aamc.org/nrmp.25. National Data; medical school type/frequency. ElectronicResidency Application Service—Program Director’s Work Sta-tion, 1998–9.

v

emergency medicine standardized letter of recommendation: predictors of guaranteed match

Documents