The Journal of Emergency Medicine, Vol. 46, No. 4, pp. 544–550, 2014Copyright � 2014 Elsevier Inc.

Printed in the USA. All rights reserved0736-4679/$ - see front matter

http://dx.doi.org/10.1016/j.jemermed.2013.08.025

Drs. Beskind aOrder of authorsh

RECEIVED: 28 SeACCEPTED: 14 A

Education

DOESTHEEXPERIENCEOFTHEWRITERAFFECTTHEEVALUATIVECOMPONENTSON THE STANDARDIZED LETTER OF RECOMMENDATION IN EMERGENCY

MEDICINE?

Daniel L. Beskind, MD, MPH, FACEP, Katherine M. Hiller, MD, MPH, FACEP, Uwe Stolz, PHD, Hans Bradshaw, MD,Matthew Berkman, MD, FACEP, Lisa R. Stoneking, MD, FACEP, Albert Fiorello, MD, FACEP, Alice Min, MD, FACEP,

Chad Viscusi, MD, FAAEM, and Kristi J. H. Grall, MD, MHPE, FACEP

Department of Emergency Medicine, Arizona Emergency Medicine Research Center, University of Arizona College of Medicine, Tucson,Arizona

Reprint Address: Daniel L. Beskind, MD, MPH, FACEP, Department of Emergency Medicine, University of Arizona College of Medicine, 1501 N.Campbell Avenue, Tucson, AZ 85724

, Abstract—Background: The Standardized Letter ofRecommendation (SLOR) was developed in an attempt tostandardize the evaluation of applicants to an emergencymedicine (EM) residency. Objective: Our aim was to deter-mine whether the Global Assessment Score (GAS) and Like-lihood of Matching Assessment (LOMA) of the SLOR forapplicants applying to an EM residency are affected bythe experience of the letter writer. We describe the distribu-tion of GAS and LOMA grades and compare the GAS andLOMA scores to length of time an applicant knew the letterwriter and number of EM rotations. Methods: We conduct-ed a retrospective review of all SLORs written for all appli-cants applying to three EM residency programs for the 2012match. Median number of letters written the previous yearwere compared across the four GAS and LOMA scores us-ing an equality of medians test and test for trend to see ifhigher scores on the GAS and LOMAwere associated withless experienced letter writers. Distributions of the scoreswere determined and length of time a letter writer knewan applicant and number of EM rotations were comparedwith GAS and LOMA scores. Results: There were 917 appli-cants representing 27.6% of the total applicant pool for the2012 United States EM residency match and 1253 SLORsfor GAS and 1246 for LOMA were analyzed. The highest

nd Hiller contributed equally as first authors.ip was determined alphabetically.

ptember 2012; FINAL SUBMISSION RECEIVED: 7 Mugust 2013

544

scores on the GAS and LOMAwere associated with the low-est median number of letters written the previous year(equality of medians test across groups, p < 0.001; test fortrend, p < 0.001). Less than 3% received the lowest scorefor GAS and LOMA. Among letter writers that knew an ap-plicant for more than 1 year, 45.3% gave a GAS score of‘‘Outstanding’’ and 53.4% gave a LOMA of ‘‘Very Compet-itive’’ compared with 31.7% and 39.6%, respectively, if theletter writer knew them 1 year or less (p = 0.002; p = 0.005).Number of EM rotations was not associated with GAS andLOMA scores. Conclusions: SLORs written by less experi-enced letter writers weremore likely to have a GAS of ‘‘Out-standing’’ (p < 0.001) and a LOMA of ‘‘Very Competitive’’(p < 0.001) thanmore experienced letter writers. The overalldistribution of GAS and LOMAwas heavily weighted to thehighest scores. The length of time a letter writer knew an ap-plicant was significantly associated with GAS and LOMAscores. � 2014 Elsevier Inc.

, Keywords—emergency medicine; residency; standard-ized letter of recommendation; graduate medical education

INTRODUCTION

Every year, 4th-year medical students apply for admissioninto emergency medicine (EM) residency training pro-grams through the Electronic Residency Application

ay 2013;

Figure 1. Box plot for number of letters written in previousyear by Global Assessment Score. Box = 25th to 75th percen-tile, horizontal bar = median, whiskers = 5th to 95th percen-tile, and dots = outliers. SLOR = Standardized Letter ofRecommendation.

Standardized Letter of Recommendation 545

Service (ERAS�) and are matched via the National Res-idency Matching Program. Many components factor intothe residency match selection process, including theapplicant’s United States (US) Medical Licensing Exam-ination scores, the dean’s performance evaluation, clini-cal rotation grades, extracurricular experiences, and themedical school’s reputation (1�4).

The Council of Residency Directors in EmergencyMedicine (CORD) recommends use of the StandardizedLetter of Recommendation (SLOR) for applicants toEM residency programs (5). The SLOR was developedin 1996 in an attempt to standardize the evaluation of ap-plicants, improve inter-rater reliability of letters of rec-ommendation, and discourage the ‘‘upward creep ofsuperlatives’’ (6,7). The SLOR has been revised severaltimes over the years and a task force from CORDconvened in 1999 and recommended that only EMfaculty submit the SLOR (6). The current version of theSLOR includes the following variables: EM rotationgrade, number of the EM rotation the student was per-forming, percent of students who received honors at theletter writer’s institution, commitment to EM, work ethic,ability to develop a treatment plan, ability to interact withothers, ability to communicate with patients, amount ofguidance predicted during residency, prediction of suc-cess in residency, Global Assessment Score (GAS), andLikelihood of Matching Assessment (LOMA) (6). Eachof these variables is reported on a scale of 3 to 5, rangingfrom best performance to worst.

Although the SLOR is an attempt to standardize stu-dents across regional and experiential boundaries, oftenthe standard to which students are held is the writers’ ex-perience with other medical students. For example,writers are asked to compare this student against othersthey have known who applied for residency in EM.Although experienced writers will have a robust base todraw from, novice writers might be potentially biaseddue to their lack of a sizeable reference to use. Also,students might seek out faculty with whom they havean outstanding rapport to write an SLOR for their appli-cation, regardless of whether the writer has experiencewith the SLOR. These writers might be more likely torank students as ‘‘Outstanding’’ or ‘‘Excellent’’ on theGAS, as compared with faculty who write SLORs fora more diverse student population. Despite widespreaduse and expectation, the validity of the SLOR has notbeen well studied.

In 2011�2012, there were 2370 US applicants and 951foreign medical graduates, for a total of 3321 applicantswho participated in the match using ERAS� (8). Duringthe same time period, 917 of these applicants applied toat least one of the three EM residency programs at thisUniversity with affiliated academic and community hos-pitals. The majority of these applicants were 4th-year US

medical students. The majority of these applicants sub-mitted one or more SLORs to support their application.The cohort analyzed represented more than a quarter(27.6%) of the total applicant pool for the match in EMin 2011�2012.

The primary objective of our cross-sectional analysisof all SLORs written on behalf of applicants to EM atthe three university EM programs was to determinewhether the distribution of scores for the GAS andLOMA differed between writers who have more experi-ence writing SLORs compared to those with less letter-writing experience. We reported the number of letterswritten in the previous year by dividing them into quin-tiles and used these five categories as a proxy for letter-writing experience. We then compared the scores on theGAS and LOMA using the nonparametric equality of me-dians and a nonparametric test for trend for each quintileto test for significance. Additional objectives were to de-scribe the distribution of the grades on the GAS and theLOMA, as well as see if the length of time a letter writerknew an applicant affected the scores on the GAS andLOMA (Figures 1 and 2).

The distribution of all other categorical variables onthe SLOR are described, and differences, if any, betweenexperienced and novice writers are reported.

METHODS

This was a retrospective review of applicants applying toone of three EM residency programs at a large universityassociated with several medical centers. Eight hundredand nine applications were submitted to one universityprogram, 541 applications were submitted to the second

Figure 2. Box plot for number of letters written in previousyear by Likelihood of Matching Assessment. Box = 25th to75th percentile, horizontal bar = median, whiskers = 5th to95th percentile, and dots = outliers. SLOR = StandardizedLetter of Recommendation.

Table 1. Total Number of Standardized Letters ofRecommendation by Quintile of Letter-WriterExperience for Global Assessment Score andLikelihood of Matching Assessment

Quintiles SLOR Writer Experience Total SLORS GAS LOMA

1 (range 0�3) 266 251 2502 (range 4�11) 263 262 2563 (range 12�23) 268 268 2674 (range 24�35) 226 224 2245 (range 36�116) 249 248 249Excluded 196 19 26Total 1468 1253 1246

GAS=Global Assessment Score; LOMA=Likelihood ofMatchingAssessment; SLOR = Standardized Letter of Recommendation.

546 D. L. Beskind et al.

program, and 38 to the combined EM/pediatrics program.A total of 917 unique applicants applied to at least one ofthe programs. All SLORs written on behalf of the appli-cants were reviewed and included in the analysis (n =1468). SLORs with incomplete data (e.g., writer did notcheck a GAS or LOMA category or did not documenthow many SLORs they had completed the year before)were excluded from the study. One hundred and ninety-six did not document number of letters written in the pre-vious year, an additional 19 did not have a GAS score, and26 did not have a LOMA score and were excluded due tomissing data. SLORs were extracted from students’ERAS� applications by the program coordinators forthe residencies and submitted to the principal investiga-tor. SLORs were assigned and distributed to study per-sonnel based on the first letter of the applicants’ lastnames. For example, all applicants whose names beginwith ‘‘A’’ were assigned to an investigator. Researchersthen abstracted the necessary data from the SLORs ontoan Excel spreadsheet (Microsoft, Redmond, WA). Dupli-cate SLORs from applicants who applied to more thanone of the university EM residencies were recordedonly once. Researchers assigned a numeric code to de-scribe the scale, for example, ‘‘Outstanding’’ was codedas 1 and ‘‘Excellent’’ as 2, and so on. Three letter writerscoded both an ‘‘Outstanding’’ (i.e., 1) and an ‘‘Excellent’’(i.e., 2) for the GAS and two coded a 2 (‘‘Excellent’’) anda 3 (‘‘Very Good’’), and these 5 scores were all roundeddown (i.e., were given the lower of the two scores). Eightletter writers coded two scores for the LOMA and thesewere rounded down as well. Once the data collectionwas complete, data were deidentified by removing anypersonal identifiers, including the applicants’ ERAS�

numbers and institution, and data analysis was

performed. Data analysis included descriptive statisticsof the distribution of all variables collected. Nonparamet-ric equality of medians and test for trend of the GAS andLOMA distributions between each quintile of letters writ-ten was compared. All statistical analysis was performedusing Stata/MP v12.1 for Windows (StataCorp LP,College Station, TX). Data analysis included descriptivestatistics of the distribution of all variables collected.

Because we had some concern that letter writers withzero letters in the previous year represented a heteroge-neous group of reviewers with varying levels of experi-ence, we conducted sensitivity analyses by repeatingour comparisons for only those reviewers with one ormore letters in the previous year and by specifically com-paring writers with zero letters in the previous year tothose with one or two.

Analysis of the GAS and LOMA score based on thelength of time that an applicant knew the letter writercomparing the proportion of applicants who knew a letterwriter a year or less to those that knew an applicant longerthan 1 year was performed using a Fisher’s exact test. In-stitutional Review Board approval for this retrospectivereview was granted.

RESULTS

There were a total of 917 unique applicants with a total of1468 SLORs that were submitted to one of three univer-sity programs in EM for the 2011�2012 interview sea-son. This represents 27.6% of the total applicants to the2012 EM match based on the ERAS� statistics (9). Onehundred and ninety-six SLORs (13.3%) were excludedbecause the previous writer experience was not docu-mented. Of the 1272 remaining SLORs, 19 (1.3%) weremissing GAS data and 26 (1.8%) did not have a LOMAscore documented. These were also excluded, leavinga total of 1253 for the GAS and 1246 for the LOMAthat were analyzable (Table 1).

Of the 1468 total SLORs, 1423 SLORs had a GASscored reported and 1416 had a LOMA score reported

Table 2. Standardized Letters of RecommendationDistribution of Scores on the Global AssessmentScore and Likelihood of Matching Assessment

Total (n) %

GASOutstanding 471 33.1Excellent 641 45.0Very Good 275 19.3Good 36 2.5Excluded 45 3.1Total 1423

LOMAVery Competitive 591 41.6Competitive 605 42.6Possible Match 197 13.9Unlikely Match 27 1.9Excluded 48 3.3Total 1420

GAS = Global Assessment Score; LOMA = Likelihood of Match-ing Assessment.

Figure 4. Distribution of Likelihood of Matching Assess-ment (LOMA) scores across the quintiles of letter writerexperience.

Standardized Letter of Recommendation 547

(even though 170 of these did not have the previous year’sletter writer experience documented), and these totalswere used to describe the distribution of the scores. Ofall applicants’ SLORs, 33.1% had a GAS score of ‘‘Out-standing’’ and 41.6% had a LOMA score of ‘‘Very Com-petitive,’’ the highest category in each of these scales.Only 1.9% were rated as ‘‘Good’’ on GAS and 2.5%were rated ‘‘Unlikely Match’’ on LOMA, the lowest cat-egory in each of the scales (Table 2).

Of the 1253 SLORs analyzed for GAS and 1246 forLOMA, the highest scores on the GAS and LOMAwere associated with the lowest median number of letterswritten the previous year (equality of medians test acrossgroups [p < 0.001; test for trend, p < 0.001]) (Figures 3and 4). Removing those SLORs that werewritten by letterwriters that documented zero letters the year before fromthe analysis did not change the results and the results andassociations were the same.

Figure 3. Distribution of Global Assessment Scores (GAS)across five quintiles of letter writer experience.

The number of EM rotations for applicants applying toone of the three University EM programs completed dur-ing the 2011�2012 year is presented in Table 3. The dis-tributions of GAS and LOMA grades did not differbetween students with one EM rotation vs. those withtwo or more (p = 0.76 and p = 0.50).

The length of time a writer knew the applicant wassignificantly associated with the proportion of GAS(p = 0.002) and LOMA scores (p = 0.005). Among re-viewers that knew an applicant for more than a year,45.3% gave a GAS grade of ‘‘Outstanding’’ comparedwith 31.7% if the reviewer knew them 1 year or less.Among reviewers that knew an applicant for more than1 year, 53.4% gave a LOMA score of ‘‘Very Competi-tive,’’ compared with 39.6% if the reviewer knew themfor a year or less.

DISCUSSION

The SLOR is an effort to standardize evaluations formedical students applying to EM residency. The inconsis-tencies in the scoring of the SLORs make the interpreta-tion of applicant’s SLORs challenging. Writers with the

Table 3. Number of Emergency Medicine RotationsCompleted for Applicants Applying to Any of theThree Emergency Medicine Residencies During2011–2012 Year

No. of EM Rotations n %

1 765 58.02 490 37.23 55 4.24 6 0.55 1 0.16 1 0.1

EM = emergency medicine.

548 D. L. Beskind et al.

least experience (lowest quintile) in this analysis weremore likely to assign an ‘‘Outstanding’’ GAS score and‘‘Very Competitive’’ LOMA score to the applicants inour study. Some institutions allow for a compositeSLOR as well as an individual SLOR. A group or depart-mental letter would hopefully be more objective, butmight not reflect the personal experience that an individ-ual faculty had with the applicant. Perhaps this varianceallows the reviewers useful information that gives themmore insight into the candidate. We did complete a sensi-tivity analysis removing the SLORs that reported zero let-ters in the previous year because the zero group mightcontain letter writers who moved to a composite/depart-mental SLOR in the current season. Removing the zerogroup did not change the results and the associationswere the same.

The length of time a writer knew the applicant wassignificantly correlated to the scores on the GAS andLOMA. We hypothesized that a student might seek outa letter writer with whom they have excellent rapport. Ifthe writer knew the student for a longer period of timethey might potentially assign a higher GAS or LOMAscore. This does appear to be the case for students whoknew their letter writer for more than 1 year.

Limitations

This study was a retrospective, cross-sectional analysis ofall SLORs written on behalf of applicants applying to oneof three EM residencies at a university in the 2011�2012season, and is subject to many of the flaws of this studydesign. While we are able to report prevalence and asso-ciations, causality is not clearly established. Althoughthere is an association between the number of SLORsa faculty member wrote the previous year and the GASand LOMA scores, the difference in the distributionscan be explained by other confounders, such as the differ-ences in the population of students who request lettersfrom experienced vs. novice writers. Even if we wereable to attribute the difference in scores to the experienceof the letter writer, the reason for the effect is not clear.We can speculate as to why there is a difference betweennovice and experienced SLOR writers, however, thisstudy was not designed to determine the reason for thatdifference.

Although data collection was largely complete, therewere a significant number of incomplete SLOR entriespotentially altering the final analysis of the data. For in-stance, an SLOR might not have a GAS or LOMA num-ber recorded (1.3%, 1.8% missing, respectively), and itwas dropped from the final analysis. Similarly, many let-ter writers did not include how many letters they wrote inthe previous year and these SLORs were excluded fromthe analysis as well (n = 196 [13.3%] of the SLORs).

We also did not blind abstractors from potentially ab-stracting data from one of their own evaluations. Thiswould be a rare occurrence and we did not feel that an ab-stractor would have a reason to alter their abstraction if ithappened to be their own evaluation, we did want to com-ment on this as a potential bias.

We have also made the assumption that ‘‘less letterswritten’’ in the previous year is a proxy for ‘‘less experi-ence evaluating students.’’ This might not reflect the ex-perience of the letter writer. There are many educatorswho have a lot of experience evaluating EM residentsbut may not have written many SLORs. It might be, inpart, an issue of the letter writer becoming more familiarwith the SLOR document.

This analysis is based solely on the applicant pool ofthe three EM residency programs at one university. It ispossible that this subpopulation is not the same as thepopulation of interest, i.e., the total applicant pool toEM residencies. However, these three EM residenciesat this University did receive applications from 27.6%of the total application pool, so we feel this should repre-sent a fairly accurate estimate because this is an academictertiary care and a community hospital setting.

CONCLUSIONS

SLORs written by less experienced faculty were statisti-cally more likely to have a GAS of ‘‘Outstanding’’ anda LOMA of ‘‘Very Competitive’’ compared to more expe-rienced letter writers. The distribution of the scores washeavily weighted to the highest two grades for bothGAS and LOMA and the lowest score of ‘‘Good’’ wasused only 2.5% of the time on the GAS and ‘‘UnlikelyMatch’’ was used only 1.9% of the time on the LOMA.There was no difference in GAS or LOMA scores inSLORs for applicants on their first rotation as comparedwith students’ scores on GAS or LOMA on SLORsfrom their second and subsequent rotations. HigherGAS and LOMA scores were associated with letterwriters that knew their applicant for more than 1 yearcompared with letter writers that knew their applicantfor 1 year or less. Missing data were fairly common. Res-idency program directors should consider the experienceof the letter writer when reviewing SLORs from appli-cants to their programs.

REFERENCES

1. Balentine J, Gaeta T, Spevack T. Evaluating applicants to emergencymedicine residency programs. J Emerg Med 1999;17:131–4.

2. Crane JT, Ferraro CM. Selection criteria for emergency medicineresidency applicants. Acad Emerg Med 2000;7:54–60.

3. Hayden SR, Hayden M, Gamst A. What characteristics of applicantsto emergency medicine residency programs predict future success asan emergencymedicine resident? Acad EmergMed 2005;12:206–10.

Standardized Letter of Recommendation 549

4. Oyama LC, Kwon M, Fernandez JA, et al. Inaccuracy of the globalassessment score in the emergency medicine standard letter ofrecommendation. Acad Emerg Med 2010;17(Suppl. 2):S38–41.

5. Council of Emergency Medicine Residency Directors. Availablefrom: http://www.cordem.org. Accessed March 30, 2013.

6. Keim SM, Rein JA, Chisholm C, et al. A standardized letter of rec-ommendation for residency application. Acad Emerg Med 1999;6:1141–6.

7. Girzadas DV Jr, Harwood RC, Dearie J, Garrett S. A comparison ofstandardized and narrative letters of recommendation. Acad EmergMed 1998;5:1101–4.

8. Association of American Medical Colleges. Electronic ResidencyApplication Service (ERAS�) data. Available from: https://www.aamc.org. Accessed March 30, 2013.

9. Association of American Medical Colleges. Available from: https://www.aamc.org. Accessed March 30, 2013.

550 D. L. Beskind et al.

ARTICLE SUMMARY

1. Why is this topic important?Standardized Letters of Recommendation (SLORs) are

an important part of the application for medical studentsapplying to an emergency medicine residency. Residencyprograms need to understand if the grading components ofthe SLOR’s are an accurate assessment of student perfor-mance and or how bias may be introduced into the SLOR.2. What does this study attempt to show?

This study is a retrospective review of all SLOR’s sub-mitted to three emergency medicine (EM) residency pro-grams during the 2011�2012 season. It evaluates if theglobal assessment components of the SLOR are affectedby the experience of the letter writer. It also looks at thedistribution of those scores on the global assessmentcomponents.3. What are the key findings?

Less experienced letter writers gave statistically highermarks on the Global Assessment Score (GAS) and Likeli-hood of Matching Assessment (LOMA) than experiencedletter writers. It also showed that the length of time a letterwriter knew an applicant was significantly associated withthe GAS and LOMA scores. Thirdly, it demonstrates thatthere was no difference in the GAS or LOMA scores ofstudents doing one rotation compared to students whocompleted two or more rotations in EM.4. How does this affect patient care?

It really has no impact on patient care.