how accurate is the acgme resident survey? comparison between acgme and in-house gme survey

How Accurate is the ACGME Resident Survey?Comparison Between ACGME and

In-House GME SurveyBridget N. Fahy1, S. Rob Todd1, Judy L. Paukert2,

Melanie L. Johnson1, Barbara L. Bass1

1 Department of Surgery, The Methodist Hospital, Weill Cornell Medical College2 The Methodist Hospital Education Institute

Association of Program Directors in Surgery20 April 2010

Accuracy of ACGME Survey

Disclosure• No financial conflict of interest


Background• Purpose of annual Accreditation Council for

Graduate Medical Education (ACGME) Resident/ Fellow Survey:– monitor graduate medical education– provide early warning of potential non-compliance

• Not administered in conjunction with program site visit-- results are used at the time of the visit

• Used to prompt early site visits for programs with significant areas of non-compliance


Background• 15/32 questions allow only “yes/no” responses• Wording of some questions is confusing resident

misunderstanding of the question*• Graduate Medical Education (GME) Department administers

Annual Survey of Residents– similar content to ACGME survey– responses given using a 5-point Likert scale– opportunities for narrative responses for each question are solicited

• Differences in the results obtained from the two surveys identified

*Holt et al. Acad Med;85, 2010.

How Accurate is the ACGME Resident Survey?


Purpose of Study• Examine systematically the results of 15 questions

on the two surveys that assessed similar information – identify areas of significant disparity


Subjects• 26 general surgery residents

– 4 categorical residents at each PGY level– 2 preliminary PGY2 residents– 4 preliminary PGY1 residents

• Tertiary referral academic training center


Survey Instruments• ACGME

– www.acgme.org/acWebsite/resident_survey– 32-question survey administered annually – Response options: “yes/no” = 15, 3-point scale = 4,

5-point Likert scale = 9– Residents assured of anonymity

http://www.acgme.org/acWebsite/resident_survey


Survey Instruments• GME

– 53-question survey administered annually– Administered to residents from all programs– Response options: majority given on 5-point Likert

scale + option for narrative responses– Resident assured of anonymity


Common Survey ItemsFaculty support1. Do the faculty spend sufficient time teaching residents/fellow in your program?2. Do the faculty spend sufficient time supervising residents/fellows in your program?3. Do your faculty members regularly participate in organized clinical discussions, rounds, journal clubs, conferences?4. Do you have the opportunity to confidentially evaluate your faculty, in writing, electronically, at least once a year?Rotation / program evaluation5. Do you receive written or electronic feedback on your performance for each rotation? 6. Do you have the opportunity to confidentially evaluate your program, in writing, electronically, at least once a year?7. Has your program provided you access to, either by hard copy or electronically, written goals and objectives for the

program overall and for each rotation and major assignment?8. Have residents/fellows had the opportunity to assess the program for the purposes of program improvement?Working / educational environment9. Does your program and/or institution have a system through which you are able to raise and resolve issues without

fear of intimidation or retaliation?10. How often are you able to access, either in print or electronic format, the specialty specific and other reference

materials that you need?11. To what extent to trainees who are not part of your program (such as residents from other specialties, subspecialty

fellows, Ph.D. students and nurse practitioners) interfere, in a negative way, with your education?12. Do your rotations and other major assignments emphasize clinical education over any other concerns, such as

fulfilling service obligations?13. Does your program offer you the opportunity to participate in research or scholarly activities?Duty hours14. Have you had sufficient education (from your program, your hospital(s), your institution, or your faculty) to recognize

and counteract the signs of fatigue and sleep deprivation?15. Have you met the ACGME duty hour requirements?


Subject DemographicsGME Survey

N=26Age Yrs. (median) 30Male:Female 18:8Postgraduate year 1 8 2 6 3 4 4 4 5 4Categorical:Preliminary 20:6


Faculty SupportYes on

ACGME (%)Yes on

GME (%)P-Value

Faculty spend sufficient time teaching

12/25 (48) 22/26 (85) 0.008

Faculty spend sufficient time supervising

24/25 (96) 25/26 (96) NS

Faculty participation in didactics 25/25 (100) 25/26 (96) NSFaculty provide feedback 22/25 (88) 23/26 (88) NS


Rotation/Program EvaluationYes on

ACGME (%)Yes on GME

(%)P-Value

Opportunity to evaluate program 25/25 (100) 26/26 (100) NSOpportunity to evaluate faculty 25/25 (100) 25/26 (96) NSReceive goals/objectives for rotation and program

25/25 (100) 25/26 (96) NS

Opportunity to assess program for improvement

23/25 (92) 25/26 (96) NS


Work / Educational EnvironmentYes on

ACGME (%)Yes on

GME (%)P-Value

Raise issues in program/institution

23/25 (92) 25/26 (96) NS

Ability to access reference materials

22/25 (88) 25/26 (96) NS

Interference by other trainees 6/25 (24) 21/26 (81) 0.0005

Rotations emphasize education over other concerns

7/25 (28) 19/26 (73) 0.001

Opportunity for research 25/25 (100) 25/26 (96) NS


Duty Hours ComplianceYes on

ACGME (%)Yes on

GME (%)P-Value

Education regarding sleep deprivation and fatigue

22/25 (88) 23/26 (88) NS

Compliance with ACGME duty hour requirements

* 24/26 (92)

80 hours/week, averaged over a four week period 23/25 (92) N/A

1 day in 7 free from all educational and clinical responsibilities, averaged over a four-week period

24/25 (96) N/A

10-hour time period provided between all daily duty periods and after in-house call

22/25 (88) N/A

In-house call must occur no more frequently than every third night, averaged over a four-week period

25/25 (100) N/A

Continuous on-site duty must not exceed 24 consecutive hours 25/25 (100) N/A

No new patients may be accepted after 24 hours of continuous duty 23/25 (92) N/A

At-home call must not be so frequent as to preclude rest and reasonable personal time

23/25 (92) N/A


Summary• Differences in responses found for 3 items:

–Faculty time spent teaching– Interference of other trainees–Rotation emphasis of education over other

concerns• Responses on ACGME survey were less favorable

than on GME survey


Conclusions• Two factors may contribute to inaccurate responses on

ACGME survey:– Limited range of available responses– Confusing wording of questions

• Inaccurate responses elicited on the ACGME survey may result in erroneously elevated rates of non-compliance

• Further validation of the current ACGME survey is warranted

how accurate is the acgme resident survey? comparison between acgme and in-house gme survey

Documents