a whole new world: institutional and program accreditation ... · a whole new world: institutional...
TRANSCRIPT
A Whole New World:
Institutional and Program
Accreditation Oversight under
NAS
Catherine M Eckart MBA
Associate DIO and Executive Director, GME
Stony Brook Medicine
Immediate Past President, AHME
1
If you think it was cold this week…………
3
• 603 beds
• 32,000 discharges
• 96,000 emergency visits
• 225,000 hospital-based OP visits
• 3,800 births/ year
• 5,800 employees
• 1,100 full-time physicians
• 60 accredited training programs
• 600 residents and fellows
• 496 medical students
• $97 million in annual research
funding
• 740 volunteers
• By the end of the session, the learner will be able to understand:
– Annual Institutional Review: • What is expected of the GME Office under NAS
• Partnering with programs on oversight
• Use of templates to assure program and institutional oversight compliance
– Annual Program Review: • The use of anonymous Resident and Faculty
Survey Information
• ACGME Categorization of requirements
• Citations vs. concerning trends
• Documentation and special reviews of underperforming programs
TODAY!!!
5
• How many represent osteopathic programs
and institutions?
• How many have been in GME for 20
years?
• 10 years?
• Less than 5 years?
• Even after 27 years of GME, it’s new every
day!
TELL US ABOUT YOURSELVES!
NAS
The New Accreditation System:
A Celebration of Culture Change and Innovation
The Next Accreditation System: Background and Rationale
MedPAC
COGME
Robert Wood Johnson
Foundation
Macy Foundation
8
Goals of NAS
• Realize the promise of Outcomes
• Prepare doctors for 21st century practice
• Free good programs to innovate
• Assist poor programs to improve
• Reduce the burden of accreditation
• Provide accountability for outcomes
to the public
Goals of NAS
9
Phase 1 Network
Has already started
• Intern
10 year cycles
10
The “Next Accreditation System” in a Nutshell
• CLER Visits to Sponsoring Institutions every 18-24 months
• Continuous Accreditation Model annually updated based on annual data submitted, other data requested, and program trends
• Scheduled Site Visits replaced by 10 year Self Study Visit
– Only if there are no trends negatively reported annually from resident or faculty surveys, deficient case loads, lack of Innovation
– Standards revised every 10 years Standards Organized by Structure / Resources /Processes /Outcomes
– Demonstration of Key Faculty Development
11
12
Phase 1 Network
Has already started
• Intern
13
• Continuous Accreditation Model
• 10 year self study visit replaces current site visits
• Program requirements revised every 10 years.
Based on annual data submitted
ACGME also takes into account…
ACGME complaints
Verified public information
Historical accreditation decisions/citations
Institutional quality and safety metrics
14
Terminology
Core Requirements:
Statements that define structure, resource, or process elements essential to
every graduate medical educational program.
Outcome Requirements:
Statements that specify expected measurable or observable attributes
(knowledge, abilities, skills, or attitudes) of residents or fellows at key stages of
their graduate medical education.
Detail Requirements:
Statements that describe a specific structure, resource, or process, for achieving
compliance with a Core Requirement.
15
What Happens during Review of my Program?
Citations will still be issued (if necessary) Programs have to provide response to citations in ADS
annually
Areas of non-compliance
Citations will not be considered resolved until the Review
Committee determines that they have been corrected
“Concerning Trends” (not a citation):
General concern(s) identified from annual review Written response not required
Will not have to be documented in ADS
Chair, the Program Director and the DIO/GMEC should act
on these areas
16
What happens after
my program has been reviewed? Annual data submission
ACGME Survey every spring
Self-Study visit every 10 years
Possible actions following Review
Committee:
Clarify information
Progress reports for potential problems
Focused site visit
Full site visit
Site visit for potential egregious violations
17
What is a Focused Site Visit?
Assesses selected aspects of a
program and may be used:
to address potential problems
identified during review of annually
submitted data
to diagnose factors underlying
deterioration in a program’s
performance
to evaluate a complaint against a
program
30-day notification given
18
When would there be a Full Site Visit? Application for a new core program At the end of the initial accreditation period Re-applications (withheld or withdrawn) Review Committee identifies broad
issues/concerns Other serious conditions/situations identified by
the Review Committee 60-day notification given Minimal document preparation Team of site visitors
19
© 2013 Accreditation Council for Graduate Medical Education Information Current as of December 2, 2013
Decisions on Program Standing in the NAS
STANDARDS
Outcomes Core Process
Detail Process
Continued Accreditation
Withdrawal of Accreditation
Accreditation
with Warning
Probationary
Accreditation
2-4% 10-15% 75-80%
<1%
Application for New Program
1. NAS: No Cycle Length
2. All programs with 1-2y cycles in the old system –
placed in Continued Accreditation with Warning Status
3. Percentages represent approximations based on
accreditation status received by programs in the past
BRINGING ALL OVERSIGHT….HOME
Next Accreditation System:
How do we prepare at home?
• The GME Office becomes the ACGME
• The GME Office becomes the Site Visitor
• The GME Office needs to have an ANNUAL
PROGRAM REVIEW
• Oversight is being redefined…
22
Annual Institutional Review
• All ACGME Survey Results: – Resident
– Faculty
• Institutional Site Visit Results
• CLER Results
• APE Oversight including program performance indicators, milestones and annual progress on action plans
• Oversight of all program self-study activity
• All RRC letters and progress reports
• Management of all Special and Focused Reviews
• Best practices and resolution of common concerns/issues
• An executive summary of the AIR must be submitted annually to Sponsoring Institution’s Governing Body
23
Resident Survey • Emphasis on themes rather than on individual questions
• High level view to minimize single resident impact
• Only significant deviation from compliance are indicators
• Trend data
• Domains: Duty hours, Faculty, Educational Content, Evaluations,
Resources, Patient safety, Teamwork
• Questions about:
– EMR
– Patient Safety
– Quality Improvement
– Handoffs
– Inter-professional Teams
– Backup when fatigued
24
ACGME Resident Survey 1) 80 hours a week, averaged over 4 weeks 5A (2+/3-)
5) In-house call no more often than q3 5A (2+/3-)
8) 24 + 3 (4 elsewhere) hours maximum shift 5A (2+/3-)
10) Sufficiency of supervision (unfortunately) 5A (2+/3-)
11) How often supervision @ appropriate level 5A (2+/3-)
12) Sufficiency of instruction (defined) 5A (2+/3-)
13) Faculty/staff interest in your education? 5A (2+/3-)
14) Effective in creating scholarly environment? 5A (2+/3-)
15) Can you access your evaluations? Y/N
16) Do you evaluate faculty >= once/year? Y/N
17) satisfied they’re treated confidentially? 5A (2+/3-)
18) Do you evaluate overall program >= annually? Y/N
19) satisfied they’re treated confidentially? 5A (2+/3-)
25
ACGME Resident Survey
20) Satisfied w/ how program uses evals to improve? 5A (2+/3-)
21) Satisfied with feedback after each major rotation? 5A (2+/3-)
22) Given G/O for major rotations? Y/N
23) Adequacy of instruction on negative effects of fatigue? Y/N
24) Opportunity participate in research/scholarly activity? 5A (2+/3-)
25) Balance between education and clinical demands? 5A (2+/3-)
26) Education compromised by service? 5A (2+/3-)
31) Reference materials available? Y/N
37) Satisfied with program’s process to deal confidentially with
resident problems/concerns?
5A (2+/3-)
38) Ability to learn been compromised by other learners? 5A (2+/3-)
39) Raise problems without fear of intimidation/retaliation? 5A (2+/3-)
26
ACGME Resident Survey
Question:
FIRST QUESTION: Are you: Beginning? Intermediate? Final?
Answer determines which questions you’ll receive. ~46 altogether
2) Rule: 1 day free in 7 How often did you break the rule that… 1 day in 7 free, averaged over 4 weeks
no longer underline “break the rule” 5 answers (2+/3-) * NOTE: NYS has never allowed averaging over 4 weeks. Must be 1 day off in 7.
3) Ever taken in-house call? Y/N
4) Ever been assigned night float? Y/N
7) Have >= 8 hours off between shifts? 5 answers (2+/3-)
27) Appropriate delegations to you…neither too easy nor too difficult/overwhelming?
5 answers (2+/3-)
27
ACGME Resident Survey
Question: 2012
28) Have you been provided with data such as patient survey results or clinical comparative data that show your effectiveness?
Y/N
29) Do you provide direct clinical patient care as a part of your duties? Y/N
30) Do you see patients across a spectrum of settings appropriate to your specialty?
Y/N
32) Do you use an EMR in primary hospital setting? Y/N
33) Do you use an EMR in primary ambulatory setting? Y/N
34) Are the 2 EMRs integrated? Y/N
35) How effective is EMR in allowing you to do your daily work? 5A (2+/3-)
36) Does your program allow you to transition care to another qualified provider when too fatigued?
Y/N/DK
28
ACGME Resident Survey
Questions: 2012
40) How often do you personally tell patients about the role you and others play in their care?
5A (2+/3-)
41) How effective is your program in creating a culture that reinforces your personal responsibility for patient safety?
5A (2+/3-)
42) Have you personally participated in program or institutional Quality Improvement or Patient Safety?
Y/N
43) How often is essential information lost or not communicated during handoffs?
5A (2+/3-)
44) How often do you work in interprofessional teams (defined)? 5A (2+/3-)
45) How effectively do these team members work together? 5A (2+/3-)
29
ACGME Resident Survey
46) Which of the following best summarizes your opinion of your residency program? A very positive experience – if I had to select again, I’d
definitely choose this one.
A positive experience – if I had to select again, I’d probably choose this one.
A neutral experience – If I had to select again, I might or might not choose this one.
A negative experience – If I had to select again, I would probably not choose this one.
A very negative experience – If I had to select again, I would definitely not choose this one.
30
• ACGME surveys all
residents and fellows
• Report available
ONLY if >=4 trainees
• Institutional Results
(shown) are given as
compared to US
• Program Results can
be compared to just
your specialty across
US
ACGME Resident/Fellow Survey
31
When analyzing results…
Be sure to recognize the THREE report formats
currently available on ACGME website:
1. YOUR (medicine) PROGRAM vs. ALL US (all
specialties)
2. YOUR INSTITUTION vs. ALL US (all specialties)
3. US (medicine) programs vs. ALL US (all specialties)
What you REALLY want is:
4. YOUR (medicine) PROGRAM vs. US (medicine)
programs
You have to create this yourself – from Report 1
(column 1) above vs. Report 3 (column 1)
32
Faculty Survey
• Hours spent teaching and supervising
• Questions in similar domains as resident survey:
– Faculty supervision
– Faculty development
– Educational Content including Scholarly activity
– Program and institutional resources
– Patient safety including Fatigue
– Teamwork
• Only Core faculty will be surveyed
(presumed to be more knowledgeable about program)
• Similar timing as resident survey
33
ACGME Faculty Survey
34
• ACGME now
surveys all core
faculty
• Institutional
Results (shown)
are given as
compared to US
• Program Results
can be compared
to just your
specialty across
US
ACGME Faculty Survey
35
20
14
Nat
ion
al M
ean
- R
esid
ent
Surv
ey
20
14
% C
om
plia
nt
Nat
ion
ally
- R
esid
ent
Surv
ey
20
14
Nat
ion
al M
ean
- F
acu
lty
Surv
ey
20
14
% C
om
plia
nt
Nat
ion
ally
- F
acu
lty
Surv
ey
% G
REE
N
% Y
ELLO
W
% R
ED
An
esth
esio
logy
res
iden
ts
An
esth
esio
logy
fac
ult
y
Co
lore
ctal
Su
rger
y fe
llow
s
Co
lore
ctal
Su
rger
y fa
cult
y
Der
mat
olo
gy r
esid
ents
Der
mat
olo
gy f
acu
lty
Emer
gen
cy M
edic
ine
resi
den
ts
Emer
gen
cy M
edic
ine
facu
lty
Fam
ily M
edic
ine
res
iden
ts
Fam
ily M
edic
ine
facu
lty
Inte
rnal
Med
icin
e re
sid
ents
Inte
rnal
Med
icin
e fa
cult
y
Car
dio
logy
fel
low
s
Car
dio
logy
fac
ult
y
End
ocr
ino
logy
fel
low
s
End
ocr
ino
logy
fac
ult
y
Gas
tro
ente
rolo
gy f
ello
ws
Gas
tro
ente
rolo
gy f
acu
lty
Infe
ctio
us
Dis
ease
fel
low
s
Infe
ctio
us
Dis
ease
fac
ult
y
4.7 95% 86% 14% 0% 4.9 5.0 4.9 4.6 4.5 4.4 4.8 4.8 5.0
4.9 98% 100% 0% 0% 4.9 5.0 5.0 4.7 4.9 4.9 5.0 4.8 4.8
5.0 100% 100% 0% 0% 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
5.0 99% 100% 0% 0% 5.0 5.0 5.0 5.0 4.8 5.0 5.0 5.0 5.0
4.7 97% 97% 3% 0% 5.0 5.0 5.0 4.8 4.7 4.6 5.0 4.6 5.0
4.8 97% 97% 3% 0% 5.0 5.0 5.0 4.9 4.8 4.4 5.0 5.0 5.0
patients' needs NA NA NA NA 0% 0% 0% 7% 11% 14% 0% 0% 0%
paperwork/ admin. Work NA NA NA NA 0% 0% 0% 13% 12% 7% 0% 11% 0%
educational experience NA NA NA NA 0% 0% 0% 0% 4% 0% 0% 0% 0%
cover for someone else NA NA NA NA 0% 0% 0% 0% 2% 0% 0% 0% 0%
night-float NA NA NA NA 0% 0% 0% 0% 5% 0% 0% 0% 0%
schedule conflict NA NA NA NA 0% 0% 0% 0% 0% 0% 0% 0% 0%
other NA NA NA NA 0% 0% 0% 7% 0% 7% 0% 0% 0%
4.3 92% 4.6 4.6 59% 32% 9% 4.4 4.7 4.7 4.7 4.3 4.2 4.7 4.1 4.6 4.3 4.7 3.9 4.5 5.0 4.8 3.9 4.4 4.8 4.3
residents seek supervisory guidance NA NA NA 4.4 51% 46% 3% 4.2 4.6 4.2 4.1 4.2 4.6 4.3 4.5 4.0 4.0
4.6 96% 4.6 76% 24% 0% 4.6 4.7 4.6 4.5 4.5 4.3 5.0 4.1 4.8
4.2 86% 4.2 41% 31% 28% 4.2 4.4 4.1 3.2 4.2 3.8 5.0 3.1 4.6
4.3 85% 4.3 4.6 63% 25% 12% 4.0 4.3 4.7 4.6 5.0 4.0 4.6 3.5 4.6 4.3 4.8 3.6 4.3 5.0 4.7 3.0 4.2 4.2 4.3
NA NA NA 4.1 33% 46% 21% 4.0 4.3 4.7 4.0 3.8 4.4 3.7 4.2 4.1 4.0
4.1 79% 4.1 34% 28% 38% 4.0 4.4 4.1 3.4 4.1 3.4 4.5 2.8 4.2
4.9 99% 4.9 100% 0% 0% 5.0 5.0 4.9 5.0 5.0 5.0 5.0 5.0 5.0
5.0 99% 5.0 98 93% 4% 3% 4.9 100 100 5.0 100 4.9 100 5.0 100 5.0 100 5.0 100 5.0 100 5.0 90 5.0 100
4.3 85% 4.3 34% 55% 10% 4.1 4.7 4.3 4.1 4.3 4.0 4.5 3.3 4.4
4.9 98% 4.9 98 94% 6% 0% 5.0 100 100 5.0 100 4.9 100 4.5 100 4.9 100 5.0 100 5.0 100 5.0 100 5.0 100
4.3 86% 4.3 34% 52% 14% 4.1 4.7 4.6 3.9 4.3 3.9 4.3 3.2 4.4
4.0 73% 4.0 31% 28% 41% 3.9 4.0 4.5 3.4 4.0 3.4 4.3 2.4 4.0
3.9 71% 3.9 4.3 38% 31% 31% 4.0 4.3 4.1 4.4 4.8 4.0 4.3 3.6 4.6 3.8 4.5 3.6 3.8 4.8 4.5 2.7 3.6 4.0 3.5
4.8 95% 4.8 97% 3% 0% 5.0 5.0 4.6 5.0 4.8 5.0 5.0 5.0 5.0
4.7 93% 4.7 99 94% 3% 3% 5.0 100 100 5.0 100 4.7 100 4.7 100 4.6 100 5.0 100 5.0 100 4.6 100 5.0 50
NA 74% NA 74 18% 26% 56% 89 71 50 71 100 69 82 83 64 100
4.0 76% 4.0 17% 52% 31% 4.1 4.3 4.2 3.7 3.7 3.3 4.0 2.7 4.0
4.2 81% 4.2 34% 45% 21% 4.0 4.7 4.4 3.9 4.0 4.1 4.5 3.6 4.4
Resident workload exceeds capacity to work 4.2 46% 41% 13% 4.4 4.1 4.7 3.3 3.6 4.6 4.0 4.5 4.1 4.3
3.9 71% 3.9 4.3 43% 35% 22% 3.9 4.1 4.0 4.6 4.8 4.4 3.7 3.2 4.4 3.7 4.5 3.9 4.1 4.5 4.3 3.4 3.8 4.4 4.3
supervisors delegate appropriately 4.6 99% 4.6 69% 24% 7% 4.5 4.7 4.8 4.1 4.5 4.1 4.8 4.3 4.8
given personal clinical effectiveness data 3.4 59% 3.4 14% 21% 66% 3.8 3.9 4.1 3.9 2.5 4.1 3.0 1.0 3.4
see patients across a variety of settings 4.8 95% 4.8 98 91% 9% 0% 4.7 100 100 5.0 100 4.3 100 5.0 100 4.8 100 5.0 100 5.0 100 5.0 100 5.0 100
Graduating residents' effectiveness NA NA NA 4.5 69% 28% 3% 4.5 4.6 5.0 4.7 4.0 4.6 4.7 4.5 4.4 4.3
Outcome achvmnt: graduating residents NA NA NA 4.7 95% 5% 0% 4.7 4.9 5.0 4.6 4.2 4.8 5.0 5.0 4.9 4.7
5.0 99% 5.0 100% 0% 0% 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
*EMR hospital 4.9 96% 4.9 97% 3% 0% 4.8 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
*EMR ambulatory 4.8 95% 4.8 76% 7% 17% 4.7 1.6 4.6 5.0 5.0 5.0 5.0 5.0 5.0
*EMR integrated 4.5 81% 4.5 90% 10% 0% 5.0 5.0 4.7 5.0 4.9 5.0 5.0 5.0 5.0
EMR effective in daily clinical work 4.0 94% 4.0 28% 48% 24% 4.4 4.7 4.0 3.4 4.4 4.4 4.5 3.8 4.2
prog has way to transition care if fatigued 4.2 80% 4.2 100 81% 12% 7% 4.3 100 100 5.0 100 4.2 100 3.4 100 3.8 100 4.7 100 5.0 100 4.6 100 5.0 100
4.1 80% 4.1 4.3 51% 35% 13% 4.1 4.3 4.4 4.3 5.0 4.4 4.7 3.3 4.6 4.2 4.7 3.9 4.2 4.8 4.7 3.0 4.2 4.2 4.7
4.5 91% 4.5 76% 21% 3% 4.6 5.0 4.5 3.9 4.5 4.4 4.8 4.8 5.0
4.2 80% 4.2 34% 31% 34% 3.9 4.4 4.4 3.6 4.3 3.9 5.0 2.7 4.6
tell patients respective role of res/faculty 4.5 99% 4.5 4.4 63% 35% 1% 4.7 4.4 4.7 4.9 4.8 4.6 4.6 4.3 4.8 4.2 4.5 4.4 3.9 4.8 4.3 4.4 4.0 4.4 4.0
patient safety responsibil ity reinforced 4.5 99% 4.5 4.3 63% 34% 3% 4.6 4.3 4.4 4.7 5.0 4.6 4.9 3.9 4.4 4.3 4.5 4.1 4.2 4.8 4.5 4.3 4.5 4.4 4.3
residents participate in QI/PS activities 4.3 81% 4.3 4.2 59% 28% 13% 4.6 4.2 4.9 4.4 4.5 4.9 4.9 4.2 5.0 4.7 4.8 3.6 4.1 4.0 4.8 4.6 4.0 3.4 4.3
information not lost during shift changes 4.0 97% 4.0 4.1 32% 47% 21% 4.1 4.1 4.4 5.0 4.8 3.7 3.7 4.0 4.2 3.7 4.2 3.9 3.8 4.3 4.2 4.1 4.3 4.2 4.3
res commun. effectively when transferring care NA NA NA 4.6 95% 5% 0% 4.6 4.6 4.8 4.7 5.0 4.7 4.5 4.8 4.6 5.0
NA NA NA 4.4 49% 51% 0% 4.4 4.3 5.0 4.1 4.2 4.5 4.1 4.3 4.5 4.3
residents work in teams 4.6 98% 4.6 62% 31% 7% 4.7 4.6 4.7 4.5 4.7 4.0 5.0 4.4 4.4
work in interprof teams effectively 4.3 99% 4.3 4.4 63% 31% 6% 4.4 4.4 4.6 4.6 5.0 4.5 4.3 3.4 4.4 4.0 4.5 3.9 4.1 4.5 4.5 4.2 4.5 4.0 4.7
% POSITIVE OVERALL EVALUATION OF PROGRAM NA NA NA 100 68% 13% 19% 83 100 100 86 100 93 100 34 100 93 100 71 100 100 100 33 100 80 100
25 NA NA NA 42 25 1 7 7 6 30 9 15 5 93 16 15 14 4 6 9 12 5 3
18 NA NA NA 42 18 1 7 7 6 30 7 15 5 84 13 14 11 4 6 9 11 5 3
72 NA NA NA 100 72 100 100 100 100 100 78 100 100 90 81 93 79 100 100 100 92 100 100
Jan
- F
eb
Jan
- F
eb
No
v -
Dec
No
v -
Dec
Jan
- F
eb
Jan
- F
eb
Ap
r -
Jun
Jan
- F
eb
Ap
r -
Jun
Jan
- F
eb
Mar
- A
pr
Mar
- A
pr
Mar
- A
pr
Mar
- A
pr
Mar
- A
pr
Mar
- A
pr
Mar
- A
pr
Mar
- A
pr
Mar
- A
pr
Mar
- A
pr
residents receive fatigue education
scholarly activity satisfaction
appropriate balance for education
education (not) compromised by service
satisfied w/process to address resident concerns
educ not compromised by other trainees
fear of intimidation/retaliation
Worked on scholarly project w/ resident
ACGME Survey Completion Schedule:
Eval
uat
ion
access evalautions
evaluate faculty
confidentiality of faculty evaluations
evaluate program
confidentiality of program evlauations
program uses evals to improve
satisfaction w/ feedback after assignmentFa
cult
y
sufficient supervision
appropriate supervision
sufficient instruction
interest in residency education
environment of scholarship and inquiry
satisfied with faculty devlopment
Pro
gram
:
Emp
ty P
rogr
ams
hav
e b
een
del
eted
Du
ty H
ou
rs
80 hours/week
1/7 days free
in house call every 3rd night
night float no more than 6 nights
8 hours off
continuous hours scheduled
Ru
les
bro
ken
bec
ause
:
Edu
cati
on
al C
on
ten
t
program teaches teamwork skil ls
96%
Pat
ien
t
Safe
ty
Team
-
wo
rk
Number surveyed
Number responding
Response Rate (percentage):
Res
ou
rces
access to reference materials
provided G/O rotations/assignments
satisfied with faculty devlopment 21% FACULTY
information not lost during shift changes 21% BOTH
appropriate balance for education 21% TRAINEES
education (not) compromised by service 22% BOTH
EMR effective in daily clinical work 24% TRAINEES
sufficient instruction 28% TRAINEES
satisfaction w/ feedback after assignment 31% BOTH
scholarly activity satisfaction 31% TRAINEES
fear of intimidation/retaliation 34% TRAINEES
environment of scholarship and inquiry 38% TRAINEES
program uses evals to improve 41% TRAINEES
Worked on scholarly project w/ resident 56% FACULTY
given personal clinical effectiveness data 66% TRAINEES
= No data. Program has <4 trainees OR faculty
= score lower than 4.0 or lower than 85% "yes"
= score of 4.5 or higher OR higher than 92.6% "yes"
= score of 4.0 through 4.4 or 85% through 92.5% "yes"
= faculty survey only questions
ACGME Resident and Faculty
Survey Results
A closer look…
20
12
Nat
ion
al M
ean
20
12
% C
om
plia
nt
Nat
ion
ally
% G
REE
N
% Y
ELLO
W
% R
ED
pro
gram
1
pro
gram
2
pro
gram
3
pro
gram
4
etc
4.4 92% 59% 38% 3% 4.8 4.7 5 4.4 4.4 4.4 4.9 4.8 4.6 4.9 4.5 4.2 4.7 4.8 4.5 4.9 4.7 4.0 5.0
residents seek supervisory guidance NA NA 76% 24% 0% 4.8 5 4.5 5 5 4.7 4.8 4.6 4.4
4.7 97% 93% 7% 0% 5.0 4.6 5.0 4.9 4.9 4.6 5.0 4.8 4.7 5.0
4.2 87% 58% 34% 9% 4.8 4.9 4.8 4.4 4.5 4.4 4.7 4.8 4.8 5.0 4.8 4.0 4.9 5.0 4.5 4.9 4.7 4.8 4.8
4.3 87% 48% 39% 13% 4.9 4.6 4.8 4.8 5.0 4.2 5.0 4.2 4.8 4.8
NA NA 48% 32% 20% 4.2 4.7 3.6 4.9 4.4 4.2 4.7 4.5 4.0
4.2 81% 48% 16% 36% 99 64 83 93 85 100 100 100 100 100 92 100 56 4.8 91 59 100 80 100
5.0 99% 100% 0% 0% 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
5.0 99% 100% 0% 0% 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
4.3 86% 43% 46% 10% 4.9 4.4 4.7 4.6 4.9 4.4 4.8 4.5 4.3 5.0
4.9 98% 97% 3% 0% 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
4.3 87% 51% 43% 6% 4.8 4.5 4.8 4.8 4.9 4.4 5.0 4.6 4.3 5.0
4.0 75% 34% 36% 30% 4.6 4.2 4.8 4.6 4.6 4.4 4.5 4.1 4.5 4.8
4.0 75% 33% 30% 37% 4.5 4.1 4.7 4.6 4.9 3.8 5.0 4.3 4.5 4.8
4.9 96% 100% 0% 0% 5.0 5.0 4.6 5.0 5.0 5.0 5.0 5.0 5.0 5.0
4.8 94% 90% 9% 1% 99 100 100 100 89 100 100 100 100 100 100 100 100 4.0 5.0 100 100 100 100
4.1 79% 42% 46% 12% 4.5 4.6 4.4 4.4 4.7 4.6 4.8 4.8 4.7 4.5
4.2 83% 46% 41% 13% 4.6 4.0 4.7 4.4 3.9 4.2 4.7 4.9 4.2 4.6 4.1 3.8 4.1 4.8 4.6 4.4 4.5 4.2 4.8
4.0 73% 37% 38% 25% 4.4 4.5 4.8 4.5 4.3 4.2 4.6 4.8 4.6 4.3 4.2 4.4 4.8 4.8 4.0 4.6 4.2 4.4 4.5
supervisors delegate appropriately 4.2 90% 81% 16% 3% 4.9 4.8 4.8 4.8 4.9 4.6 5.0 4.6 4.7 5.0
given personal clinical effectiveness data 3.6 66% 22% 13% 64% 3.6 3.0 4.6 5.0 3.2 2.6 4.0 3.2 3.0 2.0
variety of patients 4.9 97% 84% 8% 9% 100 97 100 50 77 89 100 100 100 78 100 100 100 3.0 91 94 100 100 50
Edu
cati
on
al C
on
ten
t
provided G/O rotations/assignments
Pro
gram
:
Emp
ty P
rogr
ams
hav
e b
een
del
eted
Facu
lty
sufficient supervision
appropriate supervision
sufficient instruction
interest in residency education
environment of scholarship and inquiry
satisfied with faculty devlopment
Eval
uat
ion
access evalautions
evaluate faculty
confidentiality of faculty evaluations
evaluate program
confidentiality of program evlauations
program uses evals to improve
satisfaction w/ feedback after assignment
residents receive fatigue education
scholarly activity satisfaction
appropriate balance for education
education (not) compromised by service
37
• 27% of the residents/fellows in your program thought this was a problem.
• ___I don’t think that this is a problem.
• ___I agree that this is a problem.
• If yes, please provide some details so that we can identify and resolve the problem:
_________________________________
_________________________________
_________________________________
Follow up anonymous/confidential survey:
Here’s an idea…
38
© 2013 Accreditation Council for Graduate Medical Education Information Current as of December 2, 2013
10 APEs per self study cycle
Self-
Study
VISIT
Ongoing Improvement
APE
Self-
Study
Yr 0 Yr 1 Yr 3 Yr 4 Yr 5 Yr 6 Yr 7 Yr 8 Yr 9 Yr 10 Yr 2
APE APE APE APE APE APE APE APE APE
Annual Program Evaluation (PR V.C.)
• Resident performance
• Faculty development
• Graduate performance
• Program quality
• Documented improvement plan
Here’s an idea…
• Get the most out of your APE
• GME should attend/participate
• TALK ABOUT EVERYTHING!!!
• Then, for posterity…
• Develop a Minutes Template and
• Make it available on your e-system site
• Important to follow up on last year’s APE!
40
Annual program evaluation
Agenda Discussion/
Action Plan
1. Current RRC program specific, common & institutional requirements
2. Review most recent RRC Letter of Notification and all concerns/citations
3. Review your most recent Internal Review report & all concerns/ citations
4 a. Overall program educational goals
b. Competency-based goals and objectives for each assignment
c. Competency-based goals and objectives for each educational level
5. Didactic curriculum (including topics on resident stress/fatigue)
Agenda Discussion/
Action Plan
6. Clinical curriculum- effectiveness of inpatient and ambulatory teaching experience (structure, evaluation of/by residents, case mix) 7. Volume and variety of patients and procedures (review clinical/case logs) 8. Participating training sites to ensure educational objectives are being met. (PLAs are current – dates, personnel, educational elements) 9. Review program evaluations by faculty and residents/fellows
10. Results from internal or external resident surveys, if >4 trainees
11. Financial/administrative support
41
Annual program evaluation
Agenda Discussion/
Action Plan
12. Quality of supervision
13. Performance/effectiveness of faculty (clinical knowledge, teaching ability, scholarly activity, commitment and professionalism) as eval by trainees
14. Faculty Development (Review program’s efforts to attend leadership development conference and innovative ways to develop teaching portfolio) 15. Patient Safety/Quality Improvement efforts-resident involvement (track PS/QI initiatives, including resident participation on departmental and/or any hospital committees)
Agenda Discussion/
Action Plan
16. Review current Resident/Fellow Performance, including all evaluations and resident participation on (non-PS/QI) departmental and/or any hospital committees 17. Review Graduate Performance, including Board Exam results and any “alumni” data available
18. Review Recruitment results/selection process
19. Review Duty Hours standards and ensure compliance 20. Review and track all research and scholarly activities faculty & residents/fellows
42
Annual program evaluation
25. PROGRESS TO DATE ON LAST YEAR’S ACTION PLAN:
Resolved? Brief summary of resolution or progress to date
Item 1:
Item 2:
Item 3:
Item 4:
Item 5:
Please expand as necessary
• Agenda • Discussion/ Action Plan 21.Review/update program-specific policies:
o Duty hours o Moonlighting o Eligibility/selection o Supervision o Transitions of care o Discipline/probation/remediation
22. Review Milestones: preparation, • implementation, reporting, process • assessment
23. Review CCC responsibilities: • Resident/fellow semi-annual evals • Provides PD with feedback & documentation
for meetings and file
24. Review all available program resources
43
Ten-Year Self-Study Visit Not to be confused with a focused or full site visit
requested by the ACGME
Not a traditional site visit
Will review core and subspecialty programs together
Review Annual Program Evaluations
Response to citations
Faculty development
Judge program success at CQI
Learn future goals of program
Will verify compliance with Core requirements
44
Self-Study vs. Self-Study Visit
Self-Study
Conducted by the program
Annual Program Evaluation
Review of program goals and improvement efforts
SWOT Analysis required
Self-Study Visit
Conducted by ACGME Field Staff members
Pilot of Self-Study:
Will be announced by ACGME 3/11/15 by Ingrid Philibert, PhD
Non-accreditation visit
Feedback on program SS
45
SWOT Analysis
46
SIMPLE SWOT ANALYSIS
STRENGTHS:
HAVE IT
WANT IT
WEAKNESSES
HAVE IT
DON’T WANT IT
OPPORTUNITIES
DON’T HAVE IT
WANT IT
THREATS
DON’T HAVE IT
DON’T WANT IT
47
SWOT ANALYSIS Apple SWOT analysis 2013
Strengths Weaknesses
1. Customer loyalty combined with
expanding closed ecosystem
2. Apple is a leading innovator in mobile
device technology
3. Strong financial performance
($10,000,000,000 cash, gross profit
margin 43.9% and no debt)
4. Brand reputation
5. Retail stores
6. Strong marketing and advertising teams
1. High price
2. Incompatibility with different OS
3. Decreasing market share
4. Patent infringements
5. Further changes in management
6. Defects of new products
7. Long-term gross margin decline
Opportunities Threats
1. High demand of iPad mini and iPhone 5
2. iTV launch
3. Emergence of the new provider of
application processors
4. Growth of tablet and smartphone markets
5. Obtaining patents through acquisitions
6. Damages from patent infringements
7. Strong growth of mobile advertising
market
8. Increasing demand for cloud based
services
1. Rapid technological change
2. 2013 tax increases
3. Rising pay levels for Foxconn workers
4. Breached IP rights
5. Price pressure from Samsung over key
components
6. Strong dollar
7. Android OS growth
8. Competitors moves in online music
market
SWOT ANALYSIS Apple SWOT analysis 2013
Strengths Weaknesses
1. Customer loyalty combined with
expanding closed ecosystem
2. Apple is a leading innovator in mobile
device technology
3. Strong financial performance
($10,000,000,000 cash, gross profit
margin 43.9% and no debt)
4. Brand reputation
5. Retail stores
6. Strong marketing and advertising teams
1. High price
2. Incompatibility with different OS
3. Decreasing market share
4. Patent infringements
5. Further changes in management
6. Defects of new products
7. Long-term gross margin decline
Opportunities Threats
1. High demand of iPad mini and iPhone 5
2. iTV launch
3. Emergence of the new provider of
application processors
4. Growth of tablet and smartphone markets
5. Obtaining patents through acquisitions
6. Damages from patent infringements
7. Strong growth of mobile advertising
market
8. Increasing demand for cloud based
services
1. Rapid technological change
2. 2013 tax increases
3. Rising pay levels for Foxconn workers
4. Breached IP rights
5. Price pressure from Samsung over key
components
6. Strong dollar
7. Android OS growth
8. Competitors moves in online music
market
• Conduct a SWOT analysis
using the CLER Pathways
document.
• The Pathways represent
the gold standard
• All of these pathways are
desirable, eliminating the
possibility of either weaknesses or threats.
So, categorize all of these as either
Strengths or Opportunities.
Here’s an idea…
50
SWOT Analysis: CLER Pathways
Can be conducted at institutional level – or at program level
51
KEY Question: How do you determine/study whether there are disparities in
treatment/services/access/outcomes for vulnerable populations served in your specialty?
CLE INVENTORY Program
Responses
Quality
Improvement Patient Safety
Quality Impro Curriculum
Identified QI initiatives
QI Projects Patient Safety Curriculum
Current P.S
Projects
Yes 14 16 18 14 14
No 10 8 6 10 10
Transitions of Care
Supervision Professionalism
Policies in place?
Policy to determine direct to indirect supervision?
Policy used to determine when a resident is able to take on supervisory responsibility for other residents?
Policies that complement institutional policies*?
Policy regarding resident/
fellow mistreatment
?
18 15 14 14 13
6 9 10 10 10
Duty Hours and Fatigue Mitigation
Policies that complement institutional policies on Duty Hour limits?
Educational methods or modules
regarding fatigue mitigation and
management other than SAFER?
Policies that complement institutional policies on Fatigue Mitigation?
13 7 6
10 16 17
• Conduct a CLE Inventory
• Ask your programs some
basic and important
questions about the six
CLER domains
• Allows differentiation of
“CLE” from “CLER” visits
• Upload backup documentation to create a
library of best practices/examples/policies
Here’s an idea…
53
• Q3: Does your program have a Quality Improvement curriculum?
• Q4: Has your program identified QI initiatives?
• Q5: Do your residents have ongoing and active QI projects?
• Q6: Does your program have a Patient Safety curriculum?
• Q7: Has your program identified Patient Safety initiatives?
• Q8: Do your residents have ongoing and active Patient Safety projects?
CLE INVENTORY:
QUALITY IMPROVEMENT & PATIENT SAFETY
54
• Q9. How do you determine/study whether there are disparities in treatment/services/access/outcomes for vulnerable populations served in your specialty?
• 10. What initiatives have you developed (or plan to develop) to address any disparities that you have identified?
• Q11: Does your program have policies regarding Transitions of Care that address: in-hospital transfers (eg, ED to floor, floor to unit, OR to PACU, etc.) outpatient to inpatient transfers? shift changes/hand-offs? inpatient to outpatient (to nursing home or to home) etc.
CLE INVENTORY:
DISPARITIES & CARE TRANSITIONS
55
• Q12: Does your program have a policy that outlines criteria used to determine progression from direct to indirect supervision?
• Q13: Does your program have a policy that outlines criteria used to determine when a resident is able to take on supervisory responsibility for other residents?
• Q14: Does your program have policies that complement institutional policies on: personal integrity? mutually respectful relationships? disruptive physician behavior?
• Q15: Does your program have a policy regarding resident/fellow mistreatment?
CLE INVENTORY:
SUPERVISION & PROFESSIONALISM
56
• Q16: Does your program have policies that
complement institutional policies on Duty
Hour limits?
• Q17: Do you use any educational methods
or modules regarding fatigue mitigation
and management other than SAFER?
• Q18: Does your program have policies that
complement institutional policies on
Fatigue Mitigation?
CLE INVENTORY:
DUTY HOURS & FATIGUE MITIGATION
57
THE DIO AND GMEC ARE RESPONSIBLE FOR PROGRAM OVERSIGHT
DIO’s Role
• Will oversee the Annual Program
Evaluations
• Will Oversee the program annual
download to RRC
• Will oversee all or any Duty Hour
violations from ACGME
• Will decide if there needs to be a
Special Review (ACGME-defined) or
Focused Review
(1 or 2 areas of noncompliance)
• Create subcommittees to help with this work:
PS/QI Patient Safety/Quality Improvement
ARQ Accreditation Review and Quality
ROAR Recruitment/Orientation/Activities/Retention
CLE Clinical Learning Environment
PD/PC PD Program Director/Program Coordinator Professional Development
GME Office should:
59
• PS/QI: The Big Q: Patient Safety and Quality Improvement
• Chair: William Moore, MD
• This team will focus on residency program curriculum and resident and fellow integration into the Patient Safety and Quality Improvement infrastructure of the institution. Recognizing that these areas have become a priority for SBMedicine, the residents and fellows must be contributing to and extracting maximum value from these institutional activities.
60
• ARQ: Accreditation, Review and Quality
• Chair: Elaine Gould, MD
• This team will focus on all annual program evaluations (APEs) as we move forward into the era of internal oversight at the local level. With accreditation cycles approaching 10 years, SB Medicine will be reviewing each program continuously for quality and accreditation successes and concerns. This team will also lead the Annual Institutional Review (AIR) required for the first time in 2014-15.
61
• ROAR: Recruitment, Orientation, Activities and Retention
• Co-Chair: Sonya Hwang, MD
• Co-Chair: Colby Rowe
• This team will focus on selection and eligibility of residents and fellows (and monitoring of programs regarding institutional policies), interfacing with the GME Office for institutional and program level orientation, and activities and retention strategies to retain the best of our graduates as faculty in our own institution.
62
• CLE: Clinical Learning Environment
• Chair: Robyn Blair, MD
• This team will have as its focus the top institutional priorities as identified by the ACGME for the CLER program. I would envision that the top priority of this group will be establishing an institutional policy on transitions of care/handoffs. Also important to the work of this team will be healthcare disparities, supervision, duty hours/fatigue management and professionalism - all at the institutional level.
63
• PD/PC PD: Program Director/Program Coordinator Professional Development
• Co-Chair: Meenakshi Singh, MD
• Co-Chair: Mary Tanderup
• This team will be responsible for assessing what is needed in order for our program directors and program coordinators to fulfill professional career goals and make more meaningful and satisfying contributions. By considering the professional development of both of these critical roles together, we will strengthen SBMedicine's PDs and PCs and our programs simultaneously.
64
Special Review
Conducted by: 1 GME staff member
1 faculty member 1 resident/fellow
Meet with: Residents
Faculty PD & RC
Report submitted to: ARQC
Annual Program Oversight Review
Conducted by:
2 GME Staff members
Meet with: Residents
Faculty PD & RC
Report submitted to: ARQC.
IF SIGNIFICANT CONCERNS ARE
IDENTIFIED…
Oversight of ACGME Programs by GME A
CG
ME
-RE
QU
IRE
D:
we d
o t
his
because w
e m
ust
AT
DIS
CR
ET
ION
OF
GM
E:
We d
o th
is b
ecause w
e s
hould
65
• Determine the most effective methodology for
providing the necessary program oversight in
NAS
• Discuss the elements of an ideal "program report
card" to analyze available quality data
• Construct an institutional dashboard of
meaningful program data, allowing for
identification of best practices and problem areas
needing attention
PROGRAM REPORT CARDS
66
• What is it that YOU want out of a Program Report Card?
• What is it that YOU want out of an Institutional Dashboard?
• While we are required to provide oversight, the ACGME is NOT being proscriptive about how we will each do it
• So…what does:
– A community hospital in Iowa want?
– An academic medical center in New York want?
– A major health system in Arizona want?
It’s up to you…
67
• The GMEC must demonstrate effective
oversight of underperforming programs
through a Special Review process.
– I.B.6.a) The Special Review process must
include a protocol that:
• I.B.6.a).(1) establishes criteria for identifying
underperformance; and,
• I.B.6.a).(2) results in a report that describes the
quality improvement goals, the corrective actions,
and the process for GMEC monitoring of outcomes.
Special Review in
Institutional Requirements July 2013
68
• APOR: Annual Program Oversight Review
– Happens for each accredited program
– Our cycle is two per week (108, now 113 per year)
– Format is similar to (former) IR, but MAINLY focus on RED areas and YELLOW areas.
• Special Review:
– Happens for programs with highest occurrences of RED areas and YELLOW areas. Focus on these areas.
– These programs are scheduled for Special Reviews before all other programs have APORs
• Focused Review:
– May only be one issue, but causes significant concern
– Conduct a Review ASAP
A note about program reviews
69
WHY IN THE WORLD WOULD GME CONDUCT AN ANNUAL (INTERNAL) REVIEW?
• Maintains contact with residents and fellows
• Allows GME to oversee specialty-specific requirements as well as common program and institutional requirements
• Will focus on the problem areas that GME identifies upon review of program data
Annual Program Oversight Review
70
ACGME Resident
and Faculty
Survey
2012
Nati
onal
Mea
n
2012
% Co
mplia
nt N
ation
ally
% GR
EEN
% YE
LLOW
% RE
D
Pedia
trics
resid
ents
Pedia
trics
facu
lty
Adole
scen
t med
icine
fello
ws
Adole
scen
t med
icine
facu
lty
Pedia
tric c
ritica
l car
e fell
ows
Pedia
tric c
ritica
l car
e fac
ulty
Pedia
tric e
merg
ency
med
fello
ws
Pedia
tric e
merg
ency
med
facu
lty
Pedia
tric c
ardio
logy f
ellow
s
Pedia
tric c
ardio
logy f
acult
y
Pedia
tric e
ndoc
rinolo
gy fe
llows
Pedia
tric e
ndoc
rinolo
gy fa
culty
Pedia
tric h
em/o
nc fe
llows
Pedia
tric h
em/o
nc fa
culty
Pedia
tric n
ephr
ology
fello
ws
Pedia
tric n
ephr
ology
facu
lty
Neon
atolog
y fell
ows
Neon
atolog
y fea
culty
Pedia
tric p
ulmon
ology
fello
ws
Pedia
tric p
ulmon
ology
facu
lty
Pedia
tric r
heum
atolog
y fell
ows
Pedia
tric r
heum
atolog
y fac
ulty
Pedia
tric g
astro
enter
ology
fello
ws
Pedia
tric g
astro
enter
ology
facu
lty
NA NA 62 55 67 54 66 48 38 82 76 45 23
4.8 96% 94% 6% 0% 4.8 4.9 4.9 5.0 4.9 4.8 5.0 4.6 5.0 5.0 5.0
4.9 98% 100% 0% 0% 4.9 4.9 5.0 5.0 5.0 5.0 5.0 4.6 5.0 5.0 5.0
5.0 100% 100% 0% 0% 5.0 4.9 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
5.0 99% 100% 0% 0% 4.9 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
4.7 97% 93% 7% 0% 4.9 4.9 4.4 4.9 5.0 4.6 5.0 4.9 5.0 5.0 4.9
4.8 97% 99% 1% 0% 4.9 4.6 5.0 5.0 5.0 5.0 5.0 4.7 5.0 5.0 5.0
patients' needs 5% NA 0% 0% 0% 0% 0% 0% 0% 9% 0% 0% 0%
paperwork/ admin. Work 7% NA 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
educational experience 2% NA 0% 0% 11% 0% 0% 0% 0% 0% 0% 0% 0%
cover for someone else 2% NA 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
night-float 2% NA 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
schedule conflict 2% NA 1% 0% 11% 0% 0% 0% 0% 0% 0% 0% 0%
other 2% NA 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
4.4 92% 59% 38% 3% 4.8 4.7 5 4.4 4.4 4.4 4.9 4.8 4.6 4.9 4.5 4.2 4.7 4.8 4.5 4.9 4.7 4.0 5.0 5.0 4.7 4.9
residents seek supervisory guidance NA NA 76% 24% 0% 4.8 5 4.5 5 5 4.7 4.8 4.6 4.4 5.0 4.8
4.7 97% 93% 7% 0% 5.0 4.6 5.0 4.9 4.9 4.6 5.0 4.8 4.7 5.0 5.0
4.2 87% 58% 34% 9% 4.8 4.9 4.8 4.4 4.5 4.4 4.7 4.8 4.8 5.0 4.8 4.0 4.9 5.0 4.5 4.9 4.7 4.8 4.8 5.0 4.7 4.8
4.3 87% 48% 39% 13% 4.9 4.6 4.8 4.8 5.0 4.2 5.0 4.2 4.8 4.8 4.6
NA NA 48% 32% 20% 4.2 4.7 3.6 4.9 4.4 4.2 4.7 4.5 4.0 4.0 4.5
4.2 81% 48% 16% 36% 99 64 83 93 85 100 100 100 100 100 92 100 56 4.8 91 59 100 80 100 100 100 63
5.0 99% 100% 0% 0% 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
5.0 99% 100% 0% 0% 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
4.3 86% 43% 46% 10% 4.9 4.4 4.7 4.6 4.9 4.4 4.8 4.5 4.3 5.0 4.4
4.9 98% 97% 3% 0% 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
4.3 87% 51% 43% 6% 4.8 4.5 4.8 4.8 4.9 4.4 5.0 4.6 4.3 5.0 4.6
4.0 75% 34% 36% 30% 4.6 4.2 4.8 4.6 4.6 4.4 4.5 4.1 4.5 4.8 4.4
4.0 75% 33% 30% 37% 4.5 4.1 4.7 4.6 4.9 3.8 5.0 4.3 4.5 4.8 4.4
4.9 96% 100% 0% 0% 5.0 5.0 4.6 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
4.8 94% 90% 9% 1% 99 100 100 100 89 100 100 100 100 100 100 100 100 4.0 5.0 100 100 100 100 100 100 100
4.1 79% 42% 46% 12% 4.5 4.6 4.4 4.4 4.7 4.6 4.8 4.8 4.7 4.5 5.0
4.2 83% 46% 41% 13% 4.6 4.0 4.7 4.4 3.9 4.2 4.7 4.9 4.2 4.6 4.1 3.8 4.1 4.8 4.6 4.4 4.5 4.2 4.8 5.0 4.9 4.1
4.0 73% 37% 38% 25% 4.4 4.5 4.8 4.5 4.3 4.2 4.6 4.8 4.6 4.3 4.2 4.4 4.8 4.8 4.0 4.6 4.2 4.4 4.5 4.3 4.7 4.8
supervisors delegate appropriately 4.2 90% 81% 16% 3% 4.9 4.8 4.8 4.8 4.9 4.6 5.0 4.6 4.7 5.0 5.0
given personal clinical effectiveness data 3.6 66% 22% 13% 64% 3.6 3.0 4.6 5.0 3.2 2.6 4.0 3.2 3.0 2.0 4.4
variety of patients 4.9 97% 84% 8% 9% 100 97 100 50 77 89 100 100 100 78 100 100 100 3.0 91 94 100 100 50 100 71 100
beginning residents' clinical effectiveness NA NA 21% 63% 17% 4.5 3.7 4.0 4.6 4.3 4.2 4.2 4.2 5.0 4.1
intermediate residents' clinical effectiveness NA NA 54% 46% 0% 4.7 4.3 4.5 4.6 4.4 4.6 4.3 4.6 5.0 4.5
advanced residents' clinical effectiveness NA NA 100% 0% 0% 4.9 4.8 4.8 5.0 4.8 4.6 4.8 4.8 4.8 5.0 4.8
5.0 99% 100% 0% 0% 5.0 4.7 5.0 5.0 5.0 5.0 5.0 4.6 5.0 5.0 5.0
*EMR hospital 4.6 90% 100% 0% 0% 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
*EMR ambulatory 4.5 88% 82% 13% 4% 4.9 4.4 4.6 5.0 5.0 5.0 5.0 3.9 5.0 5.0 5.0
*EMR integrated 4.7 76% 90% 7% 3% 4.3 5.0 5.0 5.0 5.0 5.0 5.0 4.3 5.0 5.0 5.0
EMR effective in daily clinical work 4.1 95% 46% 40% 13% 4.5 4.3 4.3 4.5 4.3 4.8 4.5 4.3 4.3 3.8 4.4
prog has way to transition care if fatigued 4.2 80% 61% 18% 21% 85 100 100 86 100 100 100 100 100 100 100 100 100 2.0 100 100 100 100 100 100 100 100
4.2 81% 50% 36% 14% 4.8 4.8 4.7 4.5 4.2 4.9 5.0 4.8 4.8 4.9 4.6 4.4 4.8 5.0 4.3 4.8 4.5 4.8 4.8 4.0 4.9 4.9
4.5 92% 66% 25% 9% 4.6 4.4 4.4 4.9 4.9 4.6 5.0 4.1 5.0 5.0 4.9
4.2 82% 43% 40% 16% 4.9 4.4 4.8 4.9 4.8 4.8 5.0 4.5 4.7 5.0 4.3
tell patients respective role of res/faculty 4.5 98% 66% 32% 2% 4.8 4.6 4.8 4.6 4.3 4.7 4.4 4.8 3.8 4.9 4.8 4.6 4.4 5.0 4.4 4.7 4.7 4.3 4.5 5.0 4.9 4.5
patient safety responsibil ity reinforced 4.5 99% 72% 27% 1% 4.8 4.8 4.8 4.6 4.4 4.8 4.9 4.6 4.8 4.7 4.8 4.8 4.7 4.8 4.5 4.6 4.5 4.6 4.5 5.0 4.9 4.8
residents participated in QI/PS activities 4.0 76% 62% 25% 13% 4.7 4.7 5.0 4.7 4.6 5.0 4.9 5.0 4.8 4.1 4.7 4.2 4.7 4.0 4.6 4.7 4.3 4.0 4.0 5.0 4.4 4.9
information not lost during shift changes 4.0 97% 23% 46% 32% 4.0 3.9 4.5 3.6 3.6 4.3 4.0 4.4 3.8 4.6 4.1 4.0 3.9 4.5 3.9 3.9 4.2 4.2 4.3 5.0 4.3 4.0
res commun. effectively when transferring care NA NA 100% 0% 0% 4.7 5.0 4.7 4.9 5.0 4.9 4.8 4.7 5.0 5.0 4.9
NA NA 64% 36% 0% 4.6 4.8 4.4 5.0 4.6 4.8 4.7 4.7 4.4 4.8 4.6
residents work in teams 4.6 98% 73% 24% 3% 4.9 5.0 5.0 4.9 5.0 4.8 5.0 4.5 4.8 4.3 4.7
work in interprof teams effectively 4.4 99% 63% 36% 1% 4.7 4.6 5.0 4.6 4.7 4.6 5.0 4.8 4.8 4.8 4.8 4.8 4.7 5.0 4.5 4.9 4.8 5.0 4.5 4.0 4.6 4.6
% POSITIVE OVERALL EVALUATION OF PROGRAM 4.4 88% 73% 15% 12% 99 100 100 93 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
82 38 1 6 14 17 9 7 8 5 9 12 6 10 4 3 11 17 6 5 4 4 7 8
82 36 1 6 14 13 9 7 8 5 9 12 5 9 4 3 11 17 6 5 4 4 7 8
100 95 100 100 100 76 100 100 100 100 100 100 83 90 100 100 100 100 100 100 100 100 100 100
ACGME Survey Completion Schedule:
Feb -
Mar
Feb -
Mar
Feb -
Mar
Feb -
Mar
Feb -
Mar
Feb -
Mar
Feb -
Mar
Feb -
Mar
Feb -
Mar
Feb -
Mar
Feb -
Mar
Feb -
Mar
Feb -
Mar
Feb -
Mar
Apr -
May
Feb -
Mar
Feb -
Mar
Feb -
Mar
Feb -
Mar
Feb -
Mar
Apr -
May
Feb -
Mar
Feb -
Mar
Feb -
Mar
residents receive fatigue education
scholarly activity satisfaction
appropriate balance for education
education (not) compromised by service
deal (confidentially) w/ resident concerns
educ not compromised by other trainees
fear of intimidation/retaliation
Evalu
ation
access evalautions
evaluate faculty
confidentiality of faculty evaluations
evaluate program
confidentiality of program evlauations
program uses evals to improve
satisfaction w/ feedback after assignment
Facu
lty
sufficient supervision
appropriate supervision
sufficient instruction
interest in residency education
environment of scholarship and inquiry
satisfied with faculty devlopment
Prog
ram:
Empt
y Pro
gram
s hav
e bee
n dele
ted
Duty
Hour
s
80 hours/week
1/7 days free
in house call every 3rd night
night float no more than 6 nights
8 hours off
continuous hours scheduled
Rules
brok
en be
caus
e:
hours/month teaching/supervising residents
Educ
ation
al Co
nten
t
program teaches teamwork skil ls
115,145
107,389
93%
Patie
nt
Safet
y
Team
-
work
Facu
lty
Ques
tio
Number surveyed
Number responding
Response Rate (percentage):
Reso
urce
s
access to reference materials
provided G/O rotations/assignments
71
Annual Progress Report to GME/DIO
• GME needs to monitor every program it
sponsors
• In my 4 institutions, we accomplish this by
the Annual Progress Report to GME/the DIO
• It collects in one place annual data GME
needs to determine success of each program
• ** USE THE DATA SUBMITTED**
• It’s always been important, but increasingly
emphasized in NAS 72
Metric
ACGME resident survey
ACGME faculty survey
Annual Report to the DIO
Annual Program
Evaluation/ Patient Safety
Survey
webADS Annual Update
Clinical Competency Committee (>Dec 2013)
Last Internal review
report, if applicable
Last RRC letter of
notification, if applicable
Adequacy of resources X X X X X X
Board Pass Rate X X X X
Caselogs/volume X X X X X X
Citation Status X
Duty Hours X X X X X
Educational content X X X X X X
Evaluations X X X X X
Faculty Development X X X
Faculty Scholarly Activity X X X X X
Fatigue Education X X X X
Learning Environment X X X X X X X
Patient Safety/Quality Imprv X X X X X
Policies X X
Preparation for and Achievement of Milestones
X X X X X
Recruitment/Retention X X
Resident/Fellow Scholarly Activity
X X X X
Supervision X X X X X X X
Teaching X X v
Teamwork X X X X
Transitions of Care X X X X x
Sources of Data for GME to review
73
Expected Distribution of Programs
Programs failing in
several areas (-)
Programs succeeding in
several areas (+)
Programs in the middle
PROGRAMS
SCHEDULED FOR
FIRST ROUND OF
APORs
PROGRAMS
REQUIRING
SPECIAL
REVIEWS
PROGRAMS
SCHEDULED FOR
SECOND ROUND
OF APORs
74
Advantages/Disadvantages of Dashboards
• Provides summary of data (overload)
• Look at rows and columns…
– COLUMNS = PROGRAM RESULTS
– ROWS = SYSTEM RESULTS
• Track progress of resolution (red/yellow/green)
• Should be used as a first alert to program and institutional problems
• But, is sometimes a superficial look at status
• Must always update each time a new event occurs
75
• Presence of… – ACGME Citation (resolved only when RRC
removes)
• ACGME Survey:
• Patient Safety Culture Survey • Disagree or Strongly Disagree
What warrants “red”? Some examples
76
= No data. Program has <4 residents /fellows OR faculty
= score lower than 4.0 or % lower than 85% "yes"
= score 4.5 or higher than 92.6% "yes"
= score of 4.0 through 4.4 or 85% through 92.5% "yes"
= faculty survey only questions
Actual report card – best program
Metric ACGME resident survey
ACGME faculty survey
Annual Report to
DIO
Culture of Patient Safety Survey
Last Internal review
report, if applicable
Last RRC letter of
notification, if applicable
webADS, including Annual
Update and caselogs
Clinical Competency Committee
(>Dec 2013)
Adequacy of resources 0 0 -1 x -2 x
Board Pass Rate 0 x x x
Caselogs/volume 0 0 0 x x -1
Citation Status 0 (NAS)
Duty Hours 0 0 x x x
Educational content 0 -1 0 -1 x x
Evaluations 0 0 x x x
Faculty Development -1 0 x
Faculty Scholarly Activity 0 0 x x x
Fatigue Education 0 0 0 x
Learning Environment 0 0 0 -1 x x
Patient Safety/Quality Imp 0 0 0 -1 x
Policies 0
Preparation for and Achievement of Milestones
0 0 x x x
Recruitment/Retention 0 x
Resident/Fellow Scholarly Activity 0 x x x
Supervision 0 0 0 0 x x x
Teaching 0 0 -1
Teamwork 0 0 0 -1
Transitions of Care 0 0 0 -1 x 77
Longitudinal Dashboard Creates Institutional Quality Improvement Model
78
Expected Distribution of Programs
Programs failing in
several areas (-)
Programs succeeding in
several areas (+)
Programs in the middle
PROGRAMS
SCHEDULED FOR
FIRST ROUND OF
APORs
PROGRAMS
REQUIRING
SPECIAL
REVIEWS
PROGRAMS
SCHEDULED FOR
SECOND ROUND
OF APORs
It’s a relative scale – not a true “report card”.
Need to determine who most needs our help
79
• The red programs receive a Special Review ASAP
• The yellow programs are scheduled for the first
APORs
• The final green programs are scheduled for the
second round of APORs
• We will conduct Special Reviews for all programs
for which the APOR results warrant further
intervention
Who gets our attention?
80
Information Current as of December 2, 2013
• We will meet with 1 program/week EVERY week…
• We will review:
– ACGME Resident Survey, ACGME Faculty Survey
– Annual Program Evaluation minutes
– webADS data update
– CLE discussion
• We will meet:
– With residents – don’t want to sacrifice face to face
– Discuss “red” areas from dashboards
– Will focus on specialty-specific challenges, too
Weekly Meetings with Programs
• Annual Oversight becomes a Challenge with Many Programs and Fellowships
– Make Core PD responsible for fellowships
– Involve the Chair in Annual reviews
– Involve the Dean/ CEO with AIR executive summary
• Finding Resources for additional work needed for Annual Oversight
– A subcommittee helps to tackle the workload
– Increase support staff for GME office (responsible for Annual oversight)
• Concentrate on quality and improvement as the goal for Annual oversight
– Hold departments and Program Directors accountable
– Identify who is responsible for Patient Safety and Quality
• Become expert at accessing WebAds data uploaded by program
Insight and Challenges
82
• Determine how to include (qualitative data):
– Annual Program Evaluation data
– webADS data submission
– Clinical Competency Committee data
– Milestones data
Will they/can they be quantitative enough to be incorporated into program report card?
Institutional Dashboards…compilation of each tool – due to our size…
Still left to do…
83
What Should You Do Now?
• Create continuous improvement projects for the residency program.
• Make Patient Safety, Quality Improvement, and Patient Centered Care an absolute priority.
• Create protected time for residents to be engaged in Patient Safety and Quality.
• Faculty must model and champion Patient Safety, i.e. hand washing, transition of care, and proper supervision.
• Learn the Milestones and decide how to assess for the milestones.
• Constantly re-assess your evaluation tools that work for you.
• Engage residents in quality outcomes that will improve care for patients (especially operative outcomes).
• Faculty Development! Faculty Development! Faculty Development and more Faculty Development
84
THANK YOU!!! • EXHALE!
• We are all in this together, and we will
all figure it out…
• Remember, we have all felt this way
before – when the competencies were
introduced – and we all survived!
• Questions and Feedback?
85
"If we can recognize that change and
uncertainty are basic principles, we can
greet the future and the transformation we
are undergoing with the understanding
that we do not know enough to be
pessimistic." - Hazel Henderson
86
87
