department of graduate medical education (gme) now that we have an institutional report card---how...

Department of Graduate Medical Education (GME)

Now That We Have an Institutional Report Card---How Do We Use It?

Conference Session: SES010

2012 ACGME Annual Education Conference

Ann Dohn, MA, DIO,

and

Nancy Piro, PhD,

Program Manager/Education Specialist

Department of Graduate Medical Education

Stanford Hospital & Clinics


Disclosure

• No conflicts of interest to report


Session Objectives

At the end of this session, participants will be able to:

1) Effectively use objective data in Balanced Report Cards to make Institutional and Programmatic decisions.

2) Understand strategies for gaining Institutional Administration (C-Suite) support for DIO/GMEC Decisions based on balanced report cards

3) Predict future ACGME review outcomes based on empirical data from balanced report cards


Setting the Stage

• How does GME administration function in 2012?

• What are the barriers?

• What are the challenges?

• How can we be heard by the C-Suite?

• What language should we speak?

• How do we optimize success?


How does GME Administration Function?

• How do we know?

• What do we measure?

• What management strategies do we use?


Historically…..

• GME has not used objective data in making many decisions

• Instead, decisions were often made based on:– Politics– Financial considerations– Special deals– Hunches– Ability to scream loudly and frequently– Fear


But what is going on in the environment?

• Evidenced based– Balanced score cards– Increased accountability for resources– Transparency


Fitting GME within the overall business culture

• Remember…your “C-Suite” colleagues often have that magical MBA …..

• They think differently

• They use a different language


Fitting GME within the overall business culture

• We need to understand how the C-Suite thinks

• AND learn to think the same way…

AND speak their language…

To gain their support!!!


What Considerations does the C-Suite have with Respect to Data?

• Data must have:– An Organizational focus– Regular Means of Data Collection – Include External Assessments as well as Internal

Assessments

• Data used must be– Standardized (comparable) / lean– Capable of transformation – Aligned with mission statement


DIOs need to be able to make decisions based on comparative data

• Data must:– Be Organizationally Focused– Have Regular Means of Data Collection – Include External Assessments as well as Internal

Assessments– Standardized– Capable of transformation – Aligned with mission statement


Conclusion

• CEOs and DIOs have the same requirements with respect to their data needs


But what models for data do we have?

• Few data models exist for GME

• The concept of Institutional Accountability is relatively new. – Until the ACGME Outcome Project, there was no

centralized curriculum oversight in GME, unlike medical schools / UME.


The “Report Card” Vision

• In 2005, Stanford hired its first PhD in GME– The vision was to

develop tools to construct evidence based decision-making for Graduate Medical Education consistent with our mission

– “We needed a Report Card”…


SIGH….

• It wasn’t as easy as first thought!


Our First Attempt …


Background on InstitutionalReport Cards

• Government and Industry Models– Multiple models exist and can be used as per specific

purpose:• GRPA (Government Performance and Results Act)• Organizational Report Cards• Balanced Scorecard• Benchmarking• Program Evaluations• Social Indicators• No Child Left Behind• C BEST• College Performance Testing


Which Model Should We Use?

• We needed a model that:– was organizationally focused and managed – had a track record of effective use– fit our existing structure with multiple programs and

organizations– was flexible enough to be adapted for use on an

annual basis – not an accreditation cycle-Regular Data Collection

– was “Easily Digestible” – had internal and external measurement dimensions


Our Choice

Balanced Scorecard Framework in an Organizational and/or

Report CardTool

Best of Both Worlds


The Balanced Scorecard Approach

• The Balanced Scorecard is a performance measurement and performance management system developed by Robert Kaplan and David Norton (1992, 1996) – adopted by a wide range of leading edge

organizations, both public and private.

(“The Balanced Scorecard--Measures That Drive Performance,” Harvard Business Review, Jan-Feb 1992; and “The Balanced Scorecard-Translating Strategy into Action,” Harvard Business School Press, 1996)


Stanford’s Report Card – Three Programs


Key to the Report Card


Balanced Scorecard Strategic Perspectives

Institutional / Financial Growth

MissionVision

Strategy

Learning

Program Processes

Resident

Do we continue to improve (outcomes)?

Are our programs excelling?

How do our residents see us?

Are we putting our resources in the right

places?


Selection of Balanced Report Card Measures

PRE PERI POSTINTERNAL EXTERNAL

RESIDENT PERCEPTIONS“Voice of the Resident”

PROGRAM PROCESSES

• Faculty Eval of program• Duty Hr Violations• # Internal Citations

LEARNING OUTCOMES

FINANCIAL / GROWTH

• Resources

• GME Internal HS Survey Recommendation of Program• GME Internal HS Survey Overall Satisfaction • Resident Evaluation of Program

• Match Depth• ACGME Survey Responses

• Teaching• Supervision• Scholarship• Non Hostile/Intimidating Environment• Address Concerns Confidentially

• Training Exams • Board Pass Rates• Career Placement

• Grants

• ACGME Cycle Length• # ACGME Citations


Balanced ScoreCards “Report Cards”

Had– Organizational focus

– Regular Means of Data Collection

– Include External Assessments as well as Internal Assessments

Were– Standardized

– Capable of transformation

– Aligned with the Mission of Stanford Graduate Medical Education (GME)


Stanford Hospital & Clinics Use of the “Balanced Report Card”

• The Stanford Report Card is built on the Balanced Scorecard conceptual framework for translating an organization’s vision into a set of performance indicators distributed among four perspectives adapted for GME: 1. Resident Perception Measurements2. Program Processes3. Learning Outcomes4. Financial/Growth


Stanford Background

Stanford University Medical Center currently sponsors 85 ACGME-accredited training programs with over 1100 enrolled residents and fellows.


Stanford University Medical Center Mission

• Dedication to pursuing the highest quality of patient care and graduate medical education, recognizing that one of its major responsibilities is the provision of organized educational programs. – Support of quality graduate medical education programs and

excellence in residency training and research. – Guidance and supervision of the resident while facilitating the

resident’s professional and personal development and ensuring safe and appropriate care for patients.

– Provision of adequate funding of graduate medical education to ensure support of its faculty, residents, ancillary staff, facilities, and educational resources to achieve this important mission.

– Ensuring that all of its graduate medical education programs meet or exceed the Institutional and Program Requirements promulgated by the Accreditation Council for Graduate Medical Education.


Translating the Stanford Hospital & Clinics’ GME Mission to the Balanced Report Card

1. Resident Perception Measurements“Guidance and supervision of the resident while

facilitating the resident’s professional and personal development and ensuring safe and appropriate care for patients.”

2. Program Processes“Ensuring that all of its graduate medical education

programs meet or exceed the Institutional and Program Requirements promulgated by the Accreditation Council for Graduate Medical Education.”


Stanford Hospital & Clinics Report Card

3. Learning Outcomes

“Support of quality graduate medical education programs and excellence in residency training and research.”

4. Financial/Growth

“Provision of adequate funding of graduate medical education to ensure support of its faculty, residents, ancillary staff, facilities, and educational resources to achieve this important mission.”


Stanford Hospital & Clinics Balanced Report Card

• Indicators are designed to measure SHC’s progress toward achieving its vision; other indicators are designed to measure the long term drivers of success.

• Through the balanced report card SHC :– Monitors its current performance (finances, resident

satisfaction, learning outcomes and program process results)

– Monitors its efforts to improve processes, educate residents– Enhances its ability to grow, learn and improve the quality

of its fellowship and residency educational programs.


Why GME Thinks We Need This

• ACGME and Institutions are increasingly holding DIOs and GME Committees accountable for their utilization of institutional resources.

• Actions / decisions must be based on documented real-time analyses of needs.


Selection of Balanced Report Card Measures

PRE PERI POSTINTERNAL EXTERNAL

RESIDENT PERCEPTIONS“Voice of the Resident”

PROGRAM PROCESSES

• Faculty Eval of program• Duty Hr Violations• # Internal Citations

LEARNING OUTCOMES

FINANCIAL / GROWTH

• Resources

• GME Internal HS Survey Recommendation of Program• GME Internal HS Survey Overall Satisfaction • Resident Evaluation of Program

• Match Depth• ACGME Survey Responses

• Teaching• Supervision• Scholarship• Non Hostile/Intimidating Environment• Address Concerns Confidentially

• Training Exams • Board Pass Rates• Career Placement

• Grants

• ACGME Cycle Length• # ACGME Citations


Findings

• After several years of data collection….– Some data is more valuable – Some data drives change


VOICE OF THE RESIDENT

Internal Measures

• Resident Overall Evaluation of the Program

• Duty Hour Violations

• GME/HS Survey - Overall Satisfaction ?

• GME/HS Survey - Recommend Program ?

External Measures

• Match Depth• % Top Medical Schools• ACGME Survey - %

Compliant Responses•Teaching

•Supervision

•Scholarship

•Non Hostile/Intimidating Environment

•Address Concerns Confidentially


PROGRAM PROCESSES

Internal

• Faculty Overall Evaluation of the Program

• # Internal Citations

External

• ACGME Cycle Length

• # ACGME Citations


Case Study - Stanford

• How the DIO uses the Balanced Report Card:– A Tale of Three Programs…– Starting from the bottom….– Assuming that being at the “bottom” may mean

“needs more help”


How Do We Use this Data?

• Look at Indicators that are Resident Driven – “Voice of the Resident”– Is there a discrepancy between the voice of the

resident and the other indicators?• Would the majority of the residents not choose the

program again yet the program receives a five year accreditation cycle with one or no citations?


How Do We Use this Data?

• How Do the Programs Compare Against Each Other?

• How do they compare against their ACGME Cycles?


Stanford’s Report Card – Three Programs


What does this tell the DIO? (Program A)

• Setting the stage:– 2008-ACGME gave a 5 year cycle with 1

citation

– 2009-ACGME resident survey shows

“tanking program”

– 2009-GME House Staff survey shows

“program in trouble”

– Duty Hour violations are > 15 times greater than the Institutional Average

» Over a three year period


Stanford’s Program A


What does this tell the DIO?

– Educational Milieu• Intimidation and Fear of Retaliation - Increasing

• Lack of confidence in the program to confidentially deal with problems or concerns - Increasing

• Overall resident satisfaction with the program – decreasing

• Would recommend: extremely low in 2008/09 (lowest of any SHC program)

Definitely an early warning….


How Did We Use the Data

• Validated data with resident interviews (single and group) You know you have a problem when the residents call and ask to

meet you at Starbucks…


Setting up an “Action Plan”

• Met with program leadership – Shared data– Discussed “their” interpretation of the data– “Brain-stormed” with program leadership

• “How can GME help you?”


Program View on Situation

• Need more help…more residents…more MD extenders….

• Residents are the problem…”not tough” and they “whine”….


GME Evaluation of the Situations

• Qualitative analysis of every comment on the internal house staff survey

• Trend analysis of surveys

• DIO used her training in conflict resolution

• Report developed to define the problems– Shared with C-Suite


Action Plan

• Used data regarding duty hours with the C-Suite to:– justify hiring additional physician extenders– reconfigure rotations

• Used resident program satisfaction data with the Dean to:– leverage program change and “motivate”

program leadership


Results - Success ?

• Duty Hours have improved

• Waiting on resident sat from 2012 survey (May)


Program B

• Setting the stage:– Internal Review in 2008-09 found a program substantially

out of compliance» Problems were largely identified as structural ie:

formal policies/curriculum» Acting program director» Small, internationally known fellowship» Very competitive» Trainees pleased with program» Trainees interviewed 2 months after start of the program


Program B


Stanford’s Program B


• Assigned a GME staff professional to work with the program director on the missing “structural elements” in the program.

• DIO monitored progress

Setting up an “Action Plan”


Progress…

• Internal review citations corrected

• 2010 ACGME review gave 5 years and a single citation


However….the voices of the residents

• ACGME survey begins to show concerns over training environment

• DIO starts receiving individual concerns from trainees

• Program becomes large enough to qualify for its own internal survey (smaller programs are grouped)– Internal survey shows major concern areas on

Report Card…a sea of “RED”


Power of the Data

• Concerning data is shared with program leadership


Report Card Speaks

• Changes are implemented– Service is reduced– Associate program director appointed– Protected time for education enforced– Department leadership engaged– Administration support in place


Disconnect

• Report card allowed GME to implement intervention which successfully addressed the areas of concern

• BUT…..ACGME (not having access to the most recent data) called for an early site visit.


Program “C”: Setting the Stage

• The program was already experiencing difficulties before their most recent ACGME site visit– Tough transition to a new program director

when former director abruptly left

• Problems identified on both the ACGME and internal GME house staff surveys


Program “C”


Stanford’s Program C


Action Items

• GME assigned a staff member to work closely with program– Note: program director readily accepted any

and all assistance-knew he had “inherited” a “train wreck”.


Progress?

• Program continued to spiral down……– Lack of C-Suite “buy in”


ACGME Outcomes

• ACGME site visit– Proposed probation which resulted in:

• HUGE wake-up call for the C-Suite, residents, and four major affiliates

• Data invaluable in presenting problems


Action Items

• Data presented to– Department chair– Dean– DIOs at major affiliates– CEOs– RESIDENTS


Consensus

• Decision to “save the program”– C-Suite– Program leadership– GME


Results

• Appealed the probation– 2 year cycle– Program continues to improve


What does this tell the DIO?

Report card allows for easy monitoring of multiple factors

With longer ACGME accreditation cycles in our futures…imperative to be able to monitor programs frequently


Lessons Learned…

• Three very different programs– But data drives change in all– In all three cases the final outcome was data

driven– In all three cases the C-Suite was needed to

resolve issues-and was “moved” to help ONLY after being presented with data


Lessons Learned (continued)

• As good as we think our “hunches” are…– The C-Suite

• wants facts, figures, stats, etc

• You need to “prove” your case

• The C-Suite is VERY sensitive to the “norm”, average, scores of other programs, benchmarks, etc.

• Your argument needs to be evidenced-based

• Don’t forget the outcomes!!


A Tale of Three Programs


Questions

department of graduate medical education (gme) now that we have an institutional report card---how...

Documents

data needs slide

report card slide

gme administration function

comparative data data

data models

balanced report cards

medical schools ume

objective data