Measuring Progress in Patient Safety

Peter Pronovost, MD, PhD, FCCMJohns Hopkins University

ExercisePlease answer each question with a score of 1 to 5.

1 is below average, 3 is average and 5 is above average

• How smart am I• How hard do I work• How kind am I• How tall am I• How good is the quality of care we provide

Improving Sepsis Care

(n= 19 ICUs)Mortality

13.1

21.9

41.8

0.0

10.0

20.0

30.0

40.0

50.0

Oct - Dec2003

Mar - May2004

July - Sept2004

%

69% Reduction (p < 0.001)

ICU LOS

6.27.6

10.0

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

Oct - Dec2003

Mar - May2004

July - Sept2004

Day

s

36% Reduction (NS)

Improving Sepsis Care

(n= 19 ICUs)Mortality

13.1

21.9

41.8

0.0

10.0

20.0

30.0

40.0

50.0

Oct - Dec2003

Mar - May2004

July - Sept2004

%

69% Reduction (p < 0.001)

ICU LOS

6.27.6

10.0

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

Oct - Dec2003

Mar - May2004

July - Sept2004

Day

s

36% Reduction (NS)

Central Mandate

Local Wisdom

Scientifically Sound Feasible

xx

Structure

ContextHave we created a culture of safety?

Process Outcome

Have we reduced the likelihood of harm?

How often do we do what we are supposed to?

How often do we harm?

Conceptual model for measuring safety

IT

Adapted from Donebedian

Keystone ICU Safety Dashboard

2004 2006

How often did we harm (BSI) 2.8/1000 0

How often do we do what we should

66% 95%

How often did we learn from mistakes

100s 100s

% Needs improvement in Safety climateTeamwork climate

84%82%

43%42%

Pronovost JAMA 2007

Translating EvidenceInto Practice

* Envision the problemwithin the larger health

care system

* Engage collaborativemulti-disciplinaryteams centrally(stages 1,2,&3)

and locally(stage 4)

1. Summarize the Evidence

Convert interventions to behaviors

2. Identify local barriers toimplementation: understandthe process and context of

work

3. Measure Performance

4. Ensure all patientsreceive the interventions

Identify Interventions associatedwith improved outcomes

Select interventions with the largestbenefit and lowest barriers to use

Enlist all stakeholders to shareconcerns and identify potentialgains/losses associated withintervention implementation

Observe staff performing theinterventions

"Walk the process" to identifydefects in each step of intervention

implementation

Measure Baseline Performance

Develop and pilot test measures

Select Measures(process and/or outcome)

Engage

Explain why the interventions are

important

Execute

Design an intervention “toolkit” targeted to barriers employing standardization,

independent checks and reminders, and learning from mistakes

Educate

Share the evidence supporting the interventions

Evaluate

Regularly assess performance

measures

Pronovost BMJ 2008

Comprehensive Unit-based Safety Program (CUSP)

1. Educate staff on science of safety http://www.jhsph.edu/ctlt/training/patient_safety.html

2. Identify defects3. Assign executive to adopt unit4. Learn from one defect per quarter and implement

teamwork tools

Pronovost J, Pronovost J, Patient Safety,Patient Safety, 20052005

What can be measured as a valid rate?

• Rate requires– Numerator- event– Denominator- those at risk for event– Surveillance for events and those at risk

• Minimal and Known Error– Random error– Systematic error

Sources Variation in Safety measures

• True variation in Safety• V data quality/definition/methods of

collection• V case mix• V historical rates• Chance

Measuring Preventable Harm

• Measure rate or counts directly– High sensitivity low specificity

• Estimate observed/expected (O/E)– Low sensitivity and specificity

• Link process and outcome– High specificity and moderate sensitivity

Process Measures

• Validity of the construct• Validity of how we measure construct

It is Ok to have non-rate measures

Self reported measures are generally not valid as rates

A common mistake is interpreting a non-rate measure as a valid rate

Learning from Mistakes

• What happened?• Why did it happen (system lenses)• What could you do to reduce risk• How to you know risk was reduced

– Create policy/process/procedure– Ensure staff know policy– Evaluate if policy is used correctly

Pronovost 2005 JCJQI

•Identify Hazards•(

3. Mitigate Risks

2. Analyze & Prioritize Hazards

4. Evaluate Effectiveness of Risk Reduction

Patient Safety Learning Communities

Patient safety learning communities relate to each other in a gear like fashion: as the identified hazards require stronger levels of intervention to achieve mitigation, the next learning community is engaged in action, eventually feeding back to the group that provided the initial thrust. Each group (unit, hospital, industry) follows the same four- step process, but they engage unique matrices of stakeholders to mitigate hazards that are within their locus of control.

Pronovost Health affairs in press

GYN/OB JHOC Medicine Neurosciences Oncology Ophthalmology

FAC: Fetal Assessment Center/OB Ultrasound 

GSS - Shared Specialty Suite 

Asthma & Allergy - Allergy & Clinical Immunology  BRU  GSS - Medical Oncology  GSS - Wilmer 110 

GSS - GYN/OB 420  JHOPC - Express Testing  Asthma & Allergy - Pulmonary  EMU  IPOP Clinic - HIPOP

Location  GSS - Wilmer Laser Center 

GSS - GYN/REI  JHOPC - OR  Asthma & Allergy - Rheumatology  JHOPC Neurosciences  IPOP Clinic - IPOP

Location  WECP & ER 

HAL-2  JHOPC - PACU  Blalock 4 - Endoscopy  MEY 8 (12) Weinberg OPD - 1st Floor  Wilmer OR 

JHOPC GYN/OB  WM - Shared Specialty Suite  Blalock 5 Echo Lab (2) MEY 9 (5) Weinberg OPD - 2nd Floor  Wilmer PACU 

MCE    Cardiac CT  NCCU7  WGA 5 (5) Wilmer White Marsh 

NEL-2 Nursery    CCP-5 (5) WGB 5  Wilmer: Other - E Balt Divisions 

NEL-2 Obstetric OR    CCU-5 (7) WGC-5 (3) Wilmer: Other - Satellites 

NEL-2 PACU    CVC  WGD 5 

Nelson Harvey 2    CVIL- CardioVascular Interventional Lab 

OSL-2    Dialysis Unit 

OSL-3 Nursery    GSS - Internal Medicine 

OSL-3    HAL-5 (5)

WGB-4    HAL-8 (7)

Hospitalist Unit (5)

JHOPC - Exec Health & Travel Clinic 

JHOPC - Medicine Clinics 

CAST• Each contributing factor rate

– importance of the problem and contributing factors in causing the accident

– importance of the problem and contributing factors in future accidents

• Each Intervention rate– How well the intervention solves the problem or mitigates

the contributing factors for the accident– Rates the team belief that the intervention will be

implemented and executed as intended

What is Culture*?:

“The way we do things around here”

*aka Climate

1 attitude = opinion…everyone’s attitude = culture

–Disagreements in the ICU are appropriately resolved (i.e., not who is right, but what is best for the patient)–Our doctors and nurses work together as a well coordinated team

Teamwork climate: perceived quality of collaboration between the personnel in this unit

–Trainees in my discipline are adequately supervised–This hospital deals constructively with problem personnel

Working conditions: perceived quality of the work environment and logistical support (staffing, training, etc.)

–I am less effective at work when fatigued–When my workload becomes excessive, my performance is impaired

Stress recognition: acknowledgement of how performance is influenced by stressors (permitted to be human)

–Hospital management supports my daily efforts in the ICU–Hospital management does not knowingly compromise the safety of patients

Perceptions of management: approval of managerial action

–I would feel safe being treated in this ICU–Medical errors are handled appropriately in this ICU

Safety climate: perceptions of a strong and proactive commitment to patient safety in this unit

–I like my job–This hospital is a good place to work

Job satisfaction: positivity about the work experience

Example itemsFactor: Definition

Executive Perceptions vs. Frontline Perceptions:

Executives overestimate:Teamwork Climate 4XSafety Climate 2.5X

Executive Confidence vs. Executive Accuracy:-Often wrong but rarely in doubt…-Currently no incoming data-streams-Halo Effects-Frontline data fills the gap

* * * * * *

* Statistically Significant

64 Teamwork Climate 200667 Teamwork Climate 2007

71 Teamwork Climate 2008

62 Teamwork Climate 2005

60 Safety Climate 200665 Safety Climate 200770 Safety Climate 2008

59 Safety Climate 2005

 

CSIC

U T1

CSIC

U T2

0

10

20

30

40

50

60

70

80

90

100

% o

f res

pond

ents

with

in a

n IC

U th

at a

gree

#4. “I Would Feel Safe Being Treated Here As A Patient.”

 

CSIC

U T1

CSIC

U T2

0

10

20

30

40

50

60

70

80

90

100

C S IC U

T 2

% o

f res

pond

ents

with

in a

n IC

U th

at a

gree

#3. “Nurse Input Is Well Received In This ICU.”

 

CSIC

U T2

CSIC

U T1

0

10

20

30

40

50

60

70

80

90

100

% o

f res

pond

ents

with

in a

n IC

U th

at a

gree

#26. “In This ICU, It Is Difficult To Speak Up If I Perceive A Problem With Patient Care.”

 

CSIC

U T1

CSIC

U T2

0

10

20

30

40

50

60

70

80

90

100

% o

f res

pond

ents

with

in a

n IC

U th

at a

gree

#32. “Disagreements In This ICU Are Resolved Appropriately (i.e. not who is right, but what is best for the patient).”

Questions for Reflection

• How do you know you are safer?• How will you become more efficient in

your measurement efforts?• How will you better tap into local wisdom?

Focus and Execute