Graphical methods for turning data into information

Martin Utley

Clinical Operational Research Unit (CORU)University College London

www.ucl.ac.uk/operational-research

Monitoring outcomes to improve outcomes

Care process

Data

Information system

Analysis of data

Feedback

Need to get every step right

Care process

Data

Information system

Analysis of data

Feedback

Steps discussed in this talk

Case study 1: monitoring outcomes of cardiac surgery

Work done by:

Steve GallivanChris Sherlaw-JohnsonJocelyn Lovegrove

Tom TreasureOswaldo Valencia

CORU

St Georges / Guy’s

Mortality data for cardiac surgery

0000000000000000100000000

0000000000000000000000000

0000000000000100000000000

0000000100000000000000000

0000000000000000000100000

0000000000000000000001010

Mortality data for cardiac surgery

0000000000000000100000000

0000000000000000000000000

0000000000000100000000000

0000000100000000000000000

0000000000000000000100000

0000000000000000000001010

6 perioperative deaths in 150 cases

0 20 40 60 80 100 120 140

Operation number

0

2

4

6

8

10

Cumulative deaths

Graphical presentation of data

First used in the context of surgery by DeLeval

0 20 40 60 80 100 120 140

Operation number

0

2

4

6

8

10

Cumulative deaths

Graphical presentation of data

Is this series of outcomes good or bad?

Co-morbidity

Emergency status

Repeat operation

LV function

Age

Risk of perioperative

death

Risk of perioperative

death

Patient factors that contribute to risk of death

To be fair, assessment of outcomes should account for case-mix

0 20 40 60 80 100 120 140

Operation number

0

2

4

6

8

10

Cumulative deaths

Expected mortality (from risk model)

Actual mortality

Net lifegain

Par for the course

Compare outcomes to expectations

Variable Life Adjusted Display (VLAD)

0 20 40 60 80 100 120 140

Operation number

0

1

2

3

4

5Net life gain

VLAD plot for a single surgeon

0 20 40 60 80 100 120 140Operation number

012345

-1-2-3-4

Net life gain

Vlad the impaler

The venerable bleed

Hawkeye Pierce

Comparing three fictitious surgeons

Unexpecteddeath

Survivoragainst

the odds

Comparing surgeons within a unit

Net life gain

Operation number

-15

-10

-5

0

5

10

15

0 50 100 150 200 250 300 350

Operation number

Net

lif

e g

ain

-1%

-5%

-10%

-25%

+25%

+10%+5%

+1% tail

Tools to assist interpretation

Net life gain

Operation number

Keys to success

• Surgeons say that visual display is intuitive

• Can be used to identify possible problems in real time

• Monitoring tool “rewards” good outcomes rather than just punish bad outcomes

• Clinical champion

VLAD adopted worldwide

Case study 2:

monitoring prescription errorsCollaborators:

Steve GallivanChristos Paschalides

Bryony Dean Franklin Ann JacklinKara O’Grady

Nick Barber

CORU

Hammersmith

London School of Pharmacy

Funded by the Trustees of Hammersmith Hospitals NHS Trust

Monitoring the prescribing process

Care process

Data

Information system

Analysis of data

Feedback

Junior doctor writes prescription

Ward pharmacist corrects any errors that he or she

identifies

Care process

Data

Information system

Analysis of data

Feedback

Prescription errors deemed sufficiently serious by

pharmacist are logged asincidents

Monitoring the prescribing process

Care process

Data

Information system

Analysis of data

Feedback

Extensive research on nature and rates of reported

prescription errors

Monitoring the prescribing process

Care process

Data

Information system

Analysis of data

No systematic feedback to prescribers

With no feedback, how can we expect prescribing practice to improve?

The problem

Feasibility study

Ward pharmacistchecks new medication orders...

...& records consultant team, number of new orders and all

errors identifiedData entered onto

spreadsheetJunior doctors write prescriptions

Graphical summaries prepared

Feedback sentto head of specialty

0

200

400

600

800

1000

1200

SpecialtyC

Specialty

New medication orders with at least one error

New medication orders without any errors

Number of new orders Your specialty93 (12%) of 773 new orders

had at least one error

Graphical summaries kept simple

0%

2%

4%

6%

8%

10%

12%

14%

16%

AMUC All other specialties

Specialty

Proportion of new orders w ith an

error

Proportion for whole directorate

Proportion of new orders with an error

Proportion for whole directorate

AdmissionsCXH

All other specialties

How much statistics?

95% confidenceinterval

SpecialtyC

16/02/0502/03/05

16/03/05

30/03/05

11/04/05

25/04/05

09/05/05

23/05/05

0

10

20

30

40

50

60

70

80

90

100

0 200 400 600 800 1000

Cumulative number of orders with an error

Cumulative number of new medication orders

Other specialties

AMUC

16/02/0502/03/05

16/03/05

30/03/05

11/04/05

25/04/05

09/05/05

23/05/05

0

10

20

30

40

50

60

70

80

90

100

0 200 400 600 800 1000

Cumulative number of orders with an error

Cumulative number of new medication orders

Other specialties

AMUCAdmissions

CXH

Performance over time

SpecialtyC

Prototype feedback pages 1 and 2

Prototype feedback page 3

Comments written by the Trust’s Principal Pharmacist highlighting any issues that arise from the data.

Representative examples of prescribing errors recorded over the period.

...does this process leadto improvement?

Care

process

Data

Information

system

Analysis of

dataFeedback

Care

process

Data

Information

system

Analysis of

dataFeedback

Hang on...

Planned study

Does monitoring and feedback reduce

errors in the prescribing process?

Time

Error rate

Monitor and feedback results

Summary

• Succinct graphical methods can be very useful in the analysis of clinical data and in feeding back information to clinical teams.

• Appropriate feedback cannot do any harm, can it?

• Evaluation of monitoring systems in terms of clinical improvement is desirable.