Portsmouth HospitalsNHS Trust

Dr David Prytherch and Dr Jim BriggsHealth Care Computing GroupUniversity of Portsmouth

Mr Paul Weaver and Dr Paul Schmidt, University of Portsmouth

Professor Gary Smith, Portsmouth Hospitals NHS Trust, University of Bournemouth

Measuring clinical performance using routinely collected clinical data.

Portsmouth HospitalsNHS Trust

………Hospitals and the NHS could tell you about throughput (number of patients treated), bed occupancy (the proportion of beds occupied in the hospital), and, latterly, the costs involved. But, generally speaking, quality of outcome was a closed book.

Why look at Clinical Outcomes?

At national level, the indicators of performance should be comprehensible to the public as well as to healthcare professionals. They should be fewer and of high quality, rather than numerous but of questionable or variable quality.

Why Mortality?

“Learning from Bristol”:The Report of the Public Inquiry into children’s heart surgery at the Bristol Royal Infirmary 1984 to 1995. I Kennedy, HMSO 2001

Portsmouth HospitalsNHS Trust

…… Variables such as case mix and where possible, in the case of surgery, operative risk must be allowed for, so that, wherever feasible, it is possible to compare like with like.

“Learning from Bristol”:The Report of the Public Inquiry into children’s heart surgery at the Bristol Royal Infirmary 1984 to 1995. I Kennedy, HMSO 2001

Why case-mix adjust?

For the future the multiple methods and systems for collecting data must be reduced. Data must be collected as the by-product of clinical care.

How to collect the data?

Portsmouth HospitalsNHS Trust

Measuring Clinical Performance

How do you measure clinical performance?

You need to know what you expect – predicted outcomes. Compare predicted with reported.

How do you predict outcomes?

Case mix adjusted models

Why case mix adjust?

To gain clinician engagementTo answer “my results are worse than … because my patients are sicker”

Essentially it provides a ruler

Portsmouth HospitalsNHS Trust

How do you case mix adjust?

Use clinical data that encapsulates the physiological state of the patient

Use this to predict a risk of “adverse outcome”

Trick is to collect necessary data in the clinical environment

Portsmouth HospitalsNHS Trust

Aim of study was to see if data stored in core hospital systems could be used to predict (case mix adjust) clinical outcomes.

Data from: PASBiochemistry and Haematology modules of pathology system

Data already collected / exists. No additional administrative or clinical burden.

BHOM: Biochemistry and Haematology Outcome Modelling

Portsmouth HospitalsNHS Trust

results adapted from: The use of routine laboratory data to predict in-hospital death in medical admissionsD R Prytherch, J S Sirl, P Schmidt, P I Featherstone, P C Weaver, G B

Smith. Resuscitation 2005; 66: 203-207

First demonstration of outcome prediction for General MedicineData from PAS and Biochemistryand Haematology modules of pathology system

1st January 2001 - 31st December 2001

9497 discharges from GM with necessary dataModel developed from Q1 and applied prospectively against Q2, Q3 and Q4

e.g., BHOM in General Medicine

Portsmouth HospitalsNHS Trust

•Urea•Albumin•Creatinine•Na•K•Haemoglobin•White Cell Count•Age on admission•Sex•Mode of admission•Mortality at discharge

Data items used in models for General Medicine:

Portsmouth HospitalsNHS Trust

Risk (%) Dis-charges

Mean Risk (%)

Predicted Deaths

Reported Deaths

2

0 to 5 1279 2.27 29 35 1.27

>5 to 7.5 265 6.49 17 20 0.48

>7.5 to 10 321 8.86 28 24 0.76

>10 to 12.5

164 11.48 19 20 0.08

>12.5 to 15

169 14.00 24 31 2.65

>15 to 20 154 17.78 27 28 0.02

>20 to 25 72 22.90 16 14 0.49

>25 to 33 78 28.69 22 20 0.36

>33 to 50 28 40.25 11 8 1.59

>50 to 100 14 64.21 9 8 0.31

           

0 to 100 2544 8.01 204 208 8.00

2 = 8.00

10 d.f

P = 0.63

no evidence of lack of fit

c-index=0.757

General Medicine StudyFinal 3 month period 1st October – 31st December 2001

Portsmouth HospitalsNHS Trust

General Medicine - Mortality at Discharge

Monthly, J anuary 1998 to December 2001

0

20

40

60

80

100

120

Predicteddeaths

Reporteddeaths

p-value

(prospective)

45 df = 0.11

General Medicine StudyTotal Mortality through time – 1st January 1998 to 31st December 2001 (37283 discharges)

Portsmouth HospitalsNHS Trust

Excess of reported vs predicted deaths per month in General Medicine Jan 98 to December 2001

-20

-15

-10

-5

0

5

10

15

20

25

30

01-9

8

03-9

8

05-9

8

07-9

8

09-9

8

11-9

8

01-9

9

03-9

9

05-9

9

07-9

9

09-9

9

11-9

9

01-0

0

03-0

0

05-0

0

07-0

0

09-0

0

11-0

0

01-0

1

03-0

1

05-0

1

07-0

1

09-0

1

11-0

1

December 99 and January 2000 show statistically significant excess. Bed availability problems and flu.

Use to identify periods when performance deviates from norm

Portsmouth HospitalsNHS Trust

Risk (%) Dis-charges

Mean Risk (%)

Predicted Deaths

Reported Deaths

2

0 to 5 1103 1.40 15 13 0.39

>5 to 7.5 119 6.16 7 6 0.26

>7.5 to 10 85 8.66 7 6 0.28

>10 to 12.5

68 11.29 8 10 0.79

>12.5 to 15

32 13.67 4 6 0.70

>15 to 20 66 17.34 11 12 0.03

>20 to 25 50 22.64 11 11 0.01

>25 to 33 61 29.20 18 20 0.38

>33 to 50 90 40.71 37 36 0.02

>50 to 100 93 74.58 69 78 4.24

 

0 to 100 1767 10.68 189 198 7.09

2 = 7.09

10 d.f

P = 0.72

no evidence of lack of fit

c-index=0.92

General Medicine StudyExternal Validation: T2 v T1

Portsmouth HospitalsNHS Trust

Clinical data obtained from a single venesection

Clinical data are used operationally in care of individuals

All data already stored on hospital core IT systems - no “extra” effort is required to collect data

Clinical data used are subject to extensive quality assurance

Key points: 1

Portsmouth HospitalsNHS Trust

Case mix adjusted and uses high quality data trusted by clinicians (no coded data) – more likely to win clinical acceptance

Data immediately available to inform decisions

Cannot be “gamed”

Performance and surveillance tool

Key points: 2

Portsmouth HospitalsNHS Trust

National Application of BHOM

Vascular Society of Great Britain and Ireland National Vascular Database (Risk adjusted predictive models of death after index arterial operations using a minimal data set.D R Prytherch, BMF Ridler, S Ashley on behalf of the Audit and Research Committee VSGBI, Br J Surg 2005; 92: 714-718)

NCEPOD (National Confidential Enquiry into

Patient Outcome and Death) study into Abdominal Aortic Aneurysm www.ncepod.org.uk

Portsmouth HospitalsNHS Trust

Portsmouth NHS R&D Consortium

Portsmouth Hospitals NHS Trust

University of Portsmouth

Sources of funding