key performance indicators, centre reports, and more stephen mcdonald
TRANSCRIPT
Key Performance Indicators, Centre Reports, and more
Stephen McDonald
Barbecue talk
20
40
60
80
100
120
per
mill
ion
per
yea
r
1970 1980 1990 2000 2010Year
Rate 95% CIIncident rate
Incident RRT, Australia only
More “good” news
5
10
15
20
30
40
50
60
70
50
100
150
200
0
500
0
200
400
600
0
100
200
300
1980 1990 2000 2010 1980 1990 2000 2010 1980 1990 2000 2010
0-24 25-34 45-54
65-74 75-84 85+
Rate 95% CI
per
mill
ion
per
yea
r
Year
Graphs by age group
AustraliaAge-specific incident RRT rates
Indigenous incidence rates
0
100
200
300
400
500
Per
mill
ion
per y
ear
1985 1990 1995 2000 2005 2010Year
Rate 95% CI
Aboriginal & TSI, Australia
Background
• A number of ongoing work themes exist within ANZDATA for generating output– Stock and flow figures– Annual Report– Contributor requests
• Responses to information needed for various projects
– Research projects (internal and external analyses)
– Outcomes reporting
Outcomes reporting
• Recent years have seen a growth of interest in outcomes reporting
• Centre reports have been part of ANZDATA for many years, with increasing emphasis in recent years– At “parent hospital level”– Limited distribution historically
Why measure outcomes?
.2
.5
1
2
4
Obs
erv
ed /
Exp
ecte
d m
ort
alit
y
0 20 40 60 80units
O/E 98% CI
All Australia & NZ Dialysis Units, 98% confidence intervals
Dialysis outcome
.2
.5
1
2
4
Adj
ust
ed r
ela
tive
risk
0 20 40 60 80
Units, ranked by RR
RR 95% CI
Mortality rate during dialysis treatment in Australia 2006-10, adjustedfor demographics and comorbidities
Variation in transplant outcomes
.25
.5
1
2
5
10
20
50R
R g
raft
failu
re
0 5 10 15 20Units
RR 95% CI
Fully adjusted 1 year graft survival, by unitAll transplant units, Australia and New Zealand, patients transplanted2005-2019, followup to 2010
What is happening to centre reports?
• Greater reporting of demographics and comorbidities
• Adjusted analyses in transplanting centre and dialysis reports– Details of models supplied
• Graphs– Funnel plots– CUSUM plots (transplant)
Centre reports – graph 1
0.00
0.25
0.50
0.75
1.00
Pat
ient
Sur
viva
l
0 1 2 3 4 5Years
CNARAustralia
New Zealand
Survival from 90th Day of Treatment
Centre reports – graph 2
0.00
0.25
0.50
0.75
1.00
Tech
niqu
e S
urvi
val
0 1 2 3 4 5Years
CNARAustralia
New Zealand
Technique Survival - PD at 90 days
But....
0
20
40
60
80
100
Ag
e (o
f pre
vale
nt p
atie
nts)
Everywhere else CNARTS
Adjusted graphs
.7
.8
.9
1
1.1
1.2
1.3
Adj
uste
d S
MR
CNAR Australia New Zealand
Adjusted SMR (95% CI)
Adjusted graphs
CNAR
0
.5
1
1.5
2
2.5
SM
R
0 50 100 150 200Expected Number of Deaths
How are reports derived?
You need a model
• Logistic regression model (transplant), Poisson model (dialysis)
• Adjusted for demographics, comorbidities (donor and XM variables)
• With this model, derive a probability of “expected” failure for each person / graft based on covariate matrix
• Compare this with actual outcomes
www.anzdata.org.au
Which predictors are important?
Recip
ient
age
gen
der &
gra
ft num
ber
+com
orbi
ditie
s
+ HLA
mat
chin
g
+ isc
haem
ic tim
e
+ do
nor a
ge
+ ca
use
dono
r dea
th0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Harrell's CSomer's D
Predictive power of multivariate Cox model predicting graft survival, all DD transplants 2001-2009, with sequential addition of covariate groups
www.anzdata.org.au
• Factors within the control of centre– These may be why a particular centre gets
good or bad results• Factors that occur as a result of treatment
decisions• For example, don’t adjust for
– Choice of dialysis modality, HD access– Use of immunosuppressives, rejection, 1
month graft function…
Don’t adjust for…
Other graphical demonstrations of output
• Funnel plots are a static measure and summarise performance (relative to a comparator) over a fixed period of time.– Lack a dynamic element– Weight recent and distant results equally
Adding time – CUSUM
0
100
200
300
400
Num
ber
of tx
-4
-2
0
2
4
Cum
ulat
ive
sum
O-E
01jan2004
01jan2005
01jan2006
01jan2007
01jan2008
01jan2009
Tx date
Twoway CUSUM for a transplant centre
Removing credit for good deeds
0
1
2
3
4
5
Cu
mul
ativ
e s
um
O-E
0 100
200
300
400
Tx number
Oneway CUSUM for for a hospital
Do we need to do more?
Why KPIs?
• Mortality is an insensitive and late indicators of problems– Hopefully rare– Outcome of complex series of events
• Incompletely ascertained
– Important to monitor as best we can• Key Process indicators
– Simpler to understand, easier to address– Need to be valid and correctable (and related
to meaningful outcomes)
KPI Project
• Dialysis KPI project commenced 2011 – At instigation of DNT committee
• 2 markers chosen – Peritonitis and HD access at first treatment– Deliberately limited to existing data collection
• NO additional data collected
– Based on real time ANZDATA data collection
Variation in HD access
8
5 5
40
31
65
18
66
28
10
57
2522
113
17
3668
810
15
7
47
19
12
2952
75434452
29
11
34344144
12
4464
20
1017
3144331923
139
2815
54
6 8
0
.2
.4
.6
.8
1
Pro
por
tion
of li
nes
0 20 40 60Centres
Proportion 95% CI
ANZDATA, access at first HD where first dialysis
Variation in peritonitis rate
27
24
12
6
4
Pat
ient
-mon
ths
per
epi
sod
e
.5
1
1.5
2
2.5
3
Epi
sod
es p
er
pat
ient
-ye
ar
Confidence intervals not shown where upper limit >3Units with <5 person-years PD over 2009 not shown
2009 onlyPeritonitis rates by treating unit
KPI reporting -- access
• Quarterly identified feedback to units
Peritonitis reporting
Where to from here?
• COMMUNICATE• Improve data collection• Improve access to results• Enhance reporting
– Add peritonitis rates– Access subdivided by late referral– Graphs etc etc
• Or is it all just too hard?
How do we view quality?
Centre reports -- SMR