engaging with data

Engaging with data

Martin Utley, Director of the Clinical Operational Research Unit,

University College London

Chair: Chris Sherlaw-Johnson, Senior Research Analyst,

Nuffield Trust

Building tools with clinical teams for use in monitoring outcomes.

Martin Utley

UCL Clinical Operational Research Unit

I received royalties from UCL Business in relation to co-authorship of the PRAiS software tool for monitoring 30-day survival following paediatric cardiac surgery.

Declaration

An insentient machine

? One can monitor outcomes to identify problems

One has absolute standards (some components to ¼ inch, others to a thousandth)

If there is a problem, the machine can be fixed.

To diagnose and fix problem, one has to stop production and open the box.

Stopping production comes at a cost, so one has to have a degree of

confidence that there is a genuine problem.

To what extent do these conditions apply to health care?

An insentient machine

Survival to 30 days following

paediatric cardiac surgery

Variable Life Adjusted Display (VLAD)

with Partial Risk Adjustment in Surgery (PRAiS)

Variable Life Adjusted Display (VLAD)

with Partial Risk Adjustment in Surgery (PRAiS)

Pagel C, Utley M, Crowe S, Witter T, Anderson D, Samson R, McLean A, Banks V, Tsang V, Brown K, Real time

monitoring of risk-adjusted paediatric cardiac surgery outcomes using variable life-adjusted display: implementation

in three UK centres, Heart (2013) 99:1445-50.

In use for local monitoring of outcomes at UK surgical centres since 2013.


Dissemination to

wider clinical community

Clinical consultation / contribution / leadership

Development of risk

model

Design of data displays


Dissemination to

wider clinical community

Clinicians led

engagement with surgical units,

National audit body & other stakeholders

All episodes in a 30% validation

sample within development set:

Mortality: 3.5%, N=7890

High risk diagnoses


HLHS

Interrupted aortic arch

Pulmonary atresia (inc PA+IVS)

Comorbidity

Low risk diagnoses


Fallot/DORV-Fallot type

VSD

Pulmonary stenosis

Aortic arch obst +/- VSD/ASD

ASD

Subaortic stenosis (isolated)

Aortic regurgitation

Procedure

Medium risk diagnoses


PDA

Acquired

EMPTY/Unknown

Functionally uni-ventricular heart

TGA+VSD/DORV-TGA type

Pulmonary atresia+VSD (inc fallot type)

Tricuspid valve abnormality (inc Ebstein)

Mitral valve abnormality (inc supra, sub)

TAPVC

AVSD

Aortic valve stenosis

Misc Cong

TGA & IVS

Common arterial truncus

Normal

All episodes in a 30% validation

sample within development set:


High risk diagnoses


HLHS

Interrupted aortic arch

Pulmonary atresia (inc PA+IVS)

Comorbidity

Low risk diagnoses


Fallot/DORV-Fallot type

VSD

Pulmonary stenosis

Aortic arch obst +/- VSD/ASD

ASD

Subaortic stenosis (isolated)

Aortic regurgitation

Procedure

Medium risk diagnoses


PDA

Acquired

EMPTY/Unknown

Functionally uni-ventricular heart

TGA+VSD/DORV-TGA type

Pulmonary atresia+VSD (inc fallot type)

Tricuspid valve abnormality (inc Ebstein)

Mitral valve abnormality (inc supra, sub)

TAPVC

AVSD

Aortic valve stenosis

Misc Cong

TGA & IVS

Common arterial truncus

Normal

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

Year

Percentage episodes with empty co-morbidity fields

New ways of using diagnostic codes

clinically led / meaningful

then simplified for purposes of

risk model

Any old way of including comorbidity

knowingly included poor quality data

acknowledging clinical nous

commitment to long game

Development of risk

model Old ways of using operation codes

accepted existing schemes

impact on model “performance”

impact of model use

Year

% missing

comorbidity

data

PRAIS 1:

• 29 Specific procedures

• 3 diagnosis groups (low, medium, high)

• UVH

• Yes/no comorbidity

• Bypass/non-bypass

• Continuous weight

• Continuous age

• Age band (neonate/infant/child)

• Epoch effect

38 parameters

PRAIS 2:

• 16 spec proc groupings

• 11 diagnosis groupings

• UVH

• 4 “other illness” indicators


• Continuous nonlinear weight

• Continuous nonlinear age

• Epoch effect

36 parameters

PRAiS2 – developed on 21838 surgical episodes 2009-2014 from NCHDA data.

Recent update

VLADs do not give an evaluation of the

effectiveness / benefit of a surgical

programme to the patient population.

VLADs are not evidence of good or bad

performance.

DATA

patient case-mix

structure / resources

care processes

professionals

Queensland

pyramid model of

investigation

Worked with 3 sites on design of software

We improved ease of data entry / use, shared strengths and

limitations, listened to concerns.

Consultation on design of data displays

Added after consultation with units

Secondary outcomes we would have discarded if not for discussions with clinical team

Weeks following release of PRAiS 2013

Christina Pagel - How can we better support appropriate interpretation

and use of risk adjusted survival data?

PRAiS 1 launch

What?

How?

Why? PRAiS2

Research team worked with clinicians, family representatives, press officers and journalists on resources to support understanding and use of risk-adjusted mortality data.

childrensheartsurgery.info

Launch childrensheartsurgery.info

What?

How?

Why?

Funding NIHR HSDR PI Victor Tsang

Health Foundation PI MU

NICOR

Funding NIHR HSDR PI Christina Pagel

Funding NIHR HSDR PI Victor Tsang & Kate Brown

Funding GOSH Charity PI Kate Brown

None of the views expressed are those of DH, NIHR, Health Foundation, GOSH Charity etc.

2014/15 - 4,207 surgical episodes with 97 deaths (mortality rate 2.3%) – 89 deaths predicted Area under ROC curve 0.86 [0.82, 0.89]

PRAiS2 performance in test data

Brown KL, Crowe S, Franklin R, McLean A, Cunningham D, Barron D, Tsang V, Pagel C, Utley M, Trends in 30-day mortality

rate and case mix for paediatric cardiac surgery in the UK between 2000 and 2010, Open Heart. 2015 Feb 14;2(1):e000157.

doi: 10.1136/openhrt-2014-000157.

Absolute survival rates completely absent from debate in 2013

Kate Brown

PRAIS 2:

• 16 spec proc groupings

• 11 diagnosis groupings

• UVH

• 4 “other illness” indicators


• Continuous nonlinear weight

• Continuous nonlinear age

• Epoch effect

36 parameters

Performance assessed using cross-validation in main data set and prospective use in 2014/15 national data set.

PRAiS risk model recently updated

Close collaboration with an expert panel including of intensive care consultants, surgeons, cardiologists, data experts, analysts

engaging with data

Healthcare