odense denmark 2014 the theory and reality of developing clinical decision rules

Odense Denmark 2014

The Theory and Reality of Developing Clinical Decision Rules

Jeff Perry MD MSc CCFP-EM

Associate Professor Dept of Emergency MedicineFaculty of Medicine Research Chair in Neurological EmergenciesSenior Scientist, Ottawa Hospital Research Institute

DisclosuresPeer-reviewed grants CIHR, HSFCThank you to Ian Stiell for providing slides

Theory and Reality of Developing Clinical Decision Rules

Introduction to CDRs

Examples from Ottawa

Methodological Standards for Derivation

Prospective Validation

Implementation Trial

Knowledge Transfer

What is a Clinical Decision Rule?Definition:

A tool that helps clinicians make diagnostic and therapeutic decisions at the bedsideDerived from original researchIncorporate 3 or more variables from history, exam, or simple tests

Examples : which patients with -ankle injury need x-rays?possible DVT need imaging?pneumonia need admission?headache need CT/LP?

What Conditions are Suitable for Clinical Decision Rules?

The Need:Common, high-volume conditions, e.g. chest pain, cough and fever, extremity injury, shortness of breathInefficient use of resources, e.g. diagnostic tests or hospitalization Variation in current practice

The Purpose:Standardize care based on evidenceImprove safetyImprove efficiency

References – Clinical Decision Rules

JAMA 2000

Theory and Reality of Developing Clinical Decision Rules

Introduction to CDRs

Examples from Ottawa

Methodological Standards for Derivation

Prospective Validation

Implementation Trial

Knowledge Transfer

Feel free to ask questions

Clinical Decision RulesDeveloped at U Ottawa

Ottawa Ankle and Knee Rules

Canadian C-Spine and CT Head Rules

Ottawa Subarachnoid Rule

Wells Criteria for PE and DVT – Phil Wells

Ottawa Heart Failure and COPD Rules

The Canadian C-Spine Rule

Imaging for Alert & Stable Trauma Patients

Potential C-Spine Injury:The Clinical Problem> 8 million potential neck injury cases per year in Canadian and US EDs

Most are alert and stable with <1% having c-spine fracture

C-spine diagnostic imaging use inefficient and variable

High volume items add to health care costs

Prolonged immobilization and ED overcrowding

Development of the Canadian C-Spine Rule

Variation and Inefficiency (N=6,855)CMAJ 1997

Derivation of the Rule (N=8,924) JAMA 2001

Prospective Validation (N=8,283) New Engl J Med 2003

Multicentre Implementation (N=11,648) Br Med J 2009

Awareness and Use (N=1,150) Acad Emerg Med 2008

Validation by Paramedics Ann Emerg Med 2009

Validation by ED Nurses (N=3,633) CMAJ 2010

Derivation of Canadian C-Spine Rule - JAMA 2001

Objective: Derive a clinical decision rule highly sensitive for acute cervical spine injury

Methods (N=8,924): Prospective cohort - 10 Canadian EDsAlert and stable, adult trauma patientsMDs assessed 20 clinical findingsClinically important c-spine injuryRecursive partitioning analyses

2003

2003

Potential Radiography Rates

66.6%

55.9%

40%

60%

80%

NEXUS Canadian

P < 0.001

Theory and Reality of Developing Clinical Decision RulesIntroduction to CDRs

Examples from Ottawa

Methodological Standards - Derivation

Prospective Validation

Implementation

Knowledge Transfer

Feel free to ask questions

Was the Rule Derived According to Methodological Standards?

Outcome MeasurePredictor VariablesReliability of PredictorsStudy SubjectsSample SizeMathematical TechniquesSensibility for CliniciansClassification Accuracy

Was the Rule Derived According to Methodological Standards? Outcome Measure

Clinically importantClearly definedAssessed blindly

Clinically Important C-Spine Injury

Serious Adverse Event for COPD: DeathIntubation or NIVMyocardial infarctionAdmission to monitored unit Relapse back to ED requiring admission < 14 days

Was the Rule Derived According to Methodological Standards?

Predictor VariablesStandardized definitionCollected prospectively with data formAssessed before outcome

20 Variables from History and Exam

30 Variables from History, Exam, Lab, ECG, Xray Innovative 3-min walk test

Was the Rule Derived According to Methodological Standards?

Reliability of Clinical VariablesInterobserver agreement beyond chance:

kappa (dichotomous / nominal data)weighted kappa (ordinal data)intraclass correlation coefficient (interval)

3. Reliability: Interobserver Agreement for C-Spine Findings

KappaCharacteristic (N=150)

Midline neck pain .69Immediate neck pain .48 Weakness in extremities .54Numbness / tingling .77Upright position .78Distracting injuries .41Facial Injury .75

Was the Rule Derived According to Methodological Standards?

Study SubjectsDefined inclusion criteriaUnbiased selectionCharacteristics describedSetting

Alert stable trauma patients with neck pain

Adults with exacerbation of COPD - both admitted and discharged

Excluded: very ill patients: O2 Sat < 85% on room air, HR > 130, SBP < 85, chest pain or acute ECG changes

Was the Rule Derived According to Methodological Standards?

Sample SizeRequires adequate number of positive outcomesMethods for determining SS:

rule of thumb – 10 outcomes / predictordegree of precision in CI around measure of accuracy

8,924 neck injury patients with 151 important c-spine injury cases (1.7%)

945 COPD patients with 74 SAE cases (7.8%)

Sample Size: C-Spine Phase I

Estimated- 100% sensitivity with 95% CI 97-100%- 120 injury cases- 1.5% incidence important injury- 8,000 patients

Actual- 100% sensitivity (98-100)- 151 injury cases- 1.7% incidence- 8,924 patients

Was the Rule Derived According to Methodological Standards?

Mathematical TechniquesUnivariate KappaMultivariate:

Chi-square recursive partitioningLogistic regression

Variables with strong univariate associations and kappa > 0.6 recursive partitioning

Univariate analyses logistic regression

Independent Predictors of SAE from Logistic Regression (N=945)

Variable βOR

Hx of PVD intervention 0.90 2.46Prior CABG 0.71 2.03Prior intubation 1.32 3.73ECG acute ischemia 1.18 3.25CXR pulmonary congestion 0.63 1.88Too ill to walk after treatment 1.25 3.50HR on ED arrival > 110 1.12 3.05HgB < 100 1.59 4.90Urea > 12 0.89 2.43Serum CO2 > 35 0.66 1.91Hosmer-Lemeshow Goodness-of-fit P = 0.70

Area under ROC curve = 0.80 (95%CI 0.74 -0.85)

Was the Rule Derived According to Methodological Standards?

Sensibility for CliniciansClinical validitySimple and easy to useYes/No vs probability

Yes/No Algorithm for C-Spine

Risk Scale for COPD

Was the Rule Derived According to Methodological Standards?

Accuracy – Classification Performance2 x 2 tables with sensitivity, specificityROC curve analysisPredicted probability

C-Spine – 2 x 2 table

COPD – probability

Theory and Reality of Developing Clinical Decision Rules

Introduction to CDRs

Examples from Ottawa

Methodological Standards for Derivation

Prospective Validation

Implementation Trials

Knowledge Transfer

Feel free to ask questions

Prospective Validation of Ottawa Decision Rules

Ottawa Ankle Rule (N=1,485)JAMA 1993

Ottawa Knee Rule (N=1,096) JAMA 1996

Canadian C-Spine Rule (N=8,283) New Engl J Med 2003

Canadian CT Head Rule (N=2,707)JAMA 2005

Validation by Paramedics (N=1,949)Ann Emerg Med 2009

Validation by ED Nurses (N=3,633) Can Med Assoc J 2010

Was the Rule Prospectively and Explicitly Validated?

New patients and settingsExplicit applicationOutcome measuresAccuracy:

The rulePhysicians

ReliabilityPhysician comfortPotential impact

Validation of Canadian C-Spine Rule - NEJM 2003

Objective: Prospectively compare the clinical performance of the CCR and the NEXUS criteria

Methods (N=8,283): Prospective cohort - 9 Canadian EDsAlert and stable, adult trauma patients349 MDs assessed patients for both rules – data forms2nd observer where feasible169 important c-spine injury cases by imaging or F/U

Other Validation Measurements

Physician Accuracy 91.2%

Reliability (kappa) 0.64

Clinical Sensibility“Uncomfortable” with rule 8.0%

Potential Impact: C-Spine Radiography Rates

71.7%

55.9%

40%

60%

80%

Actual Rate Potential Rate

P < 0.0001RR = 22.0%

232.8

123.0

100

150

200

250

Radiography No Radiography

N=4,129 N=1,779

Potential Impact: Total Time in ED in Minutes

Diff = 109.8P < 0.0001

Actual vs Potential C-Spine Clearance Rates (N=3,633)

Actual Rate Potential Rate0%

25%

50%

0%

40.6%

0%

Theory and Reality of Developing Clinical Decision Rules

Introduction to CDRs

Examples from Ottawa

Methodological Standards for Derivation

Prospective Validation

Implementation Trials

Knowledge Transfer

Feel free to ask questions

Implementation Trials of Ottawa Decision Rules

Ottawa Ankle Rules (N=2,342)JAMA 1994

Ottawa Ankle Rules (N=12,777)Br Med J 1995

Ottawa Knee Rule (N=3,907) JAMA 1997

Canadian C-Spine Rule (N=11,824 Br Med J 2009

Canadian CT Head Rule (N=4,531) Can Med Assoc J 2010

Has the Rule been Implemented into Practice to Assess Impact?

Controlled or cluster randomized trial design

Impact on clinical care:Process measures: e.g. hospital admission, use of imagingPatient outcomes: e.g. mortality, missed injuries

Other:Accuracy of the rulePhysician acceptabilityPatient acceptability

2009

Implementation of Canadian C-Spine Rule – Br Med J 2009

Objective: To evaluate the effectiveness of an active strategy to implement the CCR into multiple EDs

Methods (N=11,824): Matched pair cluster randomized trial12 university and community hospital EDsAlert and stable, adult trauma patients6 hospitals intervention, 6 controlActive strategies :

Education - rounds, handouts, posters, pocket cards, App, screen-saversPolicy Real-time reminders

Diagnostic Imaging Rates (N= 11,648)

20

70

% I

mag

ing

20%

45%

70%

BeforePeriod

AfterPeriod

6 InterventionHospitals

6 ControlHospitals

Study Sites

61.7%

53.7% 53.8%

59.8%

P < 0.01

P < 0.001

P < 0.01

The “mother of all negative trials” !!

PRIMARY OUTCOME (N=4,531)Diagnostic Imaging Rates

0

40

80

% I

mag

ing

BeforePeriod

AfterPeriod

6 InterventionHospitals

6 ControlHospitals

Study Sites

62.8%

76.2%

67.5%74.1%

P < 0.01

P = 0.64P < 0.01

80%

40%

0%

Barriers to Use: Post-Study Survey Physician Beliefs and Attitudes:• Bent rule to meet their needs• Didn’t ‘believe’ the rule• Didn’t like being directed

Electronic Ordering:• Physical restriction of accessing computer • Ability to circumvent the rule

Busy and Overcrowded EDs: • Easier to CT and discharge• Tests that speed flow are used

CT Head is Standard of Care:• Access to CT easy at this site and becoming routine care

Theory and Reality of Developing Clinical Decision Rules

Introduction to CDRs

Examples from Ottawa

Methodological Standards for Derivation

Prospective Validation

Implementation Trials

Knowledge Transfer

Feel free to ask questions

How does Clinician Uptake occur for Decision Rules?

How do we close the evidence-practice gap?

Passive DiffusionJournal articles, scientific meetings

DisseminationTargets an audience – mailouts, speakersMeta-analyses, reviews, guidelines

ImplementationActive, local, persistentAdministrative, educational strategies

Evaluation of the Dissemination and Uptake of Ottawa Decision Rules

Attitudes and Use Ankle/Knee in Canada (N=232)Graham - Acad Emerg Med 1998

Awareness and Use Ankle/Knee in 5 Countries (=1,769)Graham - Ann Emerg Med 2001

CCR and CCHR in Canada (N=262) Brehaut - Acad Emerg Med 2006

CCR and CCHR in 4 Countries (N=1,150) Eagles - Acad Emerg Med 2008

Theory and Reality of Developing Clinical Decision Rules

Introduction to CDRs

Examples from Ottawa

Methodological Standards for Derivation

Prospective Validation

Implementation Trials

Knowledge Transfer

Feel free to ask questions

Barriers to CCH Rule Use:Pre-Study Survey - Brehaut 2003

Forget details of CCH Rule 32%Trauma/NS will order anyway 30%Patient/family expectation 10%Research is flawed 6% See no advantage to no CT 6% Busy ED – can’t observe 6%Takes too much time 2%Rule not safe for patients 2%Resent concept of guidelines 0%

Theory and Reality of Developing Clinical Decision Rules

BMJ 2003

Was the Rule Derived According to Methodological Standards?

Ottawa Risk Scale for ED Patients with COPD

COPD Patients in the ED: The Clinical Problem

Common ED presentation

With bed shortages, MDs under pressure to send home

Adverse outcomes common, especially in those discharged

Little evidence to guide disposition decisions

Need risk scales for rational and safe admission decisions

MethodsDesign: Prospective cohort study

Setting: 6 EDs of large, tertiary care Canadian hospitals

Subjects: Adults with exacerbation of COPD - both admitted and discharged

Excluded: Very ill patients

Standardized Assessment: variables from history, exam, lab, chest x-ray, ECGSerious Adverse Event: death, intubation, critical care, MI, return to ED with admission

Serious Adverse Events (N=69/945)

Discharged Patients (N=591)

Admitted Patients (N=354)

Total SAEs (N=945)

Ottawa COPD Risk Scale - Identify ED Patients at High Risk for SAE

Univariate Correlation of COPD SAEs Variables from Exam and Labs (N=945)

SAE No SAEP-Value

SaO2 on arrival (%) 91.393.5 0.01CTAS level 2.5 2.7 0.02Urea 9.8 7.3 <0.01Glucose 8.2 7.1 0.03pCO2 60.243.6 0.01HGB 122.8 133.7 <0.01ECG ischemia 7.5 1.9 <0.01CXR congestion 25.7 9.1 <0.01Too ill to do walk test (%) 41.913.0 <0.0001Walk test highest HR (N=43, 749) 98.7104.0 0.05

COPD Patient Characteristics (N=945) Mean age 72.6

Male (%) 51.6Ambulance arrival (%) 48.3Duration of symptoms, hours 87.0CTAS score, mean 2.7Associated HF in ED (%) 10.2

Past Medical History (%)Heart Failure 21.1Admission for respiratory distress28.4Intubation for respiratory distress 2.9

Current smoker (N=792) 31.6Home oxygen (%) 12.5

COPD Risk Score Sensitivity Specificity

PotentialAdmission

0 1.0 0.0 100%1 0.91 0.45 57.6%2 0.81 0.60 43.2%3 0.60 0.84 20.0%4 0.52 0.905 0.25 0.976 0.19 0.997 0.07 0.9968 0.06 0.999

10 0.03 1.0

Classification Performance and Potential Admissions for Ottawa COPD Risk Scale

Calibration Between Observed vs Expected SAE Score in COPD Patients

Homer-Lemeshow goodness-of-fit p-value = 0.67

Secondary Outcomes (N=11,648)

Intervention Hospitals Control Hospitals Before After Before After

Outcomes N=3,266 N=3,624 N=2,413 N=2,345

Missed Fractures 0 0 0 0

Adverse Outcomes 0 0 0 0

Time in ED (Mins) 206 215 187 210