odense denmark 2014 the theory and reality of developing clinical decision rules
TRANSCRIPT
Odense Denmark 2014
The Theory and Reality of Developing Clinical Decision Rules
Jeff Perry MD MSc CCFP-EM
Associate Professor Dept of Emergency MedicineFaculty of Medicine Research Chair in Neurological EmergenciesSenior Scientist, Ottawa Hospital Research Institute
DisclosuresPeer-reviewed grants CIHR, HSFCThank you to Ian Stiell for providing slides
Theory and Reality of Developing Clinical Decision Rules
Introduction to CDRs
Examples from Ottawa
Methodological Standards for Derivation
Prospective Validation
Implementation Trial
Knowledge Transfer
What is a Clinical Decision Rule?Definition:
A tool that helps clinicians make diagnostic and therapeutic decisions at the bedsideDerived from original researchIncorporate 3 or more variables from history, exam, or simple tests
Examples : which patients with -ankle injury need x-rays?possible DVT need imaging?pneumonia need admission?headache need CT/LP?
What Conditions are Suitable for Clinical Decision Rules?
The Need:Common, high-volume conditions, e.g. chest pain, cough and fever, extremity injury, shortness of breathInefficient use of resources, e.g. diagnostic tests or hospitalization Variation in current practice
The Purpose:Standardize care based on evidenceImprove safetyImprove efficiency
References – Clinical Decision Rules
JAMA 2000
Theory and Reality of Developing Clinical Decision Rules
Introduction to CDRs
Examples from Ottawa
Methodological Standards for Derivation
Prospective Validation
Implementation Trial
Knowledge Transfer
Feel free to ask questions
Clinical Decision RulesDeveloped at U Ottawa
Ottawa Ankle and Knee Rules
Canadian C-Spine and CT Head Rules
Ottawa Subarachnoid Rule
Wells Criteria for PE and DVT – Phil Wells
Ottawa Heart Failure and COPD Rules
The Canadian C-Spine Rule
Imaging for Alert & Stable Trauma Patients
Potential C-Spine Injury:The Clinical Problem> 8 million potential neck injury cases per year in Canadian and US EDs
Most are alert and stable with <1% having c-spine fracture
C-spine diagnostic imaging use inefficient and variable
High volume items add to health care costs
Prolonged immobilization and ED overcrowding
Development of the Canadian C-Spine Rule
Variation and Inefficiency (N=6,855)CMAJ 1997
Derivation of the Rule (N=8,924) JAMA 2001
Prospective Validation (N=8,283) New Engl J Med 2003
Multicentre Implementation (N=11,648) Br Med J 2009
Awareness and Use (N=1,150) Acad Emerg Med 2008
Validation by Paramedics Ann Emerg Med 2009
Validation by ED Nurses (N=3,633) CMAJ 2010
Derivation of Canadian C-Spine Rule - JAMA 2001
Objective: Derive a clinical decision rule highly sensitive for acute cervical spine injury
Methods (N=8,924): Prospective cohort - 10 Canadian EDsAlert and stable, adult trauma patientsMDs assessed 20 clinical findingsClinically important c-spine injuryRecursive partitioning analyses
2003
2003
Potential Radiography Rates
66.6%
55.9%
40%
60%
80%
NEXUS Canadian
P < 0.001
Theory and Reality of Developing Clinical Decision RulesIntroduction to CDRs
Examples from Ottawa
Methodological Standards - Derivation
Prospective Validation
Implementation
Knowledge Transfer
Feel free to ask questions
Was the Rule Derived According to Methodological Standards?
Outcome MeasurePredictor VariablesReliability of PredictorsStudy SubjectsSample SizeMathematical TechniquesSensibility for CliniciansClassification Accuracy
Was the Rule Derived According to Methodological Standards? Outcome Measure
Clinically importantClearly definedAssessed blindly
Clinically Important C-Spine Injury
Serious Adverse Event for COPD: DeathIntubation or NIVMyocardial infarctionAdmission to monitored unit Relapse back to ED requiring admission < 14 days
Was the Rule Derived According to Methodological Standards?
Predictor VariablesStandardized definitionCollected prospectively with data formAssessed before outcome
20 Variables from History and Exam
30 Variables from History, Exam, Lab, ECG, Xray Innovative 3-min walk test
Was the Rule Derived According to Methodological Standards?
Reliability of Clinical VariablesInterobserver agreement beyond chance:
kappa (dichotomous / nominal data)weighted kappa (ordinal data)intraclass correlation coefficient (interval)
3. Reliability: Interobserver Agreement for C-Spine Findings
KappaCharacteristic (N=150)
Midline neck pain .69Immediate neck pain .48 Weakness in extremities .54Numbness / tingling .77Upright position .78Distracting injuries .41Facial Injury .75
Was the Rule Derived According to Methodological Standards?
Study SubjectsDefined inclusion criteriaUnbiased selectionCharacteristics describedSetting
Alert stable trauma patients with neck pain
Adults with exacerbation of COPD - both admitted and discharged
Excluded: very ill patients: O2 Sat < 85% on room air, HR > 130, SBP < 85, chest pain or acute ECG changes
Was the Rule Derived According to Methodological Standards?
Sample SizeRequires adequate number of positive outcomesMethods for determining SS:
rule of thumb – 10 outcomes / predictordegree of precision in CI around measure of accuracy
8,924 neck injury patients with 151 important c-spine injury cases (1.7%)
945 COPD patients with 74 SAE cases (7.8%)
Sample Size: C-Spine Phase I
Estimated- 100% sensitivity with 95% CI 97-100%- 120 injury cases- 1.5% incidence important injury- 8,000 patients
Actual- 100% sensitivity (98-100)- 151 injury cases- 1.7% incidence- 8,924 patients
Was the Rule Derived According to Methodological Standards?
Mathematical TechniquesUnivariate KappaMultivariate:
Chi-square recursive partitioningLogistic regression
Variables with strong univariate associations and kappa > 0.6 recursive partitioning
Univariate analyses logistic regression
Independent Predictors of SAE from Logistic Regression (N=945)
Variable βOR
Hx of PVD intervention 0.90 2.46Prior CABG 0.71 2.03Prior intubation 1.32 3.73ECG acute ischemia 1.18 3.25CXR pulmonary congestion 0.63 1.88Too ill to walk after treatment 1.25 3.50HR on ED arrival > 110 1.12 3.05HgB < 100 1.59 4.90Urea > 12 0.89 2.43Serum CO2 > 35 0.66 1.91Hosmer-Lemeshow Goodness-of-fit P = 0.70
Area under ROC curve = 0.80 (95%CI 0.74 -0.85)
Was the Rule Derived According to Methodological Standards?
Sensibility for CliniciansClinical validitySimple and easy to useYes/No vs probability
Yes/No Algorithm for C-Spine
Risk Scale for COPD
Was the Rule Derived According to Methodological Standards?
Accuracy – Classification Performance2 x 2 tables with sensitivity, specificityROC curve analysisPredicted probability
C-Spine – 2 x 2 table
COPD – probability
Theory and Reality of Developing Clinical Decision Rules
Introduction to CDRs
Examples from Ottawa
Methodological Standards for Derivation
Prospective Validation
Implementation Trials
Knowledge Transfer
Feel free to ask questions
Prospective Validation of Ottawa Decision Rules
Ottawa Ankle Rule (N=1,485)JAMA 1993
Ottawa Knee Rule (N=1,096) JAMA 1996
Canadian C-Spine Rule (N=8,283) New Engl J Med 2003
Canadian CT Head Rule (N=2,707)JAMA 2005
Validation by Paramedics (N=1,949)Ann Emerg Med 2009
Validation by ED Nurses (N=3,633) Can Med Assoc J 2010
Was the Rule Prospectively and Explicitly Validated?
New patients and settingsExplicit applicationOutcome measuresAccuracy:
The rulePhysicians
ReliabilityPhysician comfortPotential impact
Validation of Canadian C-Spine Rule - NEJM 2003
Objective: Prospectively compare the clinical performance of the CCR and the NEXUS criteria
Methods (N=8,283): Prospective cohort - 9 Canadian EDsAlert and stable, adult trauma patients349 MDs assessed patients for both rules – data forms2nd observer where feasible169 important c-spine injury cases by imaging or F/U
Other Validation Measurements
Physician Accuracy 91.2%
Reliability (kappa) 0.64
Clinical Sensibility“Uncomfortable” with rule 8.0%
Potential Impact: C-Spine Radiography Rates
71.7%
55.9%
40%
60%
80%
Actual Rate Potential Rate
P < 0.0001RR = 22.0%
232.8
123.0
100
150
200
250
Radiography No Radiography
N=4,129 N=1,779
Potential Impact: Total Time in ED in Minutes
Diff = 109.8P < 0.0001
Actual vs Potential C-Spine Clearance Rates (N=3,633)
Actual Rate Potential Rate0%
25%
50%
0%
40.6%
0%
Theory and Reality of Developing Clinical Decision Rules
Introduction to CDRs
Examples from Ottawa
Methodological Standards for Derivation
Prospective Validation
Implementation Trials
Knowledge Transfer
Feel free to ask questions
Implementation Trials of Ottawa Decision Rules
Ottawa Ankle Rules (N=2,342)JAMA 1994
Ottawa Ankle Rules (N=12,777)Br Med J 1995
Ottawa Knee Rule (N=3,907) JAMA 1997
Canadian C-Spine Rule (N=11,824 Br Med J 2009
Canadian CT Head Rule (N=4,531) Can Med Assoc J 2010
Has the Rule been Implemented into Practice to Assess Impact?
Controlled or cluster randomized trial design
Impact on clinical care:Process measures: e.g. hospital admission, use of imagingPatient outcomes: e.g. mortality, missed injuries
Other:Accuracy of the rulePhysician acceptabilityPatient acceptability
2009
Implementation of Canadian C-Spine Rule – Br Med J 2009
Objective: To evaluate the effectiveness of an active strategy to implement the CCR into multiple EDs
Methods (N=11,824): Matched pair cluster randomized trial12 university and community hospital EDsAlert and stable, adult trauma patients6 hospitals intervention, 6 controlActive strategies :
Education - rounds, handouts, posters, pocket cards, App, screen-saversPolicy Real-time reminders
Diagnostic Imaging Rates (N= 11,648)
20
70
% I
mag
ing
20%
45%
70%
BeforePeriod
AfterPeriod
6 InterventionHospitals
6 ControlHospitals
Study Sites
61.7%
53.7% 53.8%
59.8%
P < 0.01
P < 0.001
P < 0.01
The “mother of all negative trials” !!
PRIMARY OUTCOME (N=4,531)Diagnostic Imaging Rates
0
40
80
% I
mag
ing
BeforePeriod
AfterPeriod
6 InterventionHospitals
6 ControlHospitals
Study Sites
62.8%
76.2%
67.5%74.1%
P < 0.01
P = 0.64P < 0.01
80%
40%
0%
Barriers to Use: Post-Study Survey Physician Beliefs and Attitudes:• Bent rule to meet their needs• Didn’t ‘believe’ the rule• Didn’t like being directed
Electronic Ordering:• Physical restriction of accessing computer • Ability to circumvent the rule
Busy and Overcrowded EDs: • Easier to CT and discharge• Tests that speed flow are used
CT Head is Standard of Care:• Access to CT easy at this site and becoming routine care
Theory and Reality of Developing Clinical Decision Rules
Introduction to CDRs
Examples from Ottawa
Methodological Standards for Derivation
Prospective Validation
Implementation Trials
Knowledge Transfer
Feel free to ask questions
How does Clinician Uptake occur for Decision Rules?
How do we close the evidence-practice gap?
Passive DiffusionJournal articles, scientific meetings
DisseminationTargets an audience – mailouts, speakersMeta-analyses, reviews, guidelines
ImplementationActive, local, persistentAdministrative, educational strategies
Evaluation of the Dissemination and Uptake of Ottawa Decision Rules
Attitudes and Use Ankle/Knee in Canada (N=232)Graham - Acad Emerg Med 1998
Awareness and Use Ankle/Knee in 5 Countries (=1,769)Graham - Ann Emerg Med 2001
CCR and CCHR in Canada (N=262) Brehaut - Acad Emerg Med 2006
CCR and CCHR in 4 Countries (N=1,150) Eagles - Acad Emerg Med 2008
Theory and Reality of Developing Clinical Decision Rules
Introduction to CDRs
Examples from Ottawa
Methodological Standards for Derivation
Prospective Validation
Implementation Trials
Knowledge Transfer
Feel free to ask questions
Barriers to CCH Rule Use:Pre-Study Survey - Brehaut 2003
Forget details of CCH Rule 32%Trauma/NS will order anyway 30%Patient/family expectation 10%Research is flawed 6% See no advantage to no CT 6% Busy ED – can’t observe 6%Takes too much time 2%Rule not safe for patients 2%Resent concept of guidelines 0%
Theory and Reality of Developing Clinical Decision Rules
BMJ 2003
Was the Rule Derived According to Methodological Standards?
Ottawa Risk Scale for ED Patients with COPD
COPD Patients in the ED: The Clinical Problem
Common ED presentation
With bed shortages, MDs under pressure to send home
Adverse outcomes common, especially in those discharged
Little evidence to guide disposition decisions
Need risk scales for rational and safe admission decisions
MethodsDesign: Prospective cohort study
Setting: 6 EDs of large, tertiary care Canadian hospitals
Subjects: Adults with exacerbation of COPD - both admitted and discharged
Excluded: Very ill patients
Standardized Assessment: variables from history, exam, lab, chest x-ray, ECGSerious Adverse Event: death, intubation, critical care, MI, return to ED with admission
Serious Adverse Events (N=69/945)
Discharged Patients (N=591)
Admitted Patients (N=354)
Total SAEs (N=945)
Ottawa COPD Risk Scale - Identify ED Patients at High Risk for SAE
Univariate Correlation of COPD SAEs Variables from Exam and Labs (N=945)
SAE No SAEP-Value
SaO2 on arrival (%) 91.393.5 0.01CTAS level 2.5 2.7 0.02Urea 9.8 7.3 <0.01Glucose 8.2 7.1 0.03pCO2 60.243.6 0.01HGB 122.8 133.7 <0.01ECG ischemia 7.5 1.9 <0.01CXR congestion 25.7 9.1 <0.01Too ill to do walk test (%) 41.913.0 <0.0001Walk test highest HR (N=43, 749) 98.7104.0 0.05
COPD Patient Characteristics (N=945) Mean age 72.6
Male (%) 51.6Ambulance arrival (%) 48.3Duration of symptoms, hours 87.0CTAS score, mean 2.7Associated HF in ED (%) 10.2
Past Medical History (%)Heart Failure 21.1Admission for respiratory distress28.4Intubation for respiratory distress 2.9
Current smoker (N=792) 31.6Home oxygen (%) 12.5
COPD Risk Score Sensitivity Specificity
PotentialAdmission
0 1.0 0.0 100%1 0.91 0.45 57.6%2 0.81 0.60 43.2%3 0.60 0.84 20.0%4 0.52 0.905 0.25 0.976 0.19 0.997 0.07 0.9968 0.06 0.999
10 0.03 1.0
Classification Performance and Potential Admissions for Ottawa COPD Risk Scale
Calibration Between Observed vs Expected SAE Score in COPD Patients
Homer-Lemeshow goodness-of-fit p-value = 0.67
Secondary Outcomes (N=11,648)
Intervention Hospitals Control Hospitals Before After Before After
Outcomes N=3,266 N=3,624 N=2,413 N=2,345
Missed Fractures 0 0 0 0
Adverse Outcomes 0 0 0 0
Time in ED (Mins) 206 215 187 210