the abc of evidence-base medicine
TRANSCRIPT
Dr Max Mongelli 2016
http://drmaxmongelli.weebly.com
What is EBM?
““The conscientious, explicit and The conscientious, explicit and judicious use of judicious use of current best current best evidenceevidence in making decisions about in making decisions about the care of individual patients”the care of individual patients”
Prof. David L. Sackett, 1997 Prof. David L. Sackett, 1997
Dr Max Mongelli 2016
Why EBM?
Dr Max Mongelli 2016
Dr Max Mongelli 2016
Primum non nocerePrimum non nocere““First do no harm”First do no harm”
Hippocrates, Hippocrates, EpidemicsEpidemics
Dr Max Mongelli 2016
Sources of Evidence in Medicine
Traditional TeachingTraditional Teaching TextbooksTextbooks Basic sciencesBasic sciences Observational studiesObservational studies Computer simulationComputer simulation Decision AnalysisDecision Analysis Case-Control StudiesCase-Control Studies Randomised Controlled Trials (RCT)Randomised Controlled Trials (RCT)• Meta-analysesMeta-analyses
Dr Max Mongelli 2016
RCOG Classification of Evidence LevelsRCOG Classification of Evidence Levels
1++ High quality meta-analyses of 1++ High quality meta-analyses of RCT'sRCT's
1+ Meta-a. Or RCT's at low risk of bias1+ Meta-a. Or RCT's at low risk of bias 1- Meta-a. Or RCT's at high risk of bias1- Meta-a. Or RCT's at high risk of bias 2++ High quality meta-analyses of 2++ High quality meta-analyses of
CC'sCC's 2+ Well-conducted cc or cohort 2+ Well-conducted cc or cohort
studiesstudies 2- Case-control or cohort studies 2- Case-control or cohort studies
with ? biaswith ? bias 3 Case reports3 Case reports 4 Expert opinion4 Expert opinion
Dr Max Mongelli 2016
““Effectiveness and Effectiveness and Efficiency: Random Efficiency: Random Reflections of Reflections of Health ServicesHealth Services “, “, 19711971
Archie Cochrane (1909-88)Archie Cochrane (1909-88)
Dr Max Mongelli 2016
Randomized Controlled TrialsRandomized Controlled Trials
““Gold standard” in evaluating Gold standard” in evaluating new therapies or surgical new therapies or surgical techniquestechniques
May also be applied to new May also be applied to new diagnostic testsdiagnostic tests
Dr Max Mongelli 2016
Objectives of RCT:Objectives of RCT: Minimize bias by randomisationMinimize bias by randomisation Achieve statistical power through Achieve statistical power through
adequate sample sizeadequate sample size ““Blinding “ - single or doubleBlinding “ - single or double Analysis by intention to treatAnalysis by intention to treat
Randomized Controlled TrialsRandomized Controlled Trials
Dr Max Mongelli 2016
Randomized Controlled TrialsRandomized Controlled Trials
RandomisationRandomisation Several techniques availableSeveral techniques available Computer software linked to central Computer software linked to central
monitoring stationmonitoring station ““Block “ randomisationBlock “ randomisation Sealed envelope methodSealed envelope method
Dr Max Mongelli 2016
Dr Max Mongelli 2016
What about observational studies?What about observational studies?
Dr Max Mongelli 2016
RCT’s and Observational StudiesRCT’s and Observational Studies
• Two studies published in the NEJM in 2000 suggested that RCTs and observational studies overall produced similar results
• JAMA 2001: “discrepancies beyond chance do occur and differences in estimated magnitude of treatment effect are very common”
• RCTs may be unnecessary for treatments that have dramatic and rapid effects relative to the expected
Dr Max Mongelli 2016
RCT’s and Industry FundingRCT’s and Industry Funding
•RCT’s funded by industry are significantly more likely to report positive results
•Possibly due to publication bias
•RCTs may be unnecessary for treatments that have dramatic and rapid effects relative to the expected
Dr Max Mongelli 2016
RCT’s and Statistical ErrorRCT’s and Statistical Error
• Type I error – “false positive”
• Type II error – “false negative”
• Sample size calculations often inaccurate
Dr Max Mongelli 2016
Diagnostic TestsDiagnostic Tests
Dr Max Mongelli 2016
2 X 2 Table2 X 2 Table
Disease present
Disease absent
Test Positive
a b
Test Negative
c d
Dr Max Mongelli 2016
2 X 2 Table2 X 2 Table
Sensitivity = TP =Sensitivity = TP =
Disease present
Disease absent
Test Positive
a b
Test Negative
c d
Dr Max Mongelli 2016
2 X 2 Table2 X 2 Table
Sensitivity = TP = a/(a+c)Sensitivity = TP = a/(a+c)
Disease present
Disease absent
Test Positive
a b
Test Negative
c d
Dr Max Mongelli 2016
2 X 2 Table2 X 2 Table
False Positive Rate = FP = False Positive Rate = FP =
Disease present
Disease absent
Test Positive
a b
Test Negative
c d
Dr Max Mongelli 2016
2 X 2 Table2 X 2 Table
FP = b/(d+b)FP = b/(d+b)
Disease present
Disease absent
Test Positive
a b
Test Negative
c d
Dr Max Mongelli 2016
2 X 2 Table2 X 2 Table
Specificity = 1 - FP = Specificity = 1 - FP =
Disease present
Disease absent
Test Positive
a b
Test Negative
c d
Dr Max Mongelli 2016
2 X 2 Table2 X 2 Table
Specificity = 1 - FP = d/(d+b)Specificity = 1 - FP = d/(d+b)
Disease present
Disease absent
Test Positive
a b
Test Negative
c d
Dr Max Mongelli 2016
2 X 2 Table2 X 2 Table
Positive predictive value (PPV) =Positive predictive value (PPV) =
Disease present
Disease absent
Test Positive
a b
Test Negative
c d
Dr Max Mongelli 2016
2 X 2 Table2 X 2 Table
PPV = a/(a+b)PPV = a/(a+b)
Disease present
Disease absent
Test Positive
a b
Test Negative
c d
Dr Max Mongelli 2016
2 X 2 Table2 X 2 Table
Negative predictive value (NPV) = Negative predictive value (NPV) =
Disease present
Disease absent
Test Positive
a b
Test Negative
c d
Dr Max Mongelli 2016
2 X 2 Table2 X 2 Table
NPV = d/(c+d)NPV = d/(c+d)
Disease present
Disease absent
Test Positive
a b
Test Negative
c d
Dr Max Mongelli 2016
PPV and PrevalencePPV and Prevalence
Steep drop in positive predictive value as Steep drop in positive predictive value as disease prevalence decreasesdisease prevalence decreases
Dr Max Mongelli 2016
PPV and PrevalencePPV and Prevalence
PPV = PPV = (sens x prev)(sens x prev) (sens x prev +(1 - spec)x(1 - prev))(sens x prev +(1 - spec)x(1 - prev))
Dr Max Mongelli 2016
The Likelihood RatioThe Likelihood Ratio
Single value to indicate Single value to indicate the clinical utility of a testthe clinical utility of a test
Independent of Independent of prevalenceprevalence
LR = Sensitivity/(1- Spec.)LR = Sensitivity/(1- Spec.) LR >8 : tests usually LR >8 : tests usually
clinically usefulclinically usefulDr Max Mongelli 2016
The Likelihood RatioThe Likelihood Ratio
LR is an odds modifier:LR is an odds modifier:
Posterior odds =Posterior odds =prior odds x LRprior odds x LR
Dr Max Mongelli 2016
Odds and ProbabilityOdds and Probability
Inter-convertible:Inter-convertible:
Odds = p/(1-p)Odds = p/(1-p)
Dr Max Mongelli 2016
Can tests be combined ?Can tests be combined ?
Rare conditions: high rates Rare conditions: high rates of false positivesof false positives
Lead to excessive Lead to excessive unnecessary interventionunnecessary intervention
Can be reduced by Can be reduced by combining tests e.g. combining tests e.g. intrapartum monitoringintrapartum monitoring
Dr Max Mongelli 2016
Impact of new diagnostic Impact of new diagnostic test on clinical outcomes:test on clinical outcomes:
RCTRCT Cohort studyCohort study Case-control studyCase-control study Before and after studyBefore and after study
Dr Max Mongelli 2016
SYSTEMATIC REVIEWSSYSTEMATIC REVIEWS
Dr Max Mongelli 2016
"It is surely a great criticism of our "It is surely a great criticism of our profession that we have not organised profession that we have not organised a critical summary, by specialty or a critical summary, by specialty or subspecialty, adapted periodically, of subspecialty, adapted periodically, of all relevant randomized controlled all relevant randomized controlled trials." trials."
Archie Cochrane, 1972Archie Cochrane, 1972
Dr Max Mongelli 2016
Role of systematic reviewsRole of systematic reviews
Before commencing a new project: to determine Before commencing a new project: to determine whether further studies are reallywhether further studies are really indicated: ‘state-indicated: ‘state-of-the-art’ literature review.of-the-art’ literature review.
Gain in statistical power for average estimates.Gain in statistical power for average estimates. 'Cumulative' meta-analysis can determine when 'Cumulative' meta-analysis can determine when
further studies are nofurther studies are no longer indicated.longer indicated. Design of subsequent studies.Design of subsequent studies. Setting policy for treatment and health care – Setting policy for treatment and health care –
making the best use of themaking the best use of the data available.data available.
Dr Max Mongelli 2016
Can Studies be Combined?Can Studies be Combined?
Identification of optimal inclusion criteria can be Identification of optimal inclusion criteria can be difficult.difficult.
The most critical step is choosing the appropriate The most critical step is choosing the appropriate research question.research question.
A fairly general question is more preferable to a very A fairly general question is more preferable to a very specific one. specific one.
Tukey : "...far better an approximate answer to the Tukey : "...far better an approximate answer to the right question, than an exact answer to the wrong right question, than an exact answer to the wrong question.."question.."
Dr Max Mongelli 2016
Publication BiasPublication Bias
Entire research studies may fail to reach publication Entire research studies may fail to reach publication because of the nature of the results.because of the nature of the results.
Identification of unpublished trials can be very Identification of unpublished trials can be very difficult - in one study it accounted for 22% of the difficult - in one study it accounted for 22% of the papers included in the meta-analysis. papers included in the meta-analysis.
Failure to publish rests with the investigators rather Failure to publish rests with the investigators rather than editors. than editors.
Dr Max Mongelli 2016
Comparison of the meta-analyses of smaller studies Comparison of the meta-analyses of smaller studies with the corresponding result of the largest study.with the corresponding result of the largest study.
30 meta-analyses including a total of 185 randomised 30 meta-analyses including a total of 185 randomised controlled studies (RCT) obtained from the Cochrane controlled studies (RCT) obtained from the Cochrane pregnancy and childbirth database. The meta-pregnancy and childbirth database. The meta-analyses were only included if they had at least one analyses were only included if they had at least one trial with a total sample size of over 1000.trial with a total sample size of over 1000.
Calculations differ from the Cochrane database in Calculations differ from the Cochrane database in that the largest trial was excluded, this being used as that the largest trial was excluded, this being used as the 'gold standard' for outcomethe 'gold standard' for outcome
PREDICTIVE ABILITY OF META-ANALYSESPREDICTIVE ABILITY OF META-ANALYSES Villar et al, Lancet 1995 Villar et al, Lancet 1995
Dr Max Mongelli 2016
There was total agreement between the meta-analysis and the largest study in 18/30 (60% [C.I. 42-78]) of comparisons.
There was partial agreement between the meta-analysis and the largest study in 6/30 (20% ) of comparisons.
There was disagreement in 6/30 (20% [C.I. 6-34] ) of comparisons.
PREDICTIVE ABILITY OF META-ANALYSESPREDICTIVE ABILITY OF META-ANALYSES Villar et al, Lancet 1995 Villar et al, Lancet 1995
Dr Max Mongelli 2016
Dr Max Mongelli 2016
The Cochrane Collaboration The Cochrane Collaboration
• Established in 1993 by Sir Iain Chalmers• International: 100 countries• Independent• Not-for-profit• Over 27000 contributors
Dr Max Mongelli 2016
RANZCOG and EBMRANZCOG and EBM
“RANZCOG endorses the principles of Evidence-based Medicine and recognizes the NHMRC levels of evidence and grades of recommendations”
College Statement C-Gen 15, Nov. 2009
Dr Max Mongelli 2016
…a scientific idea can never be proven true, because no matter how many observations seem to agree with it, it may still be wrong. On the other hand, a single contrary experiment can prove a theory forever false…
Sir Karl Popper
Dr Max Mongelli 2016