simple hierarchies are (too) simplistic
DESCRIPTION
Simple hierarchies are (too) simplistic. STUDY DESIGN Randomized Controlled Trials Cohort Studies and Case Control Studies Case Reports and Case Series, Non-systematic observations. BIAS. Expert Opinion. Expert Opinion. Schünemann & Bone, 2003. Likelihood of and confidence in an outcome. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/1.jpg)
Simple hierarchies are (too) simplistic
STUDY DESIGN Randomized Controlled
Trials Cohort Studies and
Case Control Studies Case Reports and Case
Series, Non-systematic observations
BIAS
Expert Opinion
Expert Opinion
Schünemann & Bone, 2003
![Page 2: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/2.jpg)
Likelihood of and confidence in an outcome
![Page 3: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/3.jpg)
Determinants of confidence• RCTs • observational studies • 5 factors that can lower quality
1. limitations in detailed study design and execution (risk of bias criteria)
2. Inconsistency (or heterogeneity)3. Indirectness (PICO and applicability)4. Imprecision5. Publication bias
• 3 factors can increase quality1. large magnitude of effect2. opposing plausible residual bias or confounding3. dose-response gradient
![Page 4: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/4.jpg)
1. Design and Execution/Risk of BiasLimitation in observational studies Explanations
Failure to develop and apply appropriate eligibility criteria (inclusion of control population)
• under- or over-matching in case-control studies
• selection of exposed and unexposed in cohort studies from different populations
Flawed measurement of both exposure and outcome
• differences in measurement of exposure (e.g. recall bias in case- control studies)
• differential surveillance for outcome in exposed and unexposed in cohort studies
Failure to adequately control confounding
• failure of accurate measurement of all known prognostic factors
• failure to match for prognostic factors and/or adjustment in statistical analysis
Incomplete or inadequately short follow-up
![Page 5: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/5.jpg)
1. Design and Execution/Risk of BiasLimitations in RCTs
lack of concealment
intention to treat principle violated
inadequate blinding
loss to follow-up
early stopping for benefit
selective outcome reporting
![Page 6: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/6.jpg)
Design and Execution/RoB
From Cates , CDSR 2008
![Page 7: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/7.jpg)
Design and Execution/RoB
Overall judgment required
![Page 8: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/8.jpg)
YesNoDon’t know or undecided
Who believes the risk of bias is of concern?
![Page 9: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/9.jpg)
Detailed study design and execution
Mortality, cancer andanticoagulation
Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ. “Anticoagulation for the long term treatment of venous thromboembolism in patients with cancer”. Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006650.
![Page 10: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/10.jpg)
Five trials
![Page 11: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/11.jpg)
YesNoDon’t know or undecided
Who believes the risk of bias is of concern?
![Page 12: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/12.jpg)
2. Inconsistency of results(Heterogeneity)
• if inconsistency, look for explanation– patients, intervention, comparator, outcome
• if unexplained inconsistency lower quality
![Page 13: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/13.jpg)
Reminders for immunization uptake
![Page 14: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/14.jpg)
![Page 15: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/15.jpg)
Inconsistency
• I2
• P-value• Overlap in CI• Difference in point estimates
![Page 16: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/16.jpg)
3. Directness of Evidencegeneralizability, transferability, applicability
• differences in– populations/patients (HIC – L/MIC, patients with HIV – all
patients)– interventions (new fluroquinolones - old)– comparator appropriate (newer antibx – old)– outcomes (important – surrogate; signs and symptoms –
mortality)• indirect comparisons
– interested in A versus B– have A versus C and B versus C– Cryo + antibiotics versus no intervention versus Cryo versus
no intervention
![Page 17: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/17.jpg)
EVIDENCE PROFILEQuestion: Cyrotherapy with antibiotics vs no antibiotics for histologically confirmed CIN
1 All rates presented at 12 months with assumption that events would occur within this time frame.2 Indirect analysis between single arm observational studies
Quality assessment No of patients Effect
Quality ImportanceNo of studies Design Limitations Inconsistency Indirectness Imprecision Other
Cryotherapy with
antibioticsNo
antibioticsRelative(95% CI) Absolute
Major infection (follow-up 12 months1; requiring hospitalisation or blood transfusion)16 observation
al studiesno serious limitations
no serious inconsistency
serious2 no serious imprecision
none0/1600
(0%)10/4573 (0.22%)
RD 0 (0 to 0)
0 fewer per 1000 OOO IMPORTANT
Resource use - not measured
All severe adverse events (follow-up 12 months; (major infections and bleeding, pelvic inflammatory disease, stenosis, etc )17 observation
al studiesno serious limitations
no serious inconsistency
serious2 no serious imprecision
none0/1705
(0%)22/5142 (0.43%)
RD 0 (0 to 0)
0 fewer per 1000 OOO IMPORTANT
![Page 18: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/18.jpg)
4. Publication Bias
• Should always be suspected– Only small “positive” studies– For profit interest– Various methods to evaluate – none perfect, but
clearly a problem
![Page 19: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/19.jpg)
Egger M, Smith DS. BMJ 1995;310:752-54 19
I.V. Mg in acute
myocardial infarction
Publication bias
Meta-analysisYusuf S.Circulation 1993
ISIS-4Lancet 1995
![Page 20: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/20.jpg)
Egger M, Cochrane Colloquium Lyon 2001 20
Funnel plotS
tand
ard
Err
or
Odds ratio0.1 0.3 1 3
3
2
1
0
100.6
Symmetrical:No publication bias
![Page 21: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/21.jpg)
Egger M, Cochrane Colloquium Lyon 2001 21
Funnel plotS
tand
ard
Err
or
Odds ratio0.1 0.3 1 3
3
2
1
0
100.6
Asymmetrical:Publication bias?
0.4
![Page 22: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/22.jpg)
5. Imprecision• Small sample size
– small number of events
• Wide confidence intervals– uncertainty about magnitude of effect
![Page 23: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/23.jpg)
Example: Immunization in children
![Page 24: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/24.jpg)
![Page 25: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/25.jpg)
For systematic reviews
• If the 95% CI excludes a relative risk (RR) of 1.0 and the total number of events or patients exceeds the OIS criterion, precision is adequate. If the 95% CI includes appreciable benefit or harm (we suggest a RR of under 0.75 or over 1.25 as a rough guide) rating down for imprecision may be appropriate even if OIS criteria are met.
![Page 26: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/26.jpg)
Optimal information size
• We suggest the following: if the total number of patients included in a systematic review is less than the number of patients generated by a conventional sample size calculation for a single adequately powered trial, consider rating down for imprecision. Authors have referred to this threshold as the “optimal information size” (OIS)
![Page 27: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/27.jpg)
What can raise quality?
1. large magnitude can upgrade (RRR 50%/RR 2)– very large two levels (RRR 80%/RR 5)– criteria
• everyone used to do badly• almost everyone does well
– parachutes to prevent death when jumping from airplanes
![Page 28: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/28.jpg)
Reminders for immunization uptake
![Page 29: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/29.jpg)
What can raise quality?2. dose response relation
– (higher INR – increased bleeding)– childhood lymphoblastic leukemia
• risk for CNS malignancies 15 years after cranial irradiation• no radiation: 1% (95% CI 0% to 2.1%) • 12 Gy: 1.6% (95% CI 0% to 3.4%) • 18 Gy: 3.3% (95% CI 0.9% to 5.6%)
3. all plausible residual confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed
![Page 30: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/30.jpg)
All plausible residual confoundingwould result in an overestimate of effect
Hypoglycaemic drug phenformin causes lactic acidosis
The related agent metformin is under suspicion for the same toxicity.
Large observational studies have failed to demonstrate an association– Clinicians would be more alert to lactic acidosis in
the presence of the agent• Vaccine – adverse effects
![Page 31: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/31.jpg)
Quality assessment criteria Study design
Initial quality of a body of evidence
Lower if Higher if Quality of a body of evidence
Randomised trials
High Risk of Bias
Inconsistency
Indirectness
Imprecision
Publication bias
Large effect Dose response All plausible residual confounding & bias -Would reduce a demonstrated effect -Would suggest a spurious effect if no effect was observed
A/High (four plus: )
B/Moderate (three plus: )
Observational studies
Low C/Low (two plus: )
D/Very low (one plus: )
![Page 32: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/32.jpg)
Overall quality of a body of evidence
• The quality of evidence reflects the extent of our confidence that the estimates of an effect are adequate to support a particular decision or recommendation.
• Guideline developers must specify and determine importance of all relevant outcomes
• Overall quality of evidence is based on the lowest quality of all critical outcomes
![Page 33: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/33.jpg)
Meta-analyses of several critical and important outcomes (one PICO)
Better Relative RiskWorse
0.5 0.75 1 1.25 1.5
SAE (critical)
Nausea (important)
Myo. Infarct. (critical)
Mortality (critical) High
Low Due to imprecision and risk of bias
Moderate Due to imprecision
High
Overall Quality of Evidence: Moderate based on critical outcomes
Low
![Page 34: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/34.jpg)
Meta-analyses of several critical outcomes (one PICO)
Better Relative RiskWorse
0.5 0.75 1 1.25 1.5
SAE
Stroke
Dis. Specific QoL
Mortality
Threshold of acceptable harm for strong
recommendation based on sure benefit in mortality and stroke
High
High
Moderate Due to imprecision
High
Overall Quality of Evidence: High
![Page 35: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/35.jpg)
Meta-analyses of several critical outcomes (one PICO)
Better Relative RiskWorse
0.5 0.75 1 1.25 1.5
SAE
Stroke
Dis. Specific QoL
Mortality
Overall Quality of Evidence:
High
Moderate due to risk of bias
High
High
Moderate
Threshold of acceptable harm for strong
recommendation based on sure benefit in mortality and stroke
![Page 36: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/36.jpg)
Interpretation of grades of evidence
• /A/High: Further research is very unlikely to change confidence in the estimate of effect.
• /B/Moderate: Further research is likely to have an important impact on confidence in the estimate of effect and may change the estimate.
• /C/Low: Further research is very likely to have an important impact on confidence in the estimate of effect and is likely to change the estimate.
• /D/Very low: We have very little confidence in the effect estimate: Any estimate of effect is very uncertain.
![Page 37: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/37.jpg)
Agenda8.30-8.45 Introduction to the course8.45-9.15 Overview of guideline development9.15-10.15 The GRADE approach: introduction10.15-10.30 Break10.30-11.30 Asking a key question, specific outcomes – small group work11.30-12.30 The GRADE approach: assessing the quality of evidence12.30-13.15 Lunch13.15-14.45 Grading quality of evidence – small group work14.45-15.15 Moving from evidence to recommendations: theoretical considerations15.15-15.30 Break15.30-16 Making recommendations – small group work16-16.30 Report from small groups and practical considerations of group process
![Page 38: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/38.jpg)
Agenda8.30-8.45 Introduction to the course8.45-9.15 Overview of guideline development9.15-10.15 The GRADE approach: introduction10.15-10.30 Break10.30-11.30 Asking a key question, specific outcomes – small group work11.30-12.30 The GRADE approach: assessing the quality of evidence12.30-13.15 Lunch13.15-14.45 Grading quality of evidence – small group work14.45-15.15 Moving from evidence to recommendations: theoretical considerations15.15-15.30 Break15.30-16 Making recommendations – small group work16-16.30 Report from small groups and practical considerations of group process
![Page 39: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/39.jpg)
MOVING FROM EVIDENCE TO RECOMMENDATIONS
Holger (contributions from Yngve Falck-Ytter)
![Page 40: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/40.jpg)
Strength of recommendation
“The strength of a recommendation reflects the extent to which we can, across the range of patients for whom the recommendations are intended, be confident that desirable effects of a management strategy outweigh undesirable effects.” • Strong or conditional
![Page 41: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/41.jpg)
Evidence based healthcare decisions
Research evidence
Population/societalvalues
and preferences
(Clinical) state and circumstances
Expertise
Haynes et al. 2002
![Page 42: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/42.jpg)
Evidence based healthcare decisions
Research evidence
Population/societalvalues
and preferences
Expertise
Evidence of baseline risk and context for a given population
Schunemann et al. in prep
![Page 43: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/43.jpg)
Evidence based healthcare decisions
Research evidence
Expertise
Evidence of baseline risk and context for a given population
Evidence of population & societal values
& preferences
Schunemann et al. in prep
![Page 44: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/44.jpg)
Evidence based healthcare decisions
Expertise
Evidence of baseline risk and context for a given population
Evidence of population & societal values
& preferences
Evidence of effectsSchunemann et al. in prep
![Page 45: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/45.jpg)
Evidence based healthcare decisions
Expertise
Evidence of baseline risk and context for a given population
Evidence of population & societal values
& preferences
Evidence of effectsSchunemann et al. in prep
![Page 46: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/46.jpg)
From evidence to recommendations
RCTObser-vational study
High level recommen-
dation
Lower level recommen-
dation
Old system
Quality of evidence
Balance between benefits, harms & burdens
Patients’ values &
preferences
GRADE
Resource use
![Page 47: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/47.jpg)
Determinants of the strength of recommendation
Factors that can strengthen a recommendation
Comment
Quality of the evidence The higher the quality of evidence, the more likely is a strong recommendation.
Balance between desirable and undesirable effects
The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely weak recommendation warranted.
Values and preferences The greater the variability in values and preferences, or uncertainty in values and preferences, the more likely weak recommendation warranted.
Costs (resource allocation) The higher the costs of an intervention – that is, the more resources consumed – the less likely is a strong recommendation warranted
![Page 48: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/48.jpg)
Balancing desirable and undesirable consequences
Conditional
Strong For Against
Effects & $ x
values
Effects & $ x
values
![Page 49: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/49.jpg)
Balancing desirable and undesirable consequences
↑ Allergic reactions
↑ Local skin reactions
↑ Nausea↑ Resources
↑ QoL ↓ Death
↓ Morbidity
↑ herd immunity
Conditional
Strong For Against
Effect x value Effect x value
![Page 50: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/50.jpg)
↑ Allergic
reactions
↑ Local skin
reactions
↑ Nausea
↑ Resources
↑ QoL↓ Death
↓ Morbidity↑ herd
immunityConditional
Strong For Against
Balancing desirable and undesirable consequences
![Page 51: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/51.jpg)
↑ Allergic reactions↑ Local skin reactions
↑ Nausea
↑ Resources↑ QoL ↓ Death
↓ Morbidity
↑ herd immunity
Conditional
Strong For Against
Balancing desirable and undesirable consequences
![Page 52: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/52.jpg)
Balancing desirable and undesirable consequences
↑ Allergic
reactions
↑ Local skin
reactions
↑ Nausea
↑ Resources
↑ QoL
↓ Death
↓
Morbidity
↑ herd
immunity
Conditional
Strong For Against
![Page 53: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/53.jpg)
Balancing desirable and undesirable consequences
↑ Allergic reactions ↑ Local skin
reactions
↑ Nausea
↑ Resources
↑ QoL
↓ Death
↓ Morbidity
↑ herd immunity
Conditional
Strong For Against
![Page 54: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/54.jpg)
Implications of a strong recommendation
• Policy makers: The recommendation can be adapted as a policy in most situations
• Patients: Most people in this situation would want the recommended course of action and only a small proportion would not
• Clinicians: Most patients should receive the recommended course of action
![Page 55: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/55.jpg)
Implications of a conditional recommendation• Policy makers: There is a need for
substantial debate and involvement of stakeholders
• Patients: The majority of people in this situation would want the recommended course of action, but many would not
• Clinicians: Be more prepared to help patients to make a decision that is consistent with their own values/decision aids and shared decision making
![Page 56: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/56.jpg)
Case scenario
A 13 year old girl who lives in rural Indonesia presented with flu symptoms and developed severe respiratory distress over the course of the last 2 days. She required intubation. The history reveals that she shares her living quarters with her parents and her three siblings. At night the family’s chicken stock shares this room too and several chicken had died unexpectedly a few days before the girl fell sick.
![Page 57: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/57.jpg)
Methods – WHO Rapid Advice Guidelines for Avian Flu
Applied findings of a recent systematic evaluation of guideline development for WHO/ACHR
Group composition (including panel of 13 voting members): clinicians who treated influenza A(H5N1) patients infectious disease experts basic scientists public health officers methodologists
Independent scientific reviewers: Identified systematic reviews, recent RCTs, case series, animal
studies related to H5N1 infection
![Page 58: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/58.jpg)
Oseltamivir for Avian FluSummary of findings: • No clinical trial of oseltamivir for treatment of H5N1
patients.• 4 systematic reviews and health technology
assessments (HTA) reporting on 5 studies of oseltamivir in seasonal influenza. – Hospitalization: OR 0.22 (0.02 – 2.16)– Pneumonia: OR 0.15 (0.03 - 0.69)
• 3 published case series. • Many in vitro and animal studies. • No alternative that was more promising at present.• Cost: 40$ per treatment course
![Page 59: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/59.jpg)
Evidence profile
No of studies(Ref) Design Limitations Consistency Directness Other
considerations Oseltamivir Placebo Relative(95% CI )
Absolute(95% CI )
Mortality 0 - - - - - - - - - 9
5(TJ 06)
Randomised trial
No limitations One trial only Major uncertainty (-2)1
Imprecise or sparse data (-1)
- - OR 0.22(0.02 to 2.16)
- Very low
6
0 - - - - - - - - - - 7
5(TJ 06)
Randomised trial
No limitations One trial only Major uncertainty (-2)1
Imprecise or sparse data (-1)2
2/982(0.2%)
9/662(1.4%)
RR 0.149(0.03 to 0.69)
- Very low
8
53
(TJ 06)(DT 03)
Randomised trials
No limitations4 Important inconsistency(-1)5
Major uncertainty (-2)1
- - - HR 1.303
(1.13 to 1.50)-
Very low5
26
(TJ 06)Randomised trials
No limitations -7 Major uncertainty (-2)1
None - - - WMD -0.738
(-0.99 to -0.47)
Low4
0 - - - - - - - - - - 4
0 - - - - - - - - - - 7
09 - - - - - - - - - - 7
311
(TJ 06)Randomised trials
No limitations -12 Some uncertainty (-1)13
Imprecise or sparse data (-1)14
- - OR range15
(0.56 to 1.80)-
Low
0 - - - - - - - - - - 4
I mportance
Summary of findings
Cost of drugs
Outbreak control
Resistance
Serious adverse effects (Mention of significant or serious adverse effects)
Minor adverse effects 10 (number and seriousness of adverse effects)
Viral shedding (Mean nasal titre of excreted virus at 24h)
Duration of disease (Time to alleviation of symptoms/median time to resolution of symptoms – influenza cases only)
Duration of hospitalization
LRTI (Pneumonia - influenza cases only)
Healthy adults:
Hospitalisation (Hospitalisations from influenza – influenza cases only)
Quality assessmentNo of patients Effect
Quality
Oseltamivir - H5N1
-
-
Quality across
endpoints
![Page 60: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/60.jpg)
![Page 61: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/61.jpg)
PICO & Confidence in estimates
![Page 62: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/62.jpg)
Benefits vs harms
![Page 63: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/63.jpg)
Values and preferences
![Page 64: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/64.jpg)
Resource utilization
![Page 65: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/65.jpg)
From evidence to recommendation
Factors that can strengthen a recommendation
Comment
Quality of the evidence Very low quality evidence
Balance between desirable and undesirable effects
Uncertain, but small reduction in relative risk still leads to large absolute effect
Values and preferences Little variability and clear
Costs (resource allocation) Low cost under non-pandemic conditions
![Page 66: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/66.jpg)
Example: Oseltamivir for Avian FluRecommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (strong recommendation, very low quality evidence).
Remarks: This recommendation places a high value on the prevention of death in an illness with a high case fatality. It places relatively low values on adverse reactions, the development of resistance and costs of treatment.
Schunemann et al. The Lancet ID, 2007
?
![Page 67: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/67.jpg)
Other explanationsRemarks: Despite the lack of controlled treatment data for H5N1, this is a strong recommendation, in part, because there is a lack of known effective alternative pharmacological interventions at this time.
The panel voted on whether this recommendation should be strong or weak and there was one abstention and one dissenting vote.
![Page 68: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/68.jpg)
GRADE Grid
![Page 69: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/69.jpg)
Values and preferences
• Implicit value judgments in recommendations
• Trade-offs: example prevention of VTE in surgery– Thrombotic events
• Deep vein thrombosis, pulmonary embolism– Bleeding events
• Gastrointestinal bleeds, operative site bleeds– Inconvenience of injections
• Variability in values and preferences
![Page 70: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/70.jpg)
Case• 77 y/o patient with atrial fibrillation, mild
CHF, HTN, DM and history of stroke (fully recovered)
• Meds: warfarin, antihypertensives, statin, glyburide
• Admitted with nausea/vomiting, then hematemesis; INR 2.5; 1 U blood transfused; EGD: no active bleed, possible Mallory Weiss
• This is his second major bleed since he started warfarin one year ago
![Page 71: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/71.jpg)
CHADS2 score
![Page 72: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/72.jpg)
Acceptable additional bleeds?• Study: Patients at high risk for atrial fibrillation and high
risk of stroke (h/o CHF/MI); internists and cardiologists
• Warfarin decreases risk at cost of increased GI bleeds• Without treatment 100 patients will suffer:
– 12 strokes (six major, six minor), 3 serious GI bleeds in 2 years
• Warfarin would decrease strokes in 100 patients to 4 per 2 years (8 fewer strokes, 4 major, minor)
• How many additional bleeds would you accept in 100 patients over a year, and still be willing to administer/take warfarin?
![Page 73: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/73.jpg)
Strength of recommendation“The strength of a recommendation reflects the extent to which we can, across the range of patients for whom the recommendations are intended, be confident that desirable effects of a management strategy outweigh undesirable effects.”
Understanding values & preferences necessary to trade-off benefits and downsides
Values and preferences should ideally be informed by systematic reviews, but evidence is often sparse
![Page 74: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/74.jpg)
Example recommendation
ACCP AT9 recommendation:In patients undergoing major orthopedic surgery (e.g., total hip replacement), we suggest the use of LMWH in preference to the other agents.Patients who place a high value on avoiding bleeding complications and a low value on its inconvenience are likely to choose a compression device (IPCD) over the drug options.
![Page 75: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/75.jpg)
Considering resource use
• Allocating resources such as cost• In GRADE: cost = outcome. Challenges include:
– Who pays? Individual vs. society?– Varies widely (even within 1 kilometer) – Opportunity cost differ– Often highly political (e.g., USA)
• Resource use considered last as an additional step
![Page 76: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/76.jpg)
Example 1: ipilimumabfor metastatic melanoma
Estimated absolute effects at 2 years
Outcome Relative effect
(95% CI)
control(per 1000)
ipilimumab(per 1000)
Difference(per 1000)
Certainty of the effect
Death HR 0.68(0.55 to
0.85)850 725 125 fewer deaths
(49 to 202 fewer)
High
Serious immune-
related AEs
RR 7.37(4.42 to
12.3)39 289 249 more SAEs
(134 to 441 more)
High
Quality of life
Acquisition cost
Ipilimumab: $120,000 (12 weeks induction (4 injections))Dacarbazine: $400
![Page 77: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/77.jpg)
Simplified cost effectiveness projection
$ or $ per LY
Dacarbazine induction $400
Ipilimumab induction $120,000
Increase in median survival 3.6 mo (0.3 years)
Incremental cost-effectiveness ($/life year)
~$400,000
![Page 78: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/78.jpg)
Not only cost but also resource use
• Emergency room visits, hospitalizations• Severe immune-mediated events (grade 3 and
4): 86/627 patients with hepatic toxicity, 18/627 with enterocolitis; 5 immune-mediated deaths associated with ipilimumab
• Use of other interventions downstream• Include complete resource use in evidence
profile if possible (e.g., hospital days, nurses hours, drug units dispensed, etc.)
![Page 79: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/79.jpg)
Define perspective taken
• Example 1: single regional health system– Health plan / insurance– Country– International: e.g., WHO
• Perspective should be sufficiently broad– Should include downstream effects– Comprehensive view desirable: e.g., societal
• Includes all cost regardless who bears them
![Page 80: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/80.jpg)
Quality of evidence assessment
• Rarely assessed in the context of RCTs used to inform estimate of effect
• Often derived from other sources of evidence
• More confident in the acquisition cost than the estimate of resources used for serious AEs
![Page 81: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/81.jpg)
Modeling resource use• Modeling results in
cost per unit benefits achieved can be helpful– …but sometimes be misleading and prone to
bias– …relies heavily on assumptions – …debate whether indirect costs, such as lost
productivity should be included• Avoid including cost-utility models directly
in the GRADE evidence profile
![Page 82: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/82.jpg)
![Page 83: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/83.jpg)
![Page 84: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/84.jpg)
SummaryFrom evidence to making recommendations
1. Beyond net benefit, GRADE makes transparent the process on how values & preferences and resource use informs decision making
2. Understanding values & preferences is necessary to make trade-offs between benefits and downsides
3. If necessary, resource use considerations can inform direction and strength of recommendations
![Page 85: Simple hierarchies are (too) simplistic](https://reader036.vdocuments.net/reader036/viewer/2022062323/568165cb550346895dd8d38e/html5/thumbnails/85.jpg)
Conclusions Guidelines should be based on the best available evidence to
be evidence based GRADE is the approach used by over 65 organizations and
gaining acceptance internationally combines what is known in health research methodology and
provides a structured approach to improve communication Does not avoid judgments but provides framework Criteria for evidence assessment across questions and
outcomes Criteria for moving from evidence to recommendations Transparent, systematic
four categories of quality of evidence two grades for strength of recommendations
Transparency in decision making and judgments is key