chicago 2014 peter morley, eddy lang e3, grade expert incorporating lower levels of evidence
TRANSCRIPT
Chicago 2014
Peter Morley, Eddy LangE3, GRADE expert
Incorporating lower levels of evidence
Chicago 2014
COI #380Commercial/industry
Evidence Evaluation Expert (E3: ILCOR/AHA)
Potential intellectual conflictsActing Chair Australian Resuscitation Council (ARC)ANZCOR delegate on International Liaison Committee on Resuscitation (ILCOR)
Chicago 2014
EBP is to consider the best available evidence
Chicago 2014
Issues
EBP is to consider the best available evidence GRADE and other approaches to systematic reviews only incorporate RCTs and cohort studies
Chicago 2014
Unfortunately in Resuscitation and First Aid science
RCTs are lackingGood quality non-RCTs are also lackingSometimes only case series, case reports or animal data (previous LOE 4 and 5)
Chicago 2014GRADE allocation of quality of evidence across each outcome
RCTs start as highcan be downgraded
Non-RCTs start as lowCan be downgradedCan occasionally be upgraded
Chicago 2014
So where do Case Series fit?
Case series are observational studies
Same bias assessment tools as other observational studiesRecommended only to include if 4 or more cases are included in a manuscript
Chicago 2014Bias assessment for Case Series
Were appropriate eligibility criteria developed and applied to both the cohort of interest and the comparison cohort? NOWas measurement of both exposure and outcome appropriate and consistently applied to both the cohort of interest and the comparison cohort? NOWas confounding adequately controlled for? Could be YESWas follow up complete? Could be YES
Chicago 2014Determinants of quality
What lowers quality of evidence? 5 factors:
Methodological limitations
Inconsistency of results
Indirectness of evidence
Imprecision of results
Publication bias
• RCTs start high
• Observational studies start low
• Case series start very low
Chicago 2014
Still go on to complete a GRADE Evidence Profile table
Enter all studies you include in your final analysis for each outcome
Chicago 2014
www.guidelinedevelopment.org
Chicago 2014
Then populate the fields using the drop-down menus
Chicago 2014
Then populate the fields using the drop-down menus
Chicago 2014
Then populate the fields using the drop-down menus
Chicago 2014
Then populate the fields using the drop-down menus
No serious limitations: Most information is from studies at low risk of bias. Serious limitations: Most information is from studies at moderate risk of bias. Very serious limitations: Most information is from studies at high risk of bias.
Chicago 2014
And don’t forget to add a comment to the footnotes each time you are downgrading based on your assessment
Chicago 2014
Implications of the assessment of limitations
No serious limitations: Most information is from studies at low risk of bias
No downgrading of evidence Serious limitations: Most information is from studies at moderate risk of bias
Evidence would be normally downgraded 1 level (ie. for RCTs from high to moderate)
Very serious limitations: Most information is from studies at high risk of bias
Evidence would be normally downgraded 2 levels (ie. for RCTs from high to low)
Chicago 2014
Then populate the fields using the drop-down menus
Reviewers should document limitations when (1) point estimates vary widely across studies, (2) confidence intervals show minimal or no overlap (ie. studies appear to have different effects), or (3) statistical tests of heterogeneity are suggestive
Chicago 2014
Concerns about directness therefore arise when there are differences in: •The Population (eg. patients in cardiac arrest vs not in cardiac arrest), •Intervention (eg. standard CPR using 2010 guidelines vs standard CPR using 2005 guidelines), or •Outcomes (surrogates, eg. ROSC vs termination of VF for 5 seconds), or where •no head to head Comparisons between interventions
Chicago 2014Other concerns about indirectness
Evidence from animal studies, manikins or other models would generally be rated as having very serious limitations (but this would be dependent on the key outcomes listed).
Interventions may be delivered/implemented differently in different settings (eg. therapeutic hypothermia).
Limitations should therefore be considered if the intervention cannot be implemented with the same rigor or technical sophistication as in the studies from which the data come.
Limitations in more than one type of directness may suggest a need to rate the studies as having very serious limitations.
Chicago 2014
Concerns about imprecision arise when:•Cis overlap with a clinical decision threshold, eg. 1% absolute difference •Trials stopped early •Early publications (particularly if small)•Optimal Information Size (eg. <400 patients for continuous data)•Outcomes: importance, the adverse effects, the burden to the patient, resource use, and the difficulty of introducing a measure into practice
Chicago 2014
Then populate the fields using the drop-down menus
Chicago 2014
Then populate the fields using the drop-down menus
Chicago 2014
Publication bias
Reviewers should allocate strongly suspected when the evidence consists of a number of small studies, especially if these are industry sponsored or if the investigators share another conflict of interest. The risk of publication bias in observational studies is probably larger than RCTs
Chicago 2014
Then populate the fields using the drop-down menus
Chicago 2014
Magnitude of effect for non-RCTs
A large magnitude effect would be considered as justification to increase the rating by one level (eg. low to moderate) if:
relative risk [RR] = 2-5 or RR 0.5-0.2 with no plausible confounders; or RR very large >5 or <0.2 and no serious problems with risk of bias or precision (sufficiently narrow confidence intervals);
The reviewer would be more likely to rate up if the above size of effects were rapid and out of keeping with prior trajectory; they are usually supported by indirect evidence.If above criteria are all met, and the RR is very large (eg. >5-10), rating up by two levels (from low to high) could be considered, but obviously not if other concerns were present (such as risk of bias, imprecision, inconsistency, indirectness and publication bias).
Chicago 2014
Then populate the fields using the drop-down menus
Chicago 2014Plausible confounding for non-RCTs
If:all plausible prognostic factors are
accurately measured in observational studies, and if
all the observed residual confounders and biases would diminish the observed effect,
then the effect estimate would be strengthened. In this setting, the rating up by one level could be considered.
Chicago 2014
Chicago 2014
Dose-response gradient for non-RCTs
A dose-response gradient (such as increased effect with increased dose, or decreased time to intervention) increases the confidence in the findings of observational studies. In this setting, the rating up by one level could be considered
Chicago 2014
For dichotomous (yes/no) variables, for a single study, enter number with outcome, and total number of patients.Then click “apply”
Outcomes
Chicago 2014
No control group
For dichotomous (yes/no) variables, for a single study, enter number with outcome, and total number of patients.Then click “apply”
Chicago 2014
No relative effects
Choose desired variable: usually Relative Risk (RR): single or multiple studies if meta-analysis performed
Chicago 2014
When and how to perform a meta-analysis(interventional studies)
WhenMore than one study, ofGood methodological quality, addressingNear identical individual components of
PICOPopulation, Intervention, Comparison, Outcome
Rarely with non-RCTsHow
Send email and data for listed outcomes for studies to Peter or Eddy
Chicago 2014
GRADE evidence profile (preferred)
Chicago 2014
Chicago 2014
Chicago 2014
?
Chicago 2014
Peter Morley, Eddy Lang, Laurie MorrisonMethods Committee
Cost and resource implications
Chicago 2014
COI #380
Commercial/industryEvidence Evaluation Expert (E3: ILCOR/AHA)
Potential intellectual conflictsActing Chair Australian Resuscitation Council (ARC)ANZCOR delegate on International Liaison Committee on Resuscitation (ILCOR)
Chicago 2014
ILCOR Methods Committee Guidance on the
Consideration of Resources in the Creation of Treatment Recommendations as part
of the CoSTR
Chicago 2014
Issues
the requirement of significant methodological expertise and rigour around health economic evaluations the variability in cost and resource considerations across a myriad of settings
Chicago 2014
2 Cost/Resource Questions
Are the resources required small?Is the incremental cost small relative to the net benefits?
Chicago 2014
Overarching principles
The greater the resource requirements (cost, including but not limited to equipment price and training costs), the less likely it is that an option will be a priority.Resource costs, and the implications of these to the Councils implementing the guidelines, are likely to vary substantially across jurisdictions.
Chicago 2014
1st question
“Are the resources required small?”
Chicago 2014“Are the resources required small?” No* Probably no Uncertain Probably yes Yes*
* Need to be supported by formal cost analysis
Chicago 2014“Are the resources required small?” No* Probably no Uncertain Probably yes Yes*
* Need to be supported by formal cost analysis
Chicago 2014“Are the resources required small?” No
supported by formal cost analysis in relevant contexts
Chicago 2014“Are the resources required small?”
Probably no Significant resources are probably required, but no formal cost analysis identified. Incremental implementation costs are significant and likely to impact on the introduction of the new therapy/diagnostic technique. These could include significant drug, equipment or diagnostic test costs, significant educational requirements, or additional significant costs for employing more staff or purchase of monitoring devices. Please elaborate on the reasons you chose this option.
Chicago 2014“Are the resources required small?” Uncertain
Resource estimates vary considerably and no reliable source of information can be identified
Chicago 2014“Are the resources required small?” Probably yes
Resources probably not altered, but no formal cost analysis identified. Incremental implementation costs are small and unlikely to impact on the introduction of the new therapy/diagnostic technique.
Chicago 2014“Are the resources required small?” Yes
supported by formal cost analysis
Chicago 2014“Are the resources required small?” No* Probably no Uncertain Probably yes Yes*
* Need to be supported by formal cost analysis
Chicago 2014
2nd question
“Is the incremental cost small relative to the net benefits?”
Chicago 2014
“Is the incremental cost small relative to the net benefits?”
No* Probably no Uncertain Probably yes Yes*
* Need to be supported by formal cost-effectiveness analysis
Chicago 2014
“Is the incremental cost small relative to the net benefits?”
No* Probably no Uncertain Probably yes Yes*
* Need to be supported by formal cost-effectiveness analysis
Chicago 2014
“Is the incremental cost small relative to the net benefits?”
NoA formal cost-effectiveness analysis suggests that the intervention is not justified in terms of the resource allocated for benefit obtained.
Chicago 2014
“Is the incremental cost small relative to the net benefits?”
Probably noAlthough no formal cost-effectiveness analysis was conducted, the resources required to implement the intervention or diagnostic strategy appear to be large or resource intensive relative to the estimate of the net benefit.
Chicago 2014
“Is the incremental cost small relative to the net benefits?”
Uncertain The resource considerations are difficult to estimate and the benefits and risks/harms are uncertain so no reasonable consideration of cost-effectiveness is possible at this time.
Chicago 2014
“Is the incremental cost small relative to the net benefits?”
Probably yes Although no formal cost analysis was conducted and efficacy estimates are uncertain, the resources required are deemed small enough that implementation might be considered reasonable from a cost effectiveness perspective. Potential example - ASA in suspected ACS.
Chicago 2014
“Is the incremental cost small relative to the net benefits?”
YesA formal cost-effectiveness analysis supports the view that the costs are redeemed by improved outcomes.
Chicago 2014
“Is the incremental cost small relative to the net benefits?”
No* Probably no Uncertain Probably yes Yes*
* Need to be supported by formal cost-effectiveness analysis
Chicago 2014
A couple of examples
Chicago 2014
Example 1
Cardiogel
Chicago 2014
Cardiogel
systematic review identifying moderate quality evidence from 3 randomized controlled trials (Lang 2010 234, Morrison 2012 245, Morley 2013 341) downgraded for risk of bias demonstrating that in patients with cardiac arrest, “cardiogel” a hypothetical free radical scavenger improves survival to hospital discharge in comparison to placebo. The RR for this and other critical outcomes is 0.8 95% (CI 0.7-0.9)
Chicago 2014
Cardiogel
• cardiogel is perceived by committee members and the TFQO as expensive– “Probably no” to “cost small” Q– “Uncertain” to cost-effectiveness
Q (as no formal analysis was conducted or identified)
Chicago 2014Cardiogel: ILCOR guidance
• The treatment recommendation should not address resource considerations
• State “no formal economic analysis based on a systematic review was completed or identified” in the “research evidence” column to provide justification
• Furthermore the question of resource consumption should not weigh in the development of the treatment recommendation.
Chicago 2014
Example 2
“A novel bodywrap device”
Chicago 2014
Bodywrap
• systematic review identifying very low quality evidence (Lang 2010 234, Morrison 2012 245, Morley 2013 341) downgraded for imprecision, risk of bias and indirectness suggesting that a novel bodywrap device to assist CPR fails to improve any critical or important outcome in comparison to standard CPR; RR 1.0 95% CI 0.4 – 1.6
• no evidence of harm.
Chicago 2014
Bodywrap• the device is perceived by the committee members
and the TFQO to be significantly more expensive in all countries represented by the Taskforce than standard CPR and/or its implementation would consume significant resources in most health care settings– “Probably no” to “cost small” Q
• an economic analysis based on a systematic review of addressing the incremental cost-effectiveness of CPR bodywrap was not done– “Uncertain” to cost-effectiveness Q
(as no formal analysis was conducted or identified)
Chicago 2014Bodywrap: ILCOR guidance (1)
• In this setting it would be appropriate for the TFQO and EvRevs to incorporate consensus-based statements pertaining to resource considerations into the development of the treatment recommendation for this PICO.
• Specifically it would be appropriate to suggest against the use of bodywrap over standard CPR – weak recommendation
Chicago 2014Bodywrap: ILCOR guidance (2)
• “In making this recommendation we place greatest value on not directing health care resources that were perceived by the committee to be high towards an unproven treatment except under experimental conditions.”
Chicago 2014
If bodywrap were:• relatively inexpensive and/or already
readily available to resuscitation teams in most or all settings
• the Task Force could consider making a weak recommendation in favour of CPR bodywrap over standard CPR
• “In making this recommendation we place a higher value on a potential clinical benefit in a setting with a high rate of poor outcomes over other considerations”.
Chicago 2014
Additional comments
• when a useless or harmful therapy is in wide use, developing recommendations against a management approach is also appropriate as would recommendations to use interventions only in research.
• Task Forces may also choose to divide their recommendations into ones that address high and low resource settings.
Chicago 2014
?