James C. Coyne James C. Coyne 1,2 1,2 , Brett Thombs, Brett Thombs3 3 ,Mariët ,Mariët HagedoornHagedoorn22

11University of Pennsylvania, School of Medicine, USA University of Pennsylvania, School of Medicine, USA 22University Medical Center Groningen, the NetherlandsUniversity Medical Center Groningen, the Netherlands33McGill University, Montreal, CanadaMcGill University, Montreal, Canada

Meta-analyses to establish the Meta-analyses to establish the effectiveness of psychosocial effectiveness of psychosocial

interventions in health psychology: Is interventions in health psychology: Is the literature ready?the literature ready?

Narrative Reviews Are Narrative Reviews Are an Endangered an Endangered

Species, Soon to be Species, Soon to be Extinct.Extinct.

Systematic ReviewSystematic Review

Involves the application of Involves the application of scientific strategies, in ways scientific strategies, in ways that limit bias, to the that limit bias, to the assembly, critical appraisal, assembly, critical appraisal, and synthesis of all relevant and synthesis of all relevant studies that address a studies that address a specific clinical question.specific clinical question.

There is often a perception that the statistical combination of data across studies is the most important part of a systematic review. We take such a view cautiously. We believe that a well-reported, systematic qualitative review is much better than an inappropriately conducted and reported quantitative review or meta-analysis. Moher et al., 1998

Exaggerated or Premature Conclusion of a Meta Analysis

Discourages Committing Resources to Research—The Issue is Settled, So Why Study It!

Ultimately Gets Found Out.

Loss of Credibility to the Field.

Zimmermann et al. "Does One Size Fit All?" Moderators in psychosocial interventions for breast cancer patients: A meta-analysis. Ann

Behav Med. 2007; 34: 225-239.

First research question: Whether breast cancer patients had better outcomes when they received interventions in a study that only included breast cancer patients compared to studies that included patients with mixed diagnoses. Need to compare the outcome of studies in which breast cancer patients were treated alone to the effects for breast cancer patients in studies in which they were mixed with patients with other cancers.

Unfortunately, the separate treatment effects for breast cancer patients cannot be isolated in any of these studies.

Coyne, Thombs, & Hagedoorn. A Meta-Analysis of Psychosocial Interventions for Cancer Patients Gone Awry. Ann

Behav Med. (2009).“Psychosocial intervention” never defined.

A diversity of outcomes-- satisfaction with getting a tour, psychological distress, returning to work--considered comparable and collapsed within and across studies.

Failed to evaluate methodological quality in selecting studies.

Conclusions were influenced by numerous coding and computational errors, inexplicable omission of studies, and multiple counting of the same study.

Hoffman, B. M. et al. (2007). "Meta-analysis of psychological interventions

for chronic low back pain." Health Psychology , 26: 1-9.“Positive effects of psychological

interventions… were found for pain intensity, pain-related interference, health-related quality of life, and depression.”

“Multidisciplinary approaches that included a psychological component, when compared with active control conditions, were also noted to have positive short-term effects on pain interference and positive long-term effects on return to work.

“The robust nature of these findings should encourage confidence among clinicians and researchers alike.”

Critique of Hoffman, B. M. et al. (2007). "Meta-analysis of psychological

interventions for chronic low back pain."In 60% of the studies included in the meta-analysis,

intervention and control groups were not comparable on key variables at baseline.

Only 3 of 34 studies assessed patient adherence to activities required by the intervention and only 3 of 34 restricted outside interventions.

Less than a third of studies had manualized treatment or protocols describing session by session, and a smaller proportion monitored treatment fidelity.

Less than half of the studies provided adequate information concerning number of patients enrolled, treatment drop-out and reasons for drop-outs.

Only 15% of trials provided intent-to-treat analyses.

Critique of Hoffman, B. M. et al. (2007). "Meta-analysis of psychological

interventions for chronic low back pain."

A lack of evidence that psychological interventions are superior to other active treatments and of any enduring effects of psychological interventions beyond immediate post-treatment assessments.

Lack of benefit for depressive symptomatology.

Arguments for the distinctive contribution of psychological interventions to multiple modal treatments assume comparisons that are not possible from available studies.

Reviewed 4 meta-analyses that recently appeared in Health Psychology.

Problems with the transparency and completeness with which the meta-analyses were reported.

Dependence of the meta-analyses on small, underpowered trials of generally poor quality.

Conclusions were of questionable clinical validity and utility.

Evidence Based Medicine Reviews Evidence Based Medicine Reviews in Health Psychologyin Health Psychology

Dixon, Keefe, Scipio, Perri, & Dixon, Keefe, Scipio, Perri, & Abernethy, 2007Abernethy, 2007

Hoffman, Papas, Chatkoff, & Hoffman, Papas, Chatkoff, & Kerns, 2007Kerns, 2007

Irwin, Cole, & Nicassio, 2006Irwin, Cole, & Nicassio, 2006

Jacobsen, Donovan, Jacobsen, Donovan, Vadaparampil, & Small, 2007Vadaparampil, & Small, 2007

Like Pornography?Like Pornography?

““I shall not attempt I shall not attempt to define the kind to define the kind of material I of material I understand to be understand to be embraced within a embraced within a shorthand shorthand description…But description…But “I “I know it when I see know it when I see it”,it”, and the motion and the motion picture in question picture in question in this case is not in this case is not that.” that.”

Justice Potter Stewart Justice Potter Stewart

Was the conduct of Was the conduct of the meta-analysis the meta-analysis accurately and accurately and adequately described adequately described in the article or in the article or supplementary supplementary materials?materials?

Very Useful ResourceVery Useful Resource

Cooper, H., S. Maxwell, et al. Cooper, H., S. Maxwell, et al. (2008). "Reporting Standards for (2008). "Reporting Standards for Research in Psychology Why Do Research in Psychology Why Do We Need Them? What Might They We Need Them? What Might They Be?" Be?" American PsychologistAmerican Psychologist 63(9): 839-851.63(9): 839-851.

Was there an adequate Was there an adequate attempt to deal with the attempt to deal with the methodological quality of methodological quality of the original intervention the original intervention trials?trials?

Look for: Look for:

A scoring systemA scoring system

More than one assessorMore than one assessor

A strategy for taking A strategy for taking quality into account in quality into account in analysesanalyses

Was an adequate effort made to deal with the

methodological quality of studies?

“Wide variations in the nature of interventions, outcome measures, length of follow-up periods, and presentations of trials’ results prohibited us from using meta-analysis” (p. 561).”

To what extent did To what extent did the results of the the results of the

meta-analysis meta-analysis depend on small, depend on small,

underpowered underpowered studies?studies?

Are there enough sufficiently number of studies with adequate sample size to draw a conclusion?

Studies with small sample sizes are prone to publication bias.

Null results can be dismissed as the result of low statistical power and left unpublished, whereas positive results seem particularly impressive because they are obtained despite low statistical power and get published.

Effect sizes found in small underpowered studies tend to overestimate the true effect size.

Probability of detecting at least a moderate sized effect when it is present in a study with 35 patients per condition is unacceptably low, about 50-50.

Jacobsen et al.Jacobsen et al.

70-80% studies rated as only 70-80% studies rated as only fair. fair.

< 40% intervention and control < 40% intervention and control groups were comparable at groups were comparable at baselinebaseline

<50% indicated the number of <50% indicated the number of patients enrolled, treatment patients enrolled, treatment drop-out and reasons for drop-drop-out and reasons for drop-outout

Irwin et al. Irwin et al.

Exclusion of small trials (n Exclusion of small trials (n < 35) would have < 35) would have eliminated eliminated all all studies of studies of older adults; five of these older adults; five of these studies included 15 or studies included 15 or fewer participants per fewer participants per condition. Of the studies condition. Of the studies including younger adults, including younger adults, only one of the 15 studies only one of the 15 studies would have remained.would have remained.

Does the meta analysis Does the meta analysis adequately deal with adequately deal with clinical heterogeneity clinical heterogeneity and is there a basis for and is there a basis for

giving a meaningful giving a meaningful interpretation to a interpretation to a

single summary effect single summary effect size?size?

Clinical Heterogeneity

Combining studies may be inappropriate Combining studies may be inappropriate for a variety of the following reasons: for a variety of the following reasons: differences in patient eligibility criteria in differences in patient eligibility criteria in the included trials, different interventions the included trials, different interventions and outcomes, and other methodological and outcomes, and other methodological differences or missing information. differences or missing information.

Moher et al., 1998Moher et al., 1998

Clinical Connoisseurship

Consumers of meta-analyses should be intimately familiar Consumers of meta-analyses should be intimately familiar with clinical phenomena and the nature of interventions, with clinical phenomena and the nature of interventions, and they need to be able to go beyond the numerical data in and they need to be able to go beyond the numerical data in determining whether it is appropriate to integrate studies determining whether it is appropriate to integrate studies that differ in patient populations and likely restrictions on that differ in patient populations and likely restrictions on who will be willing and able to participate in an who will be willing and able to participate in an intervention, the nature of the intervention, and the intervention, the nature of the intervention, and the appropriateness of outcomes for determining the efficacy of appropriateness of outcomes for determining the efficacy of an intervention. They need to be prepared to make critical an intervention. They need to be prepared to make critical assessments of whether the attempted integration of data assessments of whether the attempted integration of data exceeds even the metaphor of mixing apples and oranges.exceeds even the metaphor of mixing apples and oranges.

To what extent does the meta To what extent does the meta analysis distinguish analysis distinguish interventions that explicitly interventions that explicitly targeted clinically significant targeted clinically significant levels of problems (distress, levels of problems (distress, fatigue) versus interventions fatigue) versus interventions that did not have such an aim that did not have such an aim or that did not require or that did not require clinically significant levels of clinically significant levels of the problem?the problem?

In Dixon (pain) and In Dixon (pain) and JacobsenJacobsen(fatigue) patients did not (fatigue) patients did not have to meet a threshold have to meet a threshold criterion of pain/fatigue, criterion of pain/fatigue, primary objective of the primary objective of the intervention did not have to intervention did not have to involve pain/fatigue involve pain/fatigue reduction, and pain/fatigue reduction, and pain/fatigue did not have to be a primary did not have to be a primary outcome.outcome.

Meta Analyses Have Become Meta Analyses Have Become Powerful and Authoritative ToolsPowerful and Authoritative Tools

The The hope is that meta analysis provides hope is that meta analysis provides definitive conclusionsdefinitive conclusions concerning whether concerning whether interventions work even when drawing on interventions work even when drawing on contradictory and sometimes flawed individual contradictory and sometimes flawed individual studies.studies.

Risk is that Risk is that bad meta analyses discourage bad meta analyses discourage revisiting the original studies that were revisiting the original studies that were integrated in it.integrated in it.

Risk is that Risk is that inaccurate meta analyses will be inaccurate meta analyses will be used for decisions to pay for delivery of used for decisions to pay for delivery of services to patients, override patient services to patients, override patient preference, and define future research preference, and define future research priorities.priorities.