Handling Bias in Medical Research

Acknowledge

• Cochrane Collaboration’s Bias Methods Group

Overview

• Impact of bias in medical research

• Introduction to bias and systematic error

• Sources of bias in the production and dissemination of evidence

• Meta-epidemiology and empirical evidence of bias

• Means to assess bias in primary studies

• How does this lead to good medical research?

Potential Impact of bias

The impact that biased estimates of effect have on research is

perhaps obvious but should not be understated

Bias leads to:

Inaccurate estimate of benefit and harms for an intervention

Biases can influence the results of secondary research

systematic reviews

Both bias and “spin” can directly influence what care patients

receive and contribute to quality of care

What is bias?

• A bias is a systematic error, or deviation from the truth,

in results or inferences

If you were to repeat the study over and over again

the results would always be inaccurate

• Biases can vary in magnitude and direction

• There are no current, well accepted means to estimate

magnitude of bias

Common misconceptions

• Bias may be distinguished from methodological quality

• Although the highest standards of conduct may have

been implemented the results still may not be believable

• Bias should not be confused with impression due to

random error (sampling variation)

In systematic reviews the quality of reporting of trial

reports is fundamental to assessing potential biases

(www.consort-statement.org)

Published

Conceived

Performed

Submitted

Cited

Research agenda bias

Sources of bias in the production and

dissemination of evidence

Sackett J Chron Dis 1979

Published

Conceived

Performed

Submitted

Cited

Sources of bias in the production and

dissemination of evidence

Methodological quality

Attrition bias Misclassification bias

Berkson bias Missing clinical data bias

Confounding bias Non blind diagnosis bias

Centripetal bias Non blind test interpretation bias

Diagnostic access bias Non-respondent bias

Diagnostic purity bias Partial verification bias

Diagnostic suspicion bias Performance bias

Diagnostic vogue bias Prevalence-incidence bias

Exposure suspicion bias Recall bias

Family information bias Referral filter bias

Interviewer bias Selection bias

Membership bias Unmasking bias

Published

Conceived

Performed

Submitted

Cited

Sources of bias in the production and

dissemination of evidence

Reporting Biases:

Publication Bias

Time Lag Bias

Language Bias

Multiple Publication Bias

Outcome reporting bias

Citation bias

Further evidence of the presence of bias

In 2010, 235 biases were mapped and characterised

17,265,924 PubMed items were clustered by biases and other text, use

of terms were mapped over time

Forty bias terms were used in the title or abstract of more than 100

articles each;

confounding, response bias, sampling bias, observer bias, cognitive

bias, race bias and perceptual bias all occur in the literature before

1965;

There were shown to be bias ‘cliques’, co-appearing biases and the

evolution of terminology was mapped over time

Chavalarias, Ioannidis 2010, J Clin Epi

0.2 0.4 0.6 0.8 2 1

Meta-analysis

Single large trial

Odds Ratio (95% Confidence Intervals)

Nitrates in myocardial infarction

Inpatient geriatric assessment

Magnesium in myocardial infarction

Aspirin for prevention of

pre-eclampsia

Intervention:

Egger BMJ 1997

Differences are apparent between trials

Meta-epidemiology

• Identify a large number of meta-analyses

• Record characteristic(s) of individual studies (e.g., degree of allocation

concealment, type of publication, language etc.)

• Compare treatment effects within each meta-analysis (e.g., adequate

allocation concealment versus inadequate)

• Estimate ratio of odds ratios comparing high quality and low quality trials

(“meta-meta-analysis”)

(Naylor, BMJ 1997; 315: 617-619

Meta-epidemiological studies: meta-meta-analytic method

Pooled ORs calculated for trials with and

without methodological flaw within each meta-

analysis, and difference calculated as Ratio of

ORs (ROR):

ROR = OR with flaw / OR without flaw

The RORs are then meta-analysed to estimate

overall ROR across all meta-analyses and the

between MA variability in bias

Flawed studies show more benefit Flawed studies show

less benefit

ROR

ROR (95% CI)

ROR

ROR

Empirical evidence of bias: RCTs

Schulz JAMA 1995

0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2

No

Yes

Double-blind

Yes

No

Exclusions

Inadequate/unclear

Adequate

Sequence generation

Inadequate/unclear

Adequate

Concealment of allocation

Ratio of Odds Ratios (95% CI)

Treatment effect

over-estimation under-estimation

(favoring experimental intervention)

Reference

33 meta-analyses,

250 RCTs

Prominent biases in RCTs

Bias Components Evidence

Selection bias Sequence generation

Allocation Concealment

Schulz 1995, Moher 1998,

Kjaergard 2001, Siersma 2007;

Schulz 1995a Jüni 2001

Performance bias Blinding of participants and personnel Pildal 2007, Wood 2008, Boutron

2006

Detection bias Blinding of outcome assessment Pildal 2007, Wood 2008, Boutron

2006

Attrition bias Incomplete outcome data Tierney 2005, Melander 2003,

Porta 2007

Reporting bias Selective reporting

Publication bias

Hutton 2000, Hahn 2002,

Williamson 2005, Chan 2004,

Dwan 2011

Rosenthal 1979, Sterling 1959,

Scherer 2007, Easterbrook 1991,

Dickersin 1993, Hopewell 2008

Methods to assess bias in primary studies

Cochrane Risk of Bias Tool (RCTs)

Downs and Black (NRS)

Newcastle- Ottawa Scale (NRS)

MINORS

NIH criteria

Zaza et al.

QUADAS (DTA)

PEDro

Oxford Centre of evidence-based medicine

Deschartres 2011 J Clin Epi

Single-centre versus multi-centre trials

• New potential (meta)bias (Dechartres, A. 2011)

26% larger effect size in single-centre trials

(adjusting for trial size)

• “Is conduct at a single-centre a surrogate for a process measure that

would explain the result?”

• But which class of studies provides the “truth” ?

• Could fall under the auspices of reporting bias:

Single centre trials published less frequently

More room to shape results positively in small teams

A new concept: Meta-bias

“Meta-epidemiological studies, examining many meta-

analyses on diverse topics, have been finding risk factors

for bias in those meta-analyses-

both in the methods that meta-analysts use and in the

characteristics of studies included-

that are not clearly associated with a bias producing

process in an individual study

We are calling the differences associated with such non-

process-related factors a meta-bias.”

(Goodman and Dickersin, 2011)

Publication Bias

• This concept is not new (Sterling, TD. 1959)

• Well accepted and more recent processes for assessing

asymmetry of funnel plots (Sterne, JAC. 2011)

• Impacts systematic review directly

• Impacts clinical practice guidelines indirectly

• Difficult to assess when k < 5

McAuley 2000

Egger 2003

Published vs. unpublished

0.91 (0.85 to 0.97)

Ratio of odds ratios

0.4 0.5 0.6 0.7 0.8 0.9 1 1.2 1.4 1.6 1.8 2

Impact of publication bias

Funnel plots

• If all studies come from a single underlying

population, this graph should look like an

inverted funnel, with the effect sizes homing in

on the true underlying value as n increases.

Light RJ, Pillemer DB. Summing up. The science of reviewing research. Harvard University Press, 1984.

Funnel plot: no evidence of bias S

tan

dard

Err

or

Odds ratio

0.1 0.3 1 3

3

2

1

0

10 0.6

Funnel plots

• In the presence of publication bias some studies

are missing

• asymmetry is present

Light RJ, Pillemer DB. Summing up. The science of reviewing research. Harvard University Press, 1984.

0.1 0.3 1 3 10 0.6

Asymmetrical

Funnel Plot

Reporting bias present

Odds ratio

Sta

nd

ard

Err

or

3

2

1

0

“Small study effect”

There is a tendency for smaller trials in a

meta-analysis to show greater treatment

effects than the larger trials

Small study effects need not result from bias

Sterne et al. Journal of Clinical Epidemiology 2000

Selective reporting

• This concept is not new (Hemminki 1980)

• Recent evidence that this remains a serious issue (Kirkham et

al PLoS ONE 2010)

• New reviews from 3 consecutive issues Cochrane Library

• 22% of 288 review/protocol pairings were discrepant in at

least one outcome

• Currently included in Cochrane RoB Tool;

• Limited by the need to review trial protocols, the potential

solution is registration mandated by funding agencies

Global attempts to minimise bias:

Trial registration

In 2004 the International committee of Medical Journal

Editors (ICMJE) announced that they will no longer publish

trials that were not registered at inception

• Clinical trials.gov

• WHO portal

Global attempts to minimise bias:

Trial registration

This is a great start, but with the evidence mounting

suggesting that registration is not optimal

Registering Systematic Review Protocols:

An International Initiative

Vancouver, BC, Canada

February 18, 2011

• http://www.crd.york.ac.uk/prospero/

• Launched Feb 2011

• 400+ registered protocols from 27 countries

Thank you for your attention

David Moher: dmoher@ohri.ca