handout h6 the scope of meta-analysis: meta-analysis of observational studies

11 August 2010

Handout H6

The scope of meta-analysis: Meta-analysis of observational studies

Topics in Meta-Analysis (Matthias Egger) Erasmus Summer Programme

Objectives

• Understand the importance of systematic reviews of observational studies

• Understand the limitations of meta-analysis in observational studies

• Understand the difficulties of avoiding publication bias in observational studies

11 August 2010 Topics in Meta-Analysis (Matthias Egger) Erasmus Summer Programme

Why do we need observational studies?

• Randomisation may be

– impossible

– unnecessary

– inappropriateBlack, BMJ 1996


Potentials of systematic reviews• More objective appraisal of the evidence than traditional

narrative reviews applies equally to OS & RCT

• May resolve uncertainty when original research, reviews and editorials disagree applies equally to OS & RCT

• May generate promising research questions to be addressed in future studies applies equally to OS & RCT

• Meta-analysis will enhance the precision of effect estimates, leading to reduced probability of false negative results BUT in OS may be a precise biased result


Meta-analysis

• A statistical analysis which combines the

results of several independent studies

considered by the analyst to be

‘combinable’

Huque 1988


Assumptions in meta-analysis

• “Fixed-effects model”: Underlying effect is the same value (fixed) in each study. The differences between study results are solely due to the play of chance.

• “Random-effects model”: Treatment effect for the individual studies are assumed to vary around some overall central effect


Fundamental difference in assumptions & how they apply to MA of RCT or observational studies

• In meta-analysis of observational studies confounding, residual confounding and bias:

– May introduce heterogeneity

– May lead to misleading (albeit very precise) estimates

• In well-conducted RCT there should not be confounding


Relative risk

(95% confidence interval)

0.1 0.2 0.5 1 2 5 10

Trial (Year)

Barber (1967) Reynolds (1972)

Wilhelmsson (1974) Ahlmark (1974)

Multicentre International (1975) Yusuf (1979)

Andersen (1979)

Rehnqvist (1980) Baber (1980)

Wilcox Atenolol (1980)

Wilcox Propanolol (1980) Hjalmarson (1981)

Norwegian Multicentre (1981)

Hansteen (1982) Julian (1982) BHAT (1982) Taylor (1982)

Manger Cats (1983)

Rehnqvist (1983) Australian-Swedish (1983)

Mazur (1984) EIS (1984)

Salathia (1985)

Roque (1987) LIT 91987)

Kaul (1988) Boissel (1990)

Schwartz low risk (1992)

Schwartz high risk (1992) SSSD (1993)

Darasz (1995) Basu (1997)

Aronow (1997)

Overall (95% CI) 0.80 (0.74 - 0.86)

Mortality results from 33

trials of beta-blockers in

secondary prevention after

myocardial infarction

Adapted from Freemantle et al BMJ 1999

0.2 0.5 1 2 5 10

Study

Allen Barongo Bollinger Bwayo Bwayo Cameron Carael Chao Chiasson Diallo Greenblatt Grosskurth Hira Hunter Konde-Luc Kreiss Malamba Mehendal Moss Nasio Pepin Quigley Sassan Sedlin Seed Simonsen Tyndall Urassa 1 Urassa 2 Urassa 3 Urassa 4 Urassa 5 Van de Perre

Relative risk

(95% confidence interval)

Results from 29 studies examining the association between intact foreskin

and the risk of HIV infection in men

Adapted from Van Howe Int J STD AIDS 1999

Formaldehyde exposure and lung cancer

0

50

100

150

IndustrialWorkers(14 Cohorts)

Anatomists,Pathologists(3 Cohorts)

Funeral DirectorsEmbalmers(7 Cohorts)

SM

R (

95

% C

I)

Blair et al Scan J Work Environ Health 1990

Dietary fat and breast cancer

0.6

0.8

1.0

1.2

1.4

1.6

1.8

12 Case-Control Studies

6 Cohort Studies

Rel

ativ

e R

isk

(95%

CI)

Boyd et al Br J Cancer 1993

Intermittent sunlight exposure and melanoma

0

1

2

3

7 Case-ControlStudies withBlinding

Od

ds

Rat

io (

95%

CI)

9 Case-ControlStudies withoutBlinding

Nelemans et al J Clin Epidemiol 1995

Test of homogeneity

• Examines the possibility of excess variability

between the results of the different studies

• Has low power if the number of studies is

small

• Can get a set of homogeneous but spurious

findings


Beta-carotene and cardiovascular mortality

0.1 0.5 0.75 1 1.25 1.5

Cohort Country

Male health workers

Social insurance, men

Male chemical workers

Hyperlipidaemic men

Nursing home residents

USA

Finland

Finland

Switzerland

USA

USA

Social insurance, women

Cohorts combined

Relative risk (95% CI)

Jah et al Ann Intern Med 1995

0.1 0.5 0.75 1 1.25 1.5 1.75

Male health workers

Social insurance, men

Male chemical workers

Hyperlipidaemic men

Nursing home residents

Social insurance, women

Male physicians

Male smokers

(Ex)-smokers, asbestos workers

Trials

Cohorts

Skin cancer patients

USA

Finland

Switzerland

USA

USA

Finland

Cohorts combined

Trials combined

Finland

USA

USA

USA

Relative risk (95% CI)

Beta carotene and cardiovascular disease

Egger et al. BMJ 1998;316:140-4

“Well, so much for antioxidants.”

Smoking and suicide

0.2 1 2 5 10 25

No of cigarettes

MRFIT screenees

Whitehall I

North Karelia men

Kuopio men

1-1414-24 25+

1-1414-24 25+

1-1414-24 25+

1-1414-24 25+

1-1414-24 25+

Meta-analysis

Relative rate (95% CI)

Davey Smith et al Lancet 1992

Smoking and homicide

• Non-smoker 1.00

• 1-2 packs/day 1.71 (1.29-2.28)

• 2+ packs/day 2.04 (1.32-3.15)


Fundamental difference in assumptions

• In meta-analysis of observational studies confounding, residual confounding and bias:

– May introduce heterogeneity

– May lead to misleading (albeit very precise) estimates


What is the appropriate weighting factor?

Inverse of variance?

Case-control studies of Helicobacter pylori infection and CHD

Study Cases +ve

Controls +ve

Cases -ve

Controls -ve

Crude OR

Adjusted OR

Weight

Danesh 1999a

472 272 650 850 2.3 1.9 44%

Danesh1999b

134 294 112 348 1.4 1.3 17%

Patel 1995

56 135 27 170 2.6 2.8 6%

Murray1995

102 1117 33 863 2.4 1.5 9%

McDonagh

1997

315 625 134 353 1.3 0.9 m1.0 f

25%


Danesh et al 1999 a

Danesh et al 1999 b

Numbers in study 2224 888

Response rate controls <20% 60%

Response rate cases ~60% 56%

Adjustment for social position

+ +++

Other adjustments + +++

Representative cases? No Fairly representative

Representative controls? No Fairly representative

OR sex/age adjusted 2.3 1.4

OR fully adjusted 1.9 1.3

Weight in meta-analysis 44% 17%

Two case-control studies of Helicobacter pylori infection and CHD


Study Cases +ve

Controls +ve

Cases

-ve

Controls -ve

Crude OR

Adjusted OR

Weight

Strachan 1998

204 1061 82 449 1.05 1.02 17%

Wald 1997 308 595 340 701 1.07 1.06 37%

Aromaa 1998 229 411 47 116 1.38 - 9%

Folsom 1998 111 257 106 241 0.98 0.97 13%

Ossewaarde 1998

39 84 15 24 0.74 - 2%

Whincup 2000

401 740 104 285 1.48 1.30 20%

Prospective and nested case-control studies of Helicobacter pylori infection and CHD


In RCT meta-analyses the appropriate study weights should relate to precision of effect estimates (e.g. inverse of variance).

In observational meta-analyses this may not generally be the case.


Inverse of variance weighting

• Can lead to magnification of pooled effect estimates when confounding and bias involved (e.g. H pylori)

• Can lead to under-estimation of effect estimates when measurement error is important


Huxley, Lancet 2002

Publication bias in MA / SR of observational studies

Data analysed

Paper published

Reporting bias in observational research

Data collected(e.g cohort study)

Report written

“I regret to inform you that the Journal of xxx will not be able to use your manuscript … We think the study is well-designed, with a fair follow-up and appropriate statistical

analysis, but the negative results found can only be published as a Letter to the editor …”

Rejection of ‘negative’ prospective cohort study finding of association of d-dimer with CHD. May 2006


Genetic meta-analysis may be an exception …

To the homogeneity and “spurious precision” problems…

But may be particularly prone to publication bias


0.33 0.5 0.66 1.0 1.5 2.0 3.00.33 0.5 0.66 1.0 1.5 2.0 3.0

0.0

0.1

0.2

0.3

0.4

Odds ratio

Sta

nd

ard

err

or

Funnel plot of meta–analysis of ACE I/D and CHD

Conclusions• The principles of systematic reviews are applicable to

any research design

• Reviews of observational studies should always be systematic

• Much attention should be given to exploring possible sources of heterogeneity

• HOWEVER: Meta-analysis of observational studies will often produce misleading and spuriously precise estimates

• Trial registers should solve much of the problems of publication bias in RCT, but trying to solve publication bias in observational studies impossible?


Future work

• We need to define optimal search strategies to identify epidemiological studies in the literature

• We need validated instruments to assess the study quality at the design, conduct and analysis level

• We need to improve the quality of reporting of epidemiological studies

• We need to facilitate individual patient data analyses

• We need to better define the place of meta-analysis in systematic reviews of epidemiological studies


handout h6 the scope of meta-analysis: meta-analysis of observational studies

Documents

os rctmetaanalysis

analysisa statistical

observational studies11

future studies

individual studies

independent studies

study results

os rctmay