CONFERENCIA INAUGURAL

“Credibilidad y geometría de la evidencia:

¿se basan las recomendaciones y decisiones

en estudios clínicos apropiados?”

John Ioannidis

Credibility and geometry of the evidence: Are

recommendations and decisions based on

appropriate clinical studies?

Madrid 10/2012

John P.A. Ioannidis, MD, DSc

C.F. Rehnborg Chair in Disease Prevention

Professor of Medicine and Professor of Health Research and Policy

Director, Stanford Prevention Research Center

Stanford University School of Medicine

Professor of Statistics (by courtesy)

Stanford University School of Humanities and Sciences

Recommendations and decisions

may depend on:

Experts (less reliable) Evidence (more reliable)

• However, experts may still

design, collect, analyze,

interpret, synthesize, apply,

or even enforce evidence

Uneven research in the health

sciences

• Spongiform encephalopathies: 2050

MEDLINE publications per 1000 patients

• Myasthenia gravis: 156 MEDLINE

publications per 1000 patients

• Cerebrovascular disease: 7.7 MEDLINE

publications per 1000 patients

• Severe varicose veins: 0.5 MEDLINE

publications per 1000 patients

Frankel and West 1993

Clinical

evidence

and burden

of disease:

do we

perform

research on

important

problems?

Swingler et al. BMJ 2003

Clinical research in/for Africa

Isaakidis et al. BMJ 2002

The new basic science for medicine:

evidence-based medicine

The advent of meta-analysis and

RCTs

The crisis of false positive research

Diet causes cancer

• Open a popular cookbook

• Randomly check 50 ingredients

• How many of those are associated with

significantly increased or significantly

decreased cancer risk in the scientific

literature?

Associated with cancer risk

• veal, salt, pepper spice, flour, egg, bread,

pork, butter, tomato, lemon, duck, onion,

celery, carrot, parsley, mace, sherry, olive,

mushroom, tripe, milk, cheese, coffee,

bacon, sugar, lobster, potato, beef, lamb,

mustard, nuts, wine, peas, corn, cinnamon,

cayenne, orange, tea, rum, raisin

Schoenfeld and Ioannidis, Am J Clin Nutrition, in press

Why research findings may not be

credible?

• There is bias

• There is random error (see multiple

comparisons)

• Usually there is plenty of both

Bias

• Any deviation from the truth beyond chance error

• Conscious, subconscious, or unconscious

• One may create theory (or theories) about bias or may study its consequences

• The former seem more robust, but it is the latter that we measure, witness, and eventually suffer

Chavalarias and Ioannidis, JCE 2010

Mapping 235 biases in 17 million Pub Med papers

A time array for biases

Discrepancies over time occur even in

randomized trials Myocardial infarction interventions

Cumulative sample size

40000

30000

20000

10000

5000

4000

3000

2000

1000

500

400

300

200

100

Rela

tive c

hange in

tre

atm

ent eff

ect

3

2

1

.9

.8

.7

.6

Ioannidis and Lau, PNAS 2001

Inflation in statistically significant

treatment effects of meta-analyses of

randomized trials?

Ioannidis, Epidemiology 1998

Pereira, Horwitz, Ioannidis, JAMA 2012

Post-study odds of a true finding are small

• When effect sizes are small

• When studies are small

• When fields are “hot” (many furtively

competitively teams work on them)

• When there is strong interest in the results

• When databases are large

• When analyses are more flexible

Ioannidis JP. PLoS Medicine 2005

The slow and uncertain pace of clinical translation

Contopoulos-Ioannidis et al. Science 2008

Disclosures • In my dreams I am the CEO of MMM (Make

More Money, Inc.)

• My company has successfully developed a new drug that is probably a big loser, but I want to make big money

• At best, the new drug may be modestly effective for one or two diseases/indications for one among many outcomes (most of them irrelevant to patients)

• If I test my drug in a study, even for this one or two indications, it may seem not to be worth it

• But still, I want to make big money

• What should I do?

The answer • Run many studies with many outcomes on each of many different

indications

• Ideally run trials against placebo (this is the gold standard for regulatory agencies) or straw man comparators, but registry studies or even electronic records would do, if need be

• Test 10 indications and 10 outcomes for each, just by chance you get 5 indications with statistically significant beneficial results

• A bit of selective outcome and analysis will help present “positive” results for 7-8, maybe even for all 10 indications

• There are systematic reviewers out there who will perform a systematic review based on the published data SEPARATELY for each indication proving the drug works for all 10 indications

• With $ 1 billion market share per approved indication, we can make $ 10 billion a year out of an (almost) totally useless drug

We probably all agree

• It is stupid to depend on the evidence of a

single study

• when there are many studies and a meta-

analysis thereof on the same treatment

comparison and same indication

Similarly

• It is stupid to depend on a single meta-analysis

• when there are many outcomes

• when there are many indications the same

treatment comparison has been applied to

• when there are many other treatments and

comparisons that have been considered for each of

these indications

Network definition

• Diverse pieces of data that pertain to research

questions that belong to a wider agenda

• Information on one research question may

indirectly affect also evidence on and inferences

from other research questions

• In the typical application, data come from trials on

different comparisons of different interventions,

where many interventions are available to

compare

Size of each node proportional to the

amount of information (sample size)

A c LD

M c SD

M s SD

N c

N s

N+bmab

N+lpnb

NT

O c

O s

T c

A c SD T s

T+tzmb

Ts+lpnb

A s LD

A s SD

A+tzmb SD

AN SD

ANT SD

AT SD

M c LD

Figure 2a

A network offers a wider picture than a single

traditional meta-analysis: e.g. making sense of 700

trials of advanced breast cancer treatment

Mauri et al, JNCI 2008

Size of each node reflecting the year of

first publication

A c LD

M c SD

M s SD

N c

N s

N+bmab

N+lpnb

NT

O c

O s

T c

A c SD

T s

T+tzmb

Ts+lpnb

A s LD

A s SD

A+tzmb SD

AN SD

ANT SD

AT SD

M c LD

Figure 2b

Focusing on what is most recent in the market

Main types of network geometry

Salanti, Higgins, Ades, Ioannidis, Stat Methods Med Res 2008

Polygons

Stars

Lines

Complex figures

Homophily

• OΜOΦΙΛΙΑ = Greek for “love of the same” =

birds of a feather flock together

• Testing for homophily examines whether

agents in the same class are disproportionately

more likely to be compared against each other

than with agents of other classes.

For example: Antifungal agents

agenda

• Old classes: polyenes, old azoles

• New classes: echinocandins, newer azoles

Rizos et al, J Clin Epidemiol, 2010

2

18 11

1

1 3

1

2

1

1

3

4

2

17

amphotericin B

ketoconazole

lipid amphotericin B

posaconazole

voriconazole

fluconazole

itraconazole

Figure 2

3

2

1

8

micafungin

other

anidulafungin

caspofungin

Figure 3

Figure 4

10

12

1

other

voriconazole or posaconazole

echinocandins

Auto-looping Design of clinical research: an open world or isolated city-states (company-states)?

Lathyris et al., Eur J Clin Invest, 2010

Don’t blame the Big Pharma

necessarily

• Treatments for basal cell cancer: surgical,

destructive, topical

• Specialties do not seem to communicate.

CA

PDT

Cryo

SG

IMI

L

5-FU IFN

MMS

SE

Rad

C&D

7 2

1

1

1

5 1

8

2

3

Placebo

3

1

4 6

1

1

1

1

LDE

1

1

API31510

1

Vismodegib

PEP005

1

1

Diclofen, Calcitriol, Both

Published + ClinicalTrials.Gov

Destructive Procedures

Surgical Procedures

Topical Creams or Injectables

4

25

15

1

9 1

Synthesis of the network evidence

(multiple-treatment meta-analysis)

• Incoherence

• Summary effects

• Ranking

• Bias modeling

Credible intervals and predictive

intervals in network meta-analysis

Salanti, Ades, Ioannidis, JCE, 2011

Cumulative ranking probability

Probability of not being worse than

threshold t from the best treatment

Modeling bias

Reversing the paradigm

Design networks prospectively

– Data are incorporated prospectively

– Geometry of the research agenda is pre-

designed

– Next study is designed based on enhancing,

improving geometry of the network, and

maximizing the informativity given the network

This may be happening

already?

Agenda-wide meta-analyses

BMJ 2010

Anti-TNF agents: $ 10 billion and 43 meta-analyses,

all showing significant efficacy for single indications

Indications

RA

Psoriasis Psoriatic

arthritis

Crohn’s

disease

Juvenile

idiopathic

arthritis

Ulcerative

colitis

Ankylosing

spondylitis

5 FDA-approved anti-TNF agents

Infliximab

Etanercept

Adalimumab

Golimumab

Certolizumab pegol

1998

1998

2003

1200 (and counting) clinical trials of

bevacizumab

Fifty years of research with 2,000 trials:

9 of the 14 largest RCTs on systemic steroids

claim statistically significant mortality benefits

Contopoulos-Ioannidis and Ioannidis EJCI 2011

Trial networks for neglected

tropical diseases (burden: 1 billion people)

ALB+IVM+PZQ

PZQ

Artesunate/ACT

Mefloquine

Artemether-lumefantrine

Mirazid

Oxamniquine

Micronutrients

Placebo/NT

Micronutrients+PZQ

PZQ+ALB

ALB

PZQ+calcitriol

Calcitriol

Oltipraz

Metrifonate

Niridazole

Hycanthone

Potassium antimony nitrate

Lucanthone

Oxamniquine+PZQ

Tartar emetic

Lucanthone+tartar emetic

Metrifonate+PZQ

Metrifonate+niridazole

PZQ+LEV

LEV

PIP

PyrPam

Placebo/NT

Bephenium

LEV

Phenylene-diisothiocyanate

MEB

ThiabendazoleOxantel pyrantel pamoate

Metronidazole

ALB

IVM

PZQ

MEB+education

Education

ALB+PZQ

Paico

Nitazoxanide

ALB+IVM

ALB+DEC

Micronutrients

Carica papaya

Tribendimidine

Thienpydin

Thienpydin+MEB

PyrPam+MEB

Fenbendazole

Bitoscanate

PIP+metronidazole

MEB+pyrantel oxantel pamoate

DEC

MEB+LEV

ALB+MEB

PIP+bephenium

ALB+education

Bephenium

TCEPyrantel emboate

Tetramisole

LEV

Phenylene di-isothiocyanate

PyrPam

MEB

Thiabendazole

Placebo/NT

FLUB

Oxantel pyrantel pamoate

ALB

Metrifonate

PZQ

IVM

IVM+ALB

PZQ+ALB

DEC

LEV+MEB

ALB+DEC

Carica papaya

Tribendimidine

PIP+bephenium

Bitoscanate

Fenbendazole

MEB+ALB

Neobedermin

Phenylene di-isothiocyanate+TCE

PIP

PyrPam

Placebo/NT

Bephenium

LEV

Phenylene-diisothiocyanate

MEB

ThiabendazoleOxantel pyrantel pamoate

Metronidazole

ALB

IVM

PZQ

MEB+education

Education

ALB+PZQ

Paico

Nitazoxanide

ALB+IVM

ALB+DEC

Micronutrients

Carica papaya

Tribendimidine

Thienpydin

Thienpydin+MEB

PyrPam+MEB

Fenbendazole

Bitoscanate

PIP+metronidazole

MEB+pyrantel oxantel pamoate

DEC

MEB+LEV

ALB+MEB

PIP+bephenium

ALB+education

Sitamniquine

Ampho B

Liposomal amphotericin+miltefosine

Liposomal amphotericin+paromomycin

Miltefosine+paromomycin

Paromomycin

PAs

PAs+paromomycin

Liposomal amphotericin

Miltefosine

ABLC

Pentamidine

Aminosidine

PAs+interferon gamma

Ketoconazole

PAs+aminosidine

PAs+pentamidine

PAs+allopurinol

PAs+ketoconazole

PAs+LEV

Pentamidine+allopurinol

Kappagoda and Ioannidis, BMJ, 2012

What the next study should do?

• Maximize diversity of treatment options

• Address comparisons that have not been addressed

• Break (unwarranted) homophily

• Be powered to find an effect or narrow the credible or predictive interval for comparisons of interest

• Maximize informativity across the network of information

• Some/all of the above

• Answer questions that are important to patients, not sponsors or academics necessarily

Meta-analysis=primary type of

prospective research

We need to think about how to design

prospectively large agendas of randomized

trials and their respective networks for

questions that are important to patients and

can make a difference in their lives

This in information should be considered a

public commodity, available transparently

to all in full details (raw data, protocols,

analysis codes)

• Tony Ades, University of Bristol

• Despina Contopoulos-Ioannidis, Stanford

University

• Shanthi Kappagoda, Stanford University

• Fotini Karassa, University of Ioannina

• Fainia Kavvoura, Oxford University

• David Kim, Stanford University

• Dimitris Lathyris, University of Ioannina

• Davide Mauri, University of Ioannina

• Georgia Salanti, University of Ioannina

• Jean Tang, Stanford University

• Shanthi Kappagoda, Stanford University

• Vish Nair, Stanford University

• Nazmus Saquib, Stanford University

• Juliann Saquib, Stanford University

• Despina Contopoulos-Ioannidis, Stanford

University

• Jonathan Schoenfeld, Harvard University

• Thomas Pfeiffer, Harvard University

Special thanks

• Lars Bertram, Harvard University and

Max Planck Institute

• David Chavalarias, Ecole Polytechnique,

Paris

• Fainia Kavvoura, Oxford University

• Kostas Siontis, University of Ioannina

• George Siontis, University of Ioannina

• Vangelis Evangelou, University of

Ioannina

• Muin Khoury, CDC and NCI

• Panagiotis Kyzas, University of Ioannina

• Orestis Panagiotou, University of Ioannina

• Jonathan Sterne, University of Bristol

• Alex Sutton, University of Leicester

• Daniele Fanelli, University of Edinburgh

• Julian Higgins, MRC Biostatistics Unit,

Cambridge University

• Joseph Lau, ICRHPS, Tufts University

• Tiago Pereira, U Sao Paolo