modelling stroke in the laboratory - separating fact from artefact the impact of sources of bias in...
TRANSCRIPT
Modelling Stroke in the Laboratory - Separating Fact
from ArtefactThe impact of sources of bias in animal
models of neurological disease, and what we should do about it
Malcolm MacleodSenior Lecturer, Centre for Clinical Brain Sciences
University of Edinburgh
1026
1026 interventions in experimental stroke
O’C
olli
ns
et
al A
nn N
euro
l 2006
1026603
1026 interventions in experimental stroke
Tested in focal ischaemia
O’C
olli
ns
et
al A
nn N
euro
l 2006
1026883374
1026 interventions in experimental stroke
Effective in focal ischaemia
O’C
olli
ns
et
al A
nn N
euro
l 2006
1026883550
97 18
1026 interventions in experimental stroke
Tested in clinical trial
O’C
olli
ns
et
al A
nn N
euro
l 2006
1026883550
97 171 3
1026 interventions in experimental stroke
Effective in clinical trial
O’C
olli
ns
et
al A
nn N
euro
l 2006
Where are we going wrong?
• Are animal experiments falsely positive?
• Have clinical trials tested the
conditions of maximum efficacy?
… and what, if anything, does this
mean for models of other diseases?
Back
gro
und
Control half dose full dose
Infa
rct V
olum
e
0
50
100
150
200
250
300
10-120 M 10-60 M
Animal data in stroke• There are huge
amounts of often confusing data
• Systematic review can help to make sense of it
• If you select extreme bits of the evidence you can “prove” either harm or substantial benefit
• However, if you have a precise and highly significant overall effect, then it is probably real
Hypothermia101 publications277 experiments
3353 animals
Bett
er
Wors
e
van d
er
Worp
et
al B
rain
2007
-100
0
100
200
Potential sources of bias in animal studies
• Internal validity
– Low sample size
• External validity– Publication bias – Are the models we use good models?
• Co-morbidities
Problem Solution
Selection Bias Randomisation
Performance Bias Allocation Concealment
Detection Bias Blinded outcome assessment
Attrition bias Reporting drop-outs/ ITT analysis
Cro
ssle
y e
t al, S
troke
200
8
Internal Validity NXY-059
Macl
eod
et
al, S
troke
2008
9 publications29 experiments
408 animalsImproved outcome by 44% (35-53%)
Internal ValidityHypothermia
van d
er
Worp
et
al B
rain
2007
Randomisation
Yes No
Blinded outcome
assessment
Yes No
101 publications277 experiments
3353 animalsImproved outcome by 44% (35-53%)
Internal Validity Stem Cell based therapies
• Infarct Volume
• Neurobehavioural score:
Jen L
ees,
un
publis
hed
54 publications127 experiments
2012 animalsImproved outcome by 29% (25-33%)
72 publications111 experiments
1876 animalsImproved outcome by 34% (30-39%)
RandomisationStem Cell based therapies
Jen L
ees,
un
publis
hed
Blinded outcome assessmentStem Cell based therapies
Jen L
ees,
un
publis
hed
What does this mean?Modelling the efficacy of tPA
“Standard”HealthyMale Rat
No randomisationHalothane anaesthesiaQuantify infarct volume
with TTC
25%
“Co-morbid”Hypertensive
Male Rat
No randomisationHalothane anaesthesiaQuantify infarct volume
with TTC
12%
“Randomised”Hypertensive
Male Rat
RandomisedHalothane anaesthesiaQuantify infarct volume
with TTC
0%
Em
ily S
en
a,
In P
repara
tion
Reported Efficacy 36%
Corrected Efficacy <0%
Control half dose full dose
Infa
rct V
olum
e
0
50
100
150
200
250
300
Comparing interventions Modelling efficacy under standard
conditions
nosd
onor
sno
otro
pic
estrog
ens
tPA
FK50
6tir
ilaza
dnico
tinam
ide
minoc
yclin
eno
sinhibito
rm
elat
onin
nxy0
59
othe
r th
rom
bolyt
icsIL
1-RA
hypo
ther
mia
Stan
dard
ised
effi
cacy
-10
0
10
20
30
40
50
60
70
Type II error, β, (1-power)
• The risk of falsely concluding that a biological effect is not present because the study was not large enough reliably to detect such differences.
• The smaller the experiment, the greater the risk of a Type II error
• Small studies with increased risk of Type II errors – waste animals, time and money– may lead to avenues of research being closed
down inappropriately.
Chances that data from any given animal will be non-
contributory
Number of animals Power % animals wasted
4 18.6% 81.4%
8 32.3% 67.7%
16 56.4% 43.6%
32 85.1% 14.9%
assume simple two group experiment seeking 30% reduction in infarct volume, observed SD
40% of control infarct volume
Chances of wasting an animal
Number of animals per group
0 10 20 30 40
% a
nim
als
was
ted
0
20
40
60
80
100
How does stroke compare?
Randomisation
Blinded Outcome
Assessment
Sample Size
calculation
Stroke 36% 29% 3%
MND 31% 20% <1%
PD 12% 15% 0%
EAE 8% 15% <1%
Glioma
14% 0% 0%
Sen
a et
al 2
007
TiN
S;
Am
aras
ingh
et
al,
J N
’Onc
in p
ress
Efficacy of Dopamine agonists in suppressing induced rotational activity following unilateral
6-OH-DA lesioning
Fer
guso
n et
al,
in p
repa
ratio
n
Sup
pres
sion
of
rota
tiona
l act
ivity
Quality items scored
BetterWorse
Pre
cisi
on
29 publications 109 experiments
1596 animals Improved outcome by 31% (27-35%)
External Validity Publication Bias for FK506
Macl
eod
et
al, JC
BFM
200
5
External ValidityPublication bias
Sena e
t al, a
ccepte
d f
or
ESC
200
8
All
stud
ies
Nic
otin
amid
eT
hrom
boly
ticN
XY
-059
NO
S d
onor
Nos
Inhi
bito
rF
K50
6H
ypot
herm
iaM
elat
onin
Est
roge
nsT
irila
zad
IL1-
RA
Effi
cacy
0
20
30
40
50
60ObservedAdjusted
991 publications
External Validity Hypertension in studies of NXY-059
Macl
eod
et
al, S
troke
in
pre
ss
7% of animals studied had hypertension
77% of patients in SAINT II had a history of hypertension at study entry
External Validity Hypertension in studies of tPA in
experimental stroke
Pere
l et
al B
MJ 2007
Comorbidity
“Normal” HBP
Effi
cacy
-2%25%
113 publications 212 experiments
3301 animals Improved outcome by 24% (20-28)
Summary
• Certain aspects of the design of animal experiments probably do lead to the over-statement of neuroprotective efficacy
• A substantial publication bias is present
• Neuroprotective efficacy may be substantially lower in animals with relevant co-morbidities
26%
Publication bias
Randomisation
Co-morbidity
bias
32%
Reported efficacy
How much efficacy is left?
20% 5%
Quality of Translation tPA and tirilazad
• Both appear to work in animals
• tPA works in humans but tirilazad doesn’t
• Time to treatment: tPA:– Animals – median 90 minutes– Clinical trial – median 90 minutes
• Time to treatment: tirilazad– Animals – median 10 minutes– Clinical trial - >3 hrs for >75% of patients
Sena e
t al, S
troke
2007;
Pere
l et
al B
MJ 200
7
tPA: Effect of time to treatment on efficacy
Pere
l et
al B
MJ 2007
; La
nce
t 2
004
AnimalStudies
Systematic Review
AndMeta-analysis
• how powerful is the treatment?
• what is the quality of evidence?
• what is the range of evidence?
• is there evidence of a publication bias?
• What are the conditions of maximum efficacy?
Clinical Trial
Summarising data from animal experiments
ST
AIR
VI:
pos
sibl
e de
velo
pmen
ts
A toolkit for effective translation
• Clear, rigorous SOPs for all aspects of experimental design
• On-line tools for– Sample size calculation– Random allocation to group
• Curated data repository– ? List of interventions tested (~ stroketrials.org)– ? Details of individual experiments (~ GENBANK)
• Development of experimental methods and funding streams to support multi-centre animal studies
• Adoption of CONSORT statement for animal stroke studies
ST
AIR
VI:
pos
sibl
e de
velo
pmen
ts
Mac
leod
et
al,
Str
oke/
JCB
FM
/IJS
in
pre
ss• Sample size calculation• Animals used• Inclusion and exclusion criteria• Randomisation• Allocation concealment• Reporting of animals excluded from
analysis• Blinded assessment of outcome• Reporting potential conflicts of interest and
study funding
Acknowledgements
Emily SenaPeter Sandercock
H Bart van der Worp
David HowellsTori O’CollinsGeoff Donnan
Nicolas CrossleyUlrich Dirnagl
Laura GrayPhilip Bath
Pablo PerelIan Roberts