research integrity: animal research - cebm.net · translational stroke research 7(5): 395-406....
TRANSCRIPT
CAMARADES: Bringing evidence to translational medicine
Research Integrity: animal research
Emily S Sena, PhDCentre for Clinical Brain Sciences, University of Edinburgh
@drEmilySena@camarades_
CAMARADES: Bringing evidence to translational medicine
Disclosures
• BMJ Open Science (Editor-in-Chief)
– I receive an honorarium for this role
• I have applied and received (& will continue to) grant funding for this research
CAMARADES: Bringing evidence to translational medicine
Hypotheses
• In the life sciences there are perverse incentives (publication, funding, promotion) to produce positive results with little attention paid to their validity
• In the use of animal disease models, pressure to reduce the number of animals (cost, time, ethics, feasibility) results in studies either being underpowered or of unknown power
• These factors combine to compromise the utility of animal models and contribute to translational failure
CAMARADES: Bringing evidence to translational medicine
What we have seen
• Many examples of translational failure
• Replication difficulties
• Bayer: 53 of 67 findings did not replicate
• Amgen: 47 of 53 findings did not replicate
• Cancer Biology:• 3 of 5 findings did not replicate
What are the causes?
• ? Fraud
• ? False positive studies +/- dubious research practices
• ? Meta- (sectoral) problems like perverse incentives and publication bias
• ? True biological heterogeneity of observed effects
CAMARADES: Bringing evidence to translational medicine
Reproducibility crisis
CAMARADES: Bringing evidence to translational medicine
Bias is prevalent and important
RandomisationBlinded Outcome
Assessment
Stroke 36% 29%
MND 31% 20%
AD 15% 25%
PD 12% 15%
EAE 8% 15%
Glioma 14% 0%
Sena et al TiNS 2007
Efficacy
Randomisation
CAMARADES: Bringing evidence to translational medicine
The ‘best’ institutions – RAE 1,173
CAMARADES: Bringing evidence to translational medicine
• Overall efficacy was reduced from; – 32% (95% CI 30 to 34%) to 26% (95% CI 24 to 28%)
• 16% of experiments remain unpublished
CAMARADES: Bringing evidence to translational medicine
Things are improving
McCann SK, Cramond F, Macleod MR, Sena ES (2016).
Translational stroke research 7(5): 395-406.
“The 2009 systematic review highlighted areas of weakness with respect
the lack of reporting on certain aspects of experimental design. While we
did not necessarily agree with all recommendations and also felt that not-reported
did not mean not done we did take on board that future studies did need to
more fully report details of experimental design. This change is reflected in
the positive outcome of the follow-up 2016 systematic review”
--- Professor Stuart Allan, University of Manchester
CAMARADES: Bringing evidence to translational medicine
Research improvement activity
• Things done by stakeholders to increase the usefulness of research with which they are associated– The effectiveness must be assessed (RCTs, Observational)
CAMARADES: Bringing evidence to translational medicine
What institutions can do….
• Be forward thinking/flexible
– Develop an active strategy to improve quality and value
– Unique dependent on local ambition and circumstance
• Learn from the healthcare improvement community
• Strategy may include
– Benchmarking/audit
– Prioritisation
– Tool development to measure performance
• Collaboration between meta-researchers & professional services
CAMARADES: Bringing evidence to translational medicine
Potential strategy
• Non-punitive approach
– Work on everyone to make them a bit better
• ? Fraud
• ? False positive studies +/- dubious research practices
• ? Meta- (sectoral) problems like perverse incentives and publication bias
• ? True biological heterogeneity of observed effects
CAMARADES: Bringing evidence to translational medicine
Research Improvement Activity
What performance do we
aspire to?
“95% of UoE manuscripts describing animal
research report randomisation where this would
be appropriate”
What is our current
performance?
Measure, [2009-10 = 8%]
What are we going to do
about it?
• Education sessions for PhD/post Doc/ PIs
• Highlighted component of AWERB review
• Identified factor in resource allocation (open
access publication funds, prioritisation of
research resources)
Did that make a difference? Measure
Is performance now good
enough?
Stick or twist
CAMARADES: Bringing evidence to translational medicine
Define target performance
Measure performance
Seek to improve
performance
Measure performance
Did we succeed?
Consolidate into standard
practice
CAMARADES: Bringing evidence to translational medicine
Take home…..
• Academic in vivo studies which do not report simple measures to avoid bias give larger estimates of treatment effects
• Most do not report simple measures to reduce bias
– Although improvements are happening
• Reporting/publication biases are important and prevalent
• You cannot pass responsibility to journals/funders alone
• Help is at hand but improvement strategies must be tested
• Meta-research approaches (& automation tools) can facilitate this work at scale
• Institutional/sectoral buy-in is required to effectively improve the integrity of our research
– I would advocate for a non-punitive approach
CAMARADES: Bringing evidence to translational medicine
Thanks to...........