how do we improve research methods and reporting? shai d. silberberg ninds / nih disclaimer opinions...
TRANSCRIPT
How Do We Improve Research Methods and Reporting?
Shai D. Silberberg NINDS / NIH
DisclaimerOpinions I will voice are not official opinions of NIH!
What are the problems? What happens if we don’t improve research? How can we fix the problems?
Prinz, Schlange and Asadullah
Nature Reviews Drug Discovery 2011; 10: 712-713
43 / 67
Begley and Ellis. Nature 2012, 483:531-533
“scientific findings were confirmed in only 6 (11%) cases.”
Confounding variables
Clever Hans
Problems withresources
Science at the cutting edge
What Are The Causes For Poor Reproducibility?
Lack of transparency in reporting
Deficient experimental procedures
Publication bias
Human nature
Poor reproducibility× × =×
“Human action can be modified to some extent, but human nature cannot be changed.”
Abraham Lincoln
Cooper Union AddressNew York, New York
February 27, 1860
Human Nature
“The moment one has offered an original explanation for a phenomenon which seems satisfactory, that moment affection for his intellectual child springs into existence...
...So soon as this parental affection takes possession of the mind, there is a rapid passage to the adoption of the theory. There is an unconscious selection and magnifying of the phenomena that fall into harmony with the theory and support it, and an unconscious neglect of those that fail of coincidence….
….There springs up, also, an unconscious pressing of the theory to make it fit the facts, and a pressing of the facts to make them fit the theory.”
Journal of Geology, 1897
Thomas Chrowder Chamberlin
The Method of Multiple Working Hypotheses
Chalmers et al., N Engl J Med 1983; 309: 1358-1361
Schulz et al., PLOS Medicine 2010; 7: 1–7
“Randomized trials can yield biased results if they lack methodological rigour.
To assess a trial accurately, readers of a published report need complete, clear, and transparent information on its methodology and findings.”
The CONSORT statement provides guidelines for reporting clinical trials
Among the 35 items included in the CONSORT guidelines are:
How sample size was determinedMethod used to generate the random allocation sequenceMechanism used to implement the random allocation sequenceWho was blinded after assignment to interventions and howLosses and exclusions after randomisation, together with reasons
Ignoring any one of these items can lead to bias
Chavalarias and Ioannidis, J Clin Epid 2010; 63: 1205-1215
“The reliability of a study is determined by the investigator’s choices about critical details of research design and conduct”
(David F. Ransohoff, 2010. J Clin Oncol 28: 698-704)
“Bias is unintentional and unconscious. It is defined broadly as the systematic erroneous association of some characteristic with a group in a way that distorts a comparison with another group…..”
“…..The process of addressing bias involves making everything equal during the design, conduct and interpretation of a study, and reporting those steps in an explicit and transparent way.”
The definition of bias
What about pre-clinical studies?
Malcom Macleod Emily Sena David Howells
Sena et al., JCBFM. 2014; 34: 737-742
Insufficient reporting of methodological approaches is evident for pre-clinical studies
Effect size for studies of FK506 (Tacrolimus) in experimental stroke.
Sena et al., Trends Neurosci 2007; 30: 433-439
The fewer methodological parameters are reported, the greater the apparent efficacy!
Hackam and Redelmeier, JAMA 2006; 14: 1731-1732
Journals:• Cell• Nature• Science• Nature Medicine• Nature Genetics• Nature Immunology• Nature Biotechnology
>500 citations
Inadequate reporting is widespread
“Peer review is the evaluation of work by one or more people of similar competence to the producers of the work.”
Peer Review
Wikipedia
Year
Publications(x106)
The Escalation in Scientific Reporting(Annual PubMed Publications in English)
Publish or perish! Grant support
Impact factor Innovation
Significance Novelty
Hackam and Redelmeier, JAMA 2006; 14: 1731-1732
Journals:• Cell• Nature• Science• Nature Medicine• Nature Genetics• Nature Immunology• Nature Biotechnology
>500 citations
Inadequate reporting is widespread
Amyotrophic lateral sclerosis (ALS)
Death within 5 years of diagnosis Central pathological finding is motor neuron death
3% of cases from gain of function mutations in SOD1 Rodents over-expressing SOD1 recapitulate ALS
2002: Minocycline reported by a number of groups to extend survival of SOD1 mice
2003: Randomized placebo controlled trial (412 patients treated for 9 months)
2007: Results of the trial are published - minocycline found to have a harmful effect on patients with ALS
“In the past five years we have screened more than 70 drugs in 18000 mice across 221 studies, using rigorous and appropriate statistical methodologies. While we were able to measure a significant difference in survival between males and females with great sensitivity, we observed no statistically significant positive (or negative) effects for any of the 70 compounds tested, including several previously reported as efficacious. “
Scott et al., Amyotroph Lateral Scler 2008; 9: 4-15
ALS Therapy Development Institute (ALS TDI)
“…the majority of published effects are most likely measurements of noise in the distribution of survival means as opposed to actual drug effect.“
Scott et al., Amyotroph Lateral Scler 2008; 9: 4-15
2241 SOD1G93A control mice
The probability of seeing an effect by chance alone is significant even with 10 animals per group
Publication decisions and their possible effects on inferences drawn from tests of significance — or vice versa
THEODORED . STERLINGUniversity of Cincinnati
“There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs - an "error of the first kind“ - and is published.”
Journal of the American Statistical Association, 1959; 54:30-34
Psychological Bulletin, 1979; 86: 638-641
“For any given research area, one cannot tell how many studies have been conducted but never reported.
The extreme view of the "file drawer problem" is that journals are filled with the 5% of the studies that show Type I errors, while the file drawers are filled with the 95% of the studies that show non-significant results.”
The “File Drawer Problem” and Tolerance for Null Results
ROBERT ROSENTHALHarvard University
“We evaluated 340 articles included in prognostic marker meta-analyses (Database 1) and 1575 articles on cancer prognostic markers published in 2005 (Database 2).
……..Only five articles in Database 1 (1.5%) and 21 in Database 2 (1.3%) were fully ‘negative’ for all presented results in the abstract and without efforts to expand on non-significant trends or to defend the importance of the marker with other arguments.”
European Journal of Cancer, 2007; 43: 2559 - 2579
SOD1G93A transgenic mice Started at 10 weeks of age i.p. 25 and 50 mg/kg/day 7 animals / group (females) Not randomized “The experimenter was blinded to
the treatment protocol.”
SOD1G93A transgenic mice Started at 5 weeks of age i.p. 10mg/kg/day 10 animals / group (sex?) Not randomized Not blinded
The survival benefit of minocycline in the SOD1G93A mouse model of ALS might be due to small sample size and/or Bias
What happens if we don’t improve research?
We will: Stifle advances in science and medicine Abuse valuable resources Fail current and future generations Lose credibility / public support
“Most people say that it is the intellect which makes a great scientist. They are wrong: it is character.”
Albert Einstein
Lack of transparency in reporting
Deficient experimental procedures
Publication bias
Human nature
Poor reproducibility× × =×
Education
Attentiveness to bias;Good experimental
design
How do we improve research methods and reporting?
RFA-GM-15-006
Training Modules to Enhance Data Reproducibility (R25)
“This FOA will support creative educational activities with a primary focus on developing courses for skills development, specifically, training modules for graduate students, postdoctoral fellows, and beginning investigators designed to enhance data reproducibility.”
Google: NIGMS clearinghouse
Lack of transparency in reporting
Deficient experimental procedures
Publication bias
Human nature
Poor reproducibility× × =×
Review
transparency in
reporting
Education
Attentiveness to bias;Good experimental
design
How do we improve research methods and reporting?
Evaluation of the strength of the data used in support of grant applications
June 20 – 21, 2012Washington Plaza Hotel
Washington DC
R
Editors
ReviewersInvestigators
Funders
“To ease the interpretation and improve the reliability of published results we will more systematically ensure that key methodological details are reported, and we will give more space to methods sections. We will examine statistics more closely and encourage authors to be transparent, for example by including their raw data.”
1. Rigorous statistical analysis
2. Transparency in reporting Journals should have no limit or generous limits on the length of methods sections Journals should use a checklist during editorial processing to ensure the reporting
of key methodological and analytical information to reviewers and readers.
3. Data and material sharing
4. Consideration of refutations
5. Consider establishing best practice guidelines
http://www.nih.gov/about/reporting-preclinical-research.htm
Lack of transparency in reporting
Deficient experimental procedures
Publication bias
Human nature
Poor reproducibility× × =×
Review
transparency in
reporting
Culture
Focus on rigor not
glitter
Education
Attentiveness to bias;Good experimental
design
How do we improve research methods and reporting?
New Biographical Sketch Format Required for NIH and AHRQ Grant Applications Submitted for Due Dates on or After May 25, 2015
The new format extends the page limit for the biosketch
from four to five pages.
Allows researchers to describe up to five of their most
significant contributions to science, along with the
historical background that framed their research.
Investigators can outline the central findings of prior work
and the influence of those findings on the investigator’s
field.
NOT-OD-15-032
“Experiments framed by hypotheses establish the idea of “positive” data versus “negative” data. Positive data are consistent with the hypothesis and negative data falsify the hypothesis. This creates a potential bias to amass positive data, owing to the desire to avoid falsification.”
Lack of transparency in reporting
Deficient experimental procedures
Publication bias
Human nature
Poor reproducibility× × =×
Review
transparency in
reporting
Culture
Focus on rigor not
glitter
Betterreproducibility× × =
Education
Attentiveness to bias;Good experimental
design
How do we improve research methods and reporting?
“It is not the slowness with which conclusions are arrived at that should give satisfaction to the moral sense, but the thoroughness, the completeness, the all-sidedness, the impartiality, of the investigation.”
Thomas Chrowder Chamberlin 1897