methods in research on research the peer review...
TRANSCRIPT
The Peer Review Process.
Why Evidence Based Practices Are Needed?
Isabelle Boutron
METHODS team
Research Centre of Epidemiology Biostatistics Sorbonne Paris Cité
Paris Descartes University
Methods in Research on Research
Acknowledgements
MiRoR projectJoint doctoral training program, to train 15 PhD students in Methods in Research on Research in the field of clinical research (http://miror-ejd.eu) funded by Marie Skłodowska-Curie Actions.
Projects-Planning-Conduct -Reporting -Peer review
7 European Universities and 10
International Partners
The peer review system
• Central to the scientific community
– Gatekeeper of the scientific publications
– To improve the quality of manuscripts
• The number of journals and manuscripts is increasing
– 8 million researchers publishing 2.5 million articles in 28,000 peer-
reviewed English-language journals/year
– The annual revenue is $10 billion
• A system relying mainly on work performed voluntarily by
academic researchers
3International Association of Scientific, Technical and Medical Publishers STM Report March 2015
The peer review system
2.7 billion US
dollars / year*
*Look H, Summary report: Rightscom Ltd for JISC Collections; 2010
The performance of the system is
questioned.
5
« material often obtained in the most rigourous kind
of biomedical experimentation is rated as publishable
or not by a system that has rarely been subjected to
any analysis, let alone one that is rigourous »
Franz Ingelfinger, editor of the NEJM (1967 to 1976)
Level of evidence of peer review
6
Published in Issue 2, 2008
At present, little empirical evidence is available to
support the use of editorial peer review as a
mechanism to ensure quality of biomedical
research.
The performance of the system is
questioned.
7
Submission of 304 versions of a fake paper with fatal flaw to open-access journals.
• More than half of the journals accepted the paper, failing to notice its fatal
flaws.
• For DOAJ publishers, that completed the review process, 45% accepted the
paper.
Impact of peer review on reporting of
RCTs
• Comparison of the submitted and accepted version of manuscripts
of RCTs published in BMC journals
• Changes were limited
– A median of 11% (range 1-60%) words deleted and 20% added
(range 2-88%).
• Peer reviewers often fail to detect important deficiencies in the
reporting
• Peer reviewers requested relatively few changes for reporting
8Hopewell S, BMJ, 2015
Detection of selective reporting of
outcomes
9
34% examined information registered on a trial
registry.
Can peer reviewers reduce spin ?
• Assessment of the manuscript submitted, peer reviewers comments, and final manuscript of non-randomized studies assessing a therapeutic intervention published in BMC Series journals (2011-2013).
• 55% of submitted manuscripts, peer reviewers identified at least one example of spin
• Of the spin identified by peer reviewers – 67% were completely deleted,
– 16% partially deleted
– 17% not removed in the final published article.
• For 15%, peer reviewers requested adding some spin
10Lazarus, J Clin Epidemiol. 2016
Level of spin before and after peer review in
the abstract conclusion
11
Before peer-review After peer-review
High
Moderate
Low
No
N=66 N=61
N=24 N=24
N=10 N=17
N=28 N=26
76% Peer reviewers failed to identify spin in abstract conclusions
Lazarus, J Clin Epidemiol. 2016
Two versions of a well-designed randomized controlled trial that differed only in the direction of the finding of the principal study end point
238 reviewers at 2 journals (assigned at random)
• were more likely to recommend the positive version of the test manuscript for publication than the no-difference version (97% vs 80%, P<0.001)
• detected more errors in the no-difference version than in the positive version (mean 0.85 vs 0.41, P<0.001)
• awarded higher methods scores to the positive manuscript than to the (identical) no-difference manuscript (8.24 vs 7.53, P=0.005)
12
Influence of authors’ prestige
• 119 reviewers were randomized to assess a fabricated
manuscript with the prestigious authors’ names and
institutions masked or visible
– Reviewers were more likely to recommend acceptance when
the prestigious authors’ names and institutions were visible than
when they were redacted
• 87% vs 68%;
• RR, 1.28 [95% CI, 1.06-1.39], P = .02
– They gave higher ratings for the methods.
13
The process is not clearly defined and
standardized
• Who are the reviewers?
– Who can be a reviewer?
– How are they identified?
– What are the core competencies?
– Should reviewers be trained and how?
• What documents should be peer-reviewed?
– Manuscript
– Appendices
– CONSORT checklist
– Registry
– Protocol, SAP
– …
– CSR?
– Raw data?14
• The tasks expected from a peer-reviewer are not realistic
– More than 200 different tasks identified
– The tasks involved different level of expertise and different
backgrounds
• Statistical and methodologic expertise
• Content expertise (novelty, relevance, external validity etc)
• None
– Verification (adherence to guidelines, consistency with registries)
– Formatting
15
Multiple tasks asked to reviewers
(important tasks for reviewers are not congruent
with important tasks for editors) Chauvin, BMC Medicine, 2016
• The most important task
for peer reviewers
(evaluating the risk of
bias) was clearly
requested by 5 % of
editors.
• The task most frequently
requested by editors
(provide recommendations
for publication), was rated
in the first tertile only by
21 % of all participants.
17
Several interventions are implemented
or proposed to improve the system
18
19
What is the evidence?A Systematic Review and Meta-analysis of Interventions to Improve the
Quality of Peer Review
• Only 22 reports of RCTs
• Only 7 were published over the past
10 years
• Interventions assessed
– Blinding
– Open peer review
– Training
– Use of checklist
– Adding experts
• Most are performed in one specific
journal
• Low methodologic quality
Bruce, BMC Med, 2016
RCTs and Peer review
• There are no barrier to the conduct of RCTs
• Large sample size is available (manuscripts, reviewers)
• Consent with an opt out approach facilitate recruitment
• Randomisation, allocation concealment are easy to perform
• Some outcomes are routinely collected
• Blinding outcome assessment is easily feasible
21
RCTs and Peer review : Methodological
research is needed
• What is the best study design for assessing interventions to improve the
peer review process?
– Design: RCTs, cluster RCTs, pairwise comparisons, Stepped wedge cluster RCT
of journals, time series analysis etc
– Unit of randomisation
• Manuscripts
• Reviewers
– Type of manuscript (real, fabricated)
• What will be the outcome?
– Quality of the peer review report (editor’s subjective assessment, validated scales
to assess the quality of the peer review report
– Quality of the final manuscript (how is high quality defined, who is to decide?)
– Need of CORE outcome set
• Ethical issues
– Impact on the final decision
Modeling (Agent-based model)
To approach the complexity of the scientific publication system
and compare different systems
Kovanis Plos One 2016
Kovanis Scientometrics. 2016
Explore
sustainability
of the system
Compare
different
systems
(cascade etc)
24
• A call to substantially increase the amount of research conducted by journals
in journalology and meta-research.
– Participate in Randomized Controlled Trials (eg about peer review
improvement)
– Share their empirical data to help modeling
– Share manuscripts submitted, reviews, revised manuscripts to develop
qualitative and quantitative research on the impact of peer review
Conclusions
25
• “So peer review is a flawed process, full of easily
identified defects with little evidence that it works.
Nevertheless, it is likely to remain central to science
and journals because there is no obvious alternative.”
• “The most important question with peer review is not
whether to abandon it, but how to improve it.”
(R. Smith)
“We need rigorous studies to tell us the pros and cons of
these approaches.” (D Rennie)