ciaran o'neill & amye kenall: peering into review - innovation, credit &...
DESCRIPTION
Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility. Talk 1 in the "What Bioinformaticians need to know about digital publishing beyond the PDF2" workshop at ISMB 2014, Boston, 16th July 2014TRANSCRIPT
![Page 1: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/1.jpg)
1
Peering into review
Innovation, credit & reproducibility
Ciaran Oneill&
Amye Kenall
![Page 2: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/2.jpg)
2
www.biomedcentral.com/biome
![Page 3: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/3.jpg)
3
Pitfalls of traditional peer review
Inconsistent
Bias
Favouritism
Abuse
Burden on researchers
Slow
![Page 4: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/4.jpg)
4
Open peer review
(Medical journals)
![Page 5: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/5.jpg)
5
“our goal is unapologetically ambitious:
to establish a new system of peer review to bolster productive scientific debate
and to provide scientists with useful guides to the literature”
Launch Editorial: Eugene Koonin, David Lipman, Laura Landweber
![Page 6: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/6.jpg)
6
![Page 7: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/7.jpg)
7
~ 50% reviewers disclose their name
~ 80% authors make the reports public
![Page 8: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/8.jpg)
8
![Page 9: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/9.jpg)
9
Decoupling peer review from the journal
![Page 10: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/10.jpg)
10
Post-publication peer review
![Page 11: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/11.jpg)
11
Community review
• Post-publication commenting• Open to authors already in PubMed
![Page 12: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/12.jpg)
12
![Page 13: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/13.jpg)
13
![Page 14: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/14.jpg)
14
I thought these were peer reviewed?Problems in reproducibility
![Page 15: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/15.jpg)
15
1. Ioannidis et al., (2009). Repeatability of published microarray gene expression analyses. Nature Genetics 41: 142. Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8)
Out of 18 microarray papers, resultsfrom 10 could not be reproducedSloppy Science
![Page 16: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/16.jpg)
16
#overlyhonestmethods
![Page 17: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/17.jpg)
17
How to combat this? ( . . . from the journal side)
![Page 18: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/18.jpg)
18
Dynamic Document Technology
![Page 19: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/19.jpg)
19
![Page 20: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/20.jpg)
20
Journal + database + Computational Tools
![Page 21: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/21.jpg)
21
Reproducibility Starts with Peer
Review
![Page 22: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/22.jpg)
22
• Repository of standardised and annotated multielectrode array data from mice and ferrets
• 366 recordings from 12 studies
• Authors submitted in knitr• Aided review process,
allowing reviewers to rerun analyses
• Authors reported it saved time—having a “natural record” of what you did
• Automatic updating of text you might overlook (figure legends, eg)
![Page 23: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/23.jpg)
23
Some testimonials for KnitrAuthors (Wolfgang Huber)“I do all my projects in Knitr. Having the textual explanation, the associated code and the results all in one place really increases productivity, and helps explaining my analyses to colleagues, or even just to my future self.”
Reviewers (Christophe Pouzat) “It took me a couple of hours to get the data, the few custom developed routines, the “vignette” and to REPRODUCE EXACTLY the analysis presented in the manuscript. With few more hours, I was able to modify the authors’ code to change their Fig. 4. In addition to making the presented research trustworthy, the reproducible research paradigm definitely makes the reviewer’s job much more fun!
![Page 24: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/24.jpg)
24
How to Scale?
![Page 25: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/25.jpg)
25
Back to #overlyhonestmethods
![Page 26: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/26.jpg)
26
Let’s delegate!
![Page 27: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/27.jpg)
27
![Page 28: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/28.jpg)
28
Why stop at publication?
More commenting?
Bring debate back to the journal?
DOIs for comments?
![Page 29: Ciaran O'Neill & Amye Kenall: Peering into review - Innovation, credit & reproducibility](https://reader034.vdocuments.net/reader034/viewer/2022042814/54c8ade04a79596d0a8b45d5/html5/thumbnails/29.jpg)
29
Questions?Amye Kenall
Journal Development Manager (Open Data), BioMed Central@AmyeKenall
Ciaran O’NeillAssociate Publisher, BioMed Central
@cjmoneill ciaran.o’[email protected]