tools for reproducible research in an increasingly digital world

brian m. bot | sage bionetworks | @BrianMBot

mayo clinic - 2015 jan 28

tools for reproducible research

in an increasingly digital world

sage bionetworks~40 FTEs

1/2 research - 1/3 platform - 1/6 leadership/support

sage bionetworks

focused on a world where biomedical research will fundamentally change to be more open and collaborative

production

distribution

aggregation

6%

21%

8%

11%

54%cannot reproduce

can reproduce in principle

can reproduce w/discrepancies

can reproduce from processed data w/discrepancies

can reproduce partially

the status quo tolerates poor communication of findings

Ioannidis A. et al. Nature Genetics 2009

208,294,724 datapoints

124 pages supplemental material

?? lines unobtainable source code

?? version or architecture of statistical analysis program (R)

enumerable R packages and package dependencies

key R package “ClaNC” no longer available

1231 citations

often what is in principle reproducible, is not practically reproducible

unidentified publication‣ from journal with 5 year impact factor of 27‣ article freely available for download‣ data freely available for download

“Scientists often study the past as obsessively as historians because few

other professions depend so acutely on it. Every experiment is a conversation with

a prior experiment, every new theory a refutation of the old”

-Siddhartha Mukherjee, The Emperor of All Maladies

scientific method1. define a question

2. gather information and resources (background research)

3. form a hypothesis

8. retest (frequently done by other scientists)

4. test hypothesis experimentally

5. analyze experimental data

7. publish results

6. draw conclusions based on data

7. publish results

finitein

∞...

conducting research for others to consume

(even if the ‘other’ is future you)

reproducible research


code

data

analysis


code

version control


code

version control

client-server

distributed


code

version control

client-server(e.g svn, cvs)


code

version control

client-server

distributed


code

version control

(e.g git, mercurial)distributed


code

version control

distributed


code

data

analysis


data

generic

domain repositories

results


data

digital object identifier (doi)

a unique identifier which remains fixed over the lifetime of a web-accessible object

metadata, including the object’s location, is stored in association with the doi and may change over time

referring to an online document by its doi provides more stable linking than simply referring to a url


code

data

analysis


analysis

R Sweave knitr

great if you know LaTeX


analysis

R Sweave knitr

great if you are lazy

(like me)


analysis

knitr

# Hello World Title ### Author: Brian M. Bot

This is a narrative with inline code execution to tell me that pi is equal to `r pi`. And a plot to show a simple function.

```{r} x <- 1:100 y <- log(x)/x plot(x,y) ```


analysis

ipython notebook


other tools

galaxy

docker

packrat

shiny


other tools

enables sharing of all resources (data, code, results) and their relationships to one another

Go Hawks!

mayo clinic - 2015 jan 28

in an increasingly digital world

brian m. bot ——————

[email protected] @BrianMBot

sage bionetworks


tools for reproducible research in an increasingly digital world

Science