Reproducibility of computational research: methods to avoid madness

Chair: Michael Hucka, Ph.D. Department of Computing and Mathematical Sciences

California Institute of Technology Pasadena, CA, USA

ICSB 2014, Melbourne, Australia, September 2014

Session introduction

So, what’s this about reproducibility?

“In biomedical science, at least one thing is apparently reproducible: a steady stream of studies that show the irreproducibility of many important experiments …”

— Wadman. (2013). Nature, 500(7460).

“We find it utterly unexpected that, overall, it is only a minority of articles that properly describe (in a reproducible way) the computational research performed …”

— Hübner, Sahle & Kummer. (2011). FEBS Journal 278(16).

What is the focus of this session?

Facets of reproducibility

Methodologicalissues

Reproducibility issues

Culturalissues

Motivations Policies

Incentives Funding

…

MethodsStandardsAlgorithmsInfrastructure…

Facets of reproducibility

Methodologicalissues

Reproducibility issues

Culturalissues

Motivations Policies

Incentives Funding

…

MethodsStandardsAlgorithmsInfrastructure…

Reproducibility issues

Culturalissues

Motivations Policies

Incentives Funding

…

Computational research should be easier to get rightHave greater control of what is done, and how it’s done

⇒ Greater potential for making our work reproducible

Assertion:

The methodological issues are amenable to practical interventions

Some examples:

• Define and adopt standards for data formats, ontologies, protocols

• Develop better methods for analysis, simulation, comparison

• Develop effective resources for sharing & communicating research

“… reproducibility in computational biology is aspired to, but rarely achieved. This is unfortunate since the quantitative nature of the science makes reproducibility more obtainable than in cases where experiments are qualitative and hard to describe explicitly.”

— Garijo et al. (2013), PLoS One 8(11)

“… reproducibility in computational biology is aspired to, but rarely achieved. This is unfortunate since the quantitative nature of the science makes reproducibility more obtainable than in cases where experiments are qualitative and hard to describe explicitly.”

— Garijo et al. (2013), PLoS One 8(11)

currently

What will be covered in this session?

Can’t cover all potential topics

Speaker Subject Relevance

Hucka standard formats accurate communication of models

Lovell data analysis appropriate inferences from data

Kuperstein data curation & visualization software reconciling data from multiple sources

Nahid workflow software recreating data analysis procedures