reusable science: how not to slip from the shoulders of giants
DESCRIPTION
No research is done in a void: science is constantly expanding previous hypotheses, building upon past knowledge. We live in a digital age where information is ubiquitous, yet we struggle to preserve accurate machine readable and quantitative descriptions of our research compromising our capacity to use them in our inferences. In the following talk I will show how and why we incorporate assumptions in our studies based on three experiments we have conducted: (i) dissociating metacognitive subdomains in medial and lateral anterior prefrontal cortex, (ii) relating reading comprehension to individual differences in the default mode network, and (iii) exploring neural correlates of the content and form of self-generated thoughts. This will be followed by introducing a new inference method - probabilistic Regions of Interest (pROI) - which allows the use of prior knowledge in the form of a probabilistic map. This approach provides the middle ground between ROI and full brain analysis, by giving researchers more flexibility in formalizing priors. The quality of prior probability maps based on the literature can be improved by using unthresholded statistical maps instead of peak coordinates. To facilitate this we have created NeuroVault.org - a community - wide effort to collect unthresholded statistical maps. Taking the initiative a step further I will describe the concept of data papers - publications purely dedicated to datasets. Together those three mechanisms (pROI, NeuroVault.org and data papers) are a small but significant steps towards better, more reusable and reproducible science.TRANSCRIPT
Reusable Science:How not to slip from the shoulders of giantsChris GorgolewskiMax Planck Research Group: Neuroanatomy & Connectivity
Anatomy of a giant
I. Example studies
II. Probabilistic ROIs
III.Sharing statistical maps
IV.Data papers
Anatomy of a giant
I. Example studies
II. Probabilistic ROIs
III.Sharing statistical maps
IV.Data papers
Study I
Medial and Lateral Networks in Anterior Prefrontal Cortex Support Metacognitive Ability for Memory
and Perception
Benjamin Baird, Jonathan Smallwood, Krzysztof J. Gorgolewski, and Daniel S. Margulies
Journal of Neuroscience (in press)
Meta-cognition
• Are we equally good in judging our performance of memory or perception tasks?
• Is metacognition related to medial or lateral prefrontal cortex? Does it depend on modality?
Measuring metacognition
Metacognition of memory and perception are distinct systems
Sources of seed points
Gilbert et al. 2006, Functional specialization within rostral prefrontal cortex (area 10): a meta-analysis. Journal of cognitive neuroscience
Fleming, S. M., Weil, R. S., Nagy, Z., Dolan, R. J., & Rees, G. (2010). Relating introspective accuracy to individual differences in brain structure. Science (New York, N.Y.), 329(5998), 1541–3.
Sources of seed points
Meta-cognition of MemoryMeta-cognition of Perception
Baird, Smallwood et al. (in press) JON
vs.
Double dissociation of metacognitive abilities
Study II
The Default Modes of Reading: Modulation of posterior cingulate and medial prefrontal cortex connectivity
associated with subjective and objective differences in reading experience
Jonathan Smallwood, Krzysztof J. Gorgolewski, Johannes Golchert, Florence J.M. Ruby, Haakon G. Engen, Benjamin
Baird, Melaina Vinski, Jonathan Schooler, Daniel S. Margulies
Frontiers in Neuroscience (in press)
Reading comprehension
• What is the relation between task focus and reading comprehension?
• What role does Default Mode Network play in reading comprehension and task focus?
Task focus is inversely correlated with reading comprehension
Smallwood, et al., Frontiers in Human Neuroscience
Reading by DefaultSeed locations
Andrews-hanna, J. R., Reidler, J. S., Sepulcre, J., Poulin, R., Buckner, R. L., & Temp, P. (2010). Functional-anatomic fractionation of the brain’s default network. Neuron, 65(4), 550–62.
Smallwood, et al., Frontiers in Human Neuroscience
Reading by Default
Reading by Default
Smallwood, et al., Frontiers in Human Neuroscience
Why mind-wandering may disrupt reading
Study III
A correspondence between the brain's intrinsic functional
architecture and the content and form of self-generated thoughts
Krzysztof J. Gorgolewski, Dan Lurie, Sebastian Urchs, Judy A. Kipping, R. Cameron Craddock, Michael P.
Milham, Daniel S. Margulies, and Jonathan Smallwood
PLoS One (submitted)
Mind wandering
• What is the content and form of thoughts in mind wandering?
• How does it relate to various aspects of intrinsic BOLD activity?
Questions about the content
Questions about the form
Future vs. Past = Words vs. Images
Resting state measures
Fractional Amplitude of Low Frequency Fluctuations
Regional Homogeneity
Degree Centrality
Group average
Not only Default Mode Network
Not only Default Mode Network
What these studies have in common?
Anatomy of a giant
I. Example studies
II. Probabilistic ROIs
III.Sharing statistical maps
IV.Data papers
Signal to Noise ratio
Looking in the wrong places
Lower SNR = we miss more stuff
Lower SNR = higher FDR threshold
How to improve power?
• stronger effects?• fewer null/noise samples -> ROI
What is wrong with ROI analysis?
What is wrong with ROI analysis?
Binary nature of masks
Fifty shades of grey
A probabilistic view on the ROI analysis
A probabilistic approach to ROI analysisGorgolewski et al. PRNI 2013
Fifty Shades of Grey, Matter
Extensions and disclaimers
• Kernel density estimation• Markov Random Field reguralization• Posterior maps cannot be used in
meta analysis – circularity!• Prior maps are integral part of the
analysis and need to be included in publications
Anatomy of a giant
I. Example studies
II. Probabilistic ROIs
III.Sharing statistical maps
IV.Data papers
Just coordinates?
• Databases such as Neurosynth or BrainMap rely on peak coordinates reported in papers (only strong effects)
Are we throwing money away?
Data sharing?
Data sharing?
• Ok, ok so we should share data.• We all know it’s good.• But almost no one does it.– You have to prepare data– You risk that your mistakes will be
found!
“I swear I’ve heard it before”
• In the past there were many attempts to propagate data sharing– For example fMRI DC:
• Failed because of technical issues• …and the amount of time it took to prepare data
for submission (a week, a very frustrating week)
• fMRI DC was however too ambitious for its time:– They wanted to collect raw data and all
metadata required to reproduce the analysis
Van Horn & Gazzaniga (2013). Why share data? Lessons learned from the fMRIDC. NeuroImage
Baby steps
• Everything is a question of cost and benefit– If we keep the cost low even small
benefit (or just conviction that data sharing is GOOD) will suffice
NeuroVault.org simple data sharing
• Minimize the cost!• We just want your statistical maps
with minimum description (DOI)– If you want you can put more metadata,
but you don’t have to
• We streamline login process (external services such as Google, Facebook)
Benefits?
• In return authors get interactive web based visualization of their statistical maps– Something they can embed on their lab
website
• We are keeping both cost and benefit low…–…but we also plan to work with journal
editors to popularize the idea
?
Share your stat maps!
Make science more reproducible
NeuroVault.org
Anatomy of a giant
I. Example studies
II. Probabilistic ROIs
III.Sharing statistical maps
IV.Data papers
Motivation
• Share your stat maps!
vs.
institutions scientists
Quality control
• Share your stat maps!
Complex datasets require elaborate descriptions
Solution – data papers
• Authors get recognizable credit for their work.– Even smaller contributors such as RAs
can be included.
• Acquisition methods are described in detail.
• Quality of metadata is being controlled by peer review.
• Neuroinformatics (Springer)• Frontiers in Human Brain Methods
• GigaScience (BGI, BioMed Central)• Scientific Data (Nature Publising
Group, coming soon)
(Frontiers Media)(Nature Publishing Group)
Where to publish data papers?
Read more
• Probabilistic ROISGorgolewski et al. PRNI, 2013
• NeuroVault.orgGorgolewski et al. OHBM, 2013
• Data papersGorgolewski et al. Frontiers in Brain Imaging Methods, 2012
Acknowledgements(my personal giants)
Pierre-Louie BazinHaakon EngenSatrajit Ghosh
Russell A. PoldrackJean-Baptiste Poline
Yannick SchwarzTal Yarkoni
Michael MilhamDaniel Margulies
Benjamin Baird
Jonathan SmallwoodJohannes GolchertFlorence J.M. RubyMelaina VinskiJonathan SchoolerDan LurieSebastian UrchsJudy A. KippingR. Cameron CraddockMPI CBS Resting state group
THANK YOU!
Details
Details
Details