reproducibility, preregistration, etc.: making good science even better
TRANSCRIPT
WSU_August_2016
The next MERC (MARCS Monday Afternoon Research Colloquium), will be held Monday 1 August, 2016 @ 1pm in Bankstown Campus, Bldg 3.G.55. Lunch included before at 12pm
MARCBLOCKBUSTER MARC
Assoc Prof Alex Holcombe
Title: Reproducibility, preregistration, et cetera: Making good science even better
Abstract:Reproducibility problems afflict many sciences, including psychology. The problems are, to some extent, rooted in the criteria for
and process of scientific publication. In response, many journals, funders and professional societies have begun incentivising change. For example, study preregistration, although traditionally used only by clinical trials researchers, is becoming more common. In this
seminar, you will learn how it is now used even in basic experimental psychology, and how you can take advantage of preregistration and other new practices to smooth your path to publication and dissemination of your work. Bring your laptop
(optional), walk with me through preregistering a study, and also learn how sites such as Open Science Framework facilitate project management and collaboration. One object of this seminar is to spark discussion of how we can all make our already wonderful
system of science even better.
Bio:Associate Professor Alex Holcombe has been active in several initiatives related to publishing innovations and open science,
beginning in 2006 when he joined the founding advisory board of the journal PLoS ONE. He has been involved with PsychFiledrawer.org (co-founder), CurateScience.org, and the openness article badges (https://osf.io/tvyxz/) adopted by
Psychological Science and others to reward open practices. Two years ago, he co-founded a new article type, the Registered Replication Report, at the journal Perspectives on Psychological Science.
WSU_August_2016
Reproducibility, preregistration, etc.:
Making good science even better
@ceptional
http://www.slideshare.net/holcombea/
3Slide courtesy Chris Chambers
4Thanks Deborah Aphthorp
5Slide courtesy Chris Chambers
Science publishing has an incentive problem
6
Low reproducibility because:P-hacking
Simmons et al. (2011), Psychological Science
• Collecting more data after seeing whether results were significant
• Option to report either of two DVs
• Control via ANCOVA for gender or not
• Drop or not one of 3 conditions
“False-positive psychology: Undisclosed flexibility in data collection and analysis allows
presenting anything as significant.”
Simmons et al. (2011), Psychological Science
60% false positive rate
p > .05
Schönbrodt, F. D. (2015). p-hacker: Train your p-hacking skills! http://shinyapps.org/apps/p-hacker/.
Slide from Tom Hardwicke
9Slide courtesy Chris Chambers
Science publishing has an incentive problem
Behavior Psychologists reporting
Failed to report all dependent measures 67%Selectively reported studies that “worked” 49%
Excluded data after looking at impact of doing so 42%
Claimed to have predicted an unexpected finding 32%
Failed to report all conditions 26%
John, Loewenstein, & Prelec (2012). Psychological Science, DOI: 10.1177/0956797611430953
Motivated reasoning
WSU_August_2016
"This one's Psych Science. 1998. Put your money on ‘will replicate’?”
"What, you think I was born yesterday?”
"Fine," muttered the bookie.
#AcademicNoir
WSU_August_201612
WSU_August_201613
WSU_August_2016
"Whatever happened to him, anyway? He always had great results."
“Office of Research Integrity got him,” I replied.
We all laughed.
CC-BY Alex Holcombehttps://twitter.com/ceptional/status/663850868396044288
#AcademicNoir
WSU_August_2016
Behavior Psychologists reporting
Failed to report all dependent measures 67%Selectively reported studies that “worked” 49%
Excluded data after looking at impact of doing so 42%
Claimed to have predicted an unexpected finding 32%
Failed to report all conditions 26%
John, Loewenstein, & Prelec (2012). Psychological Science, DOI: 10.1177/0956797611430953
Preregistration
It’ll soon shake your windowsAnd rattle your wallsFor the times, they are a’changin’
23 July 2016
Stopping rule and power analyses expected.
The International Committee of Medical Journal Editors (ICMJE) announces that their journals will not publish
reports of trials unless they have been registered.
2004
Preregistration https://osf.io/c8xhy/
• General methods• Participant recruitment, selection
criteria, and exclusion criteria• Power analysis• Data analysis plan
• Statistical tests• Outlier exclusion criteria
19
MAKE P-VALUES INTERPRETABLE
AGAIN!
PRE-REGISTER YOUR
ANALYSIS PLAN
WSU_August_201621
Concerns
1. Pre-registration does not intend to prevent researchers from conducting exploratory analysis or prevent serendipitous findings
2. It is solely to ensure that the distinction between exploratory and confirmatory findings is clear and transparent
For the times, they are a’changin’
Edwards, Lindman, & Savage (1963)
my other concern is more difficult to address. I has to do with your very small samples…Publication of low power studies is simply not good for the field in the long run - they inflate both the rate of false negatives and false positives in the literature (e.g., Ellis, 2010).
WSU_August_201623
https://osf.io/tvyxz/wiki/home/
100% P-hacking FreeHere, check our numbers.
Here’s how you can replicate our result.
CC-ZERO Alex Holcombe
For the times, they are a’changin’
WSU_August_201624
For the times, they are a’changin’
WSU_August_201625
Concerns
WSU_August_201626
New article formats,
WSU_August_201627
courtesy Chris Chambers (Cardiff)
WSU_August_2016
Any one of you go it alone, he’ll say you messed up. But if we first get him to approve the protocol, and then all run
the replication together…
WSU_August_2016
No
effe
ct
Hagger, M., & Chatzisarantis, N. (2016). A Multilab Preregistered Replication of the Ego-Depletion Effect. Perspectives on Psychological Science, 11 (4), 546-573
http://blogs.discovermagazine.com/neuroskeptic/2016/07/31/end-of-ego-depletion/#.V55g9pN96IZ