what's wrong with our scholarly infrastructure?
TRANSCRIPT
What’s Wrong with Us?Or:
How accountability is killing science
Björn BrembsUniversität Regensburg
http://brembs.net - @brembs
SCHOLARSHIP
Scientists produce publications, data and code
CROWN JEWELS
Scientists produce publications, data and code
PRECIOUS
Scientists produce publications, data and code
PROBLEM I
Dysfunctional scholarly literature
Literature• Limited access
Literature• Limited access
Literature• Limited access• No global search
Literature• Limited access• No global search• No functional hyperlinks
First demonstration: 1968 WWW: 1989
Stanford Research Institute: NLS Tim Berners-Lee: CERN
Literature• Limited access• No global search• No functional hyperlinks• No data visualization
Literature• Limited access• No global search• No functional hyperlinks• No data visualization• No submission standards
Literature• Limited access• No global search• No functional hyperlinks• No data visualization• No submission standards• (Almost) no statistics
Literature• Limited access• No global search• No functional hyperlinks• No data visualization• No submission standards• (Almost) no statistics• No text/data-mining
Literature• Limited access• No global search• No functional hyperlinks• No data visualization• No submission standards• (Almost) no statistics• No text/data-mining• No effective way to sort, filter and
discover
Antiquated Functionality
…it’s like the web in 1995!
• Limited access• No global search• No functional hyperlinks• No data visualization• No submission standards• (Almost) no statistics• No text/data-mining• No effective way to sort, filter and
discover• No scientific impact analysis• Lousy peer-review • No networking feature• Etc.
PROBLEM II
Scientific data in peril
Small Data – Long Tail
Report on Integration of Data and Publications, ODE Report 2011http://www.alliancepermanentaccess.org/wp-content/plugins/download-monitor/download.php?id=ODE+Report+on+Integration+of+Data+and+Publications
October 1-16, 2013
PROBLEM III
Non-existent software archives
UNIVAC (A-2) 1953
Dysfunctional Infrastructure
• Institutional email• Institutional webspace• Institutional blog• Library access card• Open access repository
• Publications?• Code?• Data?
HISTORICAL LESSON
Don‘t let someone with orthogonal interests touch your precious
They don‘t know science
They don‘t understand elementary science
http://www.sciencedirect.com/science/article/pii/S0092867415002421
They don‘t understand elementary science
http://science.sciencemag.org/content/351/6274/737.full
They support arms trade
They support collusion in torture• 2013: US$86m of US$126m annual APA budget comes from publishing
They fake journals
They fake journals
“This was an unacceptable practice, and we regret that it took place.”
Michael Hansen, CEO Of Elsevier's Health Sciences Division
They hold science hostage
They detest the scientific method
#researchparasites
They promote sexism
They pay politicians to make OA illegal
Your tax dollars at work!
They parasitize public funds
Modified from ARL: http://www.arl.org/bm~doc/arlstats06.pdf, http://www.arl.org/bm~doc/arlstat08.pdf
% C
hang
e
-50
0
50
100
150
200
250
300
350
400Subscription pricesCPI/inflationJournals purchased
They parasitize public funds
(Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. Nature 495, 426–9; Packer, A. L. (2010). The SciELO Open Access: A Gold Way from the South. Can. J. High. Educ. 39, 111–126)
Cost
s [th
ousa
nd U
S$/a
rticle
]
Legacy SciELO
They parasitize public funds
Cost
s [th
ousa
nd U
S$/a
rticle
]
Legacy SciELO(Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. Nature 495, 426–9; Packer, A. L. (2010). The SciELO Open Access: A Gold Way from the South. Can. J. High. Educ. 39, 111–126)
They parasitize public funds
+70%
They parasitize public funds
“The decision, based on market and competitor analysis, will bring Emerald’s APC pricing in line with the wider market, taking a mid-point position amongst its competitors.”
Emerald spokesperson
MONEY FOR NOTHING
Wasting billions on a parasitic industry
They sell bogus journal rankings
• Thomson Reuters: Impact Factor• Eigenfactor (now Thomson Reuters)• ScImago JournalRank (SJR)• Scopus: SNIP, SJR
Source Normalized Impact per Paper
Main Problems with the IF• Negotiable • Irreproducible • Mathematically
unsound
Negotiable• PLoS Medicine, IF 2-11 (8.4)
(The PLoS Medicine Editors (2006) The Impact Factor Game. PLoS Med 3(6): e291. http://www.plosmedicine.org/article/info:doi/10.1371%2Fjournal.pmed.0030291)
• Current Biology IF from 7 to 11 in 2003– Bought by Cell Press (Elsevier) in 2001…
Negotiable
https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/
June, 2014 (19 months)
Not Reproducible• Rockefeller University Press bought their data from Thomson Reuters• Up to 19% deviation from published records• Second dataset still not correct
Rossner M, van Epps H, Hill E (2007): Show me the data. The Journal of Cell Biology, Vol. 179, No. 6, 1091-1092 http://jcb.rupress.org/cgi/content/full/179/6/1091
Not Mathematically Sound• Left-skewed distributions• Weak correlation of individual
article citation rate with journal IF
Seglen PO (1997): Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):497http://www.bmj.com/cgi/content/full/314/7079/497
Not Mathematically Sound
https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/
‘QUALITY’
Is journal rank like astrology?
Selectivity
http://blog.frontiersin.org/2015/12/21/4782/
Citations
The weakening relationship between the Impact Factor and papers' citations in the digital age (2012): George A. Lozano, Vincent Lariviere, Yves Gingras arXiv:1205.4328
Methodology I
Macleod MR, et al. (2015) Risk of Bias in Reports of In Vivo Research: A Focus for Improvement. doi:10.1371/journal.pbio.1002273
Methodology II
Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience, 7. doi:10.3389/fnhum.2013.00291
Methodology III
Munafò, M., Stothart, G., & Flint, J. (2009). Bias in genetic association studies and impact factor Molecular Psychiatry, 14 (2), 119-120 DOI: 10.1038/mp.2008.77
‘Quality’
Brown, E. N., & Ramaswamy, S. (2007). Quality of protein crystal structures. Acta Crystallographica Section D Biological Crystallography, 63(9), 941–950. doi:10.1107/S0907444907033847
Quality
Journal Rank
Qua
lity
Berghmans et al. (2002): doi: 10.1093/annonc/mdg203, Tressoldi et al. (2013) doi:10.1371/journal.pone.0056180, Brembs et al. (2013) doi:10.3389/fnhum.2013.00291, Fraley & Vazire (2014) doi:10.1371/journal.pone.0109019, Macleod et al. (2015) doi:10.1371/journal.pbio.1002273
Journal Rank and Fraud/Error
Fang et al. (2012): Misconduct accounts for the majority of retracted scientific publications. PNAS 109 no. 42 17028-17033
Journal Rank and Fraud/Error
Fang et al. (2012): Misconduct accounts for the majority of retracted scientific publications. PNAS 109 no. 42 17028-17033
Journal Rank and Retractions
Data from: Fang, F., & Casadevall, A. (2011). RETRACTED SCIENCE AND THE RETRACTION INDEX Infection and Immunity DOI: 10.1128/IAI.05661-11
Credit: Scott Edmund
INCENTIVES
“High-Impact” journals attract the most unreliable research
Productivity
Productivity
Research questions:True:False:Significant:
100505021
44222219
PUBLISH OR PERISH
Quality & Productivity: Selecting the sloppy scientists
Freedman LP, Cockburn IM, Simcoe TS (2015) http://journals.plos.org/plosbiology/article?id=info:doi/10.1371/journal.pbio.1002165
Irreproducibility
61%(n=100)
Open Science Collaboration
Replication in Psychology
Open Science Collaboration Science 2015;349:aac4716
Replication in Psychology
Open Science Collaboration Science 2015;349:aac4716
Retractions on the Rise
Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience, 7. doi:10.3389/fnhum.2013.00291
2005
2013
2013
“Do you trust scientists?”
Status Quoan obscenely expensive anachronism
Beyond Science
If we cannot even trust the scientific literature,what information is trustworthy?