Deep Impact: The pernicious habit of ranking scientists by the

journals they publish in

Björn BrembsUniversität Regensburg

http://brembs.net

More scientists, more publications

Journal Rank

• Thomson Reuters: Impact Factor• Eigenfactor (now Thomson Reuters)• ScImago JournalRank (SJR)• Scopus: SNIP, SJR

Source Normalized Impact per Paper

Journal Rank

Only read publications from high-ranking journals

Job applications

Job application instructions

Publikationstätigkeit(vollständige Publikationsliste, darunter Originalarbeiten als Erstautor/in, Seniorautor/in, Impact-Punkte insgesamt und in den letzten 5 Jahren, darunter jeweils gesondert ausgewiesen als Erst- und Seniorautor/in, persönlicher Scientific Citations Index (SCI, h-Index nach Web of Science) über alle Arbeiten)

Publications:Complete list of publications, including original research papers as first author, senior author, impact points total and in the last 5 years, with marked first and last-authorships, personal Scientific Citations Index (SCI, h-Index according to Web of Science) for all publications.

Journal Rank

Only read publications from high-ranking journals

Journal Rank

Only publish in high-ranking journals

METRICS

Is journal rank like astrology?

Show of hands:

• Who knows what the IF is?• Who uses the IF to pick a journal

(rate a candidate, etc.)?• Who knows how the IF is calculated

and from what data?

The Impact Factor

A1 A2

C12

time

citationspublished

articlespublished

year 1 year 2 year 3

𝐼𝐹 (𝑦𝑒𝑎𝑟 3)=C12

A1+A2

Introduced in 1950’s by Eugene Garfield: ISI

The Impact Factor

40 60

100

time

citationspublished

articlespublished

year 1 year 2 year 3

𝐼𝐹 (𝑦𝑒𝑎𝑟 3)=100

4 0+60=1

Introduced in 1950’s by Eugene Garfield: ISI

The Impact Factor

Journal X IF 2010=

All citations from TR indexed journals in 2012 to papers in journal X

Number of citable articles published in journal X in 20010/11

€30,000-130,000/year subscription ratesCovers ~11,500 journals (Scopus covers ~16,500)

Main Problems with the IF

• Negotiable

• Irreproducible

• Mathematically

unsound

Negotiable

• PLoS Medicine, IF 2-11 (8.4)(The PLoS Medicine Editors (2006) The Impact Factor Game. PLoS Med 3(6): e291. http://www.plosmedicine.org/article/info:doi/10.1371%2Fjournal.pmed.0030291)

• Current Biology IF from 7 to 11 in 2003– Bought by Cell Press (Elsevier) in 2001…

Not Reproducible

• Rockefeller University Press bought their data from Thomson Reuters

• Up to 19% deviation from published records• Second dataset still not correct

Rossner M, van Epps H, Hill E (2007): Show me the data. The Journal of Cell Biology, Vol. 179, No. 6, 1091-1092 http://jcb.rupress.org/cgi/content/full/179/6/1091

Not Mathematically Sound

• Left-skewed distributions• Weak correlation of individual article citation

rate with journal IF

Seglen PO (1997): Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):497 (15 February)http://www.bmj.com/cgi/content/full/314/7079/497

Journal rank and citations

The weakening relationship between the Impact Factor and papers' citations in the digital age (2012): George A. Lozano, Vincent Lariviere, Yves Gingras arXiv:1205.4328

Journal Rank and Quality

Brown, E. N., & Ramaswamy, S. (2007). Quality of protein crystal structures. Acta Crystallographica Section D Biological Crystallography, 63(9), 941–950. doi:10.1107/S0907444907033847

Journal Rank and ‘Quality’

Munafò, M., Stothart, G., & Flint, J. (2009). Bias in genetic association studies and impact factor Molecular Psychiatry, 14 (2), 119-120 DOI: 10.1038/mp.2008.77

Journal Rank and Methodology

Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience, 7. doi:10.3389/fnhum.2013.00291

NO EVIDENCE

Journal rank is a figment of our imagination.

Journal Rank and Fraud/Error

Fang et al. (2012): Misconduct accounts for the majority of retracted scientific publications. PNAS 109 no. 42 17028-17033

Journal Rank and Retractions

Data from: Fang, F., & Casadevall, A. (2011). RETRACTED SCIENCE AND THE RETRACTION INDEX Infection and Immunity DOI: 10.1128/IAI.05661-11

Journal rank and citations

The weakening relationship between the Impact Factor and papers' citations in the digital age (2012): George A. Lozano, Vincent Lariviere, Yves Gingras arXiv:1205.4328

INCENTIVES

“High-Impact” journals attract the most unreliable research

“Do you trust scientists?”

“Who can you trust these days?”

“Politicians? Financial experts? Realtors?“

WHAT HAPPENED?

The disaster of our digital infrastructure

HISTORY

Journal rank is a relic of the print era

Distribution yesterday

Subscriptions yesterday

Meanwhile…

Modified from ARL: http://www.arl.org/bm~doc/arlstats06.pdf, http://www.arl.org/bm~doc/arlstat08.pdf

% C

han

ge

19861987

19881989

19901991

19921993

19941995

19961997

19981999

20002001

20022003

20042005

20062007

2008-50

0

50

100

150

200

250

300

350

400

Subscription pricesCPI/inflationJournals purchased

Subscriptions today

Distribution today

SCHOLARSHIP

Institutions produce publications, data and software

DISASTER I

Dysfunctional scholarly literature

Literature

• No scientific impact analysis

• Limited access• No global search• No functional hyperlinks• No flexible data

visualization• No submission

standards• (Almost) no statistics• No text/data-mining• No effective way to sort,

filter and discover• No networking feature• etc.

…it’s like the web in 1995!

DISASTER II

Scientific data in peril

DISASTER III

Non-existent software archives

Today‘s Digital Dystopia

• Institutional email• Institutional

webspace• Institutional blog• Library access card• Open access

repository

• No archiving of publications

• No archiving of software

• No archiving of data

WHAT NOW?

Science, tear down this paywall!

1. International Coordination

2. Hire software developers

Superior Access

• Harvest all Open Access Publications– Accessible via single interface– Not just from green repositories– Everything not obviously illegal

• Integrate resulting database– PubMed– Google Scholar

• Plug the gaps:

3. Cancel Subscriptions

Superior Alternative

• Global search and access for all literature, software and data

• Intelligent sort, filter and discover functionalities

• Scientific, evidence-based reputation system• Authoring tool for collaborative writing ans

single-click submission• Orders of magnitude cheaper: US$90/paper

(e.g. SciELO) vs. US$4,800/paper (subscription)