what you need to know about research assessment
TRANSCRIPT
Research Assessment: What you need to know.
A Presentation by Ian Gibson, Collections LibrarianJames A. Gibson Library
It’s all his fault
Image from http://vw.indiana.edu/places-images/authors/eugene-garfield.jpg
The Journal Impact Factor
• Where – Citations = citations in the current year to articles
published in the 2 previous years– Citable Articles = the number of articles published in the 2
previous years (in theory)
if you use impact factors you are
statistically illiterate 1
1Stephen Curry, Sick of Impact Factors http://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors
Photo from http://www.theproteinwrangler.com/wp-content/uploads/2011/10/scurry.jpg
Alternatives to Impact Factors?
• 5 Year Impact Factor• Eigenfactor• Source-Normalized Impact per Paper (SNIP)• SCImago Journal Rank (SJR)• Google Journal Rank
What Metrics Exist to Judge Individuals?
Let me hook you up with my h-index
Jorge Hirsch http://www-physics.ucsd.edu/fac_staff/fac_profile/faculty_description.php?person_id=99
h-Index
• h = n papers that have been cited at least n times.
We have a problem.
“Citation based statistics can play a ‐role in the assessment of research, provided they are used properly, interpreted with caution, and make up only part of the process. Citations provide information about journals, papers, and people. We don't want to hide that information; we want to illuminate it.”- Joint Committee on Quantitative Assessment of Research, International Mathematics Union in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS) http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf
Declaration on Research Assessment
General Recommendation1. Do not use journal-based metrics, such
as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist's contributions, or in hiring, promotion, or funding decisions.http://am.ascb.org/dora/
Altmetrics• http://altmetrics.org/manifesto• Putting the focus at the article level• Capturing broader impact:– “Hallway chats” (e.g. twitter, blog postings)– Those papers that you don’t cite but keep in that
drawer in your office (e.g. saved in a reference manager like Zotero or Mendeley)
– The papers that you read just for general interest (e.g. download statistics)
PLOS Article Level Metrics
• http://www.plosone.org/article/metrics/info%3Adoi%2F10.1371%2Fjournal.pone.0030366
• http://www.plosone.org/article/metrics/info%3Adoi%2F10.1371%2Fjournal.pone.0081461
Impact Story
• http://impactstory.org/heatherpiwowar• http://impactstory.org/IanGibson • This works best if you have a ORCID or
ResearcherID
Altmetric.comhttp://www.altmetric.com/details.php?domain=journal.frontiersin.org&citation_id=1527978
What do Altmetrics tell us?
http://mikuru.net/wp-content/uploads/2011/12/under-construction.gif
Conclusion
Questions?
Bibliography• Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience, 7, 291.
doi:10.3389/fnhum.2013.00291• Cantrill, S. (2012, June 30). Inside our Impact Factor. The Sceptical Chymist. Retrieved from
http://blogs.nature.com/thescepticalchymist/2012/06/inside-our-impact-factor.html• Colquhoun, D., & Plested, A. (2014, January 16). Why you should ignore altmetrics and other bibliometric nightmares. DC’s Improbable Science.
Retrieved April 2, 2014, from http://www.dcscience.net/?p=6369• Costas, R., Zahedi, Z., & Wouters, P. F. (2014, January 17). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators
with citations from a multidisciplinary perspective. CWTS-WP-2014-001. External research report. Retrieved April 2, 2014, fromhttps://openaccess.leidenuniv.nl/handle/1887/23041
• Curry, S. (2012, August 13). Sick of Impact Factors. Reciprocal Space. Retrieved April 7, 2014, from http://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors/
• Fenner, M. (2013). What Can Article-Level Metrics Do for You? PLoS Biol,11(10), e1001687. doi:10.1371/journal.pbio.1001687• Garfield, E. (1955). Citation Indexes for Science. Science, 122(3159), 108–111. Retrieved from http://www.jstor.org/stable/1749965• Garfield, E. (1964). “Science Citation Index”-A New Dimension in Indexing.Science, 144(3619), 649–654. Retrieved from
http://www.jstor.org/stable/1712875• Garfield, E. (2005). The Agony and the Ecstasy— The History and Meaning of the Journal Impact Factor. Presented at the International Congress
on Peer Review And Biomedical Publication, Chicago. Retrieved fromhttp://garfield.library.upenn.edu/papers/jifchicago2005.pdf• Rossner, M., Epps, H. V., & Hill, E. (2007). Show me the data. The Journal of Cell Biology, 179(6), 1091–1092. doi:10.1083/jcb.200711140• Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43(9), 628–638. doi:
10.1002/(SICI)1097-4571(199210)43:9<628::AID-ASI5>3.0.CO;2-0• Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98(2), 1131–1143. doi:10.1007/s11192-013-1117-2• The PLoS Medicine Editors. (2006). The Impact Factor Game. PLoS Medicine, 3(6), e291. doi:10.1371/journal.pmed.0030291• Thelwall, M., Haustein, S., Lariviere, V., & Sugimoto, C. R. (2013). Do Altmetrics Work? Twitter and Ten Other Social Web Services. PLoS ONE, 8(5),
e64841. doi:10.1371/journal.pone.0064841• Vanclay, J. K. (2012). Impact factor: outdated artefact or stepping-stone to journal certification? Scientometrics, 92(2), 211–238. doi:
10.1007/s11192-011-0561-0