durham leading research programme: academic impact

Contact Details

Academic Liaison Librarian (Researcher Support) [email protected] Phone: +44 (191) 334 2961

Leading Research Module 15:

Academic Impact: Research Indicators

mailto:[email protected]

Learning Outcomes for the session

• Increased awareness of the various journal and author

metrics available.

• Understanding of how some key indicators (JIF, H-index)

are calculated

• Understanding of the key issues around the use of these

metrics and what research behaviours might be

incentivised.

• Awareness of the potential opportunities for exploring

wider academic and non-academic impact of publications

from altmetric tools available.

Session outline

• Introduction: What can you measure?

• Citations

• Author metrics

• Journal metrics

• Metrics and the REF

• Altmetrics (in brief)

Quick Survey (1)

• How many of you have been advised to publish

in certain journals (eg “high impact journals”)?

• How many of you are aware of if and how any

citation metrics were used in the last REF?

Quick Survey (2)

• What does it mean if an article has a citation

count of 97?

• What does it mean if an author has an h-index

of 14?

• What does it mean if a journal has a Journal

Impact Factor of 1.317?

• What does it mean if an article has an altmetric

score (on altmetric.com) of 2734?

What can you measure?

• Article/Book impact

• One measure of the impact of

individual journal articles,

conference proceedings or books,

is the number of times they are

cited by other works.


• Publication (Journal) impact

• The perceived impact of a specific

academic journal might be

assessed by the number of times

their articles are cited (on average)

and where they are cited.


• Researcher impact

• The number of outputs and citation

a researcher generates can be an

indicator for the impact of an

individual researcher.


• Institutional impact

• The prestige of a department or

area of research within an

institution compared to those at

other institutions can be measured

by the sum of individual

researchers ‘impact’.

Impact ?

Are bibliometrics

effective & reliable

measures of

academic impact?

Part 1 Citations

and

Citation

Indices

Citations

• Links between papers that have

something in common

Citations


something in common

• Building upon, or challenging, research

Citations


something in common


• Help make a judgement about impact

an article has made

Citations


something in common


• Help make a judgement about impact

an article has made

• Sum of citations can be an indication of

the ‘impact’ of an author’s work / a

journal as a collection of articles

... The problem...

• How do you identify who has cited a

publication you have written or read?

Citation indices

• 1955: Eugene Garfield proposed

creating a citation index for science to...

Citation indices

• 1955: Eugene Garfield proposed

creating a citation index for science to...

“eliminate the uncritical citation of fraudulent,

incomplete or obsolete data by making it

possible for the conscientious scholar to be

aware of criticisms of earlier papers” Garfield, E (1955) ‘Citation Indexes for Science’

Science, New Series, Vol. 122, No. 3159, pp. 108-111

Citation indices

• 1955: Eugene Garfield - presents the idea of

measuring the ‘impact’ of journal articles

using citations

Citation indices



using citations

• 1960s: Science Citation Index developed to

highlight “formal, explicit linkages between

papers that have particular points in common”

Citation indices



using citations

• 1960s: Science Citation Index developed to

highlight “formal, explicit linkages between

papers that have particular points in common”

• 1975: Journal Citation Reports use Web of

Science data to rank journals within

disciplines

Citation indices

(2008) Taylor and Francis LibSite Newsletter,

issue 9. p. 5

Science subjects

Social-science subjects

Citation indices

“reference lists are held under copyright by

academic publishers which makes tracking

citations impossible”

The death of the reference and the re-use factor (2013) http://figshare.com/blog/The_Death_Of_The_Reference_and_the_reuse_factor/103

http://figshare.com/blog/The_Death_Of_The_Reference_and_the_reuse_factor/103

Citation indices

• Web of Science (Thomson Reuters)

• Sci-verse Scopus (Elsevier)

• Google Scholar

Things you can do

• Count citations to an article

• Link to other related articles

• Citation mapping

• Set up citation alerts

• Search for cited references

• See citation reports for authors and journals

Demo Web of Science /

Google Scholar

Over to you • Check Google Scholar and Web of Science for an

article you have already written/read or are aware

of, or try:

Zong, Y. and Tooley, M. J. (2003) 'A historical record of coastal

floods in Britain : frequencies and associated storm tracks.',

Natural hazards, 29 (1). pp. 13-36.

• Explore and compare the number of citations (if

any) listed

Citation searching

Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available at http://www.flickr.com/photos/theblackcanvas/2945878325/

http://www.flickr.com/photos/theblackcanvas/2945878325/

Citation Metrics

h-index (Hirsch, 2005)

• an author’s number of articles (h) that have

received at least h citations

• a researcher with an h-index of 10 has published

10 articles that have each been cited 10+ times

g-index (Egghe, 2006)

• the highest number (g) of papers that together

received g2 or more citations

• a researcher with a g-index of 10 has published 10

papers that, in total, have been cited at least 100

times

h-index

Author: Smith, J

Has written and published 9 articles (a-i),

which have been cited as follows:

a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3

“no. of articles (n) that have received at least n citations”

h-index

Author: Smith, J



a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3

H-index: 3 (at least 3 References with 3 or

more citations)


h-index

Author: Smith, J



a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3

H-index: not 4 (only 3 References with 4 or

more citations)


h-index: what’s in a number?

• Nobel Prize Winner 2013, Peter W

Higgs

• H-index (Google Scholar) = 12

• H-index (Web of Science) = 11

g-index

Author: Smith, J



a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3

“the highest number (g) of papers that together have received

g2 or more citations”

g-index

Author: Smith, J



a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3

G-index: 5 (5x5 = 25... Total of top 5 cited

articles citations = 31)



g-index

Author: Smith, J



a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3

G-index: not 6 (6x6 = 36... Total of top 6 cited

articles citations = 34)



Demo

Web of Science

Over to you • If you have published 1 or more papers, and if all

are indexed on Web of Science or Google Scholar,

see if you can calculate and compare your h-index

(or try to calculate your g-index)

• ... or try the same for a colleague, supervisor or

head of department.

Author metrics



Google Scholar – My Citations

• Track citations to your publications

• Check who is citing your publications. Graph your

citations over time. Calculate citation metrics.

• View publications by colleagues or co-authors

• Keep up with their work, view their citation

metrics.

• Appear in Google Scholar Search Results

• Create a public profile that can appear in Google

Scholar when someone searches for your name.

Author metrics – Issues?

• Author identification

e.g. Professor Gordon Love

A name is not unique

o Prof. Gordon Love, University of California (Earth Science)

o Dr Gordon L Love, Sacramento (Medicine and Health)

o Prof. Gordon Love, Durham University (Physics)

https://www.dur.ac.uk/physics/staff/profiles/?id=246

Author metrics – Issues?

• Author identification

e.g. Professor Gordon Love

... So you need a unique identifier (or 3)

o ORCID profile (0000-0001-5137-9434)

o Researcher ID profile (A-3071-2011)

o Google Scholar profile (3xJXtlwAAAAJ)

https://www.dur.ac.uk/physics/staff/profiles/?id=246

http://orcid.org/0000-0001-5137-9434

http://www.researcherid.com/rid/A-3071-2011

http://scholar.google.co.uk/citations?sortby=pubdate&hl=en&user=3xJXtlwAAAAJ&view_op=list_works

Part 3 Journal

Metrics

Journal Citation Reports (JCRs)

• JCRs – annual publication of journals and

their impact factors.

• Over 10,800 titles, across 232 disciplines

have JIFs in 2015 editions

• A journal that is cited once, on average, for

each article published has an JIF of 1.

Journal Citation Reports (JCRs)

• 2015 edition (published June 2015):

• is JCR year 2014 (providing 2014 Journal

Impact Factor (JIF) metrics)

• counting citation data from 2014

• for articles published in 2012/13

Journal Impact Factor (JIF)

Citations in 2014 to all

articles published by Journal

X in 2012 & 2013

Journal

X’s

2014

impact

factor

=


Citations in 2014 to all

articles published by Journal

X in 2012 & 2013

Number of articles that

were published in Journal X

in 2012 & 2013

Journal

X’s

2014

impact

factor

=


Citations in 2014 (in journals

indexed in Web of Science) to

all articles published by

Journal X in 2012 & 2013

Number of articles (of a type

deemed to be citeable) that

were published in Journal X

in 2012 & 2013

Journal

X’s

2014

impact

factor

=

Journal Ranking

Demo

JCRs

Journal Impact Factors

Other journal impact metrics

• Eigenfactor

• Calculates weightings based on ‘where’

the citation came from

• Also takes into account the ‘size’ of a

journal, to measure the ‘total importance of

a journal’

• http://www.eigenfactor.org/ (1997-2013)

http://www.eigenfactor.org/

http://www.eigenfactor.org/

Other journal impact metrics

• SCImago

• Based on data from Scopus (rather than

Web of Science)

• http://www.scimagojr.com/journalrank.php

http://www.scimagojr.com/journalrank.php




Over to you • Explore the JCRs to identify relevant subject

categories for your field of research.

• Identify journals and compare JIFs and Eigenfactor

Scores

Journal metrics



Issues with journal metrics

• No comprehensive data source

• Citation cultures vary across disciplines

• Publication cultures vary across disciplines

• Research careers have different stages

• Citations ≠ confirmation of excellence

• Scholarly communication is evolving…

• Blogs, twitter, data and open access

repositories

Metrics for Research Assessment

• REF2014

• 11 of 36 panels assessing submissions looked at

citation data

• Benchmarked against data for comparable articles

• Did not take into account Journal Impact Factors in

assessment (?)

• REF(2020?)

• Consulting on use of metrics

• “The Metric Tide” (9th July 2015)


“Placing too much emphasis on narrow,

poorly-designed indicators – such as journal

impact factors (JIFs) – can have negative

consequences”

“… correlation analysis of the REF2014

results ... Has shown that individual metrics

give significantly different outcomes from the

REF peer review process.”


“At an institutional level, HEI leaders should develop

a clear statement of principles … including the role

of quantitative indicators.”

“Individual researchers should be mindful of the

limitations of particular indicators in the the way they

present their own CVs and evaluate the work of

colleagues.”

“Publishers should reduce the emphasis on journal

impact factors as a promotional tool … [and] should

also make available a range of article-level metrics”

Journal Impact Factors and Citations

• “… the JIF is the main determinant of article citation

impact”

Didegah, F. and Thelwall, M. (2013) “Which factors help authors produce the highest impact research?

Collaboration, journal and document properties” Journal of Informetrics 7: 861-873. Available at

http://dx.doi.org/10.1016/j.joi.2013.08.006

• “… the impact factor of the journal in which articles

appeared was the primary predictor of the

citations that they accrued”

Haslam, N. and Koval, P. (2010) “Predicting long-term citation impact in articles in social and personality

psychology” Psychological Reports 106(3) Jun 2010. Available at http://dx.doi.org/10.2466/pr0.106.3.891-900

http://dx.doi.org/10.1016/j.joi.2013.08.006

http://dx.doi.org/10.2466/pr0.106.3.891-900

http://dx.doi.org/10.2466/pr0.106.3.891-900

http://dx.doi.org/10.2466/pr0.106.3.891-900

http://dx.doi.org/10.2466/pr0.106.3.891-900

http://dx.doi.org/10.2466/pr0.106.3.891-900


• “…a small but significant correlation between journal

rank and future citations can be observed … stems

from visibility effects due to the influence of the IF on

reading habits (Lozano et al., 2012), rather than from

factors intrinsic to the published articles … [but] the

correlation is so weak that it cannot alone account

for the strong correlation between retractions and

journal rank”

Björn Brembs, Katherine Button and Marcus Munafò (2013) “Deep impact: unintended consequences of journal

rank” Frontiers in Human Neuroscience 7: 291. Available at http://dx.doi.org/10.3389/fnhum.2013.00291

Citing

Lozano, G. A., Larivière, V., and Gingras, Y. (2012). “The weakening relationship between the impact factor and

papers' citations in the digital age” J. Am. Soc. Inf. Sci. Technol. 63, 2140–2145. Available at

http://dx.doi.org/10.1002/asi.22731

http://dx.doi.org/10.3389/fnhum.2013.00291

http://dx.doi.org/10.3389/fnhum.2013.00291




Fang, F. C., & Casadevall, A. (2011). Retracted Science and the Retraction Index

. Infection and Immunity, 79(10), 3855–3859. http://doi.org/10.1128/IAI.05661-11

http://doi.org/10.1128/IAI.05661-11

http://doi.org/10.1128/IAI.05661-11

http://doi.org/10.1128/IAI.05661-11

http://doi.org/10.1128/IAI.05661-11

Impact ?

Place of Publication: Open Access

• Boost the potential visibility and access of research,

to a wider audience

• Removes research from behind paywall barriers for

existing audience

• Enables authors to retain their rights and more easily

share via social media, email etc.


• 4633 articles across ecology, applied

mathematics, sociology and economics.

• 2280 were open access, and had an

average citation count of 9.04

• 2353 were subscriptions journals, and had

an average citation count of 5.76.

Norris, M. (2008) “The citation advantage of open access articles” Thesis. Available at

https://dspace.lboro.ac.uk/2134/4089




SPARC Europe: Open Access Citation Advantage Service

http://sparceurope.org/oaca/

Total number of studies so far 70

Studies that found a citation advantage 46

Studies that found no citation advantage 17

Studies that were inconclusive, found non-

significant data or measured other

things than citation advantage for articles

7




SPARC Europe: Open Access Citation Advantage Service


Total number of studies so far 70

Studies that found a citation advantage 46

Studies that found no citation advantage 17

Studies that were inconclusive, found non-

significant data or measured other

things than citation advantage for articles

7

Correlation ≠ Causation

evidence of a positive correlation between citation count

and open access (gold or green)…

but…

• [Cause] Because research is more discoverable and

accessible

• [Cause] Because authors choose to make their best

(most citable) research open access



Part 5 Altmetrics

(in brief)

Altmetrics

“Unlike the JIF, altmetrics reflect the impact

of the article itself, not its venue. Unlike

citation metrics, altmetrics will track impact

outside the academy, impact of influential

but uncited work, and impact from sources

that aren’t peer-reviewed.“

http://altmetrics.org/manifesto/

Altmetric.com

http://www.altmetric.com/details.php?citation_id=103551

Learning Outcomes for the session

• Increased awareness of the various journal and author

metrics available.

• Understanding of how some key indicators (JIF, H-index)

are calculated

• Understanding of the key issues around the use of these

metrics and what research behaviours might be

incentivised.

• Awareness of the potential opportunities for exploring

wider academic and non-academic impact of publications

from altmetric tools available.

Image Credits

[Slide 6] Via Flickr Creative Commons, by demonsub. Original available at http://www.flickr.com/photos/66397367@N05/6993902269

[Slide 13] Via Flickr Creative Commons, by Kirsty Andrews. Original available at http://www.flickr.com/photos/47745961@N08/5169765739

[Slide 31,44,59] Via Flickr Creative Commons, by © Stuti Sakhalkar. Original at http://www.flickr.com/photos/theblackcanvas/2945878325/

[Slide 49] Via Flickr Creative Commons, by Alan Cleaver. Original available at http://www.flickr.com/photos/11121568@N06/4122176776/

[Slide 22] ‘Vitae®, © 2010 Careers Research and Advisory Centre (CRAC) Limited‘ Available at www.vitae.ac.uk/rdf

http://www.flickr.com/photos/66397367@N05/6993902269

http://www.flickr.com/photos/kirstea/

http://www.flickr.com/photos/47745961@N08/5169765739


http://www.flickr.com/photos/11121568@N06/4122176776/

Measuring

Researcher

Development Vitae Researcher Development Framework [see image credits]

Durham University Library: Supporting your research

Contact Details

Academic Liaison Librarian (Researcher Support) [email protected] Phone: +44 (191) 334 2961

mailto:[email protected]

durham leading research programme: academic impact

Education