an overview of main bibliometric indicators: dag w. aksnes data sources, methods and applications...
TRANSCRIPT
An overview of main bibliometric indicators:
Dag W. Aksnes
Data sources, methods and applications
Nordic Institute for Studies in Innovation, Research and Education
Common indicators
Number of publications used as an indirect measure of knowledge production
The extent to which the publications have been cited in the subsequent scientific literature used as an indicator scientific impact and international visibility
Co-authorship used as an indicator of collaboration, e.g. the extent of international collaboration
3
Common indicators in bibliometric studies of research
Indicator Data
Research output Publications per unit
Research profile Publications per field
Scientific impact Number of citations
Research collaboration Co-authorship
4
Common data sources for bibliometric indicators
Web of Science (traditionally)
Scopus (increasingly)
Various field specific databases (less often, field studies)
Google scholar (increasingly, but controversial)
CRIS-databases (new options)
5
Database requirements for bibliometric indicators (simplified)
Requirements WoS Scopus Google Scholar
CRIStin CRIS (prior)
Adapted for bibliometric studies + + - 0 -
Complete indexing of all authors and author addresses
+ + - 0, + -
Citation counts + + + - -
Transparent and justified selection procedure of source material
0,+ 0,+ - + -
Coverage of natural sciences and medicine
+ + + + 0
Coverage of engineering 0 0 + + 0
Coverage of A&H and SS - - + + 0
Coverage of books and monographs
- - + + 0
CRIS-data Norway
Data on all types of scientific/scholarly publications in all fields of research are collected trough a documentation system (CRIStin).
In the database the publication output is reported by the institutions as ordinary bibliographic references.
A dynamic authority record of controlled scientific and scholarly publication channels ensures that references to non-scientific publications are not entered into the system.
Publication data from professional bibliographic data sources (e.g. Web of Science) are imported to the documentation system in order to facilitate the registration of publications by the employees.
7
Database requirements for bibliometric indicators (simplified)
Requirements WoS Scopus Google Scholar
CRIStin CRIS (prior)
Adapted for bibliometric studies + + - 0 -
Complete indexing of all authors and author addresses
+ + - 0, + -
Citation counts + + + - -
Transparent and justified selection procedure of source material
0,+ 0,+ - + -
Coverage of natural sciences and medicine
+ + + + 0
Coverage of engineering 0 0 + + 0
Coverage of A&H and SS - - + + 0
Coverage of books and monographs
- - + + 0
9
Database requirements for bibliometric indicators (simplified)
Requirements WoS Scopus Google Scholar
CRIStin CRIS (prior)
Adapted for bibliometric studies + + - 0 -
Complete indexing of all authors and author addresses
+ + - 0, + -
Citation counts + + + - -
Transparent and justified selection procedure of source material
0,+ 0,+ - + -
Coverage of natural sciences and medicine
+ + + + 0
Coverage of engineering 0 0 + + 0
Coverage of A&H and SS - - + + 0
Coverage of books and monographs
- - + + 0
Publication types* in five major areas, Norwegian universities
Natural Sci
Health Sci
Engineering
Social Sci
Humanities
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ISI articles Non-ISI articles Articles in books Books
*) Weighted: 1 book = 5 articles (N=24,154)
11
Coverage of 50 000 journal articles by Norwegian authors*
Health Sciences Natural Sciences Engineering Social Sciences Humanities0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Scopus Web of Science
*) Accredited scientific and scholarly journals, CRIStin-data 2005-2012, Source: Gunnar Sivertsen (preliminary analysis)
Proportion of WoS indexed publications, selected SSH-departments, University of Bergen
Dep of Biological and Medical Psychology
Dep of Economics
Dep of Education and Health
Dep of Psychosocial Science
Dep of Geography
Dep of Clinical Psychology
Dep of Social Anthropology
Dep of Sociology
Dep of Information Science and Media Studies
Dep of Archaeology, History, Culture studies and Religion
Dep of Administration and Organization Theory
Dep of Comparative Politics
Dep of Philosophy
Dep of Foreign Languages
Dep of Linguistic, Literary and Aesthetic studies
0% 10% 20% 30% 40% 50% 60% 70% 80%
13
Common indicators in bibliometric studies of research
Indicator Data
Research output Publications per unit
Research profile Publications per field
Scientific impact Number of citations
Research collaboration Co-authorship
Addresses are the basis for indicators of:
-Publication output per country or research unit
15
Total publication output 2012 (WoS)
USAChin
a
Great
Brit
ain
Germ
any
Japa
n
Franc
e
Canad
a
South
Kor
ea
Austra
lia
Nethe
rland
s
Switzer
land
Sweden
Belgium
Denm
ark
Austri
a
Norway
Finlan
d
Irelan
d0
50000
100000
150000
200000
250000
300000
350000
400000
National publication profile biology (Norway)
University of Bergen9%
University of Oslo9%
University of Tromsø10%
UMB13%
NTNU9%
Other HE-institutions10%
Industry5%
IMR7%
Nofima6%
NINA5%
Institute sector - other insti-tutes14%
Hospitals2%
19
Counting of publications – methodological issues
The question of how to handle papers involving co-authorship in general and international co-authorship in particular has been a recurrent issue in the literature on the use of bibliometric indicators
Whole counting and fractional counting have been considered complementary methods
According to Moed (2005), the integer or whole count method can be interpreted to measure participation, while the fractional counting method measures the number of papers creditable to a country.
20
Counting of publications – methodological issues
Introduction of additional weighting systems, e.g. of first-authorship
Weighting of books
Publication types to be included
Publication channels - two levels
Level 2- is giving extra weight includes only the leading and most selective international journals and publishers - 20 % of the total
Level 1 80% of publications -”normal” wheight
In the Norwegian system journals and publishers are divided into two levels
Funding model Norway: Weigted publication points based on publication level and publication channel:
Level 1 Level 2
– Articles in scientific/scholarly journals: 1 3 – Articles in scientific/scholarly books: 0.7 1 – Monographs: 5 8
23
Common indicators in bibliometric studies of research
Indicator Data
Research output Publications per unit
Research profile Publications per field
Scientific impact Number of citations
Research collaboration Co-authorship
Journal titles are the basis for indicators of:
- Allocation of publications to fields of research- Research profiles- Field normalization
25
Number of articles* by field, 2005-2009
Ag
ricu
lture
Bio
log
y
Bio
me
dic
ine
Ch
em
istr
y
En
gin
ee
rin
g &
M
ate
ria
ls S
...
Ge
osc
ien
ces
He
alth
Sci
e...
Ph
ysic
s &
M
ath
em
0
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1400
1500
1600
1700
University of Bergen
University of Oslo
*) Indexed in Web of Science
26
Relative Specialization Index for ‘traditional’ universities (blue line equals the world average) (2005-09)
UiB UiO
NTNU UMB
27
Common indicators in bibliometric studies of research
Indicator Data
Research output Publications per unit
Research profile Publications per field
Scientific impact Number of citations
Research collaboration Co-authorship
Citations are the basis for indicators of:
- Impact (field normalized)
Citation rates by field (2005, 5 year-window)
Discipline Cit/pub Uncited Discipline Cit/pub UncitedMolecular Biology & Genetics 22.4 4 %Geosciences 8.8 12 %
Immunology 20.7 3 %Physics 8.4 17 %
Microbiology 15.6 4 %Plant & Animal Science 6.9 16 %
Biology & Biochemistry 15.1 7 %Agricultural Sciences 6.9 16 %
Space Science 14.2 13 %Economics-Business 5.6 20 %
Clinical Medicine 12.5 10 %Social Sciences, general 5.0 22 %
Chemistry 10.7 13 %Engineering 4.6 26 %
Environment/Ecology 10.3 9 %Mathematics 3.3 29 %
Psychiatry/Psychology 10.2 11 %Computer Science 2.9 48 %
Number of citations per year per article
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 250
0.5
1
1.5
2
2.5
3
3.5
4
Agriculture Economics
Molecular biology
Number of years after publication
Av
g n
um
be
r o
f c
ita
tio
ns
pe
r a
rtic
lel
Cumulative share of citations and publications, % (2005 - 5 years cit. win)
0 10 20 30 40 50 60 70 80 90 1000
10
20
30
40
50
60
70
80
90
100
Share of journal articles
Sh
are
of
cit
ati
on
s
Citations counts – different sources
ARTL_TITLE JRNLFULL VOL YEARCitations - Web
of Science Citations - ScopusCitations - Google
scholarA macro study of self-citation SCIENTOMETRICS 56 2003 90 102 170
Characteristics of highly cited papersRESEARCH EVALUATION 12 2003 57 62 95
Peer reviews and bibliometric indicators: a comparative study at a Norwegian university
RESEARCH EVALUATION 13 2004 49 54 109
Does self-citation pay? SCIENTOMETRICS 72 2007 41 48 66
Scientific productivity and group size: A bibliometric analysis of Norwegian microbiological research SCIENTOMETRICS 49 2000 32 38 57
The effect of highly cited papers on national citation indicators SCIENTOMETRICS 59 2004 30 32 58
Citation rates and perceptions of scientific contribution JASIST 57 2006 29 37 74
Validation of bibliometric indicators in the field of microbiology: A Norwegian case study SCIENTOMETRICS 49 2000 16 13 18Researchers' perceptions of citations RESEARCH POLICY 38 2009 15 17 21
When different persons have an identical author name. How frequent are homonyms? JASIST 59 2008 9 12 14
The Structure and Development of Polar Research (1981-2007): a Publication-Based Approach
ARCTIC ANTARCTIC AND ALPINE RESEARCH 41 2009 4 4 8
Ranking national research systems by citation indicators. A comparative analysis using whole and fractionalised counting methods
JOURNAL OF INFORMETRICS 6 2012 4 6 10
Are Female Researchers Less Cited? A Large-Scale Study of Norwegian Scientists
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY 62 2011 2 3 9
Science policy and the driving forces behind the internationalisation of science: The case of Norway
Science and Public Policy 35 2008 4 8
Total 378 432 717
Google scholar vs Web of science
0 10 20 30 40 50 60 70 80 90 1000
20
40
60
80
100
120
140
160
180
f(x) = 1.8777801425987 xR² = 0.982384912054357
Web of Science
Go
og
le s
ch
ola
r
Relative citation index
Relative citation index field = (Observed Citation Rate/World Citation Average of Subfield)X100
Relative citation index journal = (Observed Citation Rate/Average citation rate of journal)X100
Example (shown for one article only):
– A regular journal article in Acta Chrystallographica E, 2005 has been cited 3 times.
– The articles published in Acta Chrystallographica E were in contrast cited 1.74 times on average this year.
– The Relative citation index – journal is: (3/1.74)*100 = 172. – The world-average citation rate for the subfield which this journal is
assigned to is 3.7 for articles published this year. The Relative citation index – field is: (3/3.7)*100 = 81
36
Field normalized relative citation indexes, 2008-2011
Switzerland
Denmark
Netherlands
Great Britain
USA
Sweden
Belgium
Norway
Canada
Ireland
Finland
Australia
Austria
Germany
France
China
South Korea
Japan
50 60 70 80 90 100 110 120 130 140 150 160
37
Field normalized relative citation indexes of universities and university hospitals (2005-2008) across subject fields
Norway 1.22 1.17 0.96 0.93 1.05 1.12 1.09 1.23 1.08
Norwegian University of Science and Technology
1.16 1.05 1.13 0.87 1.02 0.51 1.03 1.42 1.07
University of Bergen 1.11 1.11 0.94 0.97 1.27 1.38 1.08 1.03 1.11
University of Oslo 1.14 1.32 1.00 0.95 1.07 1.12 1.12 1.20 1.11
University of Stavanger 0.69 0.75
University of Tromsø 1.54 1.28 0.79 1.16 1.01 0.94 0.92 1.04
St.Olavs Hospital 1.08 1.07
University Hospital North Norway 1.05 1.02
University of Bergen Hospitals 0.67 1.12 1.04
University of Oslo Hospitals 1.02 1.13 1.11
University
Agri
culture
, Fis
heri
es &
Fore
stry
Bio
logy
Bio
medic
ine
Chem
istr
y
Engin
eeri
ng &
Mate
rials
Sci
ence
s
Geosc
ience
s
Hea
lth S
cien
ces
Physi
cs &
Math
em
atics
Tota
l
H-index
Articles ranked by decreasing citation numbers. A scientist with an index of h has published h papers each of which has been cited by at least h times – can be manually determined using citation databases or using
automatic tools– attempts to measure both the scientific productivity and the
apparent scientific impact of a scientist– Reduces the effect of highly cited papers – But not without shortcomings: No field normalisation. H [read: age]
index.
Journal impact factor
One of the most well-known bibliometric indicators
Originally introduced by Eugene Garfield as a measure of the frequency with which the average article in a journal has been cited
Average citation rate for 1 and 2 years old articles published in a particular journal:
Number of citations received in year T to the articles published in a journal during the years T-1 og T-2 divided by the number of articles published in the journal during the years T-1 og T-2
40
Common indicators in bibliometric studies of research
Indicator Data
Research output Publications per unit
Research profile Publications per field
Scientific impact Number of citations
Research collaboration Co-authorship
Addresses are the basis for indicators of:
-Publication activity per university-Research collaboration (combinations of addresses in publications)
Map of scientific collaborations 2005 to 2009
Norwegian articles with and without international co-authorship, 1981-2009
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
0500
1,0001,5002,0002,5003,0003,5004,0004,5005,0005,500
Without international collaboration
With international collaboration
Kilde: Citation Report for Norway
The institutional national and international collaboration profile of the University of Bergen based on co-authorship
Fac of
Math
em &
Natu
ral Sci
Fac of
Med
icine
and D
entis
try
Fac of
Psy
chology
Fac of
Soc
ial S
cienc
es
Facult
y of H
uman
ities
Fac of
Law-5%
0%5%
10%15%20%25%30%35%40%45%50%55%60%65%70%75%80%
International National
The international collaboration profile of the University of Bergen based on co-authorship
The institutional international collaboration profile of the University of Bergen based on co-authorship
Max-Planck-Gesellschaft
Universidadede São Paulo
Copenhagen Univ
Aarhus Univ
Centre national de la recherche scientifique,
France
Helsinginyliopisto
University College London
University of Oxford
National Health Service, UK
King's College London, U. of London
Vrije UniversiteitAmsterdam
Russian Academy of Sciences
Uppsala Univ
Karolinska Inst
GothenbourghUniv
Lunds Univ
Sahlgrenska UnivHospital
Umea Univ
Stockholms Univ
University of Cape Town
Univerzita Karlova v Praze, Prague
MakerereUniversity, Uganda
Harvard University
University of California, San Diego
UiB
47
Evaluation of research performance - an important application of bibliometrics
Growing, often controversial, policy interest to use bibliometric techniques as output measures of research productivity and impact
Bibliometrics evolved to a standard tool in science policy and research evaluation
Science policy applications
Monitoring research systems
Evaluations
Funding
Applications in research evaluations
Levels of analyses– Country– Institutions– Departments– Divisions/research groups– (Individuals – anonymous presentation)
Bibliometric indicators for funding purposes
In 2004, Norway implemented a bibliometric model for performance based budgeting of research institutions– The funding of these institutions is now partially based on the
measurement of their scientific and scholarly publishing – National bibliographic database (CRIStin) used for this
purpose– Similar system for research institutes and hospitals
Questions – comments?
www.nifu.no