comparing scientific performance across disciplines: methodological and conceptual challenges
TRANSCRIPT
![Page 1: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/1.jpg)
Comparing scientific performance across disciplines: Methodological and conceptual challengesLudo WaltmanCentre for Science and Technology Studies, Leiden University
7th International Conference on Information Technologies and Information SocietyNovember 5, 2015
![Page 2: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/2.jpg)
2
THE ranking
![Page 3: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/3.jpg)
3
Shanghai ranking
![Page 4: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/4.jpg)
4
Website University of Ljubljana
![Page 5: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/5.jpg)
5
Impact factors
![Page 6: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/6.jpg)
6
Centre for Science and Technology Studies (CWTS)• Research center at Leiden
University focusing on science and technology studies
• History of more than 25 years in bibliometric and scientometric research
![Page 7: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/7.jpg)
7
Key challenges in measuring scientific performance1. Defining scientific performance2. Designing sensible metrics3. Correcting for field differences4. Quantifying productivity5. Assessing validity of metrics6. Balancing simplicity, accuracy, and robustness to
gaming
What are Slovenia’s research strengths?
![Page 8: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/8.jpg)
8
Challenge 1: Defining scientific performance
![Page 9: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/9.jpg)
9
Indicators of scientific performance• Publications:
– Total– Per faculty– Per student– Interdisciplinary– International collaboration– Nature and Science
• Citations:– Total– Per publication– Per faculty– Highly cited researchers
• Reputation survey• Others:
– Nobel Prizes/Field Medals– PhDs awarded– PhDs awarded per faculty– Post-doc positions– Research income
![Page 10: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/10.jpg)
10
What do we mean by scientific performance?Size-dependent concept of scientific performance:• Overall contribution of a research unit to science• Total number of ‘performance points’ (e.g.,
publications, citations, expert recommendations, awards)
Size-independent concept of scientific performance:• Contribution of a research unit to science relative to
available resources• Number of ‘performance points’ divided by
available resources (e.g., number of faculty, research budget)
![Page 11: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/11.jpg)
11
Indicators of scientific performance• Publications:
– Total– Per faculty– Per student– Interdisciplinary– International collaboration– Nature and Science
• Citations:– Total– Per publication– Per faculty– Highly cited researchers
• Reputation survey• Others:
– Nobel Prizes/Field Medals– PhDs awarded– PhDs awarded per faculty– Post-doc positions– Research income
Size-dependent indicators Size-independent indicators
![Page 12: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/12.jpg)
12
Mixing up different concepts of scientific performance• Shanghai, THE, QS, and US News use composite
indicators• These composite indicators combine size-
dependent and size-independent indicators
It is unclear which concept of scientific performance is measured by Shanghai, THE, QS,
and US News
![Page 13: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/13.jpg)
13
Challenge 2: Designing sensible metrics
![Page 14: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/14.jpg)
Definition of the h-index
14
A scientist has index h if h of his papers have at least h citations each and the other papers have
at most h citations each
![Page 15: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/15.jpg)
15
Arbitrariness of the h-index
![Page 16: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/16.jpg)
Consistency requirement
16
If two scientists achieve the same performance improvement, their ranking relative to each other
should remain unchanged
![Page 17: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/17.jpg)
17
Inconsistency of the h-index
![Page 18: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/18.jpg)
18
Newly proposed metric: Relative Citation Ratio
This metric has the peculiar property
that receiving additional citations
may cause the metric to decrease
rather than increase
![Page 19: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/19.jpg)
19
Challenge 3: Correcting for field differences
![Page 20: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/20.jpg)
20
Differences in citation density between fields
0 1 2 3 4 5 6 7 8 9 100
5
10
15
20
25
30
35
publication age in years
aver
age
num
ber
of c
itatio
ns p
er p
ublic
atio
n
biochemistry & molecular biologycardiac & cardiovascular systemschemistry, analyticalsurgeryeconomicsphysics, appliedmathematics
![Page 21: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/21.jpg)
21
Differences in citation density between fields
![Page 22: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/22.jpg)
22
Field-normalized indicators
Elsevier SciVal
THE ranking
![Page 23: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/23.jpg)
23
Differences in citation density within fields (clinical neurology)
![Page 24: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/24.jpg)
24
About 4000 fields of science in the CWTS Leiden Ranking
Social sciences and
humanities
Biomedical and health sciences
Life and earth sciences
Physical sciences
and engineering
Mathematics and computer science
![Page 25: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/25.jpg)
25
CWTS Leiden Ranking
![Page 26: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/26.jpg)
26
CWTS Leiden Ranking
![Page 27: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/27.jpg)
27
SNIP: Source normalized impact per paper
![Page 28: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/28.jpg)
28
Differences in citation density within fields (clinical neurology)
![Page 29: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/29.jpg)
29
Differences in reference density within fields (clinical neurology)
![Page 30: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/30.jpg)
30
Citation density vs. reference density (clinical neurology)
Citation density Reference density
![Page 31: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/31.jpg)
31
SNIP: Source normalized impact per paper• Impact per paper (IPP) of journal X in 2013:
• Source normalized impact per paper (SNIP) of journal X in 2013:
C11CIPP P P
i
PR1
SNIP
C
1 i i
Number of publications in journal X in 2010–2012
Number of citations in 2013 to publications in journal X in 2010–2012
Number of references in citing publication i to publications in 2010–2012
Be careful in using SNIP in the humanities!
![Page 32: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/32.jpg)
32
Challenge 4: Quantifying productivity
![Page 33: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/33.jpg)
33
Quantifying productivity
• Same resources as Univ. B• P = 1000• P(top 10%) = 200• PP(top 10%) = 20%
• Same resources as Univ. A• P = 2000• P(top 10%) = 300• PP(top 10%) = 15%
Univ. A Univ. B
• Taking into account that both universities have the same resources, it is clear that university B has performed better
• However, according to the PP(top 10%) indicator, university A has performed better
![Page 34: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/34.jpg)
34
Challenge 5: Assessing validity of metrics
![Page 35: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/35.jpg)
35
Correlating metrics with peer review
Correlating journal impact metrics with Norwegian expert
judgments
Correlating citation metrics with F1000 recommendations (post publication peer review)
![Page 36: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/36.jpg)
36
Correlating metrics with peer review
Correlating various metrics with UK REF expert judgments
![Page 37: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/37.jpg)
37
Difficulties of using peer review to validate metrics• Biases in peer review, e.g.:
– Gender– Interdisciplinarity– Conservatism
• Influence of metrics on peer judgment– Did REF expert panels truly make their decisions without taking
into account any metrics?
• Limited precision of peer review outcomes doesn’t allow accurate comparisons of metrics:– Norwegian expert judgments: 3 classes– F1000 recommendations: 3 classes– UK REF expert judgments: 5 classes
![Page 38: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/38.jpg)
38
Challenge 6: Balancing simplicity, accuracy, and robustnessto gaming
![Page 39: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/39.jpg)
39
![Page 40: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/40.jpg)
40
Suspicious editorials: Time trend
• At least 50 citations to publications in own journal in past 2 years
• At least 75% journal self citations
![Page 41: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/41.jpg)
41
Suspicious editorials: Effect on IF
![Page 42: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/42.jpg)
42
Coercive citing
“you cite Leukemia (once in 42 references). Consequently, we kindly ask you to add
references of articles published in Leukemia to your present article”
![Page 43: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/43.jpg)
43
Balancing simplicity, accuracy, and robustness to gaming
Simplicity
Accuracy
Robustness to gaming
IF vs. SNIP?
Exclude citations from
editorials?
Exclude journal self-citations?
![Page 44: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/44.jpg)
44
How to deal with these challenges?
![Page 45: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/45.jpg)
45
![Page 46: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/46.jpg)
46
Leiden Manifesto
1. Quantitative evaluation should support qualitative, expert assessment
2. Measure performance against the research missions of the institution, group or researcher
3. Protect excellence in locally relevant research4. Keep data collection and analytical processes
open, transparent and simple5. Allow those evaluated to verify data and analysis
![Page 47: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/47.jpg)
47
Leiden Manifesto
6. Account for variation by field in publication and citation practices
7. Base assessment of individual researchers on a qualitative judgement of their portfolio
8. Avoid misplaced concreteness and false precision9. Recognize the systemic effects of assessment
and indicators10.Scrutinize indicators regularly and update them
![Page 48: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/48.jpg)
48
Slovenia’s research strengths
![Page 49: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/49.jpg)
49
Top 5 Slovenian research strengths
Social sciences and
humanities
Biomedical and health sciences
Life and earth sciences
Physical sciences
and engineering
Mathematics and computer science
![Page 50: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/50.jpg)
50
Top 5 Slovenian research strengths
Keywords:Journals:
Institute:
acrylic acid; ethylene terephthalate; polypropylenesurface & coatings technology; j. of applied polymer science; plasma processes & polymersjozef stefan institute
Keywords:Journals:Institute:
cholesteric liquid crystal; holographic polymer; nanoparticlemolecular crystals & liquid crystals; physical review e; liquid crystalsjozef stefan institute
Keywords:Journals:Institute:
evolution; dilemma game; altruismj. of econ. behavior & organization; games & econ. behavior; j. of theoretical biologyuniv. of maribor
Keywords:Journals:Institute:
acidic medium; mild steel corrosion; hydrochloric acid solutioncorrosion science; int. j. of electrochemical science; corrosionjozef stefan institute
Keywords:Journals:Institute:
wood density; wood property; treeeur. j. of wood & wood products; holzforschung; forest products journaljozef stefan institute; nat. inst. of biology; univ. of ljubljana
![Page 51: Comparing scientific performance across disciplines: Methodological and conceptual challenges](https://reader037.vdocuments.net/reader037/viewer/2022110108/587eddf31a28abdb198b7dcb/html5/thumbnails/51.jpg)
51
Thank you for your attention!