the multi-dimensional research assessment matrix
DESCRIPTION
The Multi-Dimensional Research Assessment Matrix . Henk F. Moed Elsevier, Amsterdam, the Netherlands Seminar Research Evaluation in Practice, National Geographic Society, Washington DC, 17 October 2012. Short CV Henk F. Moed. Contents. (o) Bibliometrics makes sense. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/1.jpg)
The Multi-Dimensional Research Assessment Matrix
Henk F. MoedElsevier, Amsterdam, the Netherlands
Seminar Research Evaluation in Practice, National Geographic Society, Washington DC, 17 October 2012
![Page 2: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/2.jpg)
Short CV Henk F. MoedYears Position
1981-2009
Staff member at Centre for Science and Technology Studies (CWTS), Leiden Univ.
2009 Professor of Research Assessment Methodologies at Leiden University
2010 – Sept 2012
Elsevier, SciVal Dept. Senior Scientific Advisor
As from Sept 2012
Elsevier, AGRM Dept. Head of Informetric Research Group
![Page 3: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/3.jpg)
Contents
1 Boundaries of the playing field; rules of the game
2 The multi-dimensional research assessment matrix
3 Towards more sophisticated indicators: combining big data sets
![Page 4: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/4.jpg)
(o)Bibliometrics makes sense
![Page 5: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/5.jpg)
Which country has these main collaborators?
Main collaborators
UK
China
Germany
Canada
USA
![Page 6: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/6.jpg)
Which country has these main collaborators?
Main collaborators
Argentina
USA
Portugal
France
Chile
Brazil
![Page 7: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/7.jpg)
Which country has these main collaborators?
Main collaborators
Thailand
India
Singapore
Iran
UK
Malaysia
![Page 8: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/8.jpg)
Which country has these main collaborators?
Main collaborators
France
Hungary
Germany
Italy
Bulgaria
Romania
![Page 9: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/9.jpg)
Which country has these main collaborators?
Main collaborators
UK
China
USA
Nigeria
Australia
Netherlands
South Africa
![Page 10: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/10.jpg)
Contents
1 Boundaries of the playing field; rules of the game
2 The multi-dimensional research assessment matrix
3 Towards more sophisticated indicators: combining big data sets
![Page 11: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/11.jpg)
(i) Bibliometric research assessment is
potentially a proper tool to consolidate academic freedom
![Page 12: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/12.jpg)
(ii) Bibliometric tools help establishing
a longer term perspective in research funding
![Page 13: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/13.jpg)
(iii) One must be cautious using
“societal benefit” as an assessment criterion of basic research:
it can not be measured in a politically neutral way
![Page 14: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/14.jpg)
(iv) Citations measure scientific-
scholarly impact rather than quality or validity
![Page 15: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/15.jpg)
(v) Citation counts in social sciences
and humanities may be influenced by political ideologies
![Page 16: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/16.jpg)
Citation impact and ideology
0
500
1,000
1,500
2,000
2,500
3,000
CITATION YEAR
NR
CIT
ES
MARX
LENINFall of the Berlin wall
in Nov. 1989
![Page 17: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/17.jpg)
(vi)The future of research assessment
lies in the intelligent combination of
metrics and peer review
![Page 18: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/18.jpg)
(vii)Case study on funding policies of a National Research Council reveals
biases in peer review
![Page 19: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/19.jpg)
Affinity Applicants – Evaluation Committee
0 Applicants are/were not member of any Committee
1 Co-applicant is/was member of a Committee, but not of the one evaluating
2 First applicant is/was member of a Committee, but not of the one evaluating
3 Co-applicant is member of the Committee(s) evaluating the proposal
4 First applicant is member of the Committee(s) evaluating the proposal
![Page 20: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/20.jpg)
For 15 % of SUBMITTED applications an applicant is a member of the evaluating Committee (Affinity=3, 4)
0
10
20
30
40
50
60
70
% A
PPLI
CATI
ONS
AFFINITY APPLICANTS-COMMITTEEProjects 63.2 10.2 11.5 5.9 9.1
0 1 2 3 4% S
UB
MIT
TED
A
PPLI
CAT
ION
S
AFFINITY APPLICANT - COMMITTEE
![Page 21: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/21.jpg)
Probability to be granted increases with increasing affinity applicants-Committee
30
40
50
60
70
80
% G
RA
NTE
D A
PPLI
CA
TON
S
AFFINITY APPLICANTS-COMMITTEE
Projects 37.0 46.9 60.1 62.6 74.0
0 1 2 3 4
AFFINITY APPLICANT - COMMITTEE
% G
RA
NTE
D
APP
LIC
ATIO
NS
![Page 22: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/22.jpg)
Logistic regression analysis: Affinity Applicant-Committee has a significant effect
upon the probability to be granted
MAXIMUM-LIKELIHOOD ANALYSIS-OF-VARIANCE TABLE (N=2,499) Source DF Chi-Square Prob ------------------------------------------------------------- INTERCEPT 1 18.47 0.0000 CITATION IMPACT APPLICANT 3 26.97 0.0000 ** Rel transdisc impact applicant 1 0.29 0.5926 AFFINITY APPLICANT-COMMITTEE 2 112.50 0.0000 ** Sum requested 1 45.47 0.0000 ** Institution applicant 4 25.94 0.0000 ** LIKELIHOOD RATIO 199 230.23 0.0638
![Page 23: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/23.jpg)
(viii)Data must be accurate and verifiable
![Page 24: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/24.jpg)
(ix)Assessed researchers must have the
opportunity to check data and comment on outcomes
![Page 25: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/25.jpg)
(x)A framework is needed to characterize and position
bibliometric indicators and products
![Page 26: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/26.jpg)
Contents
1 Boundaries of the playing field and rules of the game
2 The multi-dimensional research assessment matrix
3 Towards more sophisticated indicators: combining big data sets
![Page 27: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/27.jpg)
The Multi-Dimensional Research Assessment Matrix
Expert Group on the Assessment of University-Based Research (AUBR, 2010)
![Page 28: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/28.jpg)
Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment
Purpose Output dimensions
Bibliometric indicators
Other indicators
Individual Allocate resources
Research productivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journal citation impact
Patents, licences, spin offs
Department Monitor research programs
Innovation and social benefit
Actual citation impact
Invitations for conferences
Institution Increase regional engagement
Sustainabi-lity & Scale
Internat. co-authorship
External research income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com-pletion rates
![Page 29: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/29.jpg)
Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment
Purpose Output dimensions
Bibliometric indicators
Other indicators
Individual Allocate resources
Research productivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journal citation impact
Patents, licences, spin offs
Department Monitor research programs
Innovation and social benefit
Actual citation impact
Invitations for conferences
Institution Increase regional engagement
Sustainabi-lity & Scale
Internat. co-authorship
External research income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com-pletion rates
Read column-
wise
![Page 30: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/30.jpg)
Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment
Purpose Output dimensions
Bibliometric indicators
Other indicators
Individual Allocate resources
Research productivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journal citation impact
Patents, licences, spin offs
Department Monitor research programs
Innovation and social benefit
Actual citation impact
Invitations for conferences
Institution Increase regional engagement
Sustainabi-lity & Scale
Internat. co-authorship
External research income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com-pletion rates
![Page 31: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/31.jpg)
Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment
Purpose Output dimensions
Bibliometric indicators
Other indicators
Individual Allocate resources
Research productivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journal citation impact
Patents, licences, spin offs
Department Monitor research programs
Innovation and social benefit
Actual citation impact
Invitations for conferences
Institution Increase regional engagement
Sustainabi-lity & Scale
Internat. co-authorship
External research income
Research field
Promotion& hiring
Research infrastruct.
citation ‘prestige’
PhD com-pletion rates
![Page 32: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/32.jpg)
MD-RAM: Example 1
Individual Hiring/promotionProductivity &
impactPhD date, place,
supervisor;Invitations for
conferences
Publications in international jrnls;
Actual citation impact
![Page 33: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/33.jpg)
Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment
Purpose Output dimensions
Bibliometric indicators
Other indicators
Individual Allocate resources
Research productivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journal citation impact
Patents, licences, spin offs
Department Monitor research programs
Innovation and social benefit
Actual citation impact
Invitations for conferences
Institution Increase regional engagement
Sustainabi-lity & Scale
Internat. co-authorship
External research income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com-pletion rates
![Page 34: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/34.jpg)
MD-RAM: Example 2
Research group Monitoring
research program Scientific impact Collaborations
Topicality
Publications in international jrnls;
(Trend in) actual citation impact
![Page 35: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/35.jpg)
Research assessment methodologies: Important considerations
• Methodology must be fit-for-purpose
• What is the primary “problem” to be solved?
• Be aware of unintended effects
• Change a methodology every 5-10 years
• What is an acceptable “error rate”?
• Wrong in individual cases benificiary for the system as a whole
![Page 36: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/36.jpg)
Contents
1 Boundaries of the playing field and rules of the game
2 The multi-dimensional research assessment matrix
3 Towards more sophisticated indicators: combining big data sets
![Page 37: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/37.jpg)
Journal full text data
Journal usage data
Patents
Trade jrnls
Social media
Journal articles + citations
Books
Conference Procs
Newspapers
Unit of assess- ment
![Page 38: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/38.jpg)
(a)Downloads vs. Citations
What do full article downloads measure?
![Page 39: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/39.jpg)
Authors vs. readers
Authors Readers
?
![Page 40: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/40.jpg)
Hypothesis on degree of correlation between downloads and citations
Readers
Authors
ReadersAuthors
Strong Weak
![Page 41: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/41.jpg)
Usage vs. citations per main field
Societal
Scientific
?PSYCHOL
![Page 42: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/42.jpg)
(b)Patent citations to journal articles:
The technological impact of research
![Page 43: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/43.jpg)
The Technological Impact of Library Science Research: A Patent Analysis [Halevi et al, 2012]
PATENTS (TotalPatent)
42 LIBRARY SCIENCE JOURNALS (Scopus)
Citations by patent examiners and inventors
![Page 44: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/44.jpg)
The patents focus on electronic information administration, navigation, and products and services management in commercial systems.
The articles feature information retrieval and indexing, information and documents management systems which pertain to electronic and digital libraries development
Cited articles: keywords in titles
Citing patents: keywords in titles
![Page 45: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/45.jpg)
(c) International scientific migration
![Page 46: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/46.jpg)
International migration vs. co-authorship
Relationship Definition Comment
International co-authorship
Authors from institutions located in different countries jointly publish a paper
Country relates to where authors work, NOT to their nationality International
migrationA scientific author moves from one country to another
![Page 47: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/47.jpg)
Map of countries with Ratio migration/collaboration > 1.2
![Page 48: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/48.jpg)
TO COUNTRY
FROM Country
No. Co-authored papers
No. Migrating authors
Ratio % Migration / % Co-authorship
PAK IND 276 118 3.6PRT BRA 1,971 423 3.4IND PAK 276 96 3.3NLD IRN 492 80 3.1USA IND 12,013 3,307 2.8NLD PAK 145 21 2.8CHN TWN 3,979 1,048 2.6USA IRN 3,039 780 2.6BRA PRT 1,971 352 2.4MYS NGA 122 31 2.1
Language similarity drives migration stronger than it drives co-authorship
Political tensions affect migration less than they affect co-authorship
![Page 49: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/49.jpg)
(d)Citation context analysis
Combining citation data from Scopus with full text article data from ScienceDirect
![Page 50: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/50.jpg)
The use of contextual citations analysis to disclose the thematic and conceptual
flow of cross- disciplinary research:
the case of the Journal of Informetics 2007
(Gali Halevi et al., to be published, 2012)
![Page 51: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/51.jpg)
Emerging sectional themes (OUTSIDE DISCIPLINE)
The themes sequence in the word clouds below might suggest that the individual output evaluation done by structured peer review leads to an acknowledgment of
the importance and evolution of networks rather than individuals
INTRODUCTION FINDINGS & DISCUSSION
CONCLUSIONS
![Page 52: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/52.jpg)
(e)Book Citation Index
Citation flows between books and journals
![Page 53: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/53.jpg)
A Scholarly Book Citation Index: Approaches
a) Add selected book series
b) Add books from selected publishers
c) Add selected individual book titles
![Page 54: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/54.jpg)
Thank you for your attention!
![Page 55: The Multi-Dimensional Research Assessment Matrix](https://reader035.vdocuments.net/reader035/viewer/2022062810/56815cb0550346895dcab013/html5/thumbnails/55.jpg)
Elsevier Bibliometric Research Program:
www.ebrp.elsevier.com