hosted by the comenius university in bratislava · bratislava 10-11 october 2011 hosted by the...
TRANSCRIPT
ARRA’s experience in the identification
of top research teams:
©ARRA, October 2011
Ivan Ostrovský, Miroslav Medveď, Ferdinand Devínsky, Juraj Barta
the case study of the
Slovak Academy of Sciences
IREG-5 Forum
The National Rankings on the Rise
Bratislava 10-11 October 2011 Hosted by the Comenius University in Bratislava
CONCEPT:Fénix 2
2
Common and standardised system of periodic
assessment of R&D for HEIs and other research
institutions – basis of long-term institutional financing of
R&D
Concept of national and international excellence
International criteria for R&D assessment
Outputs - publications, citations, innovations........
Leading companies are hindered by the collaboration
with average/mediocre HEIs or research institutions.
Main Goals of the Project
SAS asked for the identification of
top and at the same time perspective
research teams
3
3 critical questions to achieve the goal:
How is a “research team” defined?
When is the team classified as “top”?
When the team has “good prospects”?
Principles
Team and Team Leader
Top team should be led by a top leader (leaders); however, a top
researcher is not necessarily the leader of the top team
Comparison is correct and accurate among the SAS
teams, but only in the same discipline/set of fields
No K.O. criteria applied, but complex assessment (not
“blind” scientometry), with needs for:
identifying the team‟s main scientific field
deciding whether a particular researcher is a member of the team or
not
assessing criteria in the context
4
Number of Articles (author)
> 10% of co-operative articles
Fyzio-VascDisOrd Kristek Cacanyiova Torok Cebova Zemancikova Malekova
František Kristek 27 15/56 4/15 10/37
Soňa Čačányiová (Kyselá) 15/94 16 3/19
Jozef Török 4/40 10 4/40 3/30
Martina Cebová 10/77 3/23 4/31 13
Anna Zemančíková 3/100 3
Magdaléna Maleková 5
Teams’ identification
Fyzio-VascDisOrd Kristek Cacanyiova Torok Cebova Zemancikova Malekova
František Kristek
Soňa Čačányiová (Kyselá)
Jozef Török
Martina Cebová
Anna Zemančíková
Magdaléna Maleková
Fyzio-VascDisOrd Kristek Cacanyiova Torok Cebova Zemancikova Malekova
František Kristek 27 15/56 4/15 10/37
Soňa Čačányiová (Kyselá) 15/94 16 3/19
Jozef Török 4/40 10 4/40 3/30
Martina Cebová 10/77 3/23 4/31 13
Anna Zemančíková 3/100 3
Magdaléna Maleková
Primary list: SAS‟s formal division (institutes/departments)
> 30% of co-operative articles
> 60% of co-operative articles
Level and degree of mutual collaboration
Author(s), with whom the other team members have the highest % of collaboration, are considered as team leaders
Team „Core‟ and „Non-core‟ members
Collaboration outside the institute/department, or outside SAS
AB
CD
EF
GH
IJ
KL
AB CD EF GH IJ KL PHYSIOLOGY
SAS structure
6
SLOVAK ACADEMY OF SCIENCES
PHYSICAL, SPACE,
EARTH, AND
ENGINEERING
SCIENCES
LIFE,
CHEMICAL,MEDICAL,
AND ENVIRONMENTAL
SCIENCES
SOCIAL SCIENCES,
HUMANITIES, ARTS
AND CULTURE
SECTION 1 SECTION 2 SECTION 3
First “filter”
In Sections 1 and 2 - more than 200 teams were identified
First selection criteria: (who is definitely NOT in the TOP)
Team leader has less than 10 WoK Articles since 2006
Team leader/most members have at most a part-time (no more
than 50%) contract with SAS
Modified H-index for particular scientific field (ModH) is lower
than 15
Borderline cases were also evaluated
63 teams matched these criteria 7
Basic Principles
Time scale – 2001 to 2010
9 steps to teams assessment – no K.O. criteria
Steps 1-2 focus on individual team members – their overall
performance and the contribution to the quality of the team
Steps 3-5 focus on the team‟s performance – quantity, quality
and the effectiveness of published papers
Steps 6-8 assess team‟s exceptional articles and their relative
impact
Step 9 describes the progress of the team during the
evaluated period
All comparisons were done against international benchmarks
„Avg citation per article‟ was also compared with the world‟s
best 35 universities (top 25 from: ARWU, USN&WR, TimesHE) 8
AnorChem-TeoChem Art Cit Cit/Art H-index DoB Age PhD 1st Art ActiveResm-quotientFTE
M OL [100] 70 3663 52,33 28 1956 55 1998 1990 27 1,04 100
MVG [100] 73 3526 48,30 28 1956 55 1990 1979 21 1,33 100
N J [53] 88 4856 55,18 30 1955 56 1995 1980 16 1,88 67,1
KO S [100] 8 53 6,63 4 1982 29 2009 2006 2 2,00 53,3
KE S [100] 8 144 18,00 6 1981 30 2009 2003 2 3,00 63,3
RE M [0] 10 74 7,40 5 1981 30 2009 2004 2 2,50 55,8
HO P [0] 16 160 10,00 9 1981 30 2008 2001 3 3,00 43,3
PA A [100] 1 1 1,00 1 1985 26 2010 1 1,00 13,3
MA E [outside SAV] 3 15 5,00 2 2006 5 0,40
KA M [outside SAV] 188 4941 26,28 36 1990 21 1,71
H-index 01-11 (Lead) 21
Art 01-10 Cit 01-10 Cit/Art 01-10W Avg Cit/ArtT/World BestUni ACAT/BestUniAvgCoAuthUnCit H-index 01-11 (Rest w/Lead) 9
63 1088 17,27 10,74 1,61 19,41 0,89 4,5 8 H-index 01-11 (Rest w/o Lead) 2
.1st Percentile 1st Percentile1st Decile2nd DecileAvg% 03-10H-index 01-10ModH 2 Angew. Chem. Int. Edit.
0 0 18 14 30,9 21 17 2 J. Am. Chem. Soc.
Art% Proc% Rev% 17 J. Chem. Phys.
33,17 26,46 12,59 1 J. Chem. Theor. Comput.
4
Art Cit Cit/Art W Avg Cit/ArtT/World
2001 7 175 25,00 17,62 1,42
2002 8 251 31,38 17,61 1,78
2003 7 122 17,43 16,07 1,08
2004 4 71 17,75 14,88 1,19
2005 10 178 17,80 13,09 1,36
2006 6 100 16,67 10,57 1,58
2007 6 54 9,00 8,35 1,08
2008 4 97 24,25 5,79 4,19
2009 5 31 6,20 2,98 2,08
2010 6 9 1,50 0,44 3,41
2011 3 0,00
2001-2010 63 1088 17,27 10,74 1,61
Phys. Chem. Chem. Phys.
26Top 5 Journals
0,00
0,50
1,00
1,50
2,00
2,50
3,00
3,50
4,00
4,50
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
Collected data were assessed according to four main
criteria:
publication productivity (quantity of outputs),
total influence (no. of citations, H-index),
effectiveness (avg cit/art, uncited articles),
relative impact (top papers, 1st percentile/decile)
Each of the abovementioned information is important for
different aspect of the research evaluation
Data evaluation
10
22 top teams were identified
Only Sections 1 and 2 are presented here
Section 3 (SSH) has different publication patterns (more on
Slide 13)
17 above-average teams were detected
Several “top-teams” criteria were not met
Summary 1
11
Summary 2
from 1800 assessed researchers in Sections 1 and 2,
~ 1000 are members of some team
questionnaire sent to 63 teams, covering some ~ 400
researchers
870 researchers do not belong to any team
580 researchers do not belong to top or above-average
teams
The majority are individual researchers (occasional pairs)
even the review of mutual citations has led to the same data
Problems with the performance evaluation using WoK,
InCites, ESI – due to the lack of data regarding SAS SSH
fields (except for several individuals within Psychology and
Economics, none with H>2)
Additional sources from SAS Library
Lack of proper scientometric data (average percentile, etc.)
Non-existing benchmarks (no systematic information about
works and citations outside main databases)
Section 3 (SSH)
13
Social Sciences in the world,
region and Slovakia
1994 SVK CZE HUN
2011 Art Cit top
1
top
10
Art Cit top
1
top
10
Art Cit top
1
top
10
Econ 1264 770 22 9 2927 2665 123 30 1033 2682 230 45
Psy 836 1608 41 23 1686 5941 383 72 1447 12442 466 132
2001-2010 World The most cited
Article
100th most cited
Article
Economics 883 239
Social Sciences 620 214
Psychology/Psychiatry 2037 399
Matematics 2280 144
Technical Sciences 1822 288
Due to the different publication patterns, it was impossible to
apply all the assessment methods/criteria used for Sections 1
and 2
Moreover, no benchmarks were available
Bibliometric analyses were possible only to some extent; and
the four main categories of evaluation (similar to those used
for Section 1 and 2) were applied:
Quantity: the number and the „quality‟ (categories: books,
articles, conf.papers, etc.) of publications, including articles and
books outside international databases
Global influence: number of citations for these publications
Effectiveness: an average number of citations per publication
Relative impact: publications cited more than 10/20/50/100
times (rough equivalent of papers in 20/10/1/0.1/.01 percentile)
Humanities Changes to evaluation methodology
16
Final list of teams
The final list of teams contains following information:
Name of the institute/department, where the majority of
team members are based
Team collaboration within SAS (with other
departments)
FTE of each team member
Team Leader(s) (including their H-index)
„Core‟/ „Non-core‟ team members
Team members outside SAS (if such exist)
18
Comparison of total data in a 10-year period in the
world
(selected ESI fields)
World benchmarks (average citation/article)
Fields 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 01-10
Biology&Biochem 29.64 27.36 24.84 21.97 18.12 14.21 10.98 7.22 3.62 0.52 15.85
Chemistry 17.62 17.61 16.07 14.88 13.09 10.57 8.35 5.79 2.98 0.44 10.74
Computer Science 7.66 7.93 5.35 3.99 3.51 2.51 3.26 2.13 0.98 0.15 3.75
Economics 11.5 11.72 10.39 9.25 7.43 5.58 4.03 2.22 0.97 0.2 6.33
Engineering 8.11 7.57 7.04 6.69 5.63 4.54 3.8 2.41 1.2 0.16 4.72
Geosciences 17.88 15.62 14.47 12.82 10.79 9.21 6.24 4.07 2 0.43 9.35
Mathematics 6.05 5.99 5.39 4.8 4.19 3.36 2.51 1.67 0.86 0.14 3.50
Mol Bio&Genetics 46.34 42.79 37.83 33.71 27.86 22.34 16.94 11.1 5.81 0.9 24.56
Physics 14.32 13.23 12.2 11.52 10 8.18 5.77 3.8 1.94 0.39 8.14
Psychology 21.11 18.94 18.08 15.84 12.89 10.21 7.33 4.42 1.88 0.34 11.10
Social Sciences 8.63 8.37 7.67 7.21 6.19 4.82 3.49 2.02 0.88 0.2 4.95
No. of articles with the top citation rates in
a particular field, in the last 10 years
(ESI classifications)
World benchmarks (top percentile)
Physics 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
0.01% 1427 1377 1041 966 671 717 630 336 348 244 37
0.10% 517 462 395 348 313 248 192 143 103 70 15
1.00% 148 139 121 109 100 82 66 51 36 19 6
10.00% 36 33 31 29 27 24 20 14 10 5 2
20.00% 20 19 18 17 16 14 12 9 6 3 1
50.00% 6 6 6 5 5 5 4 3 2 1 0
Summary 2 (chart)
from 1900 assessed researchers in Sections 1 and 2,
~ 1000 are members of some team
questionnaire sent to 63 teams
~ 400 researchers
Without team (870 researchers)
Other teams (660 researchers)
Above average (17 teams)
Top teams (22 teams)
Social Sciences Assessment
Oddelenie Meno Art WoSCit WoSC/A WoSH WoSm-quotient1% 10% 20% Art Cit Cit/Art100+ Cit50+ Cit30+ Cit20+ Cit10+ Cit
Teams
SocCom Bianchi G 29 133 4.59 4 0.16 1 4 1 46 201 4.37 2 2 2
Luksik I 14 132 9.43 4 0.19 1 4 1 38 201 5.29 2 2 1
Popper M 17 127 7.47 4 0.22 1 4 1 29 189 6.52 2 2 1
XPsi Adamovova L 6 77 12.83 2 0.29 2 3 21 140 6.67 1 2
Fickova E 23 146 6.35 5 0.19 2 3 28 165 5.89 1 2 1
Individuals
Progn Balaz V 47 142 3.02 8 0.30 3 2 62 249 4.02 2 1 4
Socio Gajdos P 18 14 0.78 2 0.12 3 1 28 147 5.25 1 2 1
22
Only two small teams and two other individuals
above the average Almost all involved in wider international collaboration
A
B
C
D
E
X
Y