overlooked principles of strategic management of research at a national level: transparency and...
DESCRIPTION
Video z této přednášky naleznete v repozitáři NTK: http://repozitar.techlib.cz/record/784 Chcete vědět víc? Mnoho dalších prezentací, videí z konferencí, fotografií i jiných dokumentů je k dispozici v institucionálním repozitáři NTK: http://repozitar.techlib.cz Would you like to know more? Find presentations, reports, conference videos, photos and much more in our institutional repository at: http://repozitar.techlib.cz/?ln=enTRANSCRIPT
TRANSPARENCY and BIBLIOMETRICS:
Overlooked Principles of
Strategic Management of Research at a
National Level
Dr. Franci Demšar
Director
Slovenian Research Agency
(ARRS)
Prague, October 2014
http://www.arrs.gov.si/
Contents:
•The Role of Transparency
• The Role of Bibliometrics
• Research Funding System based on Transparency and Bibliometrics
• Results – case of Slovenia
Science is a method –
a systematic way of
asking questions and
answering them
Philosophical Transactions
The oldest scientific journal (1665)
- Scientific publications need to encompass data which allow for the
reproducibility of experiments and independent tracking of cited sources
Medical Essays and Observasions (1731)
- Every article needs to be examined by experts before its publishing (peer
review).
The Role of Transparency
10 years 0
2000
4000
6000
8000
10000
12000
260 mio EUR
160 mio EUR
Transparency and Research Results
DEMŠAR et al, Journal of Librarianship and Information Science, 2013
Ge
ne
ral P
ub
lic
1. Transparency of
Instruments and
Procedures
2. Financial Transparency
3. Transparency of
Research Results
Media
Transparency Pillars
1. Transparency of Instruments and Procedures
– up to 2005 -
Article 36 - National Co-ordinators
• …
• Based on project proposal’s reviews, quantitative indicators (…) and the assessment of the managerial capacity of the research team, prepared by the Agency and reviews made by national co-ordinators, national co-ordinators prepare the rankings of the project proposals, recommended for funding in individual research areas.
• …
Official Gazette of the RS, št. 12/2005, 10. 2. 2005
RULES on co-financing basic, applied and postdoctoral research
projects
• Comparison on successful Project Leaders (PL) at calls for
proposals and those PLs who would succeed in case of using
bibliometrics only
• 2003 49%
• 2005 65%
• 2008 65%
• Južnič et al, Scientometrics, Peer review, bibliometric methods and
conflict of interest (2010)
Evaluation Procedures and the Conflict of Interest
- years 2003 - 2005 - 2008 -
Transparency of Instruments and Procedures
ARRS’ Annual Plan of Calls for Proposals with Full Documentation
2. Financial Transparency – “follow the money” - Monthly online reports on research funding
Annual reports on research funding
since 1954
Supervision on the Intended Use of Budgetary Funds
in the last 4 years
Educell, podjetje za celično biologijo, d.o.o.
Institut "Jožef Stefan"
Inštitut za kovinske materiale in tehnologije
Kemijski inštitut
Univerzitetni klinični center Ljubljana
Nacionalni inštitut za biologijo
Univerza v Ljubljani, Biotehniška fakulteta
Univerza v Ljubljani, Fakulteta za arhitekturo
Univerza v Ljubljani, Fakulteta za farmacijo
Univerza v Ljubljani, Fakulteta za kemijo in kemijsko tehnologijo
Univerza v Ljubljani, Medicinska fakulteta
Univerza v Ljubljani, Veterinarska fakulteta
Univerza v Novi Gorici
Zavod za gradbeništvo Slovenije
Inštitut za novejšo zgodovino
Univerza v Ljubljani, Filozofska fakulteta
Znanstvenoraziskovalni center Slovenske akademije znanosti in umetnosti
Univerza v Ljubljani, Fakulteta za elektrotehniko
Univerza v Ljubljani, Fakulteta za računalništvo in informatiko
Univerza na Primorskem
UP, Znanstveno raziskovalno središče Koper
UP, Inštitut Andrej Marušič
UP, Pedagoška fakulteta Koper
UP, Fakulteta za vede o zdravju
UP, Fakulteta za management
Univerza v Ljubljani
UL, Biotehniška fakulteta
UL, Fakulteta za družbene vede
UL, Fakulteta za šport
Univerza v Mariboru
UM, Medicinska fakulteta
Univerza v Novi Gorici
VISPORT, d.o.o.
Institut "Jožef Stefan"
Revision 2010/11 Revision 2012/13
Revision 2011/12 Revision 2013
Pedagoški inštitut
Geološki zavod Slovenije
UL, Fakulteta za strojništvo
UL, Fakulteta za matematiko in fiziko
UL, Ekonomska fakulteta
UL - Medicinska fakulteta
UL - Fakulteta za upravo
UM, Fakulteta za varnostne vede
UM - Fakulteta za organizacijske vede
UM - Pravna fakulteta
Fakulteta za uporabne družbene študije v Novi Gorici
DOMEL d.d.
465.532,98 EUR repayments
in the last 3 years
3. Transparency of Research Results
• International Databases: http://thomsonreuters.com/thomson-reuters-web-of-science/
http://www.scopus.com/home.url
• COBISS - http://www.cobiss.si/
• SICRIS - http://sicris.si/
• Digital Library of Slovenia, dLIB http://www.dlib.si/v2/Results.aspx?query='udc%3dZnanost'&pageSize=20)
• Videolectures.net - http://videolectures.net/arrs/
• Science Atlas
http://scienceatlas.si/?id=5594#url=%23home
= Slovenian research organisation’s directors and researchers
-----------------------------------------------------------------
- Procedures (online annual announcement of calls) 4,7
- Financing (online financial reporting / allocation of funds) 4,1
- Research results (Sicris / COBISS) 4,7
+ Satisfaction with contents of normative acts 2,8
KASTELC SELAN, Mojca. : MSc Thesis. Ljubljana: 2010
Survey on Satisfaction with Transparency of ARRS
----------------------------------------------------------------
The Role of Bibliometrics
OUTLINE PROPOSALS
REMOTE PEER REVIEW
International peers – 2 per project
REJECTED PROJECTS
FULL PROPOSALS
FINAL LIST OF APPROVED PROJECTS
Panel recommendation
Scientific Council of the Agency
Assessments are
sent to applicants to
respond
PHASE 1
PHASE 2
Funding of Research Projects
Two Step Evaluation Procedure ENTRANCE
CRITERIA
TOP 5% A”=50
or TOP 25% A`=200
or TOP 50% A1/2=400
or ALL PUBLICATIONS A=600
or CITATIONS (normalised) A2= 120
or EXTERNAL CONTRACT (EU, companies)
A3=1 FTE/year
REJECTED PROJECTS
PANEL
REMOTE PEER REVIEW
International peers – 2 per project
PANEL
REMOTE PEER REVIEW
International peers – 3 per project
PANEL
COBISS bibliographic system
COBISS.Net - Support to knowledge and intercultural dialogue
Researcher’s Online Bibliography – Bibliometric Scores
Researcher’s Online Bibliography – Bibliometric Scores
Bibliometric Scores – Further Possibilities
Scientific Excellence and Relevance of Research Programmes
Scientific criteria
Relevance criteria· funds from european projects and other international projects
/ core funds
· funds from companies / core funds
· funds from other ministries / core funds
BC 1A 2A 3A
2A· funds for projects and young
researchers / core funds
· international quality within
scientific field (citations)
1A
C: below 0,5
calculation according to average of individual scientific discipline
↓
A, B and C
A: above 1 (1 = average)
B: from 0,5 to 1
BC
Research Funding System based on
Transparency and Bibliometrics
About SICRIS Information System
SICRIS = SlovenIan Current Research Information System
• developed and maintained by the Institute of Information Science (IZUM) and ARRS
• Currently presented entities:
• Allows viewing of presentation pages of more than 500 European projects of the EU FPs directly from the ECs CORDIS system
• http://www.sicris.si/
SICRIS Information System
Digital Library - dLib.si
Research Reports at dLib.si
Videolectures
Science Atlas
Results – case of Slovenia
0
5.000
10.000
15.000
20.000
25.000
30.000
35.000
40.000
45.000
1992-1
996
1993-1
997
1994-1
998
1995-1
999
1996-2
000
1997-2
001
1998-2
002
1999-2
003
2000-2
004
2001-2
005
2002-2
006
2003-2
007
2004-2
008
2005-2
009
2006-2
010
2007-2
011
2008-2
012
2009-2
013
Nu
mb
er
per
millio
n in
hab
itan
ts
EU-27
Slovenia
1994-1998 1999-2003 2004-2008 2009-2013
Slovenia/EU-27 47,1 % 74,2 % 98,4 % 151,9 %
Number of Citations per Million Inhabitants
10/28
Source: WoS
0
50
100
150
200
250
2000-2
003
2001-2
004
2002-2
005
2003-2
006
2004-2
007
2005-2
008
2006-2
009
2007-2
010
2008-2
011
2009-2
012
2010-2
013
num
ber
of
publications
Slovenia EU-28
Number of Publications within the Top 10% of Most Cited Publications
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Slovenia/EU-
28 71 % 70 % 84 % 92 % 97 % 107 % 110 % 134 % 137% 144% 146%
10/28
0
20
40
60
80
100
120
140
160
1996-2006 1997-2007 1998-2008 1999-2009 2000-2010 2001-2011 2002-2012 2003-2013
period
num
ber
of
public
ations
Slovenia EU*
Number of Highly Cited Publications (1%) per Million Inhabitants
1996-2006 1997-2007 1998-2008 1999-2009 2000-2010 2001-2011 2002-2012 2003-2013
Slovenia/
EU* 69,5 % 71,5 % 82,7 % 92,9 % 88,0 % 103,4 % 120,2% 116,0 %
12/28
Innovation Union Scoreboard
0,2
0,3
0,4
0,5
0,6
2006 2007 2008 2009 2010 2011 2012 2013
Su
mm
ary
In
no
va
tio
n I
nd
ex
(S
II)
EU-28 Slovenia
12/28
2006 2007 2008 2009 2010 2011 2012 2013
Slovenia /
EU28 86,6% 85,2% 90,9% 91,9% 90,6% 95,5% 90,8% 92,6%
Publications and Citations of EU member states
Finland
Germany
Sweden
United KingdomAustria
Belgium
Bulgaria
CyprusCzech Republic
Denmark
Estonia
EU-27
France
Greece
Croatia
Ireland
Italy
Latvia
Lithuania
Luxembourg
Hungary
Malta
Netherlands
Poland
Portugal
Romania
Slovakia
Slovenia
Spain
0
20.000
40.000
60.000
80.000
100.000
0 2.000 4.000 6.000 8.000 10.000 12.000 14.000
Number of Publications per Million Inhabitants (2009-2013)
Nu
mb
er
of
Cit
ati
on
s p
er
Millio
n In
hab
it. (2
009-2
013)
Source: WoS
A Tool Connecting Slovenian Research
Conclusion(s)
• Transparency acts as an selfregulatory agent in the system.
• Bibliometrics serves the role of assessing the level of excellence,
but needs to be complemented by independent peer review when
distributing public funding.
• Benefit from the interconnected information society - IT supported
research management system.
• The combination of transparency and bibliometrics form a platform
for the effective research management system – leading to
improvement in research results.
Thank you for your attention!
Dr. Franci Demšar
Slovenian Research Agency (ARRS) Director