IS VaVaI as
the information tool
for the new Institutional
Evaluation Methodology
Jan Dvořák – Infoscience Praha
www.metodika.reformy-msmt.cz
IS VaVaI Key Points
• IS VaVaI documents
the public expenditures on RD&I in the Czech Republic (~26 bn. CZK / year)
and the outputs from that (~50-60K research outputs / year)
• Incl. research organisations
• Incl. active researchers
• Long record
• Project data since 1994, research outputs since 1998
• Institutional research plans (a.k.a. “research intentions”)
• The ultimate data source about RD&I in the Czech Republic
• Cornerstone for transparency
• Can support analyses
• Can support evaluations
• A one-stop shop for information about research in the Czech Republic
2
IS VaVaI: Organisations
• Funding providers
• 11 active ones, 25 total throughout the history
• Institutions
• Types:
• Academy of Sciences of the Czech Republic (53 active institutes, 12 historical ones)
• 26 public HEIs (with 150 Faculties and 170 other organisation units)
• Sectorial RTOs
• Museums, Libraries, Archives, … = Infrastructure for research
• Companies
• Roles:
• Beneficiaries in projects and/or other support
• Affiliation institutions in publications
• Owners of (rights in) applied research outputs
• With identifiers
3
IS VaVaI: Persons
4
Person role vs. data collected Fa
mily
n
ame
(s)
Giv
en
n
ame
(s)
Pe
rso
n
ide
nti
fier
Titl
es
E-m
ail
add
ress
, te
lep
ho
ne
no
.
Ho
me
ad
dre
ss
Project beneficiary ✔ ✔ ✔ ✔ ✔ ✔
Project Principal Investigator
✔ ✔ ✔ ✔ ✔
Project team member ✔ ✔ ✔ ✔
Output creator affiliated with submitting institution
✔ ✔ ✔
Output creator external to submitting institution
✔ (✔)
Call contact person (with a funder)
✔ ✔ ✔
Data submission technical contact
✔ ✔ ✔
IS VaVaI: Other RD&I Funding
(CEA)
CEA // Funder-Institution
•Institutional Funding
• RO Long-term strategic development support
• Institutional research plans (a.k.a. “research intentions”, CEZ: 1999-2013)
•Specific research at HEIs
•International collaboration in RD&I
• International RD&I programme contributions
• International RD&I institution fees
CEA // Funder
•Awards for outstanding achievements
•Costs of running programmes, calls, projects
•Funder operation costs
7
IS VaVaI: Research Outputs (RIV)
9
0
10000
20000
30000
40000
50000
60000
70000
2005 20062007
20082009
20102011
20122013
Results of applied research
Other results
J - Article in a periodical
D - Article in proceedings
C - Chapter in a book
B - Monograph
IS VaVaI: Research Outputs Data
Collection (RIV)
10
Author … Institution Funder RIV
Institutional CRIS IS VaVaI
Acad Sci CR … 15K records / yr Charles Univ … 12K records / yr
Min Edu … 50-60K records / yr GA CR … 12-15K records / yr AS CR … 8-12K records / yr
90-100K records / yr 50-60K outputs / yr
0–2 intermediate
levels
Typically handled by
libraries
checks
Comparable Research Information
Systems
National Current Research Information Systems
In production
• IS VaVaI
• FRIS (Flanders)
• CRISTIN (Norway)
• SK-CRIS (Slovakia)
Planned or under construction
• Most countries in Europe, many internationally
11
Important points
• An extensively integrated environment covering the whole research
management cycle
• Reliable macro-data (based on micro-data)
• Centralized, authoritative, public database of research outputs
• comprising a broader range of outputs than national and commercial bibliographic
resources
• Clear roles and responsibilities of actors
• Reliable protocol for data submission
• In operation for many years
• Everybody adapted to it
• Everybody relies on it
• Enables trust in the RD&I domain
12
Important points II
• RIV ≠ IS VaVaI
• Research outputs (RIV) in context (IS VaVaI)
• RIV ≠ current performance-based research funding system
• Current RVVI Metodika (since 2013)
• Pillar I – publications only
• Pillar II – peer-review assessment of selected research outputs (of any type)
• Pillar III – applied research
• Any of past RVVI Metodikas (2004, 2005, 2006, 2007, 2008, 2009, 2010-2012)
• IS VaVaI is a comprehensive information source
• Can support any evaluation
• Yes, IS VaVaI can support the IPn EM
13
IS VaVaI support for the EM
• Underlying data for the submission and the panels
• On EvUs and RUs
• Self-assessment form pre-filled with information from RIV, CEP & CEA
• The bibliometric reports based on RIV and a citation database
• Microdata (= verifiable evidence) will be aggregated to the requested levels
• Lower workload, higher data quality
• Already in place for the most part
• The Evaluation Process
• Registration of the RUs, eligibility check
• Self-reporting forms implemented in IS VaVaI, pre-filled
• Manage the evaluation panel structure
• Manage the evaluation work-flows and document flows
• Evaluation Results Dissemination
14
Extensions: “vedidk [vědík]”
Identification of researchers in the Czech Republic
• Basic support in place
• E.g. Project detail page contains:
• Researchers of foreign origin need more specific handling
• Allows:
• Reliable distinction of researchers of the same name
• Transition path to global initiatives for researcher identifiers
• Basis for automated researcher CV generation
• Incl. Prizes, Awards, Distinguished memberships and other achievements
15
Extensions: Discipline
classification refinement
• Current classification
• 123 categories
• 106 fit seamlessly to the OECD Fields of Science classification used in the IPn EM
• 17 need refinement
• E.g. AC – Archaeology, anthropology, ethnology:
• Archaeology 6.01 History and archaeology
• Anthropology and ethnology 5.04 Sociology
16
Extensions: Enhancing
connectivity
• Integration of information from citation databases
• WoS UT in place
• Scopus EID to be added
• Patent databases
• Other links
• Into OA repositories
• Into institutional CRISs
Why?
• To aid publication validation
• To enrich the information in RIV with external data (citations, …)
17
Extensions: Infrastructures
• Infrastructure Catalogue
• Facilities
• Services
• Equipment
• Tracking usage
• by whom
• in what projects
• led to which outputs
• Use of the information
• Efficient equipment sharing
• Assessment of infrastructures
• Strategic management of infrastructure
18
Conclusions
Yes, the IS VaVaI can be used as the information tool to support the new
Institutional Evaluation Methodology
•It contains the relevant data
• It is an authoritative, continuously updated source of data
• It can be extended to contain most of the rest
•It can be extended to handle the evaluation process and disseminate its
results
• It is a natural place for this type of agenda
• This is the most efficient approach
• Will keep the burden and the costs at an acceptable level
•Conclusions verified in the Small Pilot Evaluation 19
Thank you for your attention!
Děkujeme za pozornost!
www.metodika.reformy-msmt.cz