IS VaVaI as

the information tool

for the new Institutional

Evaluation Methodology

Jan Dvořák – Infoscience Praha

www.metodika.reformy-msmt.cz

IS VaVaI Key Points

• IS VaVaI documents

the public expenditures on RD&I in the Czech Republic (~26 bn. CZK / year)

and the outputs from that (~50-60K research outputs / year)

• Incl. research organisations

• Incl. active researchers

• Long record

• Project data since 1994, research outputs since 1998

• Institutional research plans (a.k.a. “research intentions”)

• The ultimate data source about RD&I in the Czech Republic

• Cornerstone for transparency

• Can support analyses

• Can support evaluations

• A one-stop shop for information about research in the Czech Republic

2

IS VaVaI: Organisations

• Funding providers

• 11 active ones, 25 total throughout the history

• Institutions

• Types:

• Academy of Sciences of the Czech Republic (53 active institutes, 12 historical ones)

• 26 public HEIs (with 150 Faculties and 170 other organisation units)

• Sectorial RTOs

• Museums, Libraries, Archives, … = Infrastructure for research

• Companies

• Roles:

• Beneficiaries in projects and/or other support

• Affiliation institutions in publications

• Owners of (rights in) applied research outputs

• With identifiers

3

IS VaVaI: Persons

4

Person role vs. data collected Fa

mily

n

ame

(s)

Giv

en

n

ame

(s)

Pe

rso

n

ide

nti

fier

Titl

es

E-m

ail

add

ress

, te

lep

ho

ne

no

.

Ho

me

ad

dre

ss

Project beneficiary ✔ ✔ ✔ ✔ ✔ ✔

Project Principal Investigator

✔ ✔ ✔ ✔ ✔

Project team member ✔ ✔ ✔ ✔

Output creator affiliated with submitting institution

✔ ✔ ✔

Output creator external to submitting institution

✔ (✔)

Call contact person (with a funder)

✔ ✔ ✔

Data submission technical contact

✔ ✔ ✔

IS VaVaI: RD&I Projects (CEP)

5

IS VaVaI: Other RD&I Funding

(CEA)

CEA // Funder-Institution

•Institutional Funding

• RO Long-term strategic development support

• Institutional research plans (a.k.a. “research intentions”, CEZ: 1999-2013)

•Specific research at HEIs

•International collaboration in RD&I

• International RD&I programme contributions

• International RD&I institution fees

CEA // Funder

•Awards for outstanding achievements

•Costs of running programmes, calls, projects

•Funder operation costs

7

IS VaVaI: Organisations’ RD&I

Funding

8

IS VaVaI: Research Outputs (RIV)

9

0

10000

20000

30000

40000

50000

60000

70000

2005 20062007

20082009

20102011

20122013

Results of applied research

Other results

J - Article in a periodical

D - Article in proceedings

C - Chapter in a book

B - Monograph

IS VaVaI: Research Outputs Data

Collection (RIV)

10

Author … Institution Funder RIV

Institutional CRIS IS VaVaI

Acad Sci CR … 15K records / yr Charles Univ … 12K records / yr

Min Edu … 50-60K records / yr GA CR … 12-15K records / yr AS CR … 8-12K records / yr

90-100K records / yr 50-60K outputs / yr

0–2 intermediate

levels

Typically handled by

libraries

checks

Comparable Research Information

Systems

National Current Research Information Systems

In production

• IS VaVaI

• FRIS (Flanders)

• CRISTIN (Norway)

• SK-CRIS (Slovakia)

Planned or under construction

• Most countries in Europe, many internationally

11

Important points

• An extensively integrated environment covering the whole research

management cycle

• Reliable macro-data (based on micro-data)

• Centralized, authoritative, public database of research outputs

• comprising a broader range of outputs than national and commercial bibliographic

resources

• Clear roles and responsibilities of actors

• Reliable protocol for data submission

• In operation for many years

• Everybody adapted to it

• Everybody relies on it

• Enables trust in the RD&I domain

12

Important points II

• RIV ≠ IS VaVaI

• Research outputs (RIV) in context (IS VaVaI)

• RIV ≠ current performance-based research funding system

• Current RVVI Metodika (since 2013)

• Pillar I – publications only

• Pillar II – peer-review assessment of selected research outputs (of any type)

• Pillar III – applied research

• Any of past RVVI Metodikas (2004, 2005, 2006, 2007, 2008, 2009, 2010-2012)

• IS VaVaI is a comprehensive information source

• Can support any evaluation

• Yes, IS VaVaI can support the IPn EM

13

IS VaVaI support for the EM

• Underlying data for the submission and the panels

• On EvUs and RUs

• Self-assessment form pre-filled with information from RIV, CEP & CEA

• The bibliometric reports based on RIV and a citation database

• Microdata (= verifiable evidence) will be aggregated to the requested levels

• Lower workload, higher data quality

• Already in place for the most part

• The Evaluation Process

• Registration of the RUs, eligibility check

• Self-reporting forms implemented in IS VaVaI, pre-filled

• Manage the evaluation panel structure

• Manage the evaluation work-flows and document flows

• Evaluation Results Dissemination

14

Extensions: “vedidk [vědík]”

Identification of researchers in the Czech Republic

• Basic support in place

• E.g. Project detail page contains:

• Researchers of foreign origin need more specific handling

• Allows:

• Reliable distinction of researchers of the same name

• Transition path to global initiatives for researcher identifiers

• Basis for automated researcher CV generation

• Incl. Prizes, Awards, Distinguished memberships and other achievements

15

Extensions: Discipline

classification refinement

• Current classification

• 123 categories

• 106 fit seamlessly to the OECD Fields of Science classification used in the IPn EM

• 17 need refinement

• E.g. AC – Archaeology, anthropology, ethnology:

• Archaeology 6.01 History and archaeology

• Anthropology and ethnology 5.04 Sociology

16

Extensions: Enhancing

connectivity

• Integration of information from citation databases

• WoS UT in place

• Scopus EID to be added

• Patent databases

• Other links

• Into OA repositories

• Into institutional CRISs

Why?

• To aid publication validation

• To enrich the information in RIV with external data (citations, …)

17

Extensions: Infrastructures

• Infrastructure Catalogue

• Facilities

• Services

• Equipment

• Tracking usage

• by whom

• in what projects

• led to which outputs

• Use of the information

• Efficient equipment sharing

• Assessment of infrastructures

• Strategic management of infrastructure

18

Conclusions

Yes, the IS VaVaI can be used as the information tool to support the new

Institutional Evaluation Methodology

•It contains the relevant data

• It is an authoritative, continuously updated source of data

• It can be extended to contain most of the rest

•It can be extended to handle the evaluation process and disseminate its

results

• It is a natural place for this type of agenda

• This is the most efficient approach

• Will keep the burden and the costs at an acceptable level

•Conclusions verified in the Small Pilot Evaluation 19

Thank you for your attention!

Děkujeme za pozornost!

www.metodika.reformy-msmt.cz