structural biology infrastructure for researchers...structural biology infrastructure for...
TRANSCRIPT
-
Structural biology infrastructure for researchers ESFRI Workshop – Monitoring of Ris, Periodic Update of Landmarks,
Use of KPIs
19-20 November 2018, Milan Ondřej Hradil, CEITEC, Instruct Centre, Czech Republic
European Research Infrastructure – Instruct is a Landmark project in the European Strategy Forum for Research Infrastructures
-
Internal and external evaluations - reflection
• Instruct-ERIC = distributed infrastructure, specific challenges
• Evaluation at multiple levels (as demanded by stakeholders): • International – NO • EU/ESFRI – YES
• Annual report to EC (ERIC Regulation requirement) • 5 year review (demanded by Instruct-ERIC statutes) • International Scientific Advisory Board • ESFRI Landmarks monitoring - ?
• National – YES, together with related national level infrastructures, incl. Instruct component
• Institutional – YES, given the distributed nature
-
EU / ESFRI level evaluation Instruct-ERIC evaluation:
• Why: committment in Instruct-ERIC statutes • Periodicity: every 5 years (2014, 2019) • How: External peer review panel reporting to the Instruct Council • KPIs:
• Strong visibility in the community • Demonstrable demand for services • Access provision to integrated structural biology infrastructure • Strong scientific output (publications/dissemination) • Key training achievements • Engagements with industry and supporting innovation • Instruct-ERIC evolving to meet future scientific and training needs • Delivers distinctive contribution to the research landscape; value for money
• Main stakeholders in the evaluation: ERIC member-states
-
National level evaluation – example CZ Example from Czech Republic
• Why: ranking of RIs, basis for funding decision • Periodicity: every 3-4 years (2014, 2017, next indicated for 2021) • How: international panel of experts for all RIs on national roadmap,
organised by ministry • KPIs: see next slide • What is evaluated: national level infrastructure, incl. its involvement in ESFRI • Main stakeholders: government that decides on funding + ministry
-
Czech evaluation – KPIs Obligatory outputs
Type of outcome Unit
1. Publications from the RI´s activities created by
the RI´s users (annually) number
2. Publications from the RI´s activities the RI´s
team members participated in (annually) number
3. National / foreign users of the RI number / percentage
4. Master students educated within the RI
/ subset from abroad number / number
5. Ph.D. students trained within the RI / subset
from abroad number / number
6. Financial income from the national resources
(public/private)
amount (million
CZK/year)
7. Financial income from the foreign resources
(public/private)
amount (million
CZK/year)
-
National level evaluation – example UK
Example from United Kingdom • Why: ranking of scientific projects, basis for funding decision • Periodicity: 5 years (mid-term review by funders/research councils) • How: internal panel review by MRC with international expert panel
members • KPIs: scientific output; training/skills development; scientific impact –
unique services; technical development impact • What is evaluated: national infrastructure provision through RI; excellence
vs competitors • Main stakeholders: government/reserac council
-
Other measures of success requested by Instruct members (national review)
• What is Instruct’s visibility in the member country community; (hard to measure) • Is access enabling sciences across the EU partners that would not otherwise
occur; (hard to measure) • Has Instruct initiated new research strategies to tackle ‘big challenges’; • Has Instruct positively influenced infrastructure developments in member
states – would these have developed without Instruct; (hard to measure) • Has Instruct engaged with technology industry, supporting innovation; • Does Instruct delivery an important and distinctive role in the RI landscape; • Does Instruct have an investment model that offers impact and value for
money; • Has Instruct stimulated new investment; • Has Instruct harmonized standards with other RIs.
-
Institutional level evaluation
Example from CEITEC, Instruct-ERIC centre in the Czech Republic • Why: improvement of performance, quality management • Periodicity: annually since 2015 • How: panel of 4 internal + 1 external member • KPIs: usage, impact (publications, patents, …), budgets, … • What is evaluated: individual core facilities (part of Instruct-ERIC centre) • Main stakeholders: CEITEC management
-
CEITEC areas of assessment
KPI / area of assessment
1 Staffing
2 Expenses and budget
3 Access/Utilization
4 Education and training
5 Expertise diversity
6 Quality assessment
7 Communication and Promotion
8 Impact/Outcomes – publications, patents, other results
9 Other: Technical benchmarking, financial benchmarking, etc.
-
KPI: Publications arising from access provision
Possible metric: % of papers acknowledging Instruct in relevant field
Journal Impact factor
No. Instruct publications 2015-2018
Nature 41.577 6 Science 37.205 1
Nature Chemistry 25.870 1 Nature Methods 25.062 1
Nature Immunology 21.506 1 Nature Protocols 15.269 1
Nature Chemical Biology 15.066 1 Molecular Cell 14.248 1
Nature Communications 12.353 14 Science Advances 11.51 1
EMBO Journal 9.792 1
Table shows the 10 highest impact journals publishing Instruct-acknowledged papers – as a measure of scientific excellence
2015: 53 acknowledged publications 2016: 62 acknowledged publications 2017: 63 acknowledged publications 2018 (to 21/8/18): 41 acknowledged publications
-
Instruct: Measured outcomes and impact - ACCESS Access to infrastructure: 1/7/15 to 30/6/18 442 proposals received, 16% rejected on scientific review.
FR 35%
UK 21%
IL 11%
NL 7%
ES 7%
IT 6%
BE 6%
CZ 4%
DE 3%
FI 0%
ACCESS REQUESTS TO INSTRUCT CENTRES (BY
COUNTRY) EM
Xtallisn
Protein prod
Biophys
Membrane
NMR
Imaging
nanobodies
X-ray and SAXS MS
TOP 10 ACCESS PLATFORM TYPES REQUESTED
-
Instruct: Measured outcomes and impact - Awards Training courses
FR 19%
UK 18%
DE 14%
ES 11%
USA 10%
IL 5%
NL 4%
PT 3%
GR 3%
CH 2%
Rest 11%
% of participants
FR 25%
PT 15%
ES 11%
UK 9%
IL 10%
IT 8%
CZ 8%
NL 6%
DE 4%
GR 2%
BE 2%
Courses hosted
FR
PT
ES
UK
IL
IT
CZ
NL
DE
GR
BE
-
Instruct: Measured outcomes and impact - Awards Internships: R&D Awards:
UK, 5
FR, 2
NL, 2
BE, 1
ES, 1
IL, 1
IT, 1
NUMBER OF INTERNS HOSTED In 2015: 39 applications were received; 7 awards made; total amount awarded €87,000; In 2016: 42 applications were received; 9 awards made; total amount awarded €90,000; In 2017: 60 applications were received; 7 awards made; total amount awarded €90,000; In years 2016 and 2017, the review panel recommended 29 proposals (rejected for R&D award) which were approved for access based on the science proposed.
-
KPIs vs. areas of assessment Internal monitoring
KPIs
• Only limited number!
• Clear quantification
• Shall we set targets up front?
Areas of assessment
• Wider scope and contextual picture
• Space for qualitative assessment
-
Data collection for KPIs
• Need to be established at the beginning
• Centralised metrics collected by Instruct Hub – number of proposals, training outputs, publications acknowledging Instruct-ERIC, …
• Otherwise rely on input from distributed Instruct centres
-
Expectations from ESFRI monitoring/periodic update
• No additional bureaucracy • Reflect the areas of assessment for ESFRI roadmap (science +
implementation) • Take account of different RI funding cycles/operational timeframes • Relevance of KPIs and output measures to different RI types and
objectives: single sited / distributed RI; basic / applied science • Include scientific impact as key KPI • National and institutional evaluations taken into account • ESFRI review of RIs should be accepted as valid by national
stakeholders
-
Suggestions for ESFRI monitoring/periodic update
• Include ESFRI WG members in the Instruct-ERIC evaluation
• Develop a single template evaluation that will satisfy all/many member state requirements
• Evaluation should not aim to compare RIs – they are not comparable
• Maintain scientific excellence as the overarching quality criteria
• Consultation on methodology at advanced stage (with stakeholders such as ESFRIs)
-
www.structuralbiology.eu @instructhub
Thank you
http://www.structuralbiology.eu/
-
Central aspects for success of RIs in the long run – useful lessons
• Sustainable finances: • 10 years into operational RIs on the ESFRI Roadmap – many different
financial models are in existence; • Core funding from members sustains Hub activities (governance,
organisation, legal etc) but not usually access; • Consider some form of core funding to underwrite sustainability for
infrastructures; • Could be based on inclusion on the ESFRI Roadmap; • Funds ringfenced in the Horizon-Europe budget.
• Finalise and implement an EC/ESFRI Long Term Sustainability action plan. • Political stability – BREXIT