snsf 31.03.2014 research creates knowledge. science europe working group ‚research policy and...

SNSF 31.03.2014 Research creates knowledge.

Science Europe Working Group‚Research Policy and Programme Evaluation‘

Work Strand C: ‚Uses of Evaluation Outputs‘

12 November 2015


What is Work Strand C about?

Objectives

• Contribute to increasing the usability and use of evaluation outputs

• Exchange of experience among members of Working Group

• Identify conditions under which high quality, independent evaluations contribute to policy implementation and definition

• Explore whether policy guidance or recommendations can be provided on this basis

Methods

• Case studies

• Stakeholder workshop


Overview of case studiesInstitution Research Field Instrument/ Programme

• Evaluation of INRA-Research Units by HCERES (Katrin)

• Mecanical engineering in Sweden (Sten)

• DFG Collaborative Research Centres (Anke)

• Swiss Group for clinical cancer research (Katrin)

• Mathematics in Sweden (Sten) • DFG Emmy Nöther Career Programme (Anke)

• Evaluation of Austrian Research Institute? (Klaus)

• Mathematics in Norway (Stig) • Swedish Research in Biodiversity (Sten)

• LMT (Eugenijus) • Plant & Soil Sciences in Estonia (Maarja-Liisa)

• SNSF Fellowship Programme (Katrin)

• One case study from FCT (Luisa) • Engineering in Estonia (Maarja-Liisa)

• Case study from MRC-UK (Beverly)

• One case study from Jordi ? • Evaluation of FWF Schrödinger Fellowships (Klaus)

• Education programmes (Stig)

• Nat. Research Progr.: Social challenges to national security (Eugenijus)

Case study completed

…in progress

…planned

…possible


Methodological issuesCase study selection

Case studies must cover evaluation studies that…

• are finished, published

• are backward looking (interim or ex-post)

• cover evaluation objects at the level of the institution, research field, instrument/programme but not individual grants, persons or projects

Case studies are selected to…

• cover at least 3 organisations per type of evaluation object (-> ensure critical mass/basis for comparison)

• cover various degrees of use, from extensive to non-use (-> enable exploration of potential factors influencing evaluation use)


Methodological issuesGuidelines for doing case studies

Case studies are prepared in accordance with case study template tested and refined in pilot case studies

They should rely on:

a. information that is publically available,

b. information from at least two different sources bringing in different perspectives with regard to the evaluation process.


Methodological issuesHow to synthesize results?

• Triangulation of perspectives on evaluation use within case studies

• Collection of factors potentially influencing evaluation use

• Critical examination within Working Group…

• …and on the occasion of workshop with additional stakeholders

• Potentially derive guidance or policy recommendations


Preliminary conclusionsHypotheses on factors influencing use

Evaluation set-up

• Acceptance of evaluator and methods (e.g. reluctance towards outsiders/consultants), though perceived quality not always linked to use

• Form of the evaluation report, Scope and level of recommendations

Institutional context

• Timing (fit with strategic cycles, transition of key players, lengthy processes conflict with impulse for change)

• Engagement of key players in relevant decision making processes• Joint evaluation by different stakeholders• Presentation by EU Commission highlights that very formal

structure for evaluations may itself become an impediment for use if it is perceived as a bureaucratic apparatus.

Research context

• Degree to which evaluands are accustomed to and prepared for being evaluated

Policy context

• Definition of scope – e.g. delimitation of research field, research field v. programme??

• Desire to promote or test an intended change, to fill a knowledge gap

• Fit of findings and recommendations with general strategic orientation and priorities

• Spotlight on a certain field, issue… may in itself produce an impact


Preliminary conclusionsGeneral points• Evaluation use may also manifest itself in non-action, if

the evaluation prevents a planned or discussed ‘negative action’, for example an economic cutback of a certain research program or research field.

• Evaluations may be used beyond their immediate purpose, years later (‘sleeping beauties’), possibly with respect to other programmes.

• Evaluation use may also manifest itself in best-practice for future evaluation studies (evaluation design, methodology…).

• Excessive focus on use can also be problematic• evaluation as a tool for change management?• instrumentalisation?


How to structure findings?

• Types of use

• Stakeholder analysis


Preliminary recommendations

• Do not conduct evaluations unless you plan to use them!

• Establish a “program theory” or a “logic model” for the evaluation exercise itself.


Validation Workshop (early June 2016)

Discussion on conclusions with “sounding board”

• decision-makers

• evaluators

• experts of evaluation use

Possible contacts for sounding board

Limit participation from Working Group to ≈ 10, so as not to create imbalance


Conference participations to share results

Participation in FtEval Conference

• Special session for whole Working Group? Or participation by individual Work Strands?

Participation in German Evaluation Society Conference

…


Coordination with Work Strands A & B

• Synergies with Work Strands A & B?

• Timeline for next Working Group meetings?

• Budget requirements ok from point of view of Work Strand C

• Assume there won’t be a joint final output/report by all Work Strands

snsf 31.03.2014 research creates knowledge. science europe working group ‚research policy and...

Documents