Erica WimbushHead of Evaluation, NHS Health Scotland

Swiss Jubilee Symposium on Public Health Evaluations – Lessons Learned and Future Directions

17 June 2011, University of Berne

Getting the most out of evaluationReflections on achievements and challenges in commissioning and managing evaluations -

A Scottish perspective

Scotland in Europe

What is distinctive? What do we offer?

A small country in Europe Strong tradition of public health Focus: Health inequalities, deprivation, rurality Public health located within NHS but strong

emphasis on ‘partnership working’ Data collection of a global standard

The national public health agency

60 staff, €6.7 million 300 staff, €30.3 million

UK Devolution from 1997 – divergent roles

Health Education Board for Scotland (1991-2002) (2003- present)

Role: Provides specialist advice to Government on improving Scotland’s health and reducing inequalities and supports policy and programme

implementation at local level

Health Scotland: Evaluation Team

Our mandate: ‘To coordinate the evaluation of health improvement policy in Scotland’ Health Improvement: The Challenge, Scottish Govt 2003

Our role: Enables a better understanding of the reach and impacts of policies and actions through outcome-focused planning, monitoring and evaluation.

3 main areas of work: Commissioning and managing evaluations Supporting others to do outcome planning, evaluation and

performance reporting Evaluation Summer School & workshops

Key principle: Utilisation-focused evaluations

Public health evaluation

75% of evaluations of health improvement interventions are commissioned by Health Scotland, the Scottish Government or local NHS boards

Health Scotland evaluations: Policy evaluations Programme evaluations Local project evaluations

Achievements & Challenges

Improving Evaluation Quality

What are the Quality issues? Research design and methods Timeliness and relevance for decision-making Responsiveness, flexibility, creativity

What worked? Development of theory-based evaluation practice Establishing a vibrant evaluation community Explicit quality standards and competencies (work in

progress)Continuing challenges

‘Institutionalisation’ of evaluation – part of the culture but no independent budget or requirement

Independence vs Utilisation

What is the issue? Maintaining independent position viz a viz

policy/program can be in tension with commitment to stakeholder involvement

What worked? Intermediary ‘bridging’ role

Continuing challenges Recognising the un-evaluable!

Evaluation Capacity & Culture

What is the Capacity-building issue? Lack of confidence and competence (practitioners) Intelligent demand (funders) No recognized training or qualification (evaluators)What did we try? Workshops, training, toolkits Importing talent from abroad Most effective – policy move to outcomes management &

accountability across public services Challenges Extension of team support role and skills Sustaining the gains if/when political environment changes

Commissioning &/or Coordinating?

What is the issue? Reduced funding available Commissioning focus – single study perspective Quality issues – making more of in-house expertise

What worked? Evaluation strategy – shift evaluation focus to policy Coordination of a portfolio of inter-linked studies

Continuing challenges In-house capacity - Further extension of role Internal infrastructure to support data collection and

analysis

Future – Sharing practice

What can we bring?Theory-based evaluation practiceVibrant evaluation communityBridging role

What do we need?QualificationsStandards & Competencies