Welcome toPatient and Public Involvement in Evaluation Workshop

Programme for the WorkshopWelcome and Introduction and purpose of the workshop Andrew Rix, Chair of the Public Delivery Board at Health and Care Research Wales and South West Evaluator Forum MemberPatient and Public Involvement in Evaluation - Survey Findings Jo Coulson, Project CoordinatorDr Danielle Sapsford, Evaluation Officer at APCRCKey Note Presentation: Patient and Public Involvement in Evaluation Dr Andy Gibson, Associate Professor in Patient and Public Involvement at UWE and Member of People in Research West of EnglandTable and Group discussion: What is, or should be, considered good practice when involving patients and the public in evaluation?Celebrating good practice Naomi Salisbury, Helpline Co-ordinator & Development WorkerEvaluating A&E Services for People who Self HarmResources and tools to supportDr Andy Gibson, Associate Professor in Patient and Public Involvement at UWE and Member of People in Research West of EnglandNext Steps and wrap upSouth West Evaluator Forum Member

PPI in Evaluation Workshop9 December 2015Andrew RixChair Public Delivery BoardHealth and Care Research Wales

Welcome

•Why this workshop•UN Year of Evaluation•Raise Profile of Evaluation• Encourage a network•Address an issue of importance

Thanks to

• South West Evaluator Forum • Elly Bernard, Emma Gibbard, Paul Lewis, Andrew

Rix. Claire Rugman – steering group• Jo Coulson and Anne-Laure Donskoy – who did

all the work• UK Evaluation Society – for providing impetus

(and money)• University of West of England, Bristol for

allowing us to hijack their event

Why public involvement is important

• The bottom line

• Evaluation affects the public• Is largely paid for by them and• Benefits from public input

Involvement

Public and Patient Involvement (PPI) is where evaluation is carried out actively ‘with’ or ‘by’

people - rather than ‘to’, ‘about’ or ‘for’ them1 (http://www.invo.org.uk/)

• Such participatory approaches involve service-users, carers, patients (including ‘expert-patients,’ survivors, volunteers, peers, lay representatives and other members of the public and community). • PPI is about incorporating consultation, collaboration and

user-led approaches.

Reference 1: http://www.invo.org.uk/find-out-more/what-is-public-involvement-in-research-2/

What is evaluation?

Noun • The making of a judgement about the amount,

number, or value of something; .....(OED) • which one agency translates as

Our definition

“Evaluation is study in which research procedures are used in a systematic way to judge the quality

or value of a service or intervention, providing evidence that can be used to improve it”

West of England Evaluation Strategy Group, 2013.

From Definition to Explanation

Definition• An evaluation is a systematic assessment of whether the stated aims and

objectives of an intervention have been met.

Purpose• Evaluations allow questions about the effectiveness, efficiency and acceptability

(to clients and to the community) of projects and programmes to be answered.

Scope• Evaluation can consider the context and process of implementation as well as the

outcomes achieved.

Process• Evaluation involves the collection and analysis of reliable, relevant and valid data.

Outcome• Evaluation should allow better and more informed decisions to be made about the

future of an intervention. For example, when to use a particular treatment type, how to improve a service, or whether to expand, reduce or discontinue a project.

Ref: Effective Interventions Unit Substance Misuse Division Scottish Executive (2009)

Models

• Often portrayed as a linear logical process or a cycle

Models

The reality

• In practice its messy – iterative, an unplanned afterthought, under resourced, not linked to policy decisions, applied to projects driven by expediency • Therefore difficult to engage the public, find ways of

active participation and full involvement • Easier in research because that has many of the hall

marks of a linear logical process • Co-production should make this easier – but it hasn’t yet • ’An afterthought to an afterthought’

Questions

• How do we get better at involving the public in this messy process?• What kind of guidance would help?• What would it look like?

• Which is what this project is about

Project outline and updateJo CoulsonProject Coordinator

South West Evaluator Forum

• Started in September 2014 by the Avon Primary Care Research Collaborative (APCRC) and Care Forum

• Forum where local evaluators can meet, share ideas, knowledge and experience, solve problems and support each other.

• Currently 30 members, which we are keen to grow

• Meets quarterly

• Successfully bid to the UK Evaluation Society for Year of Evaluation funds to support this project

Putting Public & Patient (PPI) Involvement first• To celebrate ‘Year of Evaluation 2015.’

Its aim: “to advocate and promote evaluation and evidence-based policy-making at international, regional, national and local levels.”

• International Year of Evaluation: http://mymande.org/evalyear/Declaring_2015_as_the_International_Year_of_Evaluation

Project Objectives

• South West Evaluator Forum Project Objectives:- Identify and promote examples of good practice of PPI

in Evaluation in South-West.- Co-design a local ‘PPI in Evaluation Charter’ with the

public that commissioners, health and social care actors, and SWEF members can use to guide their work.

Project outcomes• Evaluators in the region better able to involve service-users,

patients and other members of the public in evaluations.• Increased knowledge/understanding/interest in the benefits

and challenges of their meaningful involvement in evaluation activities.• Dissemination and celebration of good practice.• Co-production of a ‘PPI in Evaluation Charter.’

Progress so far:

Phase 1:• Plan and operationalise project and workshop • Design and disseminate survey• Analysed survey results and fed them into workshop

planning• Publicised and develope workshop• Today - Deliver workshop!

Survey distribution

• n=180 + cascading• Sent to a range of people across the South West

including:• Clinical Commissioning Groups• Providers• Universities• Networks• Community and Voluntary Sector organisations• Patient, carer and public representatives

Progress so far….

Phase 2: Today (‘workshop’)• Start to explore first draft of ‘PPI in Evaluation Charter’ • Identification and engagement of leads of future

developmentPhase 3 (‘sustainability,’ Jan-June 2016) With your help please…!• Development and review • Engagement and dissemination• Further ideas for follow-up work/directions welcome…

Survey ResultsDr Danielle SapsfordEvaluation Officer, APCRCSWEF Coordinator

Public and Patient Involvement (PPI) in evaluation: A survey to identify good practice

• 99 responses

• 86 responses had been directly involved in PPI in evaluation

Who Responded?

37%

10%

27%

11%

4%

4%

7%

Charity or other Voluntary, Community or Social En-terprise (VCSE) organisation

Local authority (including public health and social services)

NHS or other health-care provider

University

Freelancer or commercial consultancy

Regional or national ‘umbrella organisation,’ or representative body

Other

Drivers

Approaches to PPI

Support Provided to Enable PPI

Barriers and Challenges

Unsure of Value AddedInsufficient Funding

Insufficient Time Insufficient Capacity

Lack of Expertise

Lack of Guidelines

Lack of Organisational Engagement

Tokenism and Representativeness

Solutions and FacilitatorsPractical advice / tools

available to support evaluations with an

element of PPI (e.g. training, specialist advice, codes of practice, document

templates)

Raising awareness of PPI activity for evaluators and

PPI contributors. Better Joined-up working, sharing

innovative approaches and best practice

Higher quality of opportunities for the public to get involved

(including widening access, more meaningful roles within

evaluations, and seeing the benefits of their involvement)

More senior-level leadership within

organisations (including ensuring the necessary

funding and resources are allocated, ensuring PPI is

"mandatory“ and sound policies are in place).

Benefits of PPIWithout the patient

perspective your project is incomplete

It ensured we were asking the right questions in the

first place

You can head off problems in the future

Perspectives that would otherwise totally be

missed become apparentConnects people to the reality of what's been

happeningBetter insight into the issues and needs being

evaluated

Adds weight to the evaluation as it is driven

by the people who use the service rather than those

who run the serviceA "fresh set of eyes”

Resources and Good Practice

Patient and Public Involvement in Evaluation

Dr Andy GibsonAssociate Professor in Patient and Public Involvement at UWE Member of People in Research West of England

Table and Group discussion: What is, or should be, considered good practice when involving patients and the public in evaluation?

Group work (45 minutes)Question to be considered:• What is (or should be) considered good practice

when involving patients and the public in Evaluation?• Working in tables we are going to:• Discuss the question and brainstorm ideas • Cluster these ideas into common themes• Feedback • You have post-its, pens, flip chart paper,

scissors, sticky tape, existing guidance

Celebrating good practice

Evaluating A&E Services for People who Self HarmNaomi Salisbury, Helpline Co-ordinator & Development Worker

Resources and tools for supportDr Andy GibsonAssociate Professor in Patient and Public Involvement at UWE Member of People in Research West of England

Next StepsSouth West Evaluator Forum Member

Next Steps

We will pull together the findings from the workshop and survey to consider:• How we will further develop and consult on the draft

Charter• Do we need to do a further workshop or Delphi? • Do the findings mirror existing guidance we can adopt

and adapt?• How we will engage people in the development and

disseminate the charter

Working together• We would like to work together to take this forward• This could be as little or as much as you like• For example, you could• join a working group to further develop the content• comment on a draft • consult within your own organisation about the charter• help share and champion the charter• give practical support around patient and public

involvement in Evaluation

• If this is of interest please sign up using the form in your pack, leaving your contact details

Evaluation Form