www.opm.co.ukrestricted external

Evaluating the vinspired 24/24 programme

Ewan King, director OPM

30 September

www.opm.co.uk

2

restricted external

Overview of 24/24 programme

• 24 hours a week for 24 weeks: a structured volunteering and social action intervention programme for NEET+1 young people

• Delivered by 13 organisations across England, in three separate cohorts

• Designed to meet DfE policy areas (Positive for Youth)– Support young people’s personal development– All young people to be able to participate in and achieve in

education and training– Raise young people’s aspirations

• Participants receive a personalised development plan, volunteer, train for Level 2 qualification, engage in social action activity and have targeted support around employability/CV skills

Challenges to the evaluation

• Programme is delivered through 13 VCOs, with different experiences of working with young people, across different regions of England– Challenge of understanding ‘what works’ when delivery

is through diverse organisations

• NEET+1 entry requirement implies a diverse and challenging cohort of young people:– Young people have specific and very different needs as

well as not in employment, education and training– Begin the programme with different starting points, have

different journeys and leave with different outcomes

• Limited budget and resources

Summary of the evaluation approach

• Phase 1: Scoping, development of ‘theory of change’ model, provider workshop, and survey design

• Phase 2: V administer panel participant survey • Phase 3: Field work: Longitudinal case studies • Phase 4: Integrated analysis and reporting

Why the theory of change approach?

Provides a high level visual map of the programme that articulates the:

• Inputs – the things that are invested in the programme• Activities – the project activities, what actually happens • Outputs - the tangible ‘things’ that are created as a result

of the project• Outcomes – long list of potential results of the programme:

these are short and long term outcomes

• Change mechanisms – the specific processes which lead to change, the link between activities and outcomes

Inputs Activities Outputs Outcomes

Vinspired 24/24 theory of change model

Input from v including finance, branding, marketing, structure, evaluation, capacity building and ongoing support

Young people•Commitment to 24 week placement

•Time and capacity to participate

13 Delivery organisations•Established partnerships with referral

agencies and community groups

•Staff time and capacity

•Staff skills and expertise

•Service delivery infrastructure

•Physical building/assets

Ac

tivit

ies

Inp

uts

Identify,

engage and

recruit

participants

Tailor 24/24

activity by

individual and

cohort

Network with

local community

groups/partners

Provide ongoing

tailored support

per participant

Ongoing

management and

governance of

programme

Ou

tpu

ts Individual and

cohort-led

social action

plans

‘24 hours per

week for 24

weeks’

voluntary hours

Level 2

qualification

Individual

participant

development

plans

375 24/24

participants exit

the programme

Positive progression

•Skills development

•Level 2 qualified

•Readiness to work

•Experience of a

professional workplace

Improved life choices

Ou

tco

me

s

Transition to adulthood

•Time keeping

•Increased responsibility

•Active citizenship

Enhanced well being

•Improved confidence

and self esteem

•Social resilience

Improved

ability

to engage and

support young people

Improved

service/offer

Long term sustainability

How this approach worked for 24/24

• Provided a framework that guided the development of research tools for use during fieldwork and an analytical framework to guide the synthesis and analysis of findings

• Enabled us to target data collection on the most ‘explanatory’ data

• Allowed us to build a narrative of how the programme was intended to work and make a difference and then monitor the extent to which this theory was delivered in practice

• Flexible enough to incorporate individual participant journeys and outcomes within the evaluation– We could include case studies which explored a wide range of

impacts that resulted from the programme

Powerful panel surveys analysis

Lessons learnt

• Theory of change can help build a more plausible explanation of how the programme works and makes a difference and allows a better targeting of data collection efforts

• In the absence of controls, Theory of Change – combined with triangulation of different data – can help towards explaining attribution

• Self administered validated surveying of participants is possible with committed project managers

• Approach is highly participative: 1. enabling participants to contribute and refine our theory of how the programme works;2. building understanding and commitment to evaluation

• In depth case studies provided compelling accounts of the participant journeys

• Tailored findings to inform future programmes‘We have used this evaluation to create our new structured volunteering

and learning programme, vInspired Talent. This year we are working with 10 local authorities and FE colleges to offer placements to 16- 25 year olds.’ (Terry Ryall, vInspired CEO)