methodologies and approaches to evaluation in third sector organisations / jean ellis

Evaluation conference, Barcelona, 3 November 20111

Methodologies and approaches to

evaluation in the third sector

International conference on evaluation, Barcelona

3 November 2011

Dr Jean Ellis

Charities Evaluation Services


The policy waves

• Early 1990s – more business-like sector and efficiency focus

• Mid 1990s – focus on results and effectiveness

• From 2004 – emphasis on performance and value for money

• Current situation – emphasis on savings and social value


MonitoringThe routine, systematic collection of information for the purpose of checking progress against project plans EvaluationUsing monitoring and other information to make judgements and give value

Some definitions


Evaluation

Evaluation aims to answer agreed questions and to make a judgement against specific criteria. Like other research, for a good evaluation, data must be collected and analysed systematically, and its interpretation considered carefully.

CES (2009) Practical Monitoring and Evaluation


The first policy wave

Early 1990s – drive for a more business-like sector and efficiency focus


The purposes of evaluation

Evaluation for accountability – to demonstrate achievement and compliance

Evaluation for learning – to share findings, improve, replicate and innovate


Monitoring and evaluation:

• is good for you

• is good for users

• meets external demands

• can make your organisation better

Why do it?


Steps to self-evaluation

1- Clarify aims and outcomes, objectives and outputs

2- Set performance indicators; choose tools

5-Use findings

3- Collect information

4- Analyse and interpret


CES planning triangle © Overall

aim

ObjectivesThis is what we do

Specific aimsThis is why we do it

Outputs

Outcomes


The Family Centre

To provide opportunities

for play, drama and

dancing

Toprovide

workshops, information

and advice on parenting

To organise outings for children and their parents

Toimprove

the lives of the children who use the centre

To increase children’s

self-esteem

To enable parents to give each other practical support

Objectives

Specific aims

Overall aim

To run a drop-in centre with a

play area

To improve the

parenting knowledge and

skills of the parents who

use the centre


Second policy wave

• Mid-1990s – focus on results and effectiveness


More definitions

Outputs are an organisation’s detailed activities, services and products – what you do

Outcomes are all the changes, benefits, learning or other effects that happen as a result of an organisation’s activities – what you change


Aims, outcomes and indicators

Aims – the broad changes you plan to achieve

Outcomes – the specific changes for individuals, families, communities, organisations and policy

Indicators – well-defined and measurable information that will show change


Families engage with befriending service

Greater awareness within the family

of later life conditions

Families feel supported, less isolated

Increased communication and coping skills

Improved family relationships and dynamics

Intermediate outcomes

There are often smaller changes that need to happen before the final, desired outcome can be reached.

These steps along the way are known as intermediate outcomes.


The Outcomes Star

© Triangle Consulting 2011


Underpinned by a model of change

© Triangle Consulting 2011


Third policy wave

• From 2004 – emphasis on performance and value for money


Drowning in data

• Too much time collecting and reporting information

• Data not always analysed or used• Costly in time and resources

But monitoring and evaluation IT systems could help greatly.


Current policy wave

The current situation – emphasis on savings to the public purse and social value.


Using Social Return on Investment

Fab Pad supports young homeless people to

sustain new tenancies. The SROI evaluation

carried out on Fab Pad revealed that for every £1

invested by the government in support, £8.38 of

social return was derived in reduced health care

costs, reduced welfare benefits expenditure and

reduced costs of repeat homelessness.


Evaluation for learning

• What worked?

• What parts worked?

• Who did they work for?

• In what contexts did they work?

• What else happened?

• Were there any negative consequences?

• Did the results last?


Accountability or Learning?

• Closed enquiry

• Proving success

• Showing cause and effect

• Used for reporting and marketing

• Open enquiry

• Valuing failure as well

• Understanding context

• Used for improvement and change.


Theory of change

It makes clear

all the

assumptions

about how

change will

happen.

Activities

Short-term change

Steps along the way

Longer-term change


Challenges and opportunities

Managing data Getting

skilled

Finding meaning

Method

s

Innovation

Using findings

New technolog

y


A winning charity

The Leeds Survivor Led Crisis Service:

We are passionate about monitoring and evaluation as we believe it is the absolute right of our visitors and callers to know that we scrutinise our work in order to ensure that we are providing an excellent service which meets their needs.


CES contacts:

www.ces-vol.org.uk020 7713 5722

[email protected]

Jean Ellis: [email protected]

02070789386

methodologies and approaches to evaluation in third sector organisations / jean ellis

Education