anne kazimirski 2013 future focus workshop: benifits of monitoring & evaluation

vBENEFITS OF MONITORING AND EVALUATIONAnne Kazimirski

VAL future focus VCS conference, 18th September 2013

NPC: TRANSFORMING THE CHARITY SECTOR

2

NPC works at the nexus between charities and

funders

Charity

SectorFunder

Increasing the impact of charities

eg, impact-focused theories of change

Strengthening the partnership

eg, collaboration towards shared

goals

Increasing the impact of funders

eg, effective commissioning

ConsultancyThink tank

3

COUNTERFACTUAL

• Comparing the world with your organisation in it with what the world would be like without it.

4

WHY MEASURING IMPACT IS IMPORTANT

The Brandon Centre provides counselling and psychotherapy to

young people between the ages of 12 and 21

Influences the debate on

“what works”Improved services

Raises profile

Motivates staff

Taken from: Rickey, B, Lumley, T and Ni Ogain, E . (2011) A Journey to Greater Impact New Philanthropy Capital.

Helps secure funding

THEORY OF CHANGE

• Links activities èintermediate outcomes è final outcomes

– clarifies what the activities aim to achieve and how

– provides a structure for identifying what can be measured

– provides the case for why achieving intermediate outcomes is important

5

A conceptual map of how activities lead to outcomes

6

Children’s emotional resourcefulness

improves

Counselling

Clients’ ability to support their children's healthy development improves

Clients’ emotional or psychological

difficulties decrease

Parent / Child interaction improves

Clients’ capacity for self care increases

EXAMPLE THEORY OF CHANGEPARENTS’ COUNSELLING SERVICE

Activities Intermediate Outcomes Final outcomes

7

WHAT: BEFORE YOU THINK ABOUT COLLECTING DATA ASK YOURSELF:

Has anyone already proved the causal link you want to make?

Yes

No

Is it really important you need data on this outcome?

Do you already collect this data?

Don’t collect unless you decide it is really important

Use research to limit your data collection

Yes

No

Yes

Can you access this data?

No

Develop your own data source

Use this data (if it is fit for purpose)

Does anyone else collect this data?

NoNo

Yes

Yes

8

LEVELS OF EVIDENCE

Randomised control trial

Anecdotes / quotes

Before and after survey

Self-reported change

Case studies

Control groups

Credibility

Basic Advanced

9

LOTS OF DECISIONS

• Budget

• Stakeholders

• Timing

• Participants

• Tools

10

QUANTITATIVE AND QUALITATIVE DATA

• Quantitative data (numbers)

– Statistical estimates

– Prevalence of views, attitudes and experiences

– Admin data/ questionnaires (paper, web, etc.)

• Qualitative data (words)

– Detailed understanding

– In depth interviewing (telephone/ face to face)

How many people find our

newsletter useful?

What factors influence

people’s views of our

newsletter?

11

WHEN TO MEASURE?

• Pre-post design:

• What attribution can you claim?

• Perception of beneficiaries and those delivering the programme: important but subjective

Before measure

After measure

Programme / service

12

COMPARISON GROUPS

Before measure

After measure

Programme / service

Before measure

After measure

• Compare before and after data for participants and non-participants, ideally matching on their key characteristics

Intervention group

Comparison group

13

ALTERNATIVES

• Waiting lists

• Dosage: users who’ve received a smaller ‘dose’ of a service

• Type of service: comparing users who’ve received one type of service compared to another

• Combine pre and post data with respondents’ perspectives

• Collect data on other events and changes in circumstances which might have contributed

Comparison groups

No comparison group

14

5 TIPS FOR MEASURING YOUR IMPACT

1. Decide what’s important

2. Keep it simple

3. Know your limits

4. Consider investing in training

5. Think about the future