anne kazimirski 2013 future focus workshop: benifits of monitoring & evaluation
DESCRIPTION
Anne is Deputy Head of the Measurement and Evaluation at New Philanthropy Capital (NCP) and helps charities and funders to measure and communicate their impact. Her role includes developing tools and approaches for improving impact measurement for a wide range of organisations. VAL was delighted to welcome Anne to present a workshop during our 2013 Future Focus Conference. Anne's workshop was all about helping charities tell a compelling story about what they do and the impact they have. Specifically, Anne's workshop looked at the benefits of measuring impact, information about the 'theory of change process' to help charities understand what outcomes they are aiming to achieve, and helped groups start thinking about the type of data they need and how best to collect that data. While the 2013 Future Focus conference is now over, VAL runs trainings and workshops year-round. If you'd like to learn more about training for your organisation, visit www.Valoneline.org.uk.TRANSCRIPT
vBENEFITS OF MONITORING AND EVALUATIONAnne Kazimirski
VAL future focus VCS conference, 18th September 2013
NPC: TRANSFORMING THE CHARITY SECTOR
2
NPC works at the nexus between charities and
funders
Charity
SectorFunder
Increasing the impact of charities
eg, impact-focused theories of change
Strengthening the partnership
eg, collaboration towards shared
goals
Increasing the impact of funders
eg, effective commissioning
ConsultancyThink tank
3
COUNTERFACTUAL
• Comparing the world with your organisation in it with what the world would be like without it.
4
WHY MEASURING IMPACT IS IMPORTANT
The Brandon Centre provides counselling and psychotherapy to
young people between the ages of 12 and 21
Influences the debate on
“what works”Improved services
Raises profile
Motivates staff
Taken from: Rickey, B, Lumley, T and Ni Ogain, E . (2011) A Journey to Greater Impact New Philanthropy Capital.
Helps secure funding
THEORY OF CHANGE
• Links activities èintermediate outcomes è final outcomes
– clarifies what the activities aim to achieve and how
– provides a structure for identifying what can be measured
– provides the case for why achieving intermediate outcomes is important
5
A conceptual map of how activities lead to outcomes
6
Children’s emotional resourcefulness
improves
Counselling
Clients’ ability to support their children's healthy development improves
Clients’ emotional or psychological
difficulties decrease
Parent / Child interaction improves
Clients’ capacity for self care increases
EXAMPLE THEORY OF CHANGEPARENTS’ COUNSELLING SERVICE
Activities Intermediate Outcomes Final outcomes
7
WHAT: BEFORE YOU THINK ABOUT COLLECTING DATA ASK YOURSELF:
Has anyone already proved the causal link you want to make?
Yes
No
Is it really important you need data on this outcome?
Do you already collect this data?
Don’t collect unless you decide it is really important
Use research to limit your data collection
Yes
No
Yes
Can you access this data?
No
Develop your own data source
Use this data (if it is fit for purpose)
Does anyone else collect this data?
NoNo
Yes
Yes
8
LEVELS OF EVIDENCE
Randomised control trial
Anecdotes / quotes
Before and after survey
Self-reported change
Case studies
Control groups
Credibility
Basic Advanced
9
LOTS OF DECISIONS
• Budget
• Stakeholders
• Timing
• Participants
• Tools
10
QUANTITATIVE AND QUALITATIVE DATA
• Quantitative data (numbers)
– Statistical estimates
– Prevalence of views, attitudes and experiences
– Admin data/ questionnaires (paper, web, etc.)
• Qualitative data (words)
– Detailed understanding
– In depth interviewing (telephone/ face to face)
How many people find our
newsletter useful?
What factors influence
people’s views of our
newsletter?
11
WHEN TO MEASURE?
• Pre-post design:
• What attribution can you claim?
• Perception of beneficiaries and those delivering the programme: important but subjective
Before measure
After measure
Programme / service
12
COMPARISON GROUPS
Before measure
After measure
Programme / service
Before measure
After measure
• Compare before and after data for participants and non-participants, ideally matching on their key characteristics
Intervention group
Comparison group
13
ALTERNATIVES
• Waiting lists
• Dosage: users who’ve received a smaller ‘dose’ of a service
• Type of service: comparing users who’ve received one type of service compared to another
• Combine pre and post data with respondents’ perspectives
• Collect data on other events and changes in circumstances which might have contributed
Comparison groups
No comparison group
14
5 TIPS FOR MEASURING YOUR IMPACT
1. Decide what’s important
2. Keep it simple
3. Know your limits
4. Consider investing in training
5. Think about the future