evidence: helping understand what works
TRANSCRIPT
Presentation Title
Evidence – helping understandwhat works
Michael O’TooleChief Executive, Mentor
Presentation Title
• Introduction• Thinking about projects impact• Funding & commissioning for outcomes• Why measure outcomes• How to measure• Why good evidence is important• Standards for good evidence
Agenda
More than a numbers
game
Presentation Title
• Outputs
• Outcomes
• Impact
• Aim or mission
• ‘So What?’
• Thinking about projects impact
More than a numbers
game
Presentation Title
• Project in 3 sentences of no more than 20 words each• like an impact elevator pitch
• What the programme does• Immediate changes brought about• Longer-term change project
contributes to
• Theory of change
More than a numbers
game
Presentation Title
• Statement 1 describes what we do & who you work with• outputs & target group
• Statement 2 describes the changes we hope to make in your client group• Outcomes
• Theory of change
More than a numbers
game
Presentation Title
• Statement 3 talks about your impact• broader, longer-term changes,
often beyond your client group• could be impacts in a community,
systemic impacts, social value impact or inter-generational impacts
• This statement is like a mission statement
• Theory of change
More than a numbers
game
Presentation Title
What’s the problem?
Easy mistake to start by describing who you want to work with and what you want to do without first being clear about the need you are addressing.
Careful analysis of the need is an important first step in project planning and developing effect evaluation.
Funding & commissioning for outcomes
More than a numbers
game
Presentation Title
• Meeting needs• A problem or issue, or situation
where something needs to change to make things better for a person or a community
Funding & commissioning for outcomes
More than a numbers
game
Presentation Title
• Careful analysis of need is an important first step in planning a bid
• How do you know the unmet need exists and why?
• Provide recent and relevant evidence:• people with experience of the need• research into the scale of the need• local statistics about the population in the
area or levels of deprivation
Funding & commissioning for outcomes
More than a numbers
game
Presentation Title
Needs checklistWhat is the problem or issue the project will address?What evidence is there that this need exists?What are the reasons for the need?What research supports this?Why prioritise this need?How is your project an appropriate responseto the need?Have you consulted with the people your projectwill benefit?What other initiatives, or other projects work in this field and how your project will fit in?
Funding & commissioning for outcomes
More than a numbers
game
Presentation Title
• Measuring the effectiveness of an intervention• How do you know if a program was
effective?
• Identify effective practices• Determine which activities to continue,
build upon or modify
• Identify practices that need improvement• Change in order to improve effectiveness
or stop!
• Clarity & consensus on purpose• Prove value to existing and potential
funders
Why measure outcomes
More than a numbers
game
Presentation Title
• Funders will want to know:• scale or level of the change the project
will make• how you will measure the change in
order to track progress to outcomes
• Outcome indicators can be quantitative or qualitative
Why measure outcomes
More than a numbers
game
Presentation Title
• Consider measuring progress by imagining meeting a beneficiaries prior to the intervention then again after a month or year:• What will be different for them in terms
of their wellbeing, skills, attitude, knowledge, etc?
Why measure outcomes
More than a numbers
game
Presentation Title
• Indicators help understand change & track progress towards achieving change
• An indicator is a signal that the outcome is being achieved
• Indicators can be• the extent of...• the ability to...• the amount of...
• Should have at least one indicator for each outcome
How to measure outcomes
More than a numbers
game
Presentation Title
Presentation Title
• Evidence-based programmes provide funders reliable and validated information
• Establish “what works,” as well as what counts as good evidence
• Usually comprise of two components: • strong magnitude of impact• a fair and rigorous methodological
approach
• A causal relationship between implementation of the programme and outcomes of the intervention
Why good evidence is important
More than a numbers
game
Presentation Title
• Level of evidence pyramid
Why good evidence is important
More than a numbers
game
Presentation Title
• Randomised control trials• Gold standard• Repeated, systematic review RCTs
• recording same or similar result each time a study is repeated
• minimising biases linked to procedures of the study
• Ethical dilemmas
Why good evidence is important
More than a numbers
game
Presentation Title
• Important criteria for conducting an RCT:• Independent evaluation• Transferability and generalisability
(degree to which results go beyond the initial study. Generalisability is relevance of programme to a definable group
• Statistical power of the analysis(study design and methodology)
• Minimum bias(measures taken to eliminate biases)
• These criteria are fundamental to the reliability and appropriateness of a fair and well-conducted RCT
Why good evidence is important
More than a numbers
game
Presentation Title
• Levels of evidence grades
• Further information about repository of CAYT repository of impact studies - here
Why good evidence is important
More than a numbers
game
Presentation Title
• Twitter: otoole_michael
• Email: [email protected]
• Web: www.mentoruk.org.uk
Contact details
More than a numbers
game