Evaluation – Principles and methodsMatt Barnard
Head of Evaluation
NSPCC
Purpose of evaluation• Definition “Examine how a policy or intervention was
designed and carried out and with what results.” (Magenta Book)
• Asks objective questions– What were the impacts?– How was it delivered?– What were barriers and facilitators?– Did it deliver value for money?
• Aims to provide– ‘Scientific’ basis for policy making
Evaluation Design Process
Evaluation findings
Intervention Design
Logic model
Intervention implementation
planning
Evaluation implementation
Intervention implementation
Evaluation Design
Logic models
• Characteristics– Mechanisms not processes
– Key steps not every step
– Explanatory not descriptive
– Reflects theoretical assumptions
• Benefits– Sense check
– Identifies realistic outcomes
– Facilitates evaluation design
Strength of designWeak design
(Poor/ no counterfactual)
Strong design (Realistic counterfactual)
Low power
(Small numbers/ effect size)
Unlikely to detect difference/
Low confidence in attribution
Unlikely to detect difference/
High confidence in attribution
High power
(Large numbers/ effect size)
Likely to detect difference/
Low confidence in attribution
Likely to detect difference/
High confidence in attribution
Strength of design matrix
Strength – Evaluation design
Randomized controlled trial
Quasi-experimental design
Before and after measures
Types of design
• RCT– Individual randomization
– Cluster randomization/ roll out
– BAU/waiting list/alternative services
• Quasi-experimental designs– Matched area/ groups
– Matched individual
– Interrupted time series
– Regression discontinuity
Factors influencing methodology
• Intervention stage of development– Early exploration
– Defined and established but not proven
– Transferability
• Potential Costs and benefits– Resources
– Timescales
Key principles
• Clarity about key question– Avoid ‘default’ questions
• Methods matched to question– Ensure methods match desired questions
• Claims match evidence– Avoid over-claiming
• Have a coherent story to tell