betterevaluation - webinar number 2: define
TRANSCRIPT
Part 2 of 8 AEA Coffee Break Webinars 2013
Simon Hearn, ODI, London
DEFINE what is to be evaluated
The Rainbow Framework
Why do we need to start with a clear definition?
Source: Hobbies on a Budget / Flickr
Three evaluation tasks in DEFINING
Develop initial description
Develop program theory or
logic model
Develop initial description
1.
Thumbnail description
Vision & Mission
Peak Experience
Aims & Objectives
Develop program theory or logic model
2.
Options for representing logic models
Pipeline / results chain
Logical framework
Outcomes hierarchy / theory of change
Realist Matrix
Options for representing logic models
Pipeline / results chain
Logical framework
Outcomes hierarchy / theory of change
Realist Matrix
INPUTS• Apples • People at risk of
poor health
IMMEDIATE RESULTS• Apples eaten
SHORT-TERM RESULTS• Improved
nutritional status
LONGER-TERM RESULTS• Improved health
Options for representing logic models
Pipeline / results chain
Logical framework
Outcomes hierarchy / theory of change
Realist Matrix
RESULTS AREA INDICATOR MEANS OF VERIFICATION
ASSUMPTIONS/ RISKS
GOAL:Improved health
Decreased sick days
School and work records
Good status largely due to nutrition
PURPOSE:Improved nutritional
status
Adequate levels of
Vitamin CSample blood tests Apples retain
Vitamin C
OUTPUT: Apples eaten
Apples replaced by apple cores
Visual inspection Apples eaten on location
ACTIVITIES: Deliver apples
Required numbers delivered
Delivery receipts
ACTIVITIES: Buy apples
Required numbers bought
Purchase receiptsVisual inspection
Options for representing logic models
Pipeline / results chain
Logical framework
Outcomes hierarchy / theory of change
Realist Matrix
Outcomes chain showing possible alternative causal paths
Improved health
Improved vitamin C levels
Apples eaten (whole or juice)
Apples delivered to schools
Improved quercetin levels
Red apple skin eaten
Decreased Body Mass Index
Apples eaten instead of junk
food snacks
Options for representing logic models
Pipeline / results chain
Logical framework
Outcomes hierarchy / theory of change
Realist Matrix
Context Mechanism Outcome
Healthy people at high risk of gastric cancer
Quercetin from red apple skin (or red onion) protect cells from free radicals
Reduced risk of cancer
Smokers or people with regular exposure to radiation
Chemicals in cigarette smoke or radiation damages DNA in healthy cells
Continued increased risk of cancer
Individuals with vitamin deficiency through malnutrition
Sufficient vitamin C absorbed when juice from apples (or oranges) consumed
Stronger immune system
Individuals with vitamin deficiency through excessive alcohol
Alcohol destroys vitamin C No change in vitamin C level
Individuals with obesity due to excessive snacking on high fat, high calorie foods
Decreased calorie intake due to substitution of apples (or carrot sticks)
Reduced obesity and related conditions
Individuals with obesity due to genetic condition Decrease sugar intake through
substitution of applesNo change in obesity but lower risk of diabetes
Context Mechanism Outcome
Healthy people at high risk of gastric cancer
Quercetin from red apple skin (or red onion) protect cells from free radicals
Reduced risk of cancer
Smokers or people with regular exposure to radiation
Chemicals in cigarette smoke or radiation damages DNA in healthy cells
Continued increased risk of cancer
Individuals with vitamin deficiency through malnutrition
Sufficient vitamin C absorbed when juice from apples (or oranges) consumed
Stronger immune system
Individuals with vitamin deficiency through excessive alcohol
Alcohol destroys vitamin C No change in vitamin C level
Individuals with obesity due to excessive snacking on high fat, high calorie foods
Decreased calorie intake due to substitution of apples (or carrot sticks)
Reduced obesity and related conditions
Individuals with obesity due to genetic condition Decrease sugar intake through
substitution of applesNo change in obesity but lower risk of diabetes
http://betterevaluation.org/resource/example/rubrics-oakden
Identify potential unintended or negative results
3.
Source: www.smarttoolkit.net
Negative program theory
Risk assessment
Key informant interviews
Six Hats Thinking
Defining – Framing iterationInitial
description
Discuss evaluation purpose
Refine description
Define evaluation questions
Develop logic model
Agree scope of
evaluation
DEFINE
DEFINEDEFINE
FRAME
FRAME
FRAME
Simon HearnResearch Fellow, [email protected]
http://betterevaluation.org/plan/definefacebook.com/betterevaluation@bettereval
SWOT Analysis
Five Why’s
Backcasting
Options for developing logic models
Previous research / evaluation
Group facilitation