what is evaluation? - international labour organization · source chlorine dispensers (kenya)...
TRANSCRIPT
![Page 1: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/1.jpg)
What is Evaluation?
Samer Kherfi American University of Sharjah (AUS)
![Page 2: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/2.jpg)
Course Overview
1. What is Evaluation? 2. Outcomes, Impact, and Indicators 3. Why Randomize and Common Critiques 4. How to Randomize 5. Sampling and Sample Size 6. Threats and Analysis 7. Project from Start to Finish 8. Cost-Effectiveness Analysis and Scaling Up
![Page 3: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/3.jpg)
What is Evaluation?
Evaluation
Program Evaluation
Impact Evaluation
![Page 4: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/4.jpg)
Program Evaluation
Evaluation
Program Evaluation
Impact Evaluation
![Page 5: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/5.jpg)
What’s the difference between: Monitoring and Evaluation
A. Nothing. They are different words to describe the same activity
B. Monitoring is conducted internally, Evaluation is conducted externally
C. Monitoring is for management, Evaluation is for accountability
D. Don’t know E. Other
A. B. C. D. E.
0%
17% 17%
0%
67%
![Page 6: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/6.jpg)
Monitoring and Evaluation
Evaluation
Program Evaluation
Impact Evaluation
Monitoring
![Page 7: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/7.jpg)
Monitoring
Program Evaluation
Evaluation
Program Evaluation
Impact Evaluation
![Page 8: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/8.jpg)
5 Components of Program Evaluation
1. Needs Assessment
2. Program Theory Assessment
3. Process Evaluation
4. Impact Evaluation
5. Cost Effectiveness
A. What is the problem?
B. How, in theory, does the program fix the problem?
C. Does the program work as planned?
D. Were its goals achieved? The magnitude?
E. Given magnitude and cost, how does it compare to alternatives?
![Page 9: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/9.jpg)
Evaluation should usually be conducted:
A. Externally and independent from the implementers of the program being evaluated
B. Externally and closely integrated with program implementers
C. Internally D. Don’t know
A. B. C. D.
25%
4%0%
71%
![Page 10: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/10.jpg)
Who is this evaluation for?
A. Politicians / policymakers B. Constituents C. Donors D. Donor Politicians / policymakers/ constituents E. Academics F. Technocrats / Experts/ Think Tanks G. Implementers H. Proponents, Skeptics I. Beneficiaries
![Page 11: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/11.jpg)
Who is your most important audience for evaluation?
A. Politicians / policymakers B. Constituents C. Donor leadership D. Donor politicians /
policymakers/ constituents E. Academics F. Technocrats / Experts/
Think Tanks G. Implementers H. Proponents, Skeptics I. Beneficiaries
A. B. C. D. E. F. G. H. I.
8%
0%
4%
50%
8%
4%
21%
0%
4%
![Page 12: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/12.jpg)
Programs and their Evaluations: where do we start?
Intervention
A. Start with a problem B. Verify that the problem
actually exists C. Generate a theory of why
the problem exists D. Design the program E. Think about whether the
solution is cost effective
Program Evaluation A. Start with a question B. Verify the question hasn’t
been answered C. State a hypothesis
D. Design the evaluation E. Determine whether the
value of the answer is worth the cost of the evaluation
![Page 13: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/13.jpg)
WATER, SANITATION & HEALTH
An Example
![Page 14: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/14.jpg)
What do you think is the most cost-effective way to reduce diarrhea?
A. Develop piped water infrastructure
B. Improve existing water sources
C. Increase supply of and demand for chlorine
D. Education on sanitation and health
E. Improved cooking stoves for boiling water
F. Improve sanitation infrastructure
A. B. C. D. E. F.
3%
24%
14%
3%
31%
24%
![Page 15: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/15.jpg)
NEEDS ASSESSMENT Identifying the problem
![Page 16: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/16.jpg)
The Need
A. Nearly 2 million children die each year from diarrhea
B. 20% all child deaths (under 5 years old) are from diarrhea
![Page 17: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/17.jpg)
The Likely Problem
A. Bad Water B. 13% of world population lacks access to
“improved water sources”
![Page 18: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/18.jpg)
The Goal
A. MDG: “reduce by half the proportion of people without access to sustainable drinking water”
![Page 19: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/19.jpg)
7/2009 Spring Cleaning - SITE 19
![Page 20: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/20.jpg)
The Solution(s)
![Page 21: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/21.jpg)
Really the Problem?
A. Water quality helps little without hygiene (Esrey, 1996) A. 42% live without a toilet at home
B. Nearly 2.6 billion people lack any improved sanitation facilities (WHO)
C. Quantity of water is a better determinant of health than quality of water (Curtis et al, 2000)
D. People are more willing to pay for convenient water than clean water E. Chlorine is very cheap,
A. In Zambia, $0.18 per month for a family of six B. In Kenya, $0.30 per month
F. Yet less than 10% of households purchase treatment
Kremer, Michael, Amrita Ahuja and Alex Peterson Zwane. “Providing Safe Water: Evidence from Randomized Evaluations” Discussion Paper 2010--23, Cambridge, Mass.: Harvard Environmental Economics Program, September, 2010.
![Page 22: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/22.jpg)
Alternative Solution(s)?
![Page 23: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/23.jpg)
Devising a Solution
A. What is the theory behind your solution? B. How does that map to your theory of the
problem?
![Page 24: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/24.jpg)
PROGRAM THEORY ASSESSMENT
Blueprint for Change
![Page 25: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/25.jpg)
Program Theory Assessment
A. Logical Framework (Log Frame)
B. Theory of Change C. Results Framework D. Outcome Mapping
A. Causal chain B. Causal model C. Cause map D. Impact pathways
E. Intervention theory F. Intervention framework G. Intervention logic
H. Investment logic I. Logic model J. Outcomes chain K. Outcomes hierarchy
L. Outcome line M. Program logic N. Program theory
O. Programme theory P. Results chain Q. Theory-based evaluation R. Theory-driven evaluation
S. Theory-of-action
Source: Patricia Rogers
![Page 26: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/26.jpg)
Log Frame
Objec&ves Hierarchy
Indicators Sources of Verifica&on
Assump&ons / Threats
Impact (Goal/ Overall
objec3ve)
Lower rates of diarrhea
Rates of diarrhea
Household survey
Waterborne disease is primary cause of diarrhea
Outcome (Project
Objec3ve)
Households drink cleaner water
(Δ in) drinking water source; E. coli CFU/100ml
Household survey, water quality test at home storage
ShiO away from dirty sources. No recontamina3on
Outputs Source water is cleaner; Families collect cleaner water
E. coli CFU/100ml;
Water quality test at source
con3nued maintenance, knowledge of maintenance prac3ces
Inputs (Ac3vi3es)
Source protec3on is built
Protec3on is present, func3onal
Source visits/ surveys
Sufficient materials, funding, manpower
Source: Roduner, Schlappi (2008) Logical Framework Approach and Outcome Mapping, A constructive Attempt of Synthesis
Needs assessment
Process evaluation
Impact evaluation
![Page 27: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/27.jpg)
Program Theory Assessment
A. How will the program address the needs put forth in your needs assessment? A. What are the prerequisites to meet the needs? B. How and why are those requirements currently
lacking or failing? C. How does the program intend to target or
circumvent shortcomings? D. What services will be offered?
![Page 28: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/28.jpg)
PROCESS EVALUATION Making the program work
![Page 29: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/29.jpg)
With Process Evaluation
A. Was the program implemented as planned B. Did people respond as expected C. If it were…
A. What about the concept?
![Page 30: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/30.jpg)
Process Evaluation
A. Supply Side A. Logistics B. Management
B. Demand Side A. Assumption of knowledge, preferences B. Assumptions of response
![Page 31: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/31.jpg)
Process Evaluation: Logistics
A. Construction A. Construct spring protection B. Installing fencing C. Installing drainage
B. Maintenance A. Patch concrete B. Clean catchment area C. Clear drainage ditches
![Page 32: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/32.jpg)
Process Evaluation: Supply Logistics
![Page 33: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/33.jpg)
Process Evaluation: Demand-side
A. Do households collect water from improved source? B. Does storage become re-contaminated? C. Do people drink from “clean” water?
![Page 34: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/34.jpg)
IMPACT EVALUATION Measuring how well it worked
![Page 35: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/35.jpg)
Did we achieve our goals?
A. Primary outcome (impact): did spring protection reduce diarrhea?
B. Also distributional questions: what was the impact for households with good v. bad sanitation practices?
![Page 36: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/36.jpg)
What is the impact of this program? Pr
imar
y O
utco
me Program starts
Time
![Page 37: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/37.jpg)
What is the impact of this program?
1. Positive 2. Negative 3. Zero 4. Not enough info
1. 2. 3. 4.
33%
59%
0%
7%
![Page 38: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/38.jpg)
What is the impact of this program?
1. Positive 2. Negative 3. Zero 4. Not enough info
1. 2. 3. 4.
25% 25%25%25%
![Page 39: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/39.jpg)
How to measure impact?
Impact is defined as a comparison between:
1. the outcome some time after the program has been introduced
2. the outcome at that same point in time had the program not been introduced (the “counterfactual”)
![Page 40: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/40.jpg)
Counterfactual
• The counterfactual represents the state of the world that program participants would have experienced in the absence of the program (i.e. had they not participated in the program)
• Problem: Counterfactual cannot be observed • Solution: We need to “mimic” or construct
the counterfactual
![Page 41: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/41.jpg)
Constructing the counterfactual
• Usually done by selecting a group of individuals that did not participate in the program
• This group is usually referred to as the control group or comparison group
• How this group is selected is a key decision in the design of any impact evaluation
![Page 42: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/42.jpg)
Selecting the comparison group
• Idea: Select a group that is exactly like the group of participants in all ways except one: their exposure to the program being evaluated
• Goal: To be able to attribute differences in outcomes between the group of participants and the comparison group to the program (and not to other factors)
![Page 43: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/43.jpg)
How to measure impact?
A. What would have happened in the absence of the program?
B. Take the difference between what happened (with the program) …and
- what would have happened (without the program) = IMPACT of the program
![Page 44: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/44.jpg)
What is the impact of this program?
Time
Prim
ary
Out
com
e
Impact
Counterfactual
Program starts
![Page 45: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/45.jpg)
Impact: What is it?
Time
Prim
ary
Out
com
e
Impact Counterfactual Program starts
![Page 46: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/46.jpg)
Impact: What is it?
Time
Prim
ary
Out
com
e
Impact Counterfactual
Program starts
![Page 47: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/47.jpg)
Impact evaluation methods
1. Randomized Experiments • Also known as:
– Random Assignment Studies – Randomized Field Trials – Social Experiments – Randomized Controlled Trials (RCTs) – Randomized Controlled Experiments
![Page 48: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/48.jpg)
Impact evaluation methods
2. Non- or Quasi-Experimental Methods a. Pre-Post b. Simple Difference c. Differences-in-Differences d. Multivariate Regression e. Statistical Matching f. Interrupted Time Series g. Instrumental Variables h. Regression Discontinuity
![Page 49: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/49.jpg)
RANDOMIZED EVALUATION The “gold standard” for Impact Evaluation
![Page 50: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/50.jpg)
Randomly sample from area of interest
Random Sampling and Random Assignment
![Page 51: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/51.jpg)
Randomly sample from area of interest
Randomly assign to treatment and control
Random Sampling and Random Assignment
Randomly sample from both treatment and control
![Page 52: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/52.jpg)
Impact
A. 66% reduction in source water E coli concentration
B. 24% reduction in household E coli concentration
C. 25% reduction in incidence of diarrhea
![Page 53: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/53.jpg)
Making Policy from Evidence
Intervention Impact on Diarrhea
Spring protection (Kenya) 25% reduction in diarrhea incidence for ages 0-3
![Page 54: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/54.jpg)
Making Policy from Evidence
Intervention Impact on Diarrhea
Spring protection (Kenya) 25% reduction in diarrhea incidence for ages 0-3
Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea
Home chlorine distribution (Kenya) 20-40% reduction in diarrhea
Hand-washing (Pakistan) 53% drop in diarrhea incidence for children under 15 years old
Piped water in (Urban Morocco) 0.27 fewer days of diarrhea per child per week
![Page 55: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/55.jpg)
COST-EFFECTIVENESS ANALYSIS
Evidence-Based Policymaking
![Page 56: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/56.jpg)
Cost-Effectiveness Diagram
![Page 57: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/57.jpg)
Developing an evaluation strategy
A. Start with a question B. Verify the question hasn’t been answered C. State a hypothesis D. Design the evaluation E. Determine whether the value of the answer is worth the
cost of the evaluation
F. With key questions answered from impact evaluations, process evaluation can give your overall impact
G. A few high quality impact studies are worth more than many poor quality ones
A. If you ask the right question, you’re more likely to care
![Page 58: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/58.jpg)
Components of Program Evaluation
A. Needs Assessment
B. Program Theory Assessment
C. Process Evaluation
D. Impact Evaluation
E. Cost Effectiveness
A. What is the problem?
B. How, in theory, does the program fix the problem?
C. Does the program work as planned?
D. Were its goals achieved? The magnitude?
E. Given magnitude and cost, how does it compare to alternatives?
![Page 59: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/59.jpg)
Methodologically, randomized trials are the best approach to estimate the effect of a program
1. Strongly Disagree 2. Disagree 3. Neutral 4. Agree 5. Strongly Agree
![Page 60: What is Evaluation? - International Labour Organization · Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea Home chlorine distribution (Kenya) 20-40% reduction in diarrhea](https://reader033.vdocuments.net/reader033/viewer/2022050518/5fa26f264e36a026bf0b7e33/html5/thumbnails/60.jpg)
Some further readings