quasi experimental methods i
DESCRIPTION
Florence Kondylis. Non- Experimental Methods. Quasi Experimental Methods I. What we know so far. Aim: We want to isolate the causal effect of our interventions on our outcomes of interest Use rigorous evaluation methods to answer our operational questions - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/1.jpg)
AADAPT Workshop Latin AmericaBrasilia, November 16-20, 2009
Non-Experimental Methods
Quasi Experimental Methods I
Florence Kondylis
![Page 2: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/2.jpg)
What we know so far
Aim: We want to isolate the causal effect of our interventions on our outcomes of interest Use rigorous evaluation methods to answer our
operational questions Randomizing the assignment to treatment is
the “gold standard” methodology (simple, precise, cheap)
What if we really, really (really??) cannot use it?!
>> Where it makes sense, resort to non-experimental methods
![Page 3: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/3.jpg)
3
Objective
Find a plausible counterfactual Every non-experimental method is
associated with a set of assumptions The stronger the assumption, the
more doubtful our measure of the causal effect
▪ Question our assumptions▪ Reality check, resort to
common sense!
![Page 4: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/4.jpg)
4
Principal Objective▪ Increase maize production
Intervention▪ Fertilizer Vouchers distribution▪ Non-random assignment
Target group▪ Maize producers, land over 1 Ha &
under 5 Ha Main result indicator
▪ Maize yield
Example: Fertilizer Voucher Program
![Page 5: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/5.jpg)
Before After0
2
4
6
8
10
12
14Control GroupTreatment Group
5
(+) Impact of the program
(+) Impact of external factors
Illustration: Fertilizer Voucher Program (1)
![Page 6: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/6.jpg)
Before After0
2
4
6
8
10
12
14Control GroupTreatment Group
6
(+) BIASED Measure of the program impact
Illustration: Fertilizer Voucher Program (2)
“Before-After” doesn’t deliver results we can believe in!
![Page 7: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/7.jpg)
Before After0
2
4
6
8
10
12
14Control GroupTreatment Group
7
« After » difference btwparticipants andnon-participants
Illustration: Fertilizer Voucher Program (3)
« Before» difference btw participants and nonparticipants
>> What’s the impact of our intervention?
![Page 8: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/8.jpg)
8
Difference-in-Differences Identification Strategy (1)Counterfactual: 2 Options1.Non-participant maize yield after the
intervention, accounting for the “before” difference between participants/nonparticipants (the initial gap between groups)
2.Participant maize yield before the intervention, accounting for the “before/after” difference for nonparticipants (the influence of external factors)
1 and 2 are equivalent
![Page 9: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/9.jpg)
Difference-in-DifferencesIdentification Strategy (2)Underlying assumption:Without the intervention, maize yield for participants and non participants’ would have followed the same trend
>> Graphic intuition coming…
![Page 10: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/10.jpg)
10
Data -- Example 1
Average maize yield
(T / Ha)2007 2008 Difference
(2007-2008)
Participants (P) 1.3 1.9 0.6Non-participants (NP)
0.6 1.4 0.8
Difference (P-NP) 0.7 0.5 -0.2
![Page 11: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/11.jpg)
11
Data -- Example 1
Average maize yield
(T / Ha)2007 2008 Difference
(2007-2008)
Participants (P) 1.3 1.9 0.6Non-participants (NP)
0.6 1.4 0.8
Difference (P-NP) 0.7 0.5 -0.2
![Page 12: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/12.jpg)
12
NP2008-NP2007=0.8
Impact = (P2008-P2007) -(NP2008-NP2007)
= 0.6 – 0.8 = -0.2
2007 200800.20.40.60.8
11.21.41.61.8
2
Participants Non-Participants
P2008-P2007=0.6
![Page 13: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/13.jpg)
13
2007 200800.20.40.60.8
11.21.41.61.8
2
Participants Non-Participants
P-NP2008=0.5
Impact = (P-NP)2008-(P-NP)2007= 0.5 -
0.7 = -0.2
P-NP2007=0.7
![Page 14: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/14.jpg)
Assumption of same trend: Graphic Implication
2007 200800.20.40.60.8
11.21.41.61.8
2
Participants Non-Participants
Impact=-0.2
![Page 15: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/15.jpg)
Summary Negative Impact:
Very counter-intuitive: Increased input use should increase yield once external factors are accounted for!
Assumption of same trend very strong 2 groups were, in 2007, producing at very different
levels➤ Question the underlying assumption of same
trend!➤When possible, test assumption of same trend with
data from previous years
![Page 16: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/16.jpg)
2006 2007 20080
0.5
1
1.5
2
2.5
participantsnon-participants
Questioning the Assumption of same trend: Use pre-pr0gram data
>> Reject counterfactual assumption of same trends !
![Page 17: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/17.jpg)
17
Data – Example 2
Average maize yield
(T / Ha)2007 2008 Difference
(2007-2008)
Participants (P) 1.5 2.1 0.6Non-participants (NP)
0.5 0.7 0.2
Difference (P-NP) 1.0 1.4 0.4
![Page 18: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/18.jpg)
182007 20080
0.5
1
1.5
2
2.5
participantsnon-participants
P08-P07=0.6
NP08-NP07=0.2
Impact = (P2008-P2007) -(NP2008-NP2007)
= 0.6 – 0.2 = + 0.4
![Page 19: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/19.jpg)
Assumption of same trend: Graphic Implication
2007 20080
0.5
1
1.5
2
2.5
participantsnon-participants
Impact = +0.4
![Page 20: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/20.jpg)
Conclusion
Positive Impact: More intuitive
Is the assumption of same trend reasonable?
➤ Still need to question the counterfactual assumption of same trends !➤Use data from previous years
![Page 21: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/21.jpg)
Questioning the Assumption of same trend: Use pre-pr0gram data
>>Seems reasonable to accept counterfactual assumption of same trend ?!
2006 2007 20080
0.5
1
1.5
2
2.5
participantsnon-participants
![Page 22: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/22.jpg)
Caveats (1)
Assuming same trend is often problematic No data to test the assumption Even if trends are similar in the past…
▪ Where they always similar (or are we lucky)?
▪ More importantly, will they always be similar?▪ Example: Other project intervenes in our nonparticipant villages…
![Page 23: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/23.jpg)
Caveats (2) What to do?
>> Be descriptive! Check similarity in observable characteristics
▪ If not similar along observables, chances are trends will differ in unpredictable ways
>> Still, we cannot check what we cannot see… And unobservable characteristics might matter more than observable (ability, motivation, etc)
![Page 24: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/24.jpg)
Matching Method + Difference-in-Differences (1)Match participants with non-participants on the basis of
observable characteristicsCounterfactual: Matched comparison group
Each program participant is paired with one or more similar non-participant(s) based on observable characteristics
>> On average, participants and nonparticipants share the same observable characteristics (by construction)
Estimate the effect of our intervention by using difference-in-differences
24
![Page 25: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/25.jpg)
Matching Method (2)
Underlying counterfactual assumptions
After matching, there are no differences between participants and nonparticipants in terms of unobservable characteristics
AND/OR Unobservable characteristics do not affect
the assignment to the treatment, nor the outcomes of interest
![Page 26: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/26.jpg)
How do we do it?
Design a control group by establishing close matches in terms of observable characteristics Carefully select variables along which to
match participants to their control group So that we only retain
▪ Treatment Group: Participants that could find a match
▪ Control Group: Non-participants similar enough to the participants
>> We trim out a portion of our treatment group!
![Page 27: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/27.jpg)
Implications
In most cases, we cannot match everyone Need to understand who is left out
Example
Score
NonparticipantsParticipants
MatchedIndividuals
Wealth
Portion of treatmentgroup trimmed out
![Page 28: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/28.jpg)
28
Conclusion (1)
Advantage of the matching method Does not require randomization
![Page 29: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/29.jpg)
29
Conclusion (2)
Disadvantages: Underlying counterfactual assumption is
not plausible in all contexts, hard to test▪ Use common sense, be descriptive
Requires very high quality data: ▪ Need to control for all factors that influence
program placement/outcome of choice Requires significantly large sample size to
generate comparison group Cannot always match everyone…
![Page 30: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/30.jpg)
30
Summary
Randomized-Controlled-Trials require minimal assumptions and procure intuitive estimates (sample means!)
Non-experimental methods require assumptions that must be carefully tested
More data-intensiveNot always testable
Get creative: Mix-and-match types of methods!
![Page 31: Quasi Experimental Methods I](https://reader035.vdocuments.net/reader035/viewer/2022062310/5681667c550346895dda227f/html5/thumbnails/31.jpg)
31
Thank You