quasi-experimental methods
DESCRIPTION
Quasi-Experimental Methods. Florence Kondylis (World Bank). Objective. Find a plausible counterfactual Reality check Every method is associated with an assumption The stronger the assumption the more we need to worry about the causal effect Question your assumptions. Program to evaluate. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/1.jpg)
AIM-CDD
Quasi-Experimental Methods
Florence Kondylis (World Bank)
![Page 2: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/2.jpg)
AIM-CDD
Objective
• Find a plausible counterfactual»Reality check
• Every method is associated with an assumption
• The stronger the assumption the more we need to worry about the causal effect
»Question your assumptions
2
![Page 3: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/3.jpg)
AIM-CDD
3
Program to evaluateFertilizer vouchers Program (2007-08)–Main Objective• Increase maize production
– Intervention: vouchers distribution–Target group:• Maize producers• Farmers owning >1 Ha, <3 Ha land
– Indicator: Yield (Maize)
![Page 4: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/4.jpg)
AIM-CDD
I. Before-after identification strategy
Counterfactual:
Yield before program started
» EFFECT = After minus Before
Counterfactual assumption:
There is no other factor than the vouchers affecting yield from 2007 to 2008
years
4
![Page 5: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/5.jpg)
AIM-CDD
5
![Page 6: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/6.jpg)
AIM-CDD
6
Questioning the counterfactual assumption
Question: what else might have happened in 2007-2008 to affect maize yield ?
![Page 7: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/7.jpg)
AIM-CDD
7
Examine assumption with prior data
Assumption of no change over time not so great ! >> There are external
factors (rainfall, pests…)
![Page 8: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/8.jpg)
AIM-CDD
II. Non-participant identification strategy
Counterfactual:
Rate of pregnancy among non-participants
Counterfactual assumption:
Without vouchers, participants would as
productive as non-participants in a given year
8
![Page 9: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/9.jpg)
AIM-CDD
9
![Page 10: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/10.jpg)
AIM-CDD
Questioning the counterfactual assumption
10
Question: how might participants differ from non-participants?
![Page 11: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/11.jpg)
AIM-CDD
Test assumption with pre-program data
11
REJECT counterfactual hypothesis of same productivity
![Page 12: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/12.jpg)
AIM-CDD
III. Difference-in-Difference identification strategy
Counterfactual:
1.Non-participant maize yield, purging pre-program differences between participants/nonparticipants
2.“Before vouchers” maize yield, purging before-after change for nonparticipants (external factors)
• 1 and 2 are equivalent
12
![Page 13: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/13.jpg)
AIM-CDD
13
![Page 14: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/14.jpg)
AIM-CDD
14
57.50 - 46.37 = 11.13
66.37 – 62.90 = 3.47
Non-participants
Participants
Effect = 3.47 – 11.13 = - 7.66
![Page 15: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/15.jpg)
AIM-CDD
15
After
Before
Effect = 8.87 – 16.53 = - 7.66
66.37 – 57.50 = 8.87
62.90 – 46.37 = 16.53
![Page 16: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/16.jpg)
AIM-CDD
16
Counterfactual assumption:
Without intervention participants and nonparticipants’ pregnancy rates follow same trends
![Page 17: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/17.jpg)
AIM-CDD
17
74.0
16.5
![Page 18: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/18.jpg)
AIM-CDD
18
74.0 -7.6
![Page 19: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/19.jpg)
AIM-CDD
19
Questioning the assumption
• Why might participants’ trends differ from that of nonparticipants?
![Page 20: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/20.jpg)
AIM-CDD
20
Examine assumption with pre-program data
counterfactual hypothesis of same trends doesn’t look so believable
Average rate of teen pregnancy in
2004 2008 Difference (2004-2008)
Participants (P) 54.96 62.90 7.94
Non-participants (NP) 39.96 46.37 6.41
Difference (P=NP) 15.00 16.53 +1.53 ?
![Page 21: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/21.jpg)
AIM-CDD
21
IV. Matching with Difference-in-Difference identification strategy
Counterfactual:
Comparison group is constructed by pairing each program participant with a “similar” nonparticipant using larger dataset – creating a control group from similar (in observable ways) non-participants
![Page 22: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/22.jpg)
AIM-CDD
22
Counterfactual assumption:
Question: how might participants differ from matched nonparticipants?
Unobserved characteristics do not affect outcomes of interest
Unobserved = things we cannot measure (e.g. ability) or things we left out of the dataset
![Page 23: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/23.jpg)
AIM-CDD
23
73.36
66.37Matched
nonparticipant
Participant
Effect = - 7.01
![Page 24: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/24.jpg)
AIM-CDD
24
Can only test assumptionwith experimental data
Apply with care – think very hard about unobservables
Studies that compare both methods (because they have experimental data) find that:
unobservables often matter!
direction of bias is unpredictable!
![Page 25: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/25.jpg)
AIM-CDD
Summary
• Randomization requires minimal assumptions needed and procures intuitive estimates (sample means !)
• Non-experimental requires assumptions that must be carefully assessed
»More data-intensive
25
![Page 26: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/26.jpg)
AIM-CDD
Example: Irrigation for rice producers + Enhanced Market Access• Impact of interest measured by:
– Input use & repayment of irrigation fee– Rice yield– (Cash) income from rice– Non-rice cash income (spillovers to other value chains)
• Data: 500 farmers in project area / 500 random sample farmers– Before & after treatment
»Can’t randomize irrigation so what is the counterfactual?
![Page 27: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/27.jpg)
AIM-CDD
Plausible counterfactuals• Random sample difference in difference
– Are farmers outside the scheme on the same trajectory ?
• Farmers in the vicinity of the scheme but not included in scheme– Selection of project area needs to be carefully documented
(elevation…)
– Proximity implies “just-outside farmers” might also benefit from enhanced market linkages
» What do we want to measure?
• Propensity score matching
– Unobservables determining on-farm productivity ?
27
![Page 28: Quasi-Experimental Methods](https://reader033.vdocuments.net/reader033/viewer/2022051114/5681351a550346895d9c722f/html5/thumbnails/28.jpg)
AIM-CDD
28
Thank You