non-experimental methods

Non-experimental methods

Markus GoldsteinMarkus Goldstein

The World BankThe World Bank

DECRG & AFTPMDECRG & AFTPM

Objective

• Find a plausible counterfactual

• Every method is associated with an assumption

• The stronger the assumption the more we need to worry about the causal effect

» Question your assumptions

Reality check

Program to evaluate

Hopetown HIV/AIDS Program (2008-2012)

Objectives Reduce HIV transmission

Intervention: Peer education

Target group: Youth 15-24

Indicator: Pregnancy rate (proxy for unprotected sex)

I. Before-after identification strategy (aka reflexive comparison)

Counterfactual:

Rate of pregnancy observed before program started

EFFECT = After minus Before

Year Number of areas

Teen pregnancy rate (per 1000)

2008 70 62.90

2012 70 66.37

Difference +3.47

66.37

62.9

50525456586062646668

2008 2012

Tee

n p

reg

nan

cy

(per

100

0)

Effect = +3.47

Intervention

Counterfactual assumption: no change over time

Question: what else might have happened in 2008-2012 to affect teen pregnancy?

Examine assumption with prior data

Number of areas

Teen pregnancy (per 1000)

2004 2008 2012

70 54.96 62.90 66.37

Assumption of no change over time looks a bit shaky

II. Non-participant identification strategy

Counterfactual:

Rate of pregnancy among non-participants

Teen pregnancy rate (per 1000) in 2012

Participants 66.37

Non-participants 57.50

Difference +8.87

Counterfactual assumption:Without intervention participants have same pregnancy rate as non-participants

66.4

57.5

40

60

80

100

2008 2012

tee

n p

reg

na

nc

y(p

er

10

00

)

Effect = +8.87

Participants

Non-participants

Question: how might participants differ from non-participants?

Test assumption with pre-program data

?66.4

62.9

46.37

57.5

40

50

60

70

80

2008 2012

tee

n p

reg

na

nc

y(p

er

10

00

)

REJECT counterfactual hypothesis of same pregnancy rates

III. Difference-in-Difference identification strategy

Counterfactual:

1.Nonparticipant rate of pregnancy, purging pre-program differences in participants/nonparticipants

2.“Before” rate of pregnancy, purging before-after change for nonparticipants

1 and 2 are equivalent

Average rate of teen pregnancy in

2008 2012 Difference (2008-2012)

Participants (P) 62.90 66.37 3.47

Non-participants (NP) 46.37 57.50 11.13

Difference (P-NP) 16.53 8.87 -7.66

66.462.9

46.37

57.5

40

50

60

70

80

2008 2012

tee

n pr

egna

ncy

57.50 - 46.37 = 11.13

66.37 – 62.90 = 3.47

Non-participants

Participants

Effect = 3.47 – 11.13 = - 7.66

66.462.9

46.37

57.5

40

50

60

70

80

2008 2012

tee

n p

reg

na

nc

y (

pe

r 1

00

0)

After

Before

Effect = 8.87 – 16.53 = - 7.66

66.37 – 57.50 = 8.87

62.90 – 46.37 = 16.53

Counterfactual assumption:

Without intervention participants and nonparticipants’ pregnancy rates follow same trends

66.462.9

46.37

57.5

40

50

60

70

80

2008 2012

teen

pre

gn

ancy

(per

100

0)

74.0

16.5

66.462.9

46.37

57.5

40

50

60

70

80

2008 2012

teen

pre

gn

ancy

(per

100

0)

74.0-7.6

Questioning the assumption

• Why might participants’ trends differ from that of nonparticipants?

Examine assumption with pre-program data

counterfactual hypothesis of same trends doesn’t look so believable

Average rate of teen pregnancy in

2004 2008 Difference (2004-2008)

Participants (P) 54.96 62.90 7.94

Non-participants (NP) 39.96 46.37 6.41

Difference (P=NP) 15.00 16.53 +1.53 ?

IV. Matching with Difference-in-Difference identification strategy

Counterfactual:

Comparison group is constructed by pairing each program participant with a “similar” nonparticipant using larger dataset – creating a control group from similar (in observable ways) non-participants

Counterfactual assumption:

Question: how might participants differ from matched nonparticipants?

Unobserved characteristics do not affect outcomes of interest

Unobserved = things we cannot measure (e.g. ability) or things we left out of the dataset

56

58

60

62

64

66

68

70

72

74

76

2008 2012

Tee

m p

reg

nam

cy r

ate

(per

100

0)

73.36

66.37

Matched nonparticipant

Participant

Effect = - 7.01

Can only test assumptionwith experimental data

Apply with care – think very hard about unobservables

Studies that compare both methods (because they have experimental data) find that:

unobservables often matter!

direction of bias is unpredictable!

V. Regression discontinuity identification strategy

Applicability:

When strict quantitative criteria determine eligibility

Counterfactual:

Nonparticipants just below the eligibility cutoff are the comparison for participants just above the eligibility cutoff

Counterfactual assumption:

Question: Is the distribution around the cutoff smooth?

Then, assumption might be reasonable

Question: Are unobservables likely to be important (e.g. correlated with cutoff criteria)?

Then, assumption might not be reasonable

However, can only estimate impact around the cutoff, not for the whole program

Nonparticipants just below the eligibility cutoff are the same (in observable and unobservable ways) as participants just above the eligibility cutoff

• Target transfer to poorest schools

• Construct poverty index from 1 to 100

• Schools with a score <=50 are in

• Schools with a score >50 are out

• Inputs transfer to poor schools

• Measure outcomes (i.e. test scores) before and after transfer

Example: Effect of school inputs on test scores

6065

7075

80O

utco

me

20 30 40 50 60 70 80Score

Regression Discontinuity Design - Baseline

6065

7075

80O

utco

me

20 30 40 50 60 70 80Score

Regression Discontinuity Design - Baseline

Non-Poor

Poor

6570

7580

Out

com

e

20 30 40 50 60 70 80Score

Regression Discontinuity Design - Post Intervention

6570

7580

Out

com

e

20 30 40 50 60 70 80Score

Regression Discontinuity Design - Post Intervention

Treatment Effect

Applying RDD in practice: Lessons from an HIV-nutrition program

• Lesson 1: criteria not applied well– Multiple criteria: hh size, income level,

months on ART– Nutritionist helps her friends fill out the form

with the “right” answers– Now – unobservables separate treatment

from control…

• Lesson 2: Watch out for criteria that can be altered (e.g. land holding size)

• Gold standard is randomization – minimal assumptions needed, intuitive estimates

• Nonexperimental requires assumptions – can you defend them?

Summary

Different assumptions will give you different results

• The program: ART treatment for adult patients• Impact of interest: effect of ART on children of

patients (are there spillover & intergenerational effects of treatment?)– Child education (attendance)– Child nutrition

• Data: 250 patient HHs 500 random sample HHs– Before & after treatment

• Can’t randomize ART so what is the counterfactual

Possible counterfactual candidates

• Random sample difference in difference– Are they on the same trajectory?

• Orphans (parents died – what would have happened in absence of treatment)– But when did they die, which orphans do you observe,

which do you not observe?

• Parents self report moderate to high risk of HIV– Self report!

• Propensity score matching– Unobservables (so why do people get HIV?)

Estimates of treatment effects using alternative comparison groups

Standard errors clustered at the household level in each round.Includes child fixed effects, round 2 indicator and month-of-interview indicators.

(1) (2) (3) (4) (5) (6)

Comparison group:

Orphans in Random sample

High/Mod. HIV Risk

households

Orphans in Random sample

High/Mod. HIV Risk

households

Orphans in Random sample

High/Mod. HIV Risk

households

ARV hh (<100 days) * Rd. 2 10.675 10.787 15.686 14.561 10.805 10.397(3.262)*** (2.720)*** (4.877)*** (3.832)*** (4.676)** (3.979)**

ARV hh (>100 days) * Rd. 2 5.808 5.316 10.930 9.302 2.503 1.652(3.133)* (2.638)** (4.467)** (3.513)*** (4.566) (4.036)

Constant 14.723 15.836 13.073 8.307 17.526 23.553(5.583)*** (4.753)*** (6.510)** (5.693) (10.406)* (7.712)***

Observations 334 424 164 210 170 214R-squared 0.86 0.85 0.84 0.87 0.90 0.86

All kids (8-18 years) All boys (8-18 years) All girls (8-18 years)

• Compare to around 6.4 if we use the simple difference in difference using the random sample

Estimating ATT using propensity score matching

• Allows us to define comparison group using more than one characteristic of children and their households

• Propensity scores defined at household level, with most significant variables being single-headed household and HIV risk

Probit regression results

Coefficient z-value

Single-headed household 0.8917932 3.06Amt of land owned (acres) -0.0153242 -0.83Household size 0.0060359 0.12Value of livestock owned (shillings) 9.36E-07 0.4Travel time to main road (mins.) 0.0034674 1.4Value of durables owned (shillings) -9.35E-08 -0.01House with tin roof 0.2535599 0.58House with non-mud roof 0.2180698 0.7

Household with respondent who reported high/moderate risk of having HIV/AIDS 2.76405 6.88Constant -3.250733 -4.87Observations 225Pseudo R-squared 0.5151

• Dependent variable: household has adult ARV recipient

ATT using propensity score matching

Random Sample ARV households Difference T-statHours of school attendance Nearest neighbor matching -10.97 -3.69 7.28 1.94 neighbors=2 Kernel matching -7.82 -3.69 4.12 1.65 bandwidth=.06

Mean change between rounds 1 and 2

Nutritional impacts of ARV treatment

Includes child fixed effects, age controls, round 2 indicator, interviewer fixed effects, and month-of-interview indicators.

(1) (2) (3) (4)Dependent variable:Sample:

ARV household * Round 2 0.315 -0.098(0.202) (0.043)**

ARV household (<100 days in rd 1) 0.570 -0.071 * Round 2 (0.277)** (0.058)ARV household (>100 days in rd 1) -0.003 -0.111 * Round 2 (0.252) (0.053)**Constant -0.498 -0.481 0.076 0.077

(0.386) (0.386) (0.082) (0.082)Observations 772 772 772 772R-squared 0.87 0.87 0.70 0.70

WHZ WHZ<=-2All children 0-5 in round 1

Nutrition with alternative comparison groups

Includes child fixed effects, age controls, round 2 indicator, interviewer fixed effects, and month-of-interview indicators.

(1) (2) (3) (4)Dependent variable:Comparison Group: RS Mod/High Risk

ARV household * Round 2 1.038 0.521(0.733) (0.327)

ARV household (<100 days in rd 1) 1.195 0.768 * Round 2 (0.785) (0.392)*ARV household (>100 days in rd 1) 0.773 0.220 * Round 2 (0.859) (0.419)Constant 0.864 0.904 -0.339 -0.314

(1.567) (1.588) (0.819) (0.818)Observations 96 96 250 250R-squared 0.92 0.92 0.88 0.88

WHZRS Orphans

Summary: choosing among non-experimental methods

• At the end of the day, they can give us quite different estimates (or not, in some rare cases)

• Which assumption can we live with?

Thank you