treatment evaluation. identification graduate and professional economics mainly concerned with...
TRANSCRIPT
Treatment Evaluation
Identification
• Graduate and professional economics mainly concerned with identification in empirical work.
• Concept of understanding what is the causal relationship behind empirical results.
Selection Bias
Example 1:• Do hospitals make people healthier?
Selection Bias
National Health Interview Survey (NHIS)• “During the past 12 months, was the
respondent a patient in a hospital overnight?”• “Would you say your health in general is
excellent, very good, good, fair, poor?”(1 is excellent; 5 is poor)
Selection Bias
Selection Bias
• Going to the hospital makes people sicker• It’s not impossible: hospitals are full of other
sick people who might infect us, and dangerous machines and chemicals that might hurt us.
Selection Bias
• People who go to the hospital are probably less healthy to begin with.
• Even after hospitalization, people who have sought medical care are not as healthy, on average, as those who never get hospitalized.
• Though they may well be better after hospitalization than they otherwise would have been.
Selection Bias
Example 2:• Does college education increase wage?
Selection Bias
• College graduates earn 84% more than high school graduates.
Selection Bias
• Selection into college: higher ability, smarter, work harder, etc.
• College graduates would have earned more even without college education.
• Simple comparison can not identify the causal impact of college education on wage.
Solution 1: Randomization
• Random assignment makes the treatment independent of potential outcomes.
• It eliminates selection bias and reveals true treatment effect.
• Treatment effect: compare post-treatment outcome between those who get the treatment and those who don’t.
Solution 1: Randomization
• Example 1: hormone replacement therapy (HRT)• Recommended for middle-aged women to reduce
menopausal symptoms.
Solution 1: Randomization
• Nurses Health Study (non-experimental survey of nurses): better health among the HRT users.
• Randomized trial: few benefits; serious side effects(see, e.g., Women’s Health Initiative [WHI], Hsia, et al., 2006).
Solution 1: Randomization
• Example 2: government-subsidized training programs.
• Provide a combination of classroom instruction and on-the-job training for groups of disadvantaged workers such as the long-term unemployed, drug addicts, and ex-offenders.
• Aim: increase employment and earning.
Solution 1: Randomization
• Non-experimental studies: trainees earn less than comparison groups (see, e.g., Ashenfelter, 1978; Ashenfelter and Card, 1985; Lalonde 1995).
• Evidence from randomized evaluations of training programs generate mostly positive effects (see, e.g., Lalonde, 1986; Orr, et al, 1996).
Solution 1: Randomization
Problems of randomization:• Randomly offers, but people don’t want to be
part of the game• High costs• Small sample size
Solution 2: Difference-in-difference
• Panel data available
Solution 2: Difference-in-difference
• New problem: time trend• Compare change in outcomes between
treatment group and control group• Impact is the difference in the change in
outcome
Impact = (Yt1-Yt0
) - (Yc1-Yc0
)
Solution 2: Difference-in-difference
Pre Post
Solution 2: Difference-in-difference
Effect of program using only pre- & post- data from T group (ignoring general time trend).
Pre Post
Solution 2: Difference-in-difference
Effect of program using only T & C comparison from post-intervention (ignoring pre-existing differences between T & C groups).
Pre Post
Solution 2: Difference-in-difference
• Whatever happened to the control group over time is what would have happened to the treatment group in the absence of the program.
Pre Post
Effect of program difference-in-difference (taking into account pre-existing differences between T & C and general time trend).
Solution 2: Difference-in-difference
• Example:
Schooling and labor market consequences of school construction
in Indonesia: evidence from an unusual policy experiment
Esther Duflo, MITAmerican Economic Review, Sept 2001
Solution 2: Difference-in-difference
School infrastructure
Educational achievement
Educational achievement?
Salary level?
Solution 2: Difference-in-difference• 1973-1978: The Indonesian government built
61,000 schools equivalent to one school per 500 children between 5 and 14 years old
• The enrollment rate increased from 69% to 85% between 1973 and 1978
• The number of schools built in each region depended on the number of children out of school in those regions in 1972, before the start of the program.
Solution 2: Difference-in-difference• 2 sources of variations in the intensity of the
program for a given individual• By region:
simplify the intensity of the program: high or low
• By age:Young cohort of children who benefittedOlder cohort of children who did not benefit
Solution 2: Difference-in-difference
Intensity of the Building Program
Age in 1974 High Low
2-6 (young cohort)
8.49 9.76
12-17 (older cohort)
8.02 9.4
Difference 0.47 0.36 0.12 DD(0.089)
Solution 2: Difference-in-difference
• Fundamental assumption that trends (slopes) are the same in treatments and controls (sometimes true, sometimes not)
TimeTreatment
Outcome
EstimatedAverage Treatment Effect
Average Treatment Effect
Treatment Group
Control Group
Solution 2: Difference-in-difference
• Need a minimum of three points in time (age of cohort in the example) to verify this and estimate treatment (two pre-intervention)
TimeTreatment
Outcome
Treatment Group
Control Group
Average Treatment Effect
First
observation
Second
observation
Third
observation
Solution 2: Difference-in-difference
Intensity of the Building Program
Age in 1974 High Low
12-17 8.02 9.40
18-24 7.70 9.12
Difference 0.32 0.28 0.034 DD(0.098)
Solution 3: Matching
• Panel data NOT available• Controls: non-participants with same
characteristics as participants • The matches are selected on the basis of
similarities in observed characteristics
Solution 3: Matching
• Instead of aiming to ensure that the matched control for each participant has exactly the same value of X, same result can be achieved by matching on the probability of participation
Solution 3: Matching
• For each participant find a sample of non-participants that have similar propensity scores (prob. of treatment)
• Compare the outcome
Solution 3: Matching
• Common support0
.1.2
.3.4
-5 0 5 10x
kdensity treatment kdensity control
Solution 3: Matching
• Assumes no selection bias based on unobserved characteristics