AADAPT Workshop South AsiaGoa, December 17-21, 2009

Making the Most out of Discontinuities

Bilal Zia World Bank

Introduction• Many times random assignment is not possible

– Universal take-up – Non-excludable intervention– Treatment already assigned

• When randomization is not feasible, how can we exploit implementation features of the program to “measure” its impact?

• Answer: Quasi-experiments– Example: Regression Discontinuity Design.

Regression Discontinuity Design

• RDD closer to randomized experiments than other quasi-experimental methods

• Relies on knowledge of the selection process– Need to know quantifiable selection criteria – a “score”

• Assignment to “treatment” depends discontinuously on this “score”– Example: A matching grants policy that applies to firms with annual

sales less than or equal to $5,000. – A firm with sales = $5,001 would not be treated but would be very

similar to a firm with sales = $5,000. – RDD would compare firms just above and just below the $5k

threshold.

Another RDD Example

• Policy: US minimum legal drinking age is 21 alcohol consumption illegal for people younger than 21

• Observation: –People aged 20 years, 11 months and 29 days–21 year olds

–Treated differently under policy because of arbitrary age cut off–But not inherently different (likelihood to go to parties, obedience, propensity to engage in risky behavior, etc)

Effect of Alcohol on Mortality In effect: This policy rule assigns people to “treatment” and “comparison” groups– Treatment group: People between ages 20 years and 11 months and 20 years 11 months and 29 days– Comparison group: individuals who just turned 21 and can now legally drink alcohol.–They should be similar in terms of observable and unobservable characteristics that affect outcomes (mortality rates).

This way, it is possible to isolate the causal impact of alcohol consumption on mortality rates among young adults.

Graphical Depiction

Increased alcohol consumption causes higher mortality rates around the age of 21All deaths

All deaths associated with injuries, alcohol or drug use

All other deaths

Sharp and Fuzzy Discontinuities

• Sharp discontinuity – Discontinuity precisely determines treatment status

• All people 21 and older drink alcohol and no one else does• All firms with less than $5,000 in sales receive vouchers and larger

firms do not.

• Fuzzy discontinuity– Percentage of participants changes discontinuously at cut-off, but not

from 0% to 100% (or from 100% to 0%)• Some people younger than 21 end up consuming alcohol and/or

some older than 21 don’t consume at all• Policy rule determines eligibility but among firms with less than

$5,000 in sales, there is only partial compliance / take-up

0.2

5.5

.75

1

tre

atm

en

t pro

bab

ility

assignment variable

Sharp Design for Voucher receipt

0.2

5.5

.75

1

assignment variable

Fuzzy Design for Voucher receipt

Probability of Participation: Sharp ruleand “fuzzy” rule

100%

0%

75%

0%

Internal Validity• General idea

– If cut-off is arbitrary, individuals to the immediate left and right of the cut-off should be very similar.

– Differences in outcomes can be attributed to the policy.

• Major assumption

– Nothing else is happening: in absence of policy, we would not observe a discontinuity in the outcomes around this particular cut off.

– Might not be the case if • Bike-helmet policy stops also stops applying at age 21.• Another policy gives equipment to firms with sales less than

$5,000

Outcome Profile Before and After the Intervention

outc

om

e

assignment variable

Baseline

assignment variable

Follow-up

External Validity• Would the results generalize past the two groups you are

comparing?

• Counterfactual group in RDD– Individuals marginally excluded from benefits– Examples: people less than 21 but older than 20 years and 10

months, firms with sales less than $5,000 but more than $4,500.

• Causal conclusions are limited to individuals, households, villages, near the cut-off– The effect estimated is for individuals marginally or just eligible

for benefits– Extrapolation beyond this point needs additional, often

unwarranted, assumptions (or multiple cut-offs)

• Fuzzy designs exacerbate the problem

RDD Implementation: Nuts and Bolts

• Major advantage of the RDD – Transparency– Illustrated possible using graphical methods

• Major disadvantage– Requires many observations around cut-off – All observations away from the cut-off should have less

weight– Why?

• Only near the cut-off can we assume that people find themselves by chance to the left and to the right of the cut-off.

• Think about firm with annual sales = $5,000 vs firm with sales = $500.

• Or a 16 year old vs a 25 year old.

Graphical Analysiso

utc

om

e

assignment variable

Wrap Up

• RDD as a powerful tool to identify causal effects – Pros

• RDDs share the same properties as an experiment locally at the cut-off

• Can be used to evaluate interventions ex post by treating cut-offs as “natural experiments” (approximate properties of an experiment but not designed as experiments)

– Cons• Estimated program effects are representative only of

individuals/firms near the cut off• RDDs require a lot of data• Individuals/firms may adjust their behavior in subsequent years in

response to the cut-off– Example: Stop reporting sales above $5,000.

Wrap Up

• Can be used to design a prospective evaluation when randomization is not feasible– The design applies to all means tested programs– Multiple cut-offs to enhance external validity

• Can be used to evaluate ex-post interventions using discontinuities as “natural experiments”.

Thank You

17