PolMeth2009: Freedman Panel

Regression Adjustments to Experimental Data: Do David Freedman’s Concerns

Apply to Political Science?

Donald P. Green

Yale University

Using covariates in the analysis of experimental results: the conventional view

• Benefit #1: addresses random imbalance• Benefit #2: increases precision by reducing

disturbance variance

• Drawback #1: burns up degrees of freedom• Drawback #2: increases discretion,

particularly in the absence of an ex ante analysis plan

Freedman’s critique of covariate adjustment

• Doesn’t follow from the experimental design

• Asymptotically unbiased but may be severely biased in finite samples

• Conventional regression estimates of standard errors may be severely biased

Freedman’s setup

• Assign a population of size n to treatment and control groups of size m and n-m, respectively

• Potential outcomes model, with responses that are deterministic functions of experimental assignments

• When assessing unbiasedness, consider the average estimate across all possible random assignment

Why the fuss?

• Experiments are becoming increasingly common, and covariate adjustment using regression is regarded as benign standard operating procedure

• Freedman’s claim that finite-sample bias is appreciable for n < 500 encompasses a large proportion of experimental studies published in political science

Aims of my paper

• Evaluate the magnitude of the bias for varying n

• Simulated data• Real data (from experiments that have been

reconfigured so that treatment and control are latent potential outcomes)

• Assess when biases are large and whether the symptoms of bias are detectable

Results

• Simulated examples: although it is possible to construct examples with severe biases, these tend to involve n<20 and noticeably heterogeneous treatment effects

• Analysis of actual experimental data shows very little bias in estimated treatment effects and fairly accurate estimated standard errors

Bottom line

• Freedman’s legacy is to challenge unreflective use of off-the-shelf statistical methods

• Regression is not unproblematic if applied to small populations with heterogeneous treatment effects, but now we have a clearer idea of what “small” means as a practical matter