Environmental Policy Evaluation

MADE Program

30/10/2009

Policy Evaluation

• Oversight agencies increasingly concerned about effectiveness of policies (both mandatory and voluntary)

• Voluntary programs/International Agreements (ISO 9001/Kyoto Protocol) are greatly increasing in number and size

• Need for rigorous, scientfic evaluations of policies

Treatment

• Policy is known as the treatment

• Observations of unaffected entities is called the control

• An effective policy will alter the outcome of those treated (given the policy) relative to the control

Policy Story

• It is worthwhile to learn the policy background before starting an evaluation

• Sometimes the proper outcome variable to measure is not clear

• Be willing to listen to those involved in the policy but remember their interests are different than yours – No Stockholm Syndrome

Ideal Story

• You find data that:– Starts before policy went into affect– Continues after policy went into affect– Covers those affected by policy and those

unaffected

• The policy– Randomly assigned those involved– Staggered its starting date

Issue to Control for

• The big issues to control for are:– Pre-policy behavior/trend– Behavior of those unaffected by policy– Selection of those affected by policy– Other variables that affect outcome

• Ideally the policy will have a staggered start so that the analysis is using both time and space variation to determine policies affect

Panel Data

• Usually evaluations occur using panel data methods

• Panel data is when you follow the same entity over time

• Methods often used are fixed effects, random effects, dynamic panel, logit

Voluntary Programs

• Generally between government agency and a group of firms

• Firms join the program (become a partner) and pledge to make specified environmental improvement

• Partners potentially receive benefit in form of confidential information

Standard Analysis

• Conventional interpretation of partnership programs is that information given to partners bring exclusive benefits

• Selection bias likely to occur; needs to be controlled for

• Typical View - Program effective if partners had statistically better environmental outcome than non-partners

Estimation• Difference in Difference Model

– Post Policy Dummies (T)– Partner Dummy (P)– Partner & Policy Dummy Interaction (TP)– Other controls that alter dependent variable

(X, S, N)

• β9 is the relevant variable

itiititititjtitit vPTTPNSXR ^

987654

Diff-in-Diff Variable

• β9 tells us whether the group affected by the policy acted differently than the group unaffected, once the policy went into affect

• If it is positive & statistically significant, the policy is associated with an increase in dependent variable

• Vice Versa for negative coefficients

Other Interpretations

• Policy variable (β8) tells us whether the dependent variable changed when policy went into affect

• Partner variable (β7) tells us whether partners are inherently different with respect to dependent variable

13

Example: C2P2 program• Coal Combustion Partnership Program (C2P2) started

in 2001, part of Resource Conservation Challenge– promotes beneficial uses of Coal Combustion Products (CCP)

as opposed to lanfilling them– CCP can be reused in concrete, asphalt, & more

• About 170 Partners– Work with industry groups & other federal agencies to promote

education about CCP reuse– Outreach/networking/acknowledgement for CCP suppliers and

demanders – Document reuse by individual facilities, companies and

government organizations

Question• Is C2P2 associated with an increase in CCP

re-use?

• Data on coal power plants in US for 96-05 (C2P2 started in 2001)

• Dependent Variable: Fly Ash Re-Use Rates – the amount of fly-ash sold divided by the total

fly-ash generated

15

Rate of Fly Ash Reuse

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Year

Re-

Use

Rat

io

Initial Partners

Non-Partners

Late Partners

Raw Data

• Be careful with Raw Data like this

• Other co-variates not controlled for

• Policy makers tend to like these pictures even though they are not telling the whole story

17

Estimation• Random Effects Model

• Dependent Variable:– Ratio of Fly Ash Reused to Total Fly Ash Generated

• Controls:– Pre, Early, & Late C2P2 Time Dummies – Partner Variable & C2P2 Time Interactions– Total Coal Burnt– Ash Content of Coal– Price of Aggregates/Cement in State– Level of Cement Imports– Region Dummies– NOx Control Equipment Present

itiititititjtitit vPTTPNSXR ^

987654

18

Evaluation ResultsTime Period : 1996-2005Dependent Variable: Model: Fixed Effects Random EffectsExplanatory Variable Coefficient Stnd. Error Coefficient Stnd. ErrorPredicted Partner -0.18** 0,08Early-C2P2 (2001-2002) 0.06** 0,02 0.08** 0,03Late-C2P2 (2003-2005) 0.06** 0,02 0.07** 0,02Early-C2P2* Predicted Partner 0,03 0,02 0,03 0,02Late C2P2* Predicted Partner -0.01 0,02 -0.01 0,02Number of Utility's Plants Near 0.01* 0 0,01 0,01Post-C2P2* Utility's Plants Near 0,01 0,01 0,01 0,01Lag Utility Reuse 0.08** 0,04 0.11*** 0,03Aggregates Price -0.01 0,02 0.06*** 0,02Cement Price 0,01 0,01 -0,01 0,01Cement Imports 0,01 0,01 0,01 1SCR Installed 0,01 0,02 0,01 0,01Avearge Ash Content -0.01*** 0 -0.02*** 0,01Total Coal 0.01** 0,00 0.01*** 0,00N 1883 1883Plants 303 303R2 0,04 0,3Region & Year Dummies Not Shown for BrevityErrors Clustered by Utility for all specifications and Bootstrapped for the first two specifications*, **, *** indicate 10%, 5% and 1% statistical significance, respectively

Fly Ash Reuse RatioFly Ash Reuse Ratio

Other Issues

• Treatment Spillovers

• Selection Bias

• Error Specification

20

Treatment Spillovers• Programs provide information that spills-over

to non-partners; programs may be public good in nature

– Technical information-spreading is not limited to partners, thus both partners and non-partners can take advantage of program

– One country agreeing to reduce consumption of a good could lead to increases in price for other countries

Evaluation Interpretation

• If there are treatment spillovers,– Partners and non-partners do not need to be

statistically different for program to be effective– Requires evidence that spillovers exist– Requires statistically better environmental

outcomes for both groups after program initiation

22

Evidence of Spillovers: C2P2

• Non-partner data analyzed for evidence of treatment spillover by comparing non-partners in states w/ many partners to non-partners in states w/ few partners

Example: C2P2Time Period : 1996-2005Dependent Variable: Model: Fixed Effects Random EffectsExplanatory Variable Coefficient Stnd. Error Coefficient Stnd. ErrorModerate/Low C2P2 Partners Nearby 0.34*** 0,08Early-C2P2 (2001-2002) 0.10** 0,04 0.11** 0,04Late-C2P2 (2003-2005) 0.14** 0,05 0.15** 0,05Early-C2P2* Moderate/Low C2P2 Partner -0.09* 0,05 -0.09* 0,05Late-C2P2* Moderate/Low C2P2 Partner -0.14** 0,06 -0.14** 0,06Number of Utility's Plants Near -0.01 0,01 -0.01 0,01Post-C2P2* Utility's Plants Near 0,01 0,02 0,01 0,02Lag Utility Reuse 0,07 0,06 0,04 0,06Aggregates Price -0.01 0,02 -0.01 0,02Cement Price -0.01 0,01 -0.01* 0,00Cement Imports 0,01 0,01 0,01 0,01SCR Installed 0,04 0,03 0,04 0,04Avearge Ash Content -0.01** 0 -0.02** 0,01Total Coal -0.01 0,01 0.01** 0,00N 1088 1088Plants 176 176R2 0,2 0,48State & Year Dummies Not Shown for BrevityErrors Clustered by Utility*, **, *** indicate 10%, 5% and 1% statistical significance, respectively

Fly Ash Reuse Ratio Fly Ash Reuse Ratio

Selection Bias

• If entities are not randomly assigned to treatment or control, selection bias could be a problem

• If treatment is mandatory, selection bias is less likely to be a problem

• If treatment is voluntary, then selection bias is likely to be a problem

Endogeneity

• Another way to describe selection bias is to say that the choice of enetering the voluntary progam is endogenous

• This generally means that the choice of entering the program affects the outcome measure, rather than the outcome measure affecting the choice of entering the program.

Direction of Bias

• One big problem is that the direction of bias could be either positive or negative

• If entities join program because it is easy to acheive goals set, the bias is positive

• If entities join program because they really need the help, the bias is negative.

Example-Positive

• Norway joins the Oslo Protocol because it has large natural gas reserve, making reducing sulfur emissions relatively cheap

• If evaluate the effect of the Protocol on Norway (relative to Poland for example), it will look like the Protocol had a large effect when in reality the change in behavior is driven by the switch to natural gas.

Example-Negative• Poland joins the Sophia Protocol even

though it relies heavily on coal for electricity production

• If we evaluate the effect of the Protocol on Poland (relative to Sweden for example) it will look like the Protocol is having no effect when in reality it is the lack of alternatives for electricity that is driving the result.

Correcting for Endogeneity

• Generally known at Instrumental Variables

• Find other variables that predict the variable that is endogenous

• In the case of selection bias into the policy treatment, this generally means variables that are associated with the benefits or costs of recieveing the treatment

30

Partner Choice Example: C2P2• Dependent Variable:

– Partner Designation

• Controls:– Investment in Solid Waste Disposal 96-00– State w/ CCP re-use authorized– State w/o CCP Permit Requirement– Utility Size– Avg Fly and Bottom Ash Reuse 96-00– Avg Price of Agg. & Cement in State 96-00

31

Example : C2P2Dependent Variable: Model: ProbitExplanatory Variable Coefficient Stnd. ErrorAvg Fly Ash Reuse Ratio 1996-2000 -0.11 0.28Avg Bottom Ash Reuse Ratio 1996-2000 0.2 0.24Avg Aggregates Price 1996-2000 0.34*** 0.08Avg Cement Price 1996-2000 -0.01 0.01Utility Size 0.04*** 0.01Restructed Market -0.41* 0.22NOx SIP Call State 0.15 0.5Total Solid Waste Disposal Investment 1996-2000

0.03 0.02

No CCP Permit Required 0.45** 0.21Re-use Guidelines Set 1.1*** 0.3N 305R2 0.18*, **, *** indicate 10%, 5% and 1% statistical significance

Initial Partner Dummy

Error Specification

• Generally in panel methods, you would like to cluster errors by largest grouping entity possible

• These could be industry classification, firm (as opposed to plant), country, etc

• This controls for unobserved correlation better than a narrow grouping

Other Options

• Bootstrapping of error terms is a catch all – Code is quite easy in Stata to code

• Many other potential fixes available, Stata help very good for this

Policy Evaluation

• Important to:– Consider pre-policy trends– Consider control group’s actions– Other co-variates– How entities enter the policy– How the treatment affects entities

• With these issues covered, you have done as much as is possible as a data user

Questions?

• About Data

• About Methods

• About Examples

• About Information on Policies

• About dealing with policymakers

• …