Do Energy Efficiency Investments Deliver? Evidencefrom the Weatherization Assistance Program

Meredith Fowlie, Michael Greenstone, and Catherine Wolfram∗

September 28, 2014

Abstract

It is conventional wisdom that individuals and firms fail to undertake energy effi-ciency investments that are predicted by engineering models to have private returnsgreatly in excess of their costs. Policymakers have seized on this so called energyefficiency gap with a wide variety of interventions that promise private benefits andreductions in environmental damages, especially declines in the greenhouse gases thatcause climate change. This paper applies experimental and quasi-experimental tech-niques to assess the private and social returns to energy efficiency investments usingdetailed data from the application of the nation’s largest residential energy efficiencyprogram, the Federal Weatherization Assistance Program (WAP), in Michigan. Wefind that participating households’ energy consumption declined significantly. We alsopresent suggestive evidence that households modestly increased indoor temperaturesby about 0.6 degrees F, providing some of the first direct evidence on compensatoryresponse to a reduction in the price of energy services (i.e., the “rebound” effect) inthe building sector. Accounting for the energy savings and consumers’ valuations ofthe higher indoor temperatures, the data indicate that these investments have nega-tive annual returns both privately and socially (i.e., when the monteized value of thegreenhouse gas savings are included). Specifically, the private and social internal ratesof return or discount rate that would rationalize these investments are -4% and -3%,respectively.

∗We received many helpful comments from seminar participants at Carnegie Mellon, Columbia University,National University of Singapore, Resources for the Future, the University of Michigan, and the Universityof Maryland. The authors gratefully acknowledge the financial support of the Alfred P. Sloan Foundation,the Rockefeller Foundation and the UC Berkeley Energy and Climate Initative, and institutional supportfrom the Poverty Action Lab (JPAL) at MIT, the Center for Local, State, and Urban Policy (CLOSUP)at the University of Michigan, and the Energy Institute at Haas. We thank James Gillan, Walter Graf,Erica Myers, and Matthew Woerman for excellent research assistance. We are indebted to Jesse Worker foroutstanding management of a challenging project. Finally, we thank our contacts at both our partner utilityand the community action agencies, without whom this project would not have been possible.

1

Preliminary draft. Please do not cite

1 Introduction

Energy efficiency investments are widely believed to offer the rare win-win opportunity. De-

tailed engineering projections, such as those summarized by the well-known McKinsey curves

(McKinsey & Company, 2009), routinely project net present value positive investments based

on private returns alone (win #1). Moreover, by reducing the energy necessary to achieve a

given level of energy services (e.g., heating or cooling), these investments promise to decrease

greenhouse gas emissions in addition to other pollutants that compromise human health (win

#2).

Importantly, there is a large and persistent difference between the levels of investment

in energy efficiency that appear to be privately beneficial and the investments that private

individuals actually pursue. Over the last three decades, a wide variety of explanations,

have been offered for this apparent failure of consumers to avail themselves of profitable

investment opportunities (see, for example, Allcott and Greenstone, 2012; Gillingham and

Palmer, 2014).

Mounting concern about climate change has increased the urgency of understanding

this ‘energy efficiency gap’. Governments around the world are pursuing a wide range of

policies designed to narrow or close this gap. In 2013, U.S. electric utilities budgeted nearly

$7 billion for efficiency programs; these expenditures are projected to double in the next

decade (Barbose et al. 2013); energy efficiency building codes and appliance standards

entail additional costs. The case for the U.S. government’s fuel economy standards for

automobiles and light trucks and proposed rule to reduce emissions from power plant, which

are the two most important climate policies, depends crucially on win-win energy efficiency

opportunities.1

1For example, 82% of the benefits from the fuel economy standard for automobiles and light trucks comefrom consumer’s suboptimal purchases of vehicles with insufficient fuel economy. A significant share of thereductions in carbon emissions by 2030 from power plants under the proposed Clean Power Plan are expectedto come from demand-side energy efficiency improvements (CATF, 2014).

2

This paper provides some of the first field evidence on the returns to energy efficiency

investments from a randomized control trial. We focus on the residential building sector

which has been identified as a particularly promising source for cost effective opportunities

to reduce growth in energy demand (IPCC, 2014). We focus on interventions that improve

the efficiency of space heating which accounts for over 40 percent of energy consumption in

U.S. homes.2

We use experimental variation in participation in the nation’s largest residential energy

efficiency program, the Federal Weatherization Assistance Program (WAP), to identify the

average effects of these investments on household-level energy consumption. We also exam-

ine the extent to which households receiving weatherization respond to the reduced costs of

energy services by choosing a different indoor temperature (i.e., the “direct rebound” effect).

The large-scale randomized encouragement design experiment was conducted on a sample

of almost 29,000 households presumptively eligible for participation in WAP in the state

of Michigan. Over 7,500 of these households were randomly assigned to an encouragement

treatment. These households were encouraged to apply for the program and received signif-

icant application assistance. The control households were free to apply for WAP but were

not contacted or assisted in any way by our team.

There are three primary findings. First, the encouragement intervention increased WAP

participation from less than 1 percent in the control group to almost 6 percent in the en-

couraged group. The encouragement intervention was implemented by a firm with extensive

canvassing experience, including with low income communities. Field staff made almost

7,000 home visits and nearly 33,000 phone calls, and yet increased the participation rate

by less than 5 percentage points. This increase is surprisingly small given that participat-

ing households receive $5,000 worth of efficiency improvements on average, but entail no

2Estimates from the most recent Residential Energy Consumption Survey (RECS) show that 48% ofenergy consumption in U.S. homes in 2009 was for heating and cooling. For more details, see U.S. EnergyInformation Administration, Residential Energy Consumption Survey (RECS) at http://www.eia.gov.

3

monetary costs

Second, we find that efficiency investments deliver economically and statistically sig-

nificant energy savings among participating households. We estimate that weatherization

assistance reduces monthly energy consumption by 10-20 percent on average; a substantial

assist to participating low-income households in the form of reduced energy bills. However,

we show that the net present value of these energy savings (under a range of plausible as-

sumptions) are all lower than the direct cost of the efficiency investments. These results

call into question the assumption underlying the debate on the energy efficiency gap that

households are by-passing cost effective investment opportunities.

Third, we conducted a follow-up survey of indoor temperatures at weatherized and non-

weatherized homes and find some evidence that measured indoor temperatures are roughly

0.7 degrees F higher (p-value = 0.XX) in weatherized homes. This is consistent with house-

holds re-optimizing energy consumption in response to lower implicit prices per unit of energy

service (e.g., heating). Though the existence of the so-called ‘rebound effect’ has been the

subject of much debate (Gillingham et al., 2013), our findings provide some of the first direct

field evidence of this phenomenon.

In our penultimate section, we discuss the implications of these findings from a private

and social perspective. Accounting for the energy savings and consumers’ valuations of

the higher indoor temperatures, the data indicate that these investments have negative

annual returns both privately and socially (i.e., when the monteized value of the greenhouse

gas savings are included). Specifically, the private and social internal rates of return or

discount rate that would rationalize these investments are -4% and -3%, respectively. We

also calculate the average cost per ton of avoided CO2 under a range of assumptions. The

most plausible estimates are larger than $300/ton, which is much more expensive than the

the U.S. government’s estimate of the social cost of carbon of roughly $37.

This paper contributes to a burgeoning literature that examines potential explanations for

4

the energy efficiency gap. To date, this literature has emphasized the role of market failures

such as imperfect information and principal-agent problems, and behavioral phenomenon

such as bounded attention and heuristic decision-making. There has been relatively little

attention paid to an alternative explanation for the apparent gap: hidden costs (such as

transaction and process costs) and overstated benefits.

Early work by Joskow and Marron (1993) raised concerns about overstated efficiency

potential and underscored the importance of using ex post measures of consumer behavior to

estimate energy savings. Subsequent work by Metcalf and Hassett (1999) uses panel data on

household energy expenditures to estimate realized returns on investments in attic insulation.

Although projected returns on these investments were on the order of 50 percent per year,

these authors estimate a median rate of return below 10 percent. More recently,Jacobsen

and Kotchen (2013) use residential billing data to evaluate the effect of an increase in the

stringency of an energy code on both electricity and natural gas consumption in Florida.

They estimate a private payback period of approximately 6.4 years.

This work also builds on prior studies that highlight important interactions between

consumer behavior and the introduction of energy efficiency improvements that lower the

effective price of the services they provide. In a widely cited study, Dubin et al. (1986)

exploit a small field experiment conducted by a Florida utility in which efficiency improve-

ments were randomly assigned. They find that consumers with improved insulation and

more efficient heating equipment conserve 8-13% less energy than would be predicted from

engineering models. .More recently, Davis, Gertler and Fuchs (2013) use quasi-experimental

variation to measure ex post realized energy savings for an appliance replacement program

in Mexico.They find upgrading the efficiency of air conditioners actually increased energy

consumption, a result they interpret as consistent with a large rebound effect. Overall, they

find that the program is an expensive way to reduce externalities from energy use, reducing

carbon dioxide emissions at a program cost of over $500 per ton . In contrast, we find that

5

rebound effects can explain only a very small fraction of the discrepancy between projected

and ex post realized energy savings. We attribute the remainder of this “measurement gap”

to the engineering models’ overstatement of the savings.

Our paper begins by introducing a conceptual framework for energy efficiency invest-

ments, highlighting possible deviations between modeled and actual behavior. Section 3

summarizes key details on the Weatherization Assistance Program. Section 4 summarizes

the data we have collected. Section 5 describes our empirical strategy. Section 6 presents the

main results on actual savings. Section 7 describes estimation and calculations we performed

to rationalize the measurement gap. Section 8 assesses the overall cost-effectiveness of these

efficiency investments in light of our findings. Section 9 concludes.

2 Conceptual framework

In the economics literature, the most standard definition of the ‘energy efficiency gap’ refers

to the difference between actual and cost-effective energy use. In other words, a gap exists

if individuals systematically overlook energy efficiency investments that confer benefits in

excess of costs. This section provides a conceptual framework for thinking about the bene-

fits generated by efficiency improvements. This framework will serve as foundation for the

subsequent empirics.

Gains from an investment in energy efficiency are realized through two main channels:

reduced energy consumption and, possibly, increased consumption of energy services (e.g.

lighting, space heating, air conditioning). With respect to the first channel, any reduction in

dollars spent on energy can be allocated to other forms of welfare enhancing consumption.

The second channel becomes important when an efficiency-induced reduction in energy end-

use costs leads to an increase or “rebound” in the demand for the energy service. If we

assume agents act to maximize utility, any re-optimization of consumption that occurs in

6

Figure 1: Household-level re-optimization in response to an efficiency improvement

                   

                             X               

 

 

  

 

 

 

 

                                                                                                                         Budget set                     Budget set 

                                                                                                                               Status quo             Weatherization   

 

                                                                                        

 

rebound 

 

upper bound 

 

              E(H;W,Z)                                                                       Status quo          Weatherized 

 

 

 

                              H0 /H’1     H1                                                    Heating services   (H)  

 

E’1 

 

E1 

E0 

 

E’ 

 

 

  

 X’1 

 

 X0 

 

 

response to an efficiency improvement will be welfare improving.

These basic ideas are illustrated in the upper quadrant of Figure 1, which plots con-

sumption of energy services on the horizontal axis and consumption of the numeraire (i.e.,

all other goods), X, on the vertical axis.3 In particular, we focus on space heating demand.

Home heating is a particularly important end-use in our empirical setting; over 93 percent of

projected energy savings from the weatherization investments we analyze are heating related.

The two downward sloping lines in the upper quadrant of the figure reflect budget con-

straints, the lower before weatherization and the higher after weatherization. This pivot

3For ease if exposition, this figure depicts the limited range of indoor air temperatures over which energyconsumption is increasing in indoor temperatures. This range is bounded from below by the outdoor airtemperature (it the thermostat is set below the outdoor air temperature, no heating services are required).The figure also assumes that the capacity of the home heating system does not bind. When the capacityconstraint binds, energy consumption ceases to increase with the temperature differential.

7

in the budget constraint post-weatherization reflects that the price of heating services (i.e.,

the price of increasing the thermostat in winter) has fallen; energy efficiency improvements

reduce the cost of purchasing any given level of thermal comfort.

The figure also illustrates a family of indifference curves for a representative consumer,

each of which trace out the bundles of the numeraire, X, and heating services that deliver

the same level of utility. The shape of the indifference curves reflects that households do

not like to be too hot or too cold. In the status quo (i.e. absent an efficiency improve-

ment), the representative agent will maximize utility through the choice of H0 and X0.The

weatherization-induced expansion of the budget constraint allows the agent to move to a

higher level of utility associated with the bundle of H1 and X1. In the figure, status quo

consumption occurs below the satiation point for thermal comfort. Thus, when the price

of thermal comfort falls, demand for heating services increases by H1 − H0. The positive

income effect also increases consumption of the numeraire by X1 −X0.

The paper’s empirical challenge is to measure the welfare gains conferred by weatheriza-

tion investments. Our empirical setting allows us to develop a measure of willingness to pay

(WTP) for weatherization that accounts for both reductions in energy expenditures and the

increased consumption of space heating. The effect of energy efficiency improvements on

other consumption (X) can be measured using data on monthly energy expenditures which

vary one for one with the consumption of the numeraire. Put another way, a $1 decrease in

energy expenditures allows for a $1 increase in consumption of all other goods. Measuring

willingness to pay for the increase in heating services (i.e, the direct rebound effect) is more

challenging because demand for energy services (such as heating) is not readily observable

in household energy consumption or expenditure data.

To obtain an estimate of the efficiency-induced increase in demand for heating (H1−H0 in

the figure), we conduct a survey of indoor temperatures in weatherized and non-weatherized

homes. With this estimate in hand, we can construct bounds for the welfare consequences

8

of this increase in warmth if we impose some structure on the relationship between heating

energy demand and temperature.4

The bottom quadrant of Figure 1 plots a representative building-specific relationship be-

tween indoor temperature and the energy required to achieve that temperature E, holding

constant outdoor temperatures, W , and building characteristics, Z. 5 Efficiency improve-

ments to the building envelope (e.g. insulation improvements, window sealing, a furnace

upgrade) should reduce the energy required to deliver any given level of heating services.

This implies that the slope of the relationship between temperature and energy consumption

becomes less steep following an efficiency improvement. We can use rich data on energy con-

sumption and temperature variation over the study period to estimate these relationships

between temperature and heating demand in weatherized and unweatherized homes.

Building on the top quadrant, the relationship between heating demand and energy

consumption can be used to construct bounds on the utility gains from the efficiency-induced

increase in the demand for indoor temperature using revealed preference logic. Since the

agent chooses to increase heating services from H0 to H1following the efficiency improvement,

it follows that a lower bound for the increased utility derived from this increase in heating

services is the associated increase in heating costs incurred after weatherization– this is

represented by PE*(E1-E′1) in the figure. Note that this agent chose not to choose H1prior

to the efficiency improvement. Thus, the cost of this change in heating services prior to

weatherization, measured by PE*(E′0 − E0), provides an upper bound on the welfare gain.

4Our measure of the returns to weatherization investments accounts for a household’s valuation of in-creased warmth, but we do not account for any increase in comfort conditional on indoor air temperature.Researchers have noted that improvements in insulation enhances comfort by reducing drafts and heat ra-diation while retaining moisture (Schwarz and Taylor, 1995). The enhanced comfort allows the consumerto achieve comfort at a lower thermostat setting. Consequently, what we measure here is the net effect ofefficiency improvements on heating demand. We return to this point in our final discussion.

5The figure assumes a linear relationship between air temperature and energy demand over the relevantrange of indoor air temperatures. In the literature that analyzes energy use in buildings, this linearityassumption is fairly standard (e.g. Friedman (1987), Dyson et al.,2014). Other researchers have argued thatthis relationship is better described as non-linear (e.g. Dewees and Wilson (1990)). In section 5, we estimatemore flexible specifications and provide empirical support for this linearity assumption in our data.

9

Preferences revealed prior to the efficiency improvement suggests that the agent values the

increase in heating services less than this incremental cost.

The resulting measure of WTP can be compared to the costs of these investments to

estimate net returns and to gain insight into individuals’ decisions to make energy efficiency

investments. Specifically, the net present value of an energy efficiency investment is given

by:

NPV = C −T∑t=1

WTPt

(1 + r)t, (1)

whereC measures the private costs to the household of the energy efficiency improvement;

T represents the investment time horizon for the improvement; r is the discount rate. Will-

ingness to pay for an energy efficiency investment in period t is denoted WTPt which is

the sum of the change in energy expenditures and the monetized value of the change in the

consumption of energy services

A simple model of households’ investment choices assumes that a household will invest if

NPV > 0.6 It is noteworthy that the standard approach in the energy efficiency literature is

to ignore the utility value of any change in the consumption of energy services and effectively

set WTPt equal to the value of energy savings assuming no energy demand response.

Previous work points out several reasons why households’ investment choices may devi-

ate from this simple model. For one, this equation assumes households are fully informed

both about all costs associated with the retrofits and their energy usage with and without

retrofits. Homeowners may make systematic optimization mistakes if they are inattentive

to energy costs, unable to convey the precise energy usage of an improved house or apart-

ment they are selling or renting to others (Allcott and Greenstone 2012; Myers, 2014), or

6For simplicity, we are assuming that there is single energy efficiency improvement with a distinct lifetimeT . The home energy efficiency retrofits we will ultimately be interested in analyzing comprise several differentmeasures (e.g., insulation as well as duct sealing). In these investment scenarios, projected energy savingsaccount for interactions between interventions when forecasting post-retrofit energy use.

10

if they face imperfect credit markets. Households also may value a home with retrofits dif-

ferently, meaning there is some utility loss or gain associated with the retrofit that is not

captured by Equation 1. Finally, Equation 1 assumes risk neutrality, which may not hold

given uncertainty about future energy prices and future energy consumption.

With this paper, we do not attempt to disentangle the relative importance of these

explanations. We are primarily concerned with estimating the return on investments in

efficiency. With these estimates in hand, we conduct a series of exercises that shed light on

the cost effectiveness of these efficiency investments and the extent to which an efficiency

gap exists in this context. Specifically, we calculate the internal rate of return implied by

estimating equation 1that justifies these investments under a range of assumptions. Further,

we will calculate the cost per ton of CO2 abated.

3 Background and study design

3.1 Weatherization Assistance Program

The Weatherization Assistance Program (WAP) is the nation’s largest residential energy-

efficiency program. The WAP program supports improvements in the energy efficiency of

dwellings occupied by low-income families. Since its inception in 1976, over 7 million low-

income households had received weatherization assistance through the program. Proponents

credit the program with saving energy, creating jobs, reducing emissions, and assisting low-

income households. The American Reinvestment and Recovery Act PL111-5 (ARRA) dra-

matically increased the scale and scope of WAP.7 Our analysis seeks to estimate the impacts

of weatherization assistance over the ARRA-funded time period.

WAP funds are distributed to states based on a formula tied to a state’s climate, the num-

7Funding increased from $450 million annually in 2009 to almost $5 billion for the 2011-2012 programyears. Under the ARRA-funded program, all owner-occupied households at or below 200 percent of thepoverty line were presumptively eligible for assistance.

11

ber of low-income residents, and their typical energy bills. The states distribute WAP money

to over 1000 local sub-grantees, which are typically community action agencies (CAAs) or

similar nonprofit groups. These sub-grantees are then tasked with identifying and serving

eligible households. Participating WAP households receive free energy audits and a home

retrofit that typically includes some combination of insulation, window replacements, fur-

nace replacement, and infiltration reduction. The average value of measures installed at

participating households in our data is approximately $5000 per home.

The process of applying for weatherization is highly onerous and time intensive, at least

partially to prevent fraud. Extensive paperwork documenting income eligibility must be

completed and submitted in a timely fashion. This paperwork includes utility bills, earnings

documentation, social security cards for all residents of the home and deeds to the home.

Once the paperwork is completed, households are put on a list where the waiting times

can exceed one year. After rising to the top of the list, homeowners must accommodate

scheduling of energy audits and construction crews who do the retrofits.

The energy audits are a critical step in the weatherization process. Program auditors visit

an applicant’s home to collect detailed information about the building structure, heating and

cooling systems, appliances, ventilation, etc. Taking into account local weather conditions,

retrofit measure costs, fuel costs, and specific construction details of the home, auditors es-

timate energy savings and costs of different efficiency measures. The WAP program requires

that all recommended measures return a minimum of $1.00 in incremental savings for every

$1.00 expended in labor and material costs.

The process that determines which households receive weatherization assistance is not

purely random. Local agencies often identify potential applicants from the pool of households

that are receiving other social services, although walk-in clients are routinely admitted. Local

agencies screen potential applicants for eligibility. Eligible applicants are then prioritized

following guidelines that recommend CAAs rank applicants highly if the household includes

12

elderly, persons with disabilities or families with children, or where the occupants typically

face a high energy burden (energy as a share of income) or have high residential energy use

(see 10 CFR 440.16(b) (1-5)).8 Consequently, any comparisons of energy consumption across

weatherized and un-weatherized households risks confounding the effect of the program with

pre-existing differences in determinants of energy consumption.

3.2 Research Design

The paper’s empirical challenge is to obtain causal estimates of the effect of participation

in the WAP program on energy consumption and indoor heating. The difficulty is that

households that participate in WAP are not chosen randomly from the universe of households.

This paper uses quasi-experimental and experimental approaches to address the potentially

confounding effects of non-random selection into the WAP program.

We collect data from a sample of Michigan households. Michigan is one of the largest

recipients of WAP program funding on account of its cold winters and large low-income

population. Further, we were able to secure collaborative agreements with a major Michigan

utility and five WAP agencies working in this utility’s service territory. This allowed us

access to detailed, household-level energy consumption and weatherization program data.

Michigan received over $200 million in ARRA funding for weatherization assistance.

Figure 2 plots the number of weatherizations completed at households in our data set, since

2010. The solid black line in the figure shows how activity ramped up dramatically in early

2011 when ARRA funds became available. All stimulus funds had to be spent by March

2012. After that point (marked by the solid vertical line), the pace of weatherization activity

dropped precipitously.

8Given the high ARRA funding levels during our study period, the prioritization scheme was less bindingas compared to lower funding periods.

13

Figure 2: Household weatherizations (monthly) 2010-2014

Notes: This figure plots the number of weatherization retrofits completed per month. Thebroken line marks both the start of our encouragement intervention and the point at whichstimulus funding began to stimulate additional weatherization activities. The vertical solidline marks the deadline for spending stimulus funds.

14

3.3 Quasi-experimental design

The quasi-experimental research design compares energy consumption at homes weatherized

after March 2011 with energy consumption at homes that applied for weatherization during

our study period, but have yet to receive any efficiency improvements.

Among the households applying for weatherization after March 2011, only 40 percent

actually receive weatherization assistance during our study period. The road from applica-

tion to energy efficiency investments is long and there are many potential off-ramps. The

explanations for why applicant households fail to received these investments include: failure

to complete the necessary paperwork or to schedule/complete an energy audit, risks to WAP

employees (e.g., due to the presence of asbestos in the home), and a long waiting list such

that many applicant households do not receive assistance during the period covered by our

data. Some of these explanations for program participation are orthogonal to household

characteristics that determine energy consumption patterns, while others clearly are not.

We estimate the following equation designed to control for the effects of unobservable

factors that determine energy consumption trajectories at households that apply for weath-

erization assistance:

yimt = β1{WAP}imt + αim + αmt + εimt, (2)

where yimt measures the energy consumption (natural gas, electricity, or a combined

measure)of household i in month m and year t. The WAP indicator variable switches from

zero to one in the month after a household’s weatherization retrofit is complete. The equation

includes household-by-month fixed effects, αim, to account for permanent differences in a

household’s energy consumption in a given month. It is possible to include such a rich set

of fixed effects that account for permanent household-specific seasonal variation in energy

consumption due to the multiple years that households are observed in our data. The model

15

also includes month-by-year fixed effects, αmt, to adjust for the average effects of time-varying

factors (e.g., winter temperature) across households.

The parameter of interest is β and it measures the mean difference in energy consumption

subsequent to the completion of WAP energy efficiency investments, after adjustment for the

fixed effects. It is a difference-in-differences estimator that compares the change in energy

consumption after weatherization to before, relative to consumption among households that

have either not yet weatherized through WAP or never will.

In equation 2, the primary threat to consistent estimation of β is the possibility that time-

varing factors that affect household demand for energy also influence WAP participation. For

instance, households may choose to aggressively push forward their WAP application when

they anticipate an increase in their demand for energy as would be the case when the number

of people in their household increases or they lose a job and expect to spend more time at

home. While the quasi-experimental approach does not have a direct solution to this threat

to identification, we can do more to balance observable characteristics and trends across

weatherized and unweatherized households.

To evaluate the robustness of our findings, we re-estimate Equation 2, using alternative

sets of controls. In one specification, we drop all voided applicants out of concern that these

households may differ from successful applicants in ways that affect energy trajectories over

time. Our preferred quasi-experimental estimates re-weight control observations in order to

achieve covariate balance across weatherized and un-weatherized controls. We estimate the

propensity score, or the conditional probability of receiving weatherization assistance, using

a logit specification. Estimated participation probabilities are a function of historic natural

gas and electricity consumption, trends in energy consumption over the pre-treatment period,

household income, county of residence, and other factors that can play a role in determining

whether a household receives weatherization assistance (e.g. number of children). We then

re-estimate Equation 2 using estimated inverse probability weights for all unweatherized

16

applicants.

Randomized encouragement design

The paper’s primary empirical estimates are derived from an experimental design. The basis

of the experiment is a randomly assigned encouragement intervention that aims to increase

the probability of treatment household’s participation in WAP through recruitment and

significant application assistance. This research required close collaboration with both our

partner utility and community action agencies that serve households in the utility’s service

territory. Having established these partnerships, we confined our attention to households

living in the counties served by our partner agencies. Within these counties, we identified

census blocks that are associated with high rates of home ownership, natural gas heating,

and household incomes that would quality for weatherization assistance. Households living

in these census blocks comprise our population. From this population, we randomly selected

34,161 households for our study sample. Approximately one quarter of these households

were randomly assigned to our encouragement “treatment”. These treated households were

encouraged to apply for WAP and offered extensive application assistance. We provide no

recruitment, outreach, or assistance to the remaining 25,513 households. For households

assigned to the control groups, we simply observe energy consumption and program partici-

pation decisions.

To implement the recruitment and assistance, we issued a request for proposals from

organizations specializing in grassroots organizing and mobilization. We selected Field-

Works LLC, a private company that specializes in designing communications strategies,

running neighborhood canvassing operations, and managing outreach campaigns. Prior to

our project, their staff had generated millions of phone calls and knocked on millions of

doors in previous engagements. We found them to be highly focused and innovative in their

approach to educating households about weatherization assistance, and helping individuals

17

to navigate the application and enrollment process.

We worked closely with FieldWorks to develop a persuasive recruit and assist strategy.

Fieldworks personnel, together with our field manager, coordinated field operations in Michi-

gan. Individuals selected from the targeted communities were hired to conduct the bulk of

the door-to-door canvassing and outreach activities. The encouragement campaign ran from

March to May, 2011. Enrollment assistance activities lasted through February 2012. Figure

2 illustrates how our encouragement campaign got underway around the same time that

stimulus funds began accelerating the pace of weatherization activity in Michigan.

Table 1 summarizes these encouragement and enrollment activities in detail.During the

encouragement phase, field staff made almost 7,000 initial, in-person house visits.9 . These

ground operations were complemented with 23,500 targeted “robo-calls” to raise awareness

of both the weatherization program and our encouragement campaign.

After an initial encouragement phase designed to generate interest in and awareness of

the program, we transitioned to an enrollment phase. The application process required

households to provide extensive documentation, including utility bills, social security cards

for all household members, and the deed for the house. Further, a minority of applicants

were required to provide further documentation or to correct something on their initial

application. Our staff made over 9,000 personal phone calls to provide assistance and to

coordinate in-person meetings. Over the course of 2,720 home visits, our field staff helped

individuals assemble documentation and complete paperwork. In some cases, our field staff

provided transportation to and from the program agency offices.

The final row of Table 1 reports that we spent around $445, 000 on the encouragement or

roughly $50 per household in the treatment group. It is noteworthy that we did not initially

intend to devote such extensive efforts and resources to the encourage and enrollment phases.

9Most- but not all- houses assigned to the treated group were contacted. A small fraction were deemedinaccessible (e.g., because of a locked gate).

18

Table 1: Randomized encouragement intervention

Encouragement activity

Encouraged group (households) 8,648

Initial home visits 6,694

Robo-calls 23,500

Personal calls 9,171

Follow up appointments 2,720

Average cost/encouraged hh $51.50

Note: The table summarizes efforts to encourage a group of Michigan households to take up weatherizationassistance. These households were selected randomly from a sub-population of households who were locatedin the service territory of our partner utility and presumptively eligible based on ex ante available incomeinformation.

However, the early results suggested that we were failing to have a substantial impact on

applications and concerns about the ultimate precision of our estimated treatment effects

motivated us to raise additional monies to extend this phase of the project.10

Figure 2 provides some insight into how this randomized encouragement intervention af-

fected program participation over time. The dotted line shows monthly enrollment in the ex-

perimental control group. The broken line illustrates enrollment in the (smaller) encouraged

group. We use random assignment to encouragement as an instrument for weatherization

status. In a first stage, we estimate:

10On the one hand, these costs may have been higher than necessary. This was the first time that anencouragement program for WAP has been attempted; there was some learning by doing. On the otherhand, these costs do not reflect the time that the research team devoted to overseeing the encouragementeffort.

19

1{WAP}imt = θ1{Encouraged}imt + δim + δmt + ηimt, (3)

where the dependent variable is as described in the previous subsection and there are

also household by month, δim, and month by year, δmt, fixed effects. The indicator variable,

1{Encouraged}imt is set to zero for all households prior to the encouragement intervention

which began. After March 2011, this indicator switches to 1 for the 25% of the experimental

sample that are randomly assigned to the treatment group. The estimated impact of reduc-

ing barriers to participation (e.g. information and process costs) on program uptake is of

independent interest both to policymakers and researchers.

We substitute ˆ1{WAP}imt from the estimation of equation (3) to fit equation (??) to ob-

tain ˆβIV . In this instrumental variables (IV) framework, ˆβIV is identified using the variation

in program participation that is generated via the random assignment of encouragement;

this guarantees a consistent estimate of β.

If there is heterogeneity in how weatherization retrofits impact residential energy con-

sumption, the expectation of unbiased quasi-experimental and experimental estimates β need

not be equivalent.11 Note that our quasi-experimental approach is designed to provide an

estimate of the average treatment effect on all treated households (ATET), or an average

across the full distribution of treatment effects. In contrast, the randomized encouragement

design is used to estimate the so-called local average treatment effect (LATE), or the av-

erage effect for the subset of the population who must be encouraged to participate in the

program (i.e. the compliers) (Angrist, Imbens, and Rubin 1996). In other words, differences

in the quasi-experimental and experimental estimates could be due to bias in the former or

differences in the LATE and ATET.

11Note that this will only strictly be true when we limit the quasi-experimental sample to the same twoCAAs that comprise the experimental sample.

20

4 Data Sources and Summary Statistics

4.1 Data Sources

The data summarized below correspond to two overlapping groups of households. The first

group comprises the 34,160 households in our experimental sample. To conduct this field

experiment, we worked closely with two large agencies serving households in the service ter-

ritory of our partner utility. Our experimental sample is thus drawn from the counties served

by these two agencies. The second group of households corresponds to our quasi-experimental

research design. This design did not require the same degree of coordination with our agency

partners; we were able to expand the scope of our efforts to include five implementing agen-

cies. The quasi-experimental sample therefore includes the 7,304 households that applied for

weatherization assistance at these five agencies. This quasi-experimental sample is smaller

overall but has a larger number of applicants and weatherized households, relative to the

experimental sample. At the intersection of these two groups are the 1,773 of these applicant

households who are also are part of our experimental sample.

4.1.1 Energy Consumption Data

We obtained monthly natural gas and electricity consumption data over the period January

2008 to May 2014. This period includes at least three years of pre-retrofit data for all weath-

erized households in our sample.12 The utility data track monthly kilowatt-hours (kWh)

of electricity and thousand cubic feet (Mcf) of gas used at the dwelling. We convert both

of these variables to million British thermal units (MMBtu) using the standard conversion

factors employed by the WAP program.13 .

12We identify a household as a particular account number at an address. For some addresses, we havedata on multiple accounts, if one household moved out and another moved in, but we are working with theutility to fill in data for more addresses where a single account does not cover the whole time period.

13We also reconfigure the data to account for the fact that consumption records correspond to billingcycles or segments versus calendar months. The utility assigns households to one of 21 billing portions. In a

21

Energy consumption records obtained from the utility are merged with households in our

experimental sample and the applicant data which comprise our quasi-experimental sample.

Data are merged using detailed name and address information. Not all households find an

exact match in consumption records. Match rates are 85 and 69 percent in our experimental

and quasi-experimental samples, respectively.14

4.1.2 Energy efficiency measures

Before implementing a weatherization retrofit, CAA program staff conduct an energy audit

of the home. Detailed data collected during the audit are used to calibrate a computer-based

audit tool: the National Energy Audit Tool (NEAT). This tool uses engineering algorithms

to model the energy use of single-family and small multi-family residential units. NEAT is

the most widely used tool for weatherization audits; it is used by state and local WAP sub

grantees, utility companies, and home energy auditors.(EERE (2013))

The purpose of the audit is to make recommendations regarding which efficiency im-

provements (i.e. building envelope, space-heating and space-cooling system, and baseload

energy efficiency measures) should be implemented at the home. Auditors first estimate the

impacts of different combinations of efficiency measures on household energy consumption.

Projected energy savings are then translated into annual bill savings. The present value of

energy savings are calculated using a discount rate of 3 percent and an engineering estimate

of the lifespan of the measures.15

given month, each portion maps to a different set of calendar days. For example, one household’s June billmay reflect consumption in all of June, whereas another household’s June bill captures the last half of Apriland the first half of June. The utility provided us with a detailed mapping of billing segments to calendardays over the duration of our study period. For each customer and for each billing cycle, we divide meterreads evenly across days in the cycle. These “meter-day” measures are then aggregated by calendar monthto construct estimates of monthly consumption at the household.

14There are several possible reasons why our match rates are not perfect. For example, the householdmember holding the utility account may have a different surname than the household applying for weath-erization. There were also several applicant households located outside the service territory of our partnerutility.

15The 3% discount rate is consistent with OMB guidance on how to evaluate benefits of federal spending

22

Importantly, whereas program auditors are careful to calibrate efficiency audit analysis to

the characteristics of the dwelling, they do not calibrate their analysis to reflect the behavior

of the occupants. The rationale is that calibrating savings estimates to the behaviors of

the occupants would over-allocate program resources to more intensive users.16 Also, the

engineering estimates of savings do not aim to account for any behavioral responses to the

increased efficiency of energy services, such as the rebound effects outlined above.

For each audited household, auditors generate a report of recommended energy efficiency

measures that identifies—both individually and cumulatively—the projected energy savings,

installation cost, and the savings:investment ratio of the recommended measures. We have

obtained all of the information used to calibrate the simulations and generate the report for

all audited households in our sample. We have also obtained the work summaries filed once

the weatherization is complete. This provides information about what the measures actually

cost to install.

4.1.3 Indoor Temperature Data

Two years after our encouragement effort was initiated, a subset of weatherized and un-

weatherized households were selected randomly for a field survey. The primary purpose of

the survey was to collect measurements of indoor temperatures that can be used to test for

a direct “rebound effect.”

Michigan field staff attempted to contact 6,400 households on cold days (temperatures

below 45 degrees Fahrenheit) in March and early April 2013. Survey questions were de-

signed to collect information about thermostat set points and, where applicable, household’s

experience with weatherization. With the home owner’s permission, surveyors entered the

home, closed the door, moved to the center of the room, and recorded multiple indoor air

but is substantially lower than the cost of borrowing for most households, especially low income ones.16For example, a furnace replacement would appear to generate higher energy savings for a household

that was keeping its thermostat at 80 degrees during the winter than for a household that was keeping itsthermostat at 65 degrees.

23

temperature measurements using two different devices. Whenever possible, temperature

measurements were recorded in multiple rooms. Of our initially targeted sample, surveyors

spoke with1,658 home owners. Of these, 688 allowed us to enter their homes, close the door,

and collect two or more indoor thermometer readings. Survey results are discussed in detail

in Section X.

4.2 Summary Statistics

Table 2 summarizes pre-treatment information on the households in both the RED and quasi-

experimental samples. The top panel summarizes monthly energy consumption during the

immediate pre-treatment period: 2010-2011. The first two columns summarize means and

standard deviations (in parentheses) for the randomized encouragement and experimental

control groups, respectively. The third column reports differences between the control and

treatment groups. The fourth column summarizes these data for all weatherized households.

The final three columns correspond to the quasi-experimental controls.

Winter natural gas consumption (which is dominated by space heating) is significantly

higher than summer gas use (comprised primarily of hot water heating and cooking). Elec-

tricity usage, also measured in MMBtu, is fairly consistent across seasons.17

Households in the experimental sample were randomly assigned to the encouraged and

control groups. It is therefore to be expected that the differences in natural gas and electricity

consumption across these two groups, reported in column 3, are all small and statistically

indistinguishable from zero.

The table also provides an opportunity to judge the credibility of the comparisons that

underlie the quasi-experimental estimates that complement the paper’s experimental esti-

17Overall, these lower income households in our sample consume less energy on average as compared tothe larger Michigan population. In our sample, average monthly natural gas consumption is approximately7.33 MMBtu. Across the entire service territory, the annual residential natural gas consumption averages7.75 MMBtu per month.

24

Tab

le2:

Diff

eren

ces

insa

mple

mea

ns

bet

wee

ngr

oups

ofhou

sehol

ds

Exp

erim

enta

lE

xp

erim

enta

l(1

)-

(2)

All

Unw

eath

eriz

ed(3

)-

(4)

(3)

-m

atch

eden

cou

rage

dco

ntr

olw

eath

eriz

edap

pli

cants

app

lica

nts

(1)

(2)

(3)

(4)

(5)

(6)

(7)

Pre

-tre

atm

entperiod

month

lyenerg

yconsu

mption

Win

ter

gas

(MM

Btu

)10.

4010

.38

0.02

9.88

11.6

3-1

.75∗∗

0.08

(5.3

6)(5

.23)

(0.0

7)(3

.76)

(5.1

1)(0

.16)

(0.1

5)

Su

mm

ergas

(MM

Btu

)2.8

42.

790.

061.

802.

16-0

.36∗

∗0.

04(3

.87)

(1.9

3)(0

.03)

(1.7

5)(1

.94)

(0.0

5)(0

.06)

Win

ter

elec

tric

ity

(MM

Btu

)2.1

22.

100.

022.

242.

30-0

.06

0.00

(1.1

7)(1

.20)

(0.0

2)(1

.39)

(1.3

9)(0

.04)

(0.0

0)

Su

mm

erel

ectr

icit

y(M

MB

tu)

2.17

2.17

0.00

2.23

2.20

0.03

0.00

(1.3

0)(1

.28)

(0.0

2)(1

.39)

(1.3

9)(0

.04)

(0.0

0)

Demogra

phicsand

dwellingch

ara

cteristics

Hou

seh

old

inco

me

($)

19,6

1717

,509

2,10

8∗∗

-965

(12

,018

)(

11,9

48)

(417

.16)

(522

.31)

Per

cent

of

pov

erty

(%)

115

104

11∗∗

-4(

55)

(64

)(1

.96)

(2.6

7)

Hou

seh

old

size

(#)

2.56

2.47

0.09

-0.0

8(1

.68)

(1.

51)

(0.0

6)(0

.07)

Ch

ild

ren

0.24

0.15

0.10

∗∗-0

.02

(sh

are

of

hou

seh

old

s)(0

.43)

(0.

35)

(0.0

1)(0

.02)

Rep

orte

dd

isab

ilit

y0.

040.

030.

01∗

0.00

(sh

are

of

hou

seh

old

s)(0

.19)

(0.

16)

(0.0

0)(0

.01)

Eld

erly

0.23

0.13

0.09

∗∗-0

.00

(sh

are

of

hou

seh

old

s)(0

.42)

(0.

34)

(0.0

1)(0

.02)

Hig

hsc

hool

edu

cati

on

0.46

0.36

0.10

∗∗0.

08∗∗

(sh

are

of

hou

seh

old

s)(0

.50)

(0.

48)

(0.0

1)(0

.02)

Age

of

hom

e(y

ears

)59

.15

62.0

8-2

.93

2.02

(26.

63)

(21

3.28

)(5

.34)

(7.8

1)

Hou

seh

old

s7,

549

21,3

392,

074

2,97

3

Not

e:C

olu

mn

snu

mb

ered

(1),

(2),

(3),

and

(4)

rep

ort

aver

age

valu

esan

dst

an

dard

dev

iati

on

s(in

pare

nth

eses

).O

ther

colu

mn

sre

port

diff

eren

ces

inm

eans

(sta

nd

ard

erro

rsar

ein

par

enth

eses

).F

inal

colu

mn

rep

ort

sd

iffer

ence

sin

mea

ns

bet

wee

nw

eath

eriz

edh

ou

shold

san

dp

rop

ensi

ty-s

core

wei

ghte

dco

ntr

ols

(con

dit

ion

ing

onap

pli

cati

on).

Sig

nifi

cant

atth

e5

per

cent

leve

l*

Sig

nifi

cant

atth

e1

per

cent

leve

l

25

mates.Column 5 of the table summarizes data from the pool of possible quasi-experimental

controls: households that have applied for weatherization but have not received weather-

ization assistance. The mean differences in column 6 show that weatherized households

have historically consumed significantly less natural gas than the un-weatherized applicants

during both winter and summer months.

The significant differences between the weatherized and unweatherized applicants mo-

tivate us to re-weight observations in the control group so that observable factors are dis-

tributed similarly in the weatherized and unweatherized applicant groups. We use an es-

timated propensity score to balance covariates that presumably play a role in determining

program take-up such as historic energy use, income, and other demographics. This reweight-

ing exercise is discussed in detail in Section 5. The final column in Table 2 reports differences

in average covariate values across weatherized households and propensity-score weighted con-

trols. Reassuringly, this reweighting eliminates all significant differences in pre-period energy

consumption.

The lower panel in Table 2 summarizes the detailed demographic information and dwelling

characteristics that are collected as part of the application process. These data are not avail-

able for the majority of households in our experimental sample that choose not to apply to the

program. Weatherized households have higher incomes in an absolute and relative sense.18

Weatherized households report having more children, are more likely to report an elderly

resident, and less likely to report a disabled resident.

Table 3 reports on the detailed information collected during the household energy ef-

ficiency audits which inform the implementation of the weatherization assistance program.

The table summarizes data from 1,630 households receiving weatherization assistance during

18Program eligibility is based on the “percent of poverty”. The Census Bureau uses a set of moneyincome thresholds that vary by family size and composition to determine who is in poverty. To qualify forthe program, a household’s income cannot exceed 200 percent of poverty. Applicants fall well below thisthreshold on average. Incomes among applicants who ultimately receive weatherization are farther abovethe poverty line on average.

26

the period covered by our data.19

The top panel of Table 3 summarizes the projected energy savings. The simulations

predict that the average household will save over 50 MMBtu of natural gas each year as

a result of the efficiency improvements installed. Using auditor’s estimates of households’

baseline energy consumption, this translates into a quite substantial 46 percent reduction

in gas use. Weatherization investments are projected to have much smaller impacts on

electricity consumption with NEAT predicting a 16% decline in electricity consumption.

The middle panel of Table 3 summarizes information on the costs and projected savings

for the WAP projects. The average project involved more than $4,400 in direct expenditures

on energy saving measures. This includes materials, labor, and construction costs, but

does not include any program overhead. If costs on non-energy measures (such as safety

improvements, carbon monoxide detectors, etc) are included, average costs exceed $5,100.

Ex post realized expenditures are somewhat lower than expected. .

Using a 3% discount rate, the projected net present value of energy bill savings average

$10,680, yielding an average projected savings to investment ratio of over 2.4. Recall that

the program will only fund efficiency measures with projected savings to investment ratio

exceeding 1.

The bottom panel provides summarizes the measures recommended and implemented at

weatherized houses. Attic insulation occurs at 85% of households, while furnace replacement

(34%) and wall insulation (44%) are less frequent.

19Only 1,630 of the 2,194 NEAT data files could be confidently matched with weatherized households. Weapplied fairly strict matching criteria so as not to mis-match weatherized households with audit information.Occupant names were often not included in the audit files. Addresses appear to have been miscoded inseveral instances.

27

Table 3: Projected costs, savings, retrofit measures at weatherized households

Weatherizedhouseholds

Energy savings (projected)

Natural gas savings: heating (annual MMBtu) 50.37(46.03)

Heating energy savings/ baseline 0.46(0.20)

Electricity savings: cooling (annual MMBtu) 4.29(50.53)

Cooling energy savings/ baseline 0.16(0.12)

Investment costs and projected savings

Investment cost: reported ($) 4,146(2,692)

Investment cost: projected ($) 4,406(2,491)

Projected NPV savings ($) 10,680(11,862)

Projected savings:investment ratio 2.47(2.92)

Key measures

Furnace replacement 0.34(0.47)

Attic insulation 0.85(1.06)

Wall insulation 0.44(1.16)

Infiltration reduction 0.76(0.43)

Households 1,630

Notes: This table summarizes data from all weatherized households that could be exactly matched withaudit data.Average values reported, standard deviations appear in parentheses.

28

5 ResultsQuasi-experimental Estimates of Energy Sav-

ings

Table 4 presents the quasi-experimental estimates. The dependent variable in all regressions

summarized by this table is the log of monthly energy consumption (measured in MMBtu).

The first two columns use data from all weatherization applicants. Specifications differ only

in terms of the fixed-effects included, with the second specification allowing time period

effects to vary across counties. Columns (3) and (4) drop voided applications from the

sample. Columns (5) and (6) report the propensity score reweighted estimates.20 Standard

errors are clustered by household in all specifications.

The first row in Table 4 reports the estimated average treatment effect. Because the

dependent variable has been log transformed, these estimates can be interpreted in terms

of percentage reductions in monthly energy consumption. Estimates range from approxi-

mately 8-10 percent across specifications. The second row reports average monthly energy

consumption in the corresponding control group during the post-encouragement period. Im-

plied energy reductions range from 0.6 MMBtu/month (implied by specifications (3) and

(4)) to 0.92 MMBtu/month (specification (6)).

Panel B of Table 4 computes the present value of the estimated energy savings under

alternative assumptions about investment time horizons and discount rates. To express our

estimates of monthly energy savings into dollar terms, we decompose the estimates reported

in Table 4 into fuel-specific savings. More precisely, we re-estimate equation (?? ) for nat-

ural gas and electricity, respectively (See Appendix). Estimates of average monthly natural

gas savings are multiplied by the average residential retail price of natural gas (in $2013)

charged by this utility over the post-weatherization period ($7.98/MMBtu). Similarly, aver-

20Because some of the covariates included in the propensity score estimating equation (e.g. reporteddisability and number of children) are not reported by all households, this sample is somewhat smaller.

29

Table 4: Quasi-experimental estimates of the impact of weatherization on household energyconsumption

Dependent variable is monthly energy consumption (in logs)

All applicants Drop voids Matched controls(1) (2) (3) (4) (5) (6)

WAP -0.08∗∗ -0.09∗∗ -0.08∗∗ -0.08∗∗ -0.09∗∗ -0.10∗∗

(0.01) (0.01) (0.01) (0.01) (0.01) (0.01)

Average consumption 8.13 7.28 9.27(MMbtu/month) [ 7.80] [7.09] [8.27]in control group

month-of-sample FE Y N Y N Y Nmonth-of-sample x county FE N Y N Y N YP-score matched sample N N N N Y Y

R-squared 0.85 0.86 0.85 0.86 0.80 0.81

Households 5,012 5,012 4,330 4,330 3,403 3,403Observations 282,196 282,196 247,,742 247742 188,235 188,235

Present value of (discounted) savings

Time Horizon Discount rate3 percent 6 percent 10 percent

10 years $1,297 $1,119 $99415 years $1,815 $1,476 $1,15620 years $2,261 $1,743 $1,294

Note: Panel A reports estimates of the reduction in monthly energy consumption following weatherization. The unit ofobservation is a household-month. The dependent variable is monthly household energy consumption (electricity andnatural gas) measured in MMBtu. The coefficient reported is the coefficient on the weatherization indicator. Standarderrors (in parentheses) are clustered at the household level. Panel B reports the net present value of energy savingsimplied by the preferred estimate reported in. Panel B computes the net present value of estimated savings. Reductionsin energy bills associated with the estimates in column (6) are assumed to accrue over the life of the measure. using arange of discount rates and assumed time horizons.* Significant at the 5 percent level** Significant at the 1 percent level

30

age electricity savings are multiplied by the average retail electricity price ($0.11/kWh).21We

estimate that weatherized households save $7.82 per month on average in avoided gas ex-

penditures and $4.84 per month in electricity costs. This implies an average savings of $152

per year. 22

To compute the net present value of these energy savings, we assume that the effect of

weatherization on energy consumption - and energy prices- do not vary over the life of the

measure. Discounted benefits are calculated at discount rates of 3%, 6%, and 10% (see rows).

The NEAT program specifies life spans for all efficiency measures. Projected lifespans

range from 3 years (for a furnace tune-up) to 20 years (for attic insulation). The energy

savings-weighted average lifespan for installed measures in our dataset is 16 years. In the

table, we report discounted benefits over a range of time horizons(10-20 years). Estimates of

the net present value of energy savings range from $994 (high discount rate and short time

horizon) to $2,261 (low discount rate and long time horizon). Notably, all of these estimates

of discounted benefits fall below the average cost of the measures installed ($4,146). We

explore this finding in more detail in the next section.

5.1 Experimental Estimates of Energy Savings

5.1.1 First-stage: Program Take-Up

It may seem straightforward to encourage households to participate in a program that pro-

vides free efficiency retrofits worth over $4,000 that are designed to significantly reduce

energy expenditures. In our experience, that was hardly the case.

Table 5summarizes program take-up at three separate stages. In the first stage, our goal

21The NEAT program audits assume an electricity price of $0.11/kWh and a natural gas price of$11.46/MMBtu. The higher gas price is presumably based on 2006 prices which averaged around$11.50/MMBtu in this service territory in 2006.

22An alternative approach redefines the dependent variable in the IV regressions in monetary terms. Weobtained nominal retail prices for electricity and gas over the study period and impute monthly energyexpenditures using these prices. This approach yields very similar results

31

Table 5: Randomized encouragement: Return on effort

Application Efficiency Weatherizationaudit complete

Base rate 0.02∗∗ 0.01∗∗ 0.01∗∗

(0.00) (0.00) (0.00)

Encouragement 0.13∗∗ 0.05∗∗ 0.05∗∗

(0.00) (0.00) (0.00)

Households 28,889 28,889 28,889

Note: The table shows the effect of our encouragement on program applications, efficiency audits, andweatherization. Indicators of program participation status are regressed on an encouragement indicator anda constant. The unit of observation is a household. Significant at the 5 percent level* Significant at the 1 percent level

was simply to increase the share of households filing applications. The column (1) entry

shows that the encouragement intervention increased the rate of application to the program

by 11 percentage points from the control group mean of 2 percent.

Once a household’s application is approved, the energy audit is the critical next step.

Column (2) reveals that the fraction of households who received an energy audit was 4

percentage points higher in the encouraged group (off a base of about 1 percent). The

considerable drop off between submitting an application and receiving an energy audit is

due to several factors. For example, households asked for additional information to complete

their application sometimes neglected to follow through. Many households that submitted

applications during our study period are still waiting to schedule their audit as the pace of

the program dropped off precipitously after the ARRA funding ended. Figure 2 shows that

weatherization rates have dropped to fewer than ten weatherizations per month; households

32

waiting for weatherization assistance likely face a very long wait.

Column (3) of Table 5 documents that the treatment increased the fraction of households

that were successfully weatherized by about 4 percentage points, against a 1 percent rate

in the control group.23 The encouragement treatment is a sufficiently powerful predictor of

weatherization such that it can be used to instrument for program participation.

Before proceeding, it is worth underscoring that the low take-up rates in the encouraged

group are striking. Participants in the program incur no direct monetary costs. Successful

applicants receive substantive home improvements. All households in our encouraged group

received some information about the program via a phone call or door hanger. A majority of

households (i.e. those who spoke with our canvassers in person or by phone) received further

information about our offer of application assistance. It seems reasonable to surmise that

some combination of high perceived costs of applying for the program, low expectation of an

application leading to a weatherization, high unmeasured process costs, and low expected

benefits of participating in the program are impediments to WAP participation. In the end,

the average cost of encouragement per completed weatherization was about $1050, which is

more than 20% of the average costs of measures installed.

5.1.2 Instrumental Variables Estimates of Energy Savings

Panel A of Table 6 presents the experimental estimates of the impact of weatherization on

energy consumption, using random assignment to the encouragement group as an instru-

mental for program participation. In the first two columns, the dependent variable is the log

of total energy consumption (MMBtu/month). The third and fourth columns report results

separately for natural gas and electricity, respectively.

In the , monthly energy consumption is regressed directly on the WAP indicator. This

serves as a non-experimental differences-in-differences benchmark to be compared with the

23A small fraction of households get audited but not weatherized, primarily because the auditors deemthe home a possible danger to weatherization contractors (e.g., due to the presence of asbestos).

33

IV estimates. This estimate is very similar to the quasi-experimental estimates discussed in

the preceding section..

The IV estimate is reported in column (2). We estimate that weatherization reduces

monthly energy consumption by approximately 20 percent among households induced to

participate in WAP by the treatment. The experimental point estimate of energy savings is

twice as large - and statistically distinguishable from - the non-experimental estimate.

Columns (4) and (5) report local average treatment effects for natural gas and electric-

ity, respectively. Natural gas accounts for 94 percent of projected savings (measured in

MMBtu), so it is not surprising that natural gas consumption is more significantly impacted

by weatherization.24 . The effect of weatherization on gas consumption is also more precisely

estimated. This is also to be expected. Natural gas consumption is driven primarily by the

end uses targeted by weatherization (space and water heating), whereas electricity consump-

tion is derived from many end uses that are unaffected by weatherization (e.g. lighting and

appliances).

Panel B of Table 6 computes the present value of the estimated energy savings under

alternative assumptions about investment time horizons and discount rates. The approach we

take is similar to that described in section 5. One difference is that we cannot directly observe

average energy consumption among compliers assigned to the control group. To obtain a

consistent estimate of energy savings in levels, we re-estimate equation separately for natural

gas and electricity. Estimation results are summarized in the Appendix. The product of fuel

savings estimates and the corresponding fuel prices summed across natural gas and electricity

yield average annual savings of $222.40 per household. Discounted savings estimates range

from $1367 (high discount rate and short time horizon) to $3309 (low discount rate and long

time horizon).

24Note that the samples differ slightly across the columns and that the coefficients are not constrained tobe equal across columns (2) and (3) so the total energy estimate is not mechanically a weighted average ofthe gas and electricity coefficients.

34

Table 6: Estimated impacts of weatherization on household energy consumption

Panel A: Dependent variable is monthly energy consumption (log MMBtu)

Total Energy Gas Electricity(1) (2) (3) (4)

OLS-FE IV-FE IV-FE IV-FE

WAP -0.11∗∗ -0.21∗ -0.22∗∗ -0.10(0.01) (0.09) (0.08) (0.11)

month-of-sample FE Y Y Y Y

F-statistic . 263.3** 256.0** 262.3**encouragementHouseholds 27,229 27,229 25,505 26,571

Observations 1,653,583 1,653,583 1,515,117 1,624,030

Panel B: Present value of (discounted) savings

Time Horizon Discount rate3 percent 6 percent 10 percent

10 years $1928 $1663 $138815 years $2628 $2195 $171920 years $3362 $2592 $1924

Note: Dependent variable measures monthly household energy consumption. Panel A reports regressioncoefficients. With the exception of the first column, all specifications are estimated using 2SLS. Standarderrors (in parentheses) are clustered by household. Panel B reports savings projections generated by NEATaudit. See text for details. Standard deviations are in parentheses.

∗ Significant at the 5 percent level∗∗ Significant at the 1 percent level

35

The roughly 20 percent savings in overall energy consumption is more than twice as large

as the quasi-experimental estimates (and a simple DID constructed using data from the

experimental sample). These larger impacts imply larger reductions in energy expenditures

over the life of the measures. Ideally, we would like to understand what drives the differences

between the quasi-experimental and experimental estimates.

When we compare households receiving weatherization across the experimental encour-

aged and control groups, we find that WAP contractors installed fewer measures at encour-

aged households on average, and the total cost of the investment was significantly lower.

This rules out the possibility that differences in savings rates can be explained by higher

investment rates among compliers. We cannot, however, rule out the possibility is that

weatherization impacts (holding investment levels constant) are significantly larger among

compliers as compared to the households that seek out weatherization without our added

encouragement. Notably, weatherized households in the encouraged group do differ along

observable dimensions. In particular, they have significantly higher incomes and larger house-

holds. The houses do not differ significantly along observable dimensions (such as size or

age). Of course, another potential explanation for the difference in estimated savings rates

is a violation of the identifying assumption for the quasi-experiment.It may be that the tra-

jectory of average energy consumption in the quasi-experimental control groups provides a

biased (down) estimate of the counterfactual energy consumption at weatherized households

post-weatherization.

5.2 Household re-optimizing behavior, building thermal proper-

ties, and the welfare implications of rebound

In Section 2 we introduced the concept of bounding the average willingness to pay for an

efficiency-induced increase in energy services. In this section, we implement this bounding

36

exercise in three steps.

5.2.1 Does weatherization lead to temperature “take back”?

The first step tests for an effect of weatherization on household demand for space heating.

Table 7 summarizes the results from our survey of indoor air temperatures collected during

the winter of 2013. All weatherized households surveyed had received efficiency improvements

at least one year before the survey was administered, allowing plenty of time for residents to

observe how the retrofit affected winter heating costs.

Approximately half (347) of the 688 households that allowed us to enter their home

to take temperature measurements had received weatherization assistance. Anticipating

some measurement error, we used two different devices to measure indoor temperatures at

each home. We regress thermometer readings on a binary variable indicating whether the

household has been weatherized. Standard errors are clustered at the household level.

Table 7 summarizes the regression results. Column (1) reports the base specification.

The constant term measures the average indoor air temperature at unweatherized house-

holds: 67.3 degrees Fahrenheit.25 The coefficient on the weatherization dummy suggests

that weatherized homes are kept 0.73 degrees Fahrenheit warmer. This estimate borders on

statistical significance at conventional levels.

25Notably, this estimate of the baseline indoor temperature is very close 68 degrees, the set point assumedby the NEAT audit tool.

37

Table 7: Indoor temperature survey results

Dependent variable is indoor air temperature (measured) in ◦F

(1) (2) (3) (4)

Base temperature 67.27** 67.31** 72.33** 72.48**(0.36) (0.35) (0.94) (1.12)

Weatherized home 0.73 0.78 0.63 0.71(0.42) (0.44) (0.41) (0.43)

HDD . . −0.16∗∗ −0.15∗∗

(0.03) (0.04)

Largest implied 1.54 1.65 1.44 1.57increase (95%)

Propensity score N Y N Yweights

R-squared 0.003 0.003 0.03 0.03

Note: The table reports measured indoor temperature differentials across weatherized (WAP) andunweatherized households. All specifications estimated on 1359 observations across 688 households basedon two thermometer reads at most households. Columns (2) and (4) are weighted so that surveyedpopulation better represents total quasi-experimental sample. Standard errors clustered at a householdlevel. Significant at the 5 percent level* Significant at the 1 percent level

38

Survey respondents comprise a non-random subset of our sample. Only a fraction of

the targeted household were home and/or willing to open the door to receive our surveyors.

Only 41 percent of households who opened the door to our surveyors allowed us to come

in and collect temperature measurements. Pairwise comparisons of observable household

and dwelling characteristics across surveyed households and the larger quasi-experimental

sample reveal that the surveyed sub-sample is observationally similar among most- but not

all- dimensions. Survey respondents are more likely to report having children or elderly

family members, less likely to be unemployed, and are more likely to use gas as their primary

source of heat, as compared to the larger quasi-experimental sample. Among unweatherized

households, surveyed households are larger.

Given these observable differences between survey respondents and the larger sample,

we also estimate a weighted regression where household observations are weighted so as

to match the covariate distributions in the larger quasi-experimental sample of weatherized

households. These results are reported in columns (2) and (4). Weighted regression estimates

do not differ significantly. 26

Columns (3) and (4) control for the outdoor temperature on the day of the survey. The

coefficient on outdoor temperature is statistically significant, suggesting that our indoor

temperatures measure lower on colder days. Although we asked surveyors to wait several

minutes before recording temperatures, this finding suggests that cold air brought with the

surveyor could be affecting our measurements. Point estimates of the average impact of

weatherization on indoor temperatures remain small and imprecisely estimated.

In sum, we find some suggestive - but noisy- evidence of a small effect of the weatheriza-

tion intervention on indoor air temperatures. We believe this is the first field measurement

26To re-weight observations, we take a similar approach as was deployed in the quasi-experimental esti-mation of average treatment effects. In a first stage, we estimate the probability of responding to the surveyconditional on being in our quasi-experimental sample. Inverse propensity scores are then used to weightobservations for survey responders in the regression.

39

of household’s re-optimizing response to exogenous variation in heating efficiency improve-

ments.

5.2.2 Building energy performance

In order to estimate how a given increase in demand for indoor space heating translates

into an increase in monthly gas consumption, we need to estimate the thermal properties

a representative building. We use the so-called “degree day method”, quite standard in

building energy performance analysis, to model the energy required to increase temperatures

in an average home in our data (Thorpe, 2013).

In residential buildings where building envelope losses are the major determinant of

heating energy requirements, it is standard to summarize the technical relationship between

energy consumption and heating demand by regressing energy consumption against heating

degree days. Heating degree days (HDDs) represent the number of degrees over a period in

which the ambient temperature is lower than a specified base level. We collect hourly HDD

data for each county in our data and aggregate HDDs up to the county-month level.27 We

estimate the following equation:

Cimt = αi + β11{WAP}imt + β2HDDmt + β3HDDmt ∗ 1{WAP }imt (4)

+β4HDD2mt + β5HDD

2mt ∗ 1{WAP}imt,

where Cimt measures the natural gas consumption at household i in month m and year

t. We estimate this equation using data collected from program applicants during winter

27We adopt a standard base temperature of 65°F. This is consistent with the base temperature estimatedin the previous section less the typical heat gain that comes from other sources such as people and appliancesin the home.

40

months (September-March) over the entire sample period. Panel data allows us to include

household-level fixed effects in this regression. 28 We include both a linear and a quadratic

HDD term, allowing each coefficient to vary with weatherization status.

Estimates of all coefficients in equation (4 ) are highly statistically significant and very

precisely estimated. The R-squared value is 0.73. Figure 3 summarizes the relationship

between energy consumption and HDD separately for weatherized and unweatherized ob-

servations. This figure is analogous to the bottom panel of Figure 1 (flipped into the top

quadrant). As expected, the slope of this relationship (which is approximately linear) is less

steep among weatherized homes.

Before proceeding to use these estimated relationships to translate changes in indoor

temperatures into changes in energy consumption, several caveats are in order. First, this

approach to estimating the relationship between indoor heating demand and building en-

ergy consumption implicitly assumes that indoor air temperatures are set at a constant level

during the winter months. Our temperature survey data calls this assumption into question,

although another explanation for the relationship between our measured indoor tempera-

tures and outdoor temperatures could be cold air coming into the home with the surveyor.

Second, these graphs are generated using the data from our quasi-experimental sample and

are vulnerable to the selection biases discussed above.

5.2.3 Bounding the average valuation of increased indoor heat

Figure 3 shows that weatherization effectively reduces the marginal cost of space heating.

To get a sense of the average impact of weatherization on the marginal cost of heating, we

multiply the slope of the (approximately linear) relationships in Figure 3 by the average

28Although it is standard to assume a linear relationship between heating demand and energy consumption,researchers have noted the potential for a non-linear relationship. For example, Dewees and Wilson (1990)note that while heat losses per unit of air will rise linearly with the temperature difference, air exchangelosses are increasing in temperature. This implies that the marginal cost of a degree of indoor temperaturecan increase as the outdoor temperature falls.

41

Figure 3: Building energy performance at weatherized versus unweatherized homes

42

natural gas price in the post-encouragement period. We obtain an estimate of $0.072 per

heating degree day (or $2.17 per heating degree month) at unweatherized households and

$0.056 per heating degree day (or $1.67 per heating degree month) at weatherized homes.

This implies a marginal cost reduction of approximately 20 percent. 29

If we take as given our point estimate of the efficiency-induced rebound in heating demand

(0.78 °F) and the summarized relationships between heating demand and energy consump-

tion (Figure 3), we can bound the welfare gain from re-optimizing heating demand. The

lower bound is given by 0.78 °F * $1.67/degree-month. At this lower bound, the utility

gains from increased warmth are exactly offset by the increase in the energy expenditures

incurred to achieve the temperature increase. To define the upper bound, we note that by re-

vealed preference, households are not willing to pay 0.78 °F * $2.17/degree-month to achieve

this incremental increase in temperature in the unweatherized state. It follows that average

marginal benefits from this temperature increase cannot exceed $2.17. So the average net

gain from the increase in warmth following weatherization (net of increased expenditures to

achieve the increase) should not exceed $0.39per winter month. In sum, this bounding exer-

cise suggests that the welfare gains from any efficiency-induced rebound in heating demand

are quite small.

6 Interpretation

In this section, we evaluate the returns to energy efficiency investments through the WAP

program from both a private and social perspective. We also provide an alternative summary

measure of the program’s cost effectiveness, which is the cost per ton of carbon abated.

29These estimates of incremental heating costs are comparable to a “rule of thumb” popularized by theAmerican Council for an Energy-Efficient Economy. This rule states that a household will pay approximately3% on their gas bill for a degree increase in winter thermostat settings. Average natural gas bills duringwinter months are $85.95 and $57.96 at weatherized and un-weatherized homes, respectively.

43

6.1 Cost effectiveness

With our estimates of energy savings and households’ valuation of efficiency-induced increase

in winter warmth in hand, we turn to an evaluation of the overall cost effectiveness of these

weatherization investments. Panel A of Table 8 evaluates the internal rate of return (IRR) on

investment from a private perspective. More precisely, we report the discount rate at which

the discounted value of average avoided energy expenditures exactly equals the average

upfront investment. Although households in our sample did not actially incur installation

or materials costs in this program, this IRR metric provides a useful summary statistic for

thinking about the cost effectiveness of these weatherization improvements.

Column (1) computes this internal rate of return using the average installation and ma-

terials cost ($4,991 per household) and the average reductions in annual energy expenditures

projected by the WAP program audit. Over a range of time horizons, returns are quite

high; all exceed 10 percent. In other words, efficiency investments supported under the

Weatherization Assistance Program appear highly cost effective based on the savings and

cost projections that informed program implementation.

The second column of Panel A replaces the natural gas price assumed by the WAP effi-

ciency audit calculations (i.e. $11.46/MMBtu) with the average volumetric price paid by re-

tail customers in this service territory over the post-weatherization period ($7.82/MMBtu).30

This reduction in the valuation of avoided natural gas consumption notwithstanding, esti-

mated rates of return on investment are still positive across all time horizons.

The third column of Table 8 replaces the ex ante projected savings generated ex ante

during the WAP efficiency audit with the experimental estimates generated ex post using

our randomized encouragement design. The experimental estimate of average energy savings

amount to less than a third of the average projected savings. Consequently, this adjustment

30No change is made to electricity rates because the retail electricity price used in the efficiency auditsmatches the average observed retail price.

44

Table 8: Estimated returns on investments in energy efficiency

Time (1) (2) (3) (4)horizon

Panel A: Private internal rate of return on investment

10 years 13% 1% -12% -12%15 years 17% 6% -5% -4%20 years 18% 8% -1% -1%

Panel B: Social rate of return on investment

10 years 17% 2% -11% -10%15 years 20% 7% -3% -3%20 years 21% 9% -1% 0%

Panel C: CO2 abatement cost - 3 percent discount ($/ton CO2)

10 years -$87 $48 $525 $51715 years -$132 -$3 $304 $29620 years -$154 -$19 $195 $187

Panel D: CO2 abatement cost - 7 percent discount ($/ton CO2)

10 years -$53 $82 $692 $68415 years -$96 -$38 $472 $46820 years -$117 -$18 $375 $367

Energy savings projected projected experimental experimentalestimate (NEAT) (NEAT) estimate estimate

Energy value NEAT post-period post-period post-periodassumed average observed average observed average observed

Rebound N N N Yadjustment

Note: Calculations summarized by column (1) uses average retrofit costs of $4991 and average engineeringprojections of annual savings ($907.84). In Panel B, column (1) also incorporates the value of estimatedemissions reductions valued (using a social cost of carbon value of $38 per ton CO2). Column (2) replacesthe value of natural gas used in the WAP program audits with the average retail price in 2012-2104 (PanelA) or the average gas recovery charge over this period (Panels B and C). Columns (3) and (4) replaceengineering estimates of energy savings with our experimental estimates of energy savings. Column (4)incorporates the upper bound on the net welfare gain from increased heating demand.

45

to the savings estimates pushes the rate of return on these investments into the negative

domain. The final column incorporates our estimate of the upper bound on welfare gains

from the rebound in heating demand (net of the associated increase in energy costs). This

small welfare gain has little impact on estimated returns.

Panel B conducts a very similar exercise from a social (versus private) perspective. These

calculations differ from those in Panel A in two important ways. First, estimated benefits

include the value of avoided emissions.31 Avoided emissions estimates are predicated on

the assumption that burning natural gas emits 116.39 lbs CO2 per mmbtu and a marginal

operating emissions rate of 0.916 lbs CO2 per kWh in the Midwest power sector (Callaway

et al. 2014). Avoided emissions are valued at $38 per ton. The second difference between

our private versus social rate of return calculation pertains to the valuation of avoided

fuel consumption in columns (2) through (4). The retail electricity and natural gas prices

used in Panel A include a fixed cost component; these fixed costs are not avoided when

natural gas and electricity consumption is reduced. The gas recovery charge reported by the

utility in regulatory proceedings provides a more accurate measure of the economic value

per unit of natural gas over the post-weatherization period ($5.54). To value reductions

in electricity consumption, we use the average wholesale electricity price. The net effect of

these two opposing effects (a reduction in the value of reduced energy consumption, but the

added value of avoided emissions) is a slight increase in rates of return vis a vis the private

rates. Note that these calculations do not account for the social cost of public funds, the

administrative costs of the program, or the costs of our encouragement intervention.

It is important to note that there may be additional benefits to households that received

weatherization assistance that we are not measuring. For example, consumers may realize

health benefits associated with reduced draftiness. While consumers who pay out of pocket

31We consider only the value of avoided carbon dioxide emissions. Criteria pollutants such as NOx andSO2 from electricity generation in Michigan are subject to a (barely) binding cap.

46

for an energy efficiency program would internalize those expected benefits, we cannot measure

them in a government-supported program. At the same time, there are speculations in the

public health literature that the program may increase health risks as less drafty homes may

trap indoor air pollutants such as radon.

Panels C and D estimate the average cost per ton of CO2avoided at two different discount

rates (3 and 7 percent). To construct these average abatement cost estimates, the net cost

of the investments (i.e. levelized annual cost less the value of annual energy cost savings)

is divided by estimated annual emissions reductions. Using energy savings values generated

by the NEAT program audits, abatement costs are negative because emissions reducing

efficiency investments appear highly cost effective from a private perspective. In contrast,

when the experimental estimates of natural gas and electricity savings are valued using

utility-reported gas recovery charge and average wholesale electricity prices, abatement costs

turn positive. Our preferred estimates of the average cost per ton of CO2 avoided accounts

for the small welfare gain associated with increased heating demand (column (4)).

6.2 What explains the low rate of return on these efficiency in-

vestments?

It is natural to ask why the returns to energy efficiency investments through WAP are so

low. After all, the program is designed so that the only measures implemented are ones with

projected savings to costs ratios of greater than 1.

Table 8 shows that low realized returns can be partly explained by the fact that pro-

gram auditors over-value avoided natural gas consumption; auditors assume a value of

$11.46/MMBtu, almost 50 percent above the average retail price of natural gas since 2011.

This over-valuation explains the difference between columns (1) and (2) of Table 8.

Another important factor leading to negative returns on investment is the incomplete

47

realization of projected energy savings. Our preferred experimental estimates amount to less

than a third of projected savings. The quasi-experimental estimates of energy savings are

between 23 and 28 percent of the corresponding average savings projections. Importantly,

we are not the first to report ex post realized savings that fall short of ex ante projections.

Low realization rates are fairly ubiquitous in both the academic literature (see, for example,

Davis et al. (2013); Dubin et al. (1986)) and utility program evaluation reports.32

In the academic literature, low realization rates are often attributed to t behavioral

responses to efficiency improvements are rarely captured by building simulation models (see,

for example, Dubin et al. 1986; Davis et al. 2013). However, our estimates of weatherization-

induced increases in demand for heating account for less than 5 percent of the gap between

projected and ex post realized energy savings.

Having ruled out behavioral responses as the primary explanation, we conclude that the

efficiency audit tool must systematically overstate the real returns to these investments.

Along these lines, we explore some alternative sources of this projection bias. First, we com-

pare the distribution of temperatures observed during our study period against the typical

weather patterns on which engineering calculations are based. Although we do observe some

moderate spells in our time frame, on average we observe colder than average temperatures

and higher than average degree day measures in our sample.

A second source of bias concerns the over-statement of baseline energy use. Energy

efficiency program analysts and practitioners have documented major errors in standard

software for estimating natural gas use in older, inefficient homes.33 Program audit data allow

us to isolate predictions of baseline energy consumption for each audited household. When

we compare these projections with the actual billing data, wefind that a typical household is

32For example, a recent summary of realization rates documented in Massachussets reports realiza-tion rates ranging from below 10 percent to over 100 percent. See http://ma-eeac.org/wordpress/wp-content/uploads/Home-Energy-Services-Realization-Rate-Results-Memo-6-28-12.pdf.

33See, for example. http://2011.acinational.org/sites/default/files/session/81087/aci11chal1blasnikmichael.pdf.

48

projected to use 10.25 MMBtus of natural gas per month before weatherization. In the data,

we observe audited households consuming 8.11 MMBtu per month prior to weatherization.

Thus on average, the heating-related natural gas consumption assumed by the efficiency

audits are more than 25 percent higher than the observed consumption.This suggests that

the auditing tool could be under-estimating the efficiency properties of the average home

prior to weatherization, which would lead to an over-statement of the benefits of upgrading

to a given efficiency standard.

In sum, our findings suggest that the NEAT audit tool is over-estimating returns on

the energy efficiency investments we analyze by a significant margin. This is an important

finding; NEAT is widely used by state and local WAP sub grantees, utility companies, and

home energy audit firms. While more sophisticated building simulation models exist, it is

important to remember that thousand of implementers with a range of skills and technical

training must use whatever program informs WAP or similar large-scale energy efficiency

programs. The DOE cites one of the primary benefits of NEAT as its accessibility to non-

technical users (DOE-EERE).

7 Conclusion

This paper presents results from one of the first large-scale experiments designed to measure

the real-world returns on investments in residential energy efficiency improvements. We

implement a randomized encouragement design to evaluate the benefits of weatherization

retrofits provided to low-income consumers. We also present corroborating evidence from a

quasi-experimental analysis covering over twice as many weatherizations.

We find that the program led to economically significant reductions in energy expendi-

tures for weatherized households. On average, weatherization reduces energy consumption

by 10-20 percent at participating households. Neoclassical economic theory suggests that

49

consumers facing lower prices for energy services, such as lower prices to heat their homes

after weatherization, will re-optimize and potentially consume more services. As a result,

the total direct consumer welfare of an energy efficiency program includes the utility from

increased consumption, such as increased indoor heating. We implemented a follow-up sur-

vey of program participants and non-participants to measure their indoor temperatures and

find modest increases in temperatures among participants. We use a revealed preference

approach to bound the associated increases in welfare and find very small welfare gains.

Overall, we show that the net present value of the savings, accounting for both reduced

energy expenditures and increased utility from indoor heating, are lower than the direct

program costs under a range of plausible assumptions on both the average lifetimes of the

energy efficiency measures and consumer discount rates. Accounting for program overhead

costs or the marginal cost of public funds would lead to even lower estimated returns on

investment. Our preferred estimates of the average cost per ton of CO2 avoided significantly

exceed the value that federal agencies use to estimate the climate benefits associated with

avoided emissions ($38/ton).

A natural instinct is to try to generalize our experimental estimates to other contexts,

including weatherization programs in other states, and investments in energy efficiency im-

provements more generally. Any such attempt comes with an important caveat. Our results

are specific to a population of low-income households in Michigan who were encouraged to

participate in the federally funded weatherization assistance program. From a research de-

sign perspective, analyzing a federally funded program provides several benefits. For one, we

have systematic, household-specific measures of implementation costs, savings projections,

etc. Given the low-income population served by the program, we are relatively confident

that our encouragement intervention did not inspire contemporaneous complementary in-

vestments in efficiency improvements. However, the external validity or our findings is

limited by the fact that households in our study are not representative of U.S. households

50

in other income segments or other regions of the country. Our estimates of the extent to

which realized savings fall short of ex ante projections are also specific to the audit tool that

is used to inform investment decisions in the WAP program. Although this tool is widely

used by state and local WAP sub grantees, utility companies, and home energy audit firms,

the savings projections it generates may not be representative of other modeling approaches

that inform efficiency investments in other contexts.

Widely publicized engineering estimates, such as the iconic McKinsey curve, suggest

that consumers are systematically bypassing opportunities to invest in cost-effective energy

efficiency improvements that lower their energy expenditures and reduce externalities asso-

ciated with energy production. As a result, many policymakers are championing programs

designed to encourage energy efficiency as a cost-effective strategy to confront climate change

and reduce other harmful pollutants. This paper provides evidence that field savings may

be very different from engineering estimates. These findings underscore the importance of

field-testing of projected returns on energy efficiency investments, particularly given the

increasing role of efficiency programs as a policy tool.

References

Allcott, H. and Greenstone, M. (2012). Is there an energy efficiency gap? The Journal ofEconomic Perspectives, pages 3–28. 1

Davis, L., Fuchs, A., and Gertler, P. (2013). Cash for coolers: Evaluating a large-scaleappliance replacement program in Mexico. American Economic Journal: Economic Policy,forthcoming. 6.2

Dewees, D. N. and Wilson, T. A. (1990). Cold houses and warm climates revisited: Onkeeping warm in chicago, or paradox lost. Journal of Political Economy, 98(3):pp. 656–663. 5

Dubin, J. A., Miedema, A. K., and Chandran, R. V. (1986). Price effects of energy-efficienttechnologies: A study of residential demand for heating and cooling. The RAND Journalof Economics, 17(3):pp. 310–325. 1, 6.2

EERE, D. (2013). Review of selected home energy auditing tools: In support of the de-velopment of a national building performance assessment and rating program. Technical

51

Report MSU-CSE-05-29, U.S. Department of Energy, Energy Efficiency & Renewable En-ergy. 4.1.2

Friedman, D. (1987). Cold Houses in Warm Climates and Vice Versa: A Paradox of RationalHeating. Journal of Political Economy, University of Chicago Press, 95(5):1089–97. 5

Gillingham, K., Kotchen, M. J., Rapson, D. S., and Wagner, G. (2013). Energy policy: Therebound effect is overplayed. Nature, 493(7433):475–476. 1

Gillingham, K. and Palmer, K. (2014). Bridging the Energy Efficiency Gap: Policy Insightsfrom Economic Theory and Empirical Evidence. Review of Environmental Economics andPolicy, 8(1):18–38. 1

Jacobsen, G. D. and Kotchen, M. J. (2013). Are building codes effective at saving energy?Evidence from residential billing data in florida. Review of Economics and Statistics,95(1):34–49. 1

McKinsey & Company (2009). Unlocking energy efficiency in the U.S. economy. McKinsey& Company. 1

Schwarz, P. M. and Taylor, T. N. (1995). Cold hands, warm hearth?: Climate, net takeback,and household comfort. The Energy Journal, 16(1):pp. 41–54. 4document

52