Do Energy Efficiency Investments Deliver? Evidencefrom the Weatherization Assistance Program
Meredith Fowlie, Michael Greenstone, and Catherine Wolfram∗
September 28, 2014
Abstract
It is conventional wisdom that individuals and firms fail to undertake energy effi-ciency investments that are predicted by engineering models to have private returnsgreatly in excess of their costs. Policymakers have seized on this so called energyefficiency gap with a wide variety of interventions that promise private benefits andreductions in environmental damages, especially declines in the greenhouse gases thatcause climate change. This paper applies experimental and quasi-experimental tech-niques to assess the private and social returns to energy efficiency investments usingdetailed data from the application of the nation’s largest residential energy efficiencyprogram, the Federal Weatherization Assistance Program (WAP), in Michigan. Wefind that participating households’ energy consumption declined significantly. We alsopresent suggestive evidence that households modestly increased indoor temperaturesby about 0.6 degrees F, providing some of the first direct evidence on compensatoryresponse to a reduction in the price of energy services (i.e., the “rebound” effect) inthe building sector. Accounting for the energy savings and consumers’ valuations ofthe higher indoor temperatures, the data indicate that these investments have nega-tive annual returns both privately and socially (i.e., when the monteized value of thegreenhouse gas savings are included). Specifically, the private and social internal ratesof return or discount rate that would rationalize these investments are -4% and -3%,respectively.
∗We received many helpful comments from seminar participants at Carnegie Mellon, Columbia University,National University of Singapore, Resources for the Future, the University of Michigan, and the Universityof Maryland. The authors gratefully acknowledge the financial support of the Alfred P. Sloan Foundation,the Rockefeller Foundation and the UC Berkeley Energy and Climate Initative, and institutional supportfrom the Poverty Action Lab (JPAL) at MIT, the Center for Local, State, and Urban Policy (CLOSUP)at the University of Michigan, and the Energy Institute at Haas. We thank James Gillan, Walter Graf,Erica Myers, and Matthew Woerman for excellent research assistance. We are indebted to Jesse Worker foroutstanding management of a challenging project. Finally, we thank our contacts at both our partner utilityand the community action agencies, without whom this project would not have been possible.
1
Preliminary draft. Please do not cite
1 Introduction
Energy efficiency investments are widely believed to offer the rare win-win opportunity. De-
tailed engineering projections, such as those summarized by the well-known McKinsey curves
(McKinsey & Company, 2009), routinely project net present value positive investments based
on private returns alone (win #1). Moreover, by reducing the energy necessary to achieve a
given level of energy services (e.g., heating or cooling), these investments promise to decrease
greenhouse gas emissions in addition to other pollutants that compromise human health (win
#2).
Importantly, there is a large and persistent difference between the levels of investment
in energy efficiency that appear to be privately beneficial and the investments that private
individuals actually pursue. Over the last three decades, a wide variety of explanations,
have been offered for this apparent failure of consumers to avail themselves of profitable
investment opportunities (see, for example, Allcott and Greenstone, 2012; Gillingham and
Palmer, 2014).
Mounting concern about climate change has increased the urgency of understanding
this ‘energy efficiency gap’. Governments around the world are pursuing a wide range of
policies designed to narrow or close this gap. In 2013, U.S. electric utilities budgeted nearly
$7 billion for efficiency programs; these expenditures are projected to double in the next
decade (Barbose et al. 2013); energy efficiency building codes and appliance standards
entail additional costs. The case for the U.S. government’s fuel economy standards for
automobiles and light trucks and proposed rule to reduce emissions from power plant, which
are the two most important climate policies, depends crucially on win-win energy efficiency
opportunities.1
1For example, 82% of the benefits from the fuel economy standard for automobiles and light trucks comefrom consumer’s suboptimal purchases of vehicles with insufficient fuel economy. A significant share of thereductions in carbon emissions by 2030 from power plants under the proposed Clean Power Plan are expectedto come from demand-side energy efficiency improvements (CATF, 2014).
2
This paper provides some of the first field evidence on the returns to energy efficiency
investments from a randomized control trial. We focus on the residential building sector
which has been identified as a particularly promising source for cost effective opportunities
to reduce growth in energy demand (IPCC, 2014). We focus on interventions that improve
the efficiency of space heating which accounts for over 40 percent of energy consumption in
U.S. homes.2
We use experimental variation in participation in the nation’s largest residential energy
efficiency program, the Federal Weatherization Assistance Program (WAP), to identify the
average effects of these investments on household-level energy consumption. We also exam-
ine the extent to which households receiving weatherization respond to the reduced costs of
energy services by choosing a different indoor temperature (i.e., the “direct rebound” effect).
The large-scale randomized encouragement design experiment was conducted on a sample
of almost 29,000 households presumptively eligible for participation in WAP in the state
of Michigan. Over 7,500 of these households were randomly assigned to an encouragement
treatment. These households were encouraged to apply for the program and received signif-
icant application assistance. The control households were free to apply for WAP but were
not contacted or assisted in any way by our team.
There are three primary findings. First, the encouragement intervention increased WAP
participation from less than 1 percent in the control group to almost 6 percent in the en-
couraged group. The encouragement intervention was implemented by a firm with extensive
canvassing experience, including with low income communities. Field staff made almost
7,000 home visits and nearly 33,000 phone calls, and yet increased the participation rate
by less than 5 percentage points. This increase is surprisingly small given that participat-
ing households receive $5,000 worth of efficiency improvements on average, but entail no
2Estimates from the most recent Residential Energy Consumption Survey (RECS) show that 48% ofenergy consumption in U.S. homes in 2009 was for heating and cooling. For more details, see U.S. EnergyInformation Administration, Residential Energy Consumption Survey (RECS) at http://www.eia.gov.
3
monetary costs
Second, we find that efficiency investments deliver economically and statistically sig-
nificant energy savings among participating households. We estimate that weatherization
assistance reduces monthly energy consumption by 10-20 percent on average; a substantial
assist to participating low-income households in the form of reduced energy bills. However,
we show that the net present value of these energy savings (under a range of plausible as-
sumptions) are all lower than the direct cost of the efficiency investments. These results
call into question the assumption underlying the debate on the energy efficiency gap that
households are by-passing cost effective investment opportunities.
Third, we conducted a follow-up survey of indoor temperatures at weatherized and non-
weatherized homes and find some evidence that measured indoor temperatures are roughly
0.7 degrees F higher (p-value = 0.XX) in weatherized homes. This is consistent with house-
holds re-optimizing energy consumption in response to lower implicit prices per unit of energy
service (e.g., heating). Though the existence of the so-called ‘rebound effect’ has been the
subject of much debate (Gillingham et al., 2013), our findings provide some of the first direct
field evidence of this phenomenon.
In our penultimate section, we discuss the implications of these findings from a private
and social perspective. Accounting for the energy savings and consumers’ valuations of
the higher indoor temperatures, the data indicate that these investments have negative
annual returns both privately and socially (i.e., when the monteized value of the greenhouse
gas savings are included). Specifically, the private and social internal rates of return or
discount rate that would rationalize these investments are -4% and -3%, respectively. We
also calculate the average cost per ton of avoided CO2 under a range of assumptions. The
most plausible estimates are larger than $300/ton, which is much more expensive than the
the U.S. government’s estimate of the social cost of carbon of roughly $37.
This paper contributes to a burgeoning literature that examines potential explanations for
4
the energy efficiency gap. To date, this literature has emphasized the role of market failures
such as imperfect information and principal-agent problems, and behavioral phenomenon
such as bounded attention and heuristic decision-making. There has been relatively little
attention paid to an alternative explanation for the apparent gap: hidden costs (such as
transaction and process costs) and overstated benefits.
Early work by Joskow and Marron (1993) raised concerns about overstated efficiency
potential and underscored the importance of using ex post measures of consumer behavior to
estimate energy savings. Subsequent work by Metcalf and Hassett (1999) uses panel data on
household energy expenditures to estimate realized returns on investments in attic insulation.
Although projected returns on these investments were on the order of 50 percent per year,
these authors estimate a median rate of return below 10 percent. More recently,Jacobsen
and Kotchen (2013) use residential billing data to evaluate the effect of an increase in the
stringency of an energy code on both electricity and natural gas consumption in Florida.
They estimate a private payback period of approximately 6.4 years.
This work also builds on prior studies that highlight important interactions between
consumer behavior and the introduction of energy efficiency improvements that lower the
effective price of the services they provide. In a widely cited study, Dubin et al. (1986)
exploit a small field experiment conducted by a Florida utility in which efficiency improve-
ments were randomly assigned. They find that consumers with improved insulation and
more efficient heating equipment conserve 8-13% less energy than would be predicted from
engineering models. .More recently, Davis, Gertler and Fuchs (2013) use quasi-experimental
variation to measure ex post realized energy savings for an appliance replacement program
in Mexico.They find upgrading the efficiency of air conditioners actually increased energy
consumption, a result they interpret as consistent with a large rebound effect. Overall, they
find that the program is an expensive way to reduce externalities from energy use, reducing
carbon dioxide emissions at a program cost of over $500 per ton . In contrast, we find that
5
rebound effects can explain only a very small fraction of the discrepancy between projected
and ex post realized energy savings. We attribute the remainder of this “measurement gap”
to the engineering models’ overstatement of the savings.
Our paper begins by introducing a conceptual framework for energy efficiency invest-
ments, highlighting possible deviations between modeled and actual behavior. Section 3
summarizes key details on the Weatherization Assistance Program. Section 4 summarizes
the data we have collected. Section 5 describes our empirical strategy. Section 6 presents the
main results on actual savings. Section 7 describes estimation and calculations we performed
to rationalize the measurement gap. Section 8 assesses the overall cost-effectiveness of these
efficiency investments in light of our findings. Section 9 concludes.
2 Conceptual framework
In the economics literature, the most standard definition of the ‘energy efficiency gap’ refers
to the difference between actual and cost-effective energy use. In other words, a gap exists
if individuals systematically overlook energy efficiency investments that confer benefits in
excess of costs. This section provides a conceptual framework for thinking about the bene-
fits generated by efficiency improvements. This framework will serve as foundation for the
subsequent empirics.
Gains from an investment in energy efficiency are realized through two main channels:
reduced energy consumption and, possibly, increased consumption of energy services (e.g.
lighting, space heating, air conditioning). With respect to the first channel, any reduction in
dollars spent on energy can be allocated to other forms of welfare enhancing consumption.
The second channel becomes important when an efficiency-induced reduction in energy end-
use costs leads to an increase or “rebound” in the demand for the energy service. If we
assume agents act to maximize utility, any re-optimization of consumption that occurs in
6
Figure 1: Household-level re-optimization in response to an efficiency improvement
X
Budget set Budget set
Status quo Weatherization
rebound
upper bound
E(H;W,Z) Status quo Weatherized
H0 /H’1 H1 Heating services (H)
E’1
E1
E0
E’
X’1
X0
response to an efficiency improvement will be welfare improving.
These basic ideas are illustrated in the upper quadrant of Figure 1, which plots con-
sumption of energy services on the horizontal axis and consumption of the numeraire (i.e.,
all other goods), X, on the vertical axis.3 In particular, we focus on space heating demand.
Home heating is a particularly important end-use in our empirical setting; over 93 percent of
projected energy savings from the weatherization investments we analyze are heating related.
The two downward sloping lines in the upper quadrant of the figure reflect budget con-
straints, the lower before weatherization and the higher after weatherization. This pivot
3For ease if exposition, this figure depicts the limited range of indoor air temperatures over which energyconsumption is increasing in indoor temperatures. This range is bounded from below by the outdoor airtemperature (it the thermostat is set below the outdoor air temperature, no heating services are required).The figure also assumes that the capacity of the home heating system does not bind. When the capacityconstraint binds, energy consumption ceases to increase with the temperature differential.
7
in the budget constraint post-weatherization reflects that the price of heating services (i.e.,
the price of increasing the thermostat in winter) has fallen; energy efficiency improvements
reduce the cost of purchasing any given level of thermal comfort.
The figure also illustrates a family of indifference curves for a representative consumer,
each of which trace out the bundles of the numeraire, X, and heating services that deliver
the same level of utility. The shape of the indifference curves reflects that households do
not like to be too hot or too cold. In the status quo (i.e. absent an efficiency improve-
ment), the representative agent will maximize utility through the choice of H0 and X0.The
weatherization-induced expansion of the budget constraint allows the agent to move to a
higher level of utility associated with the bundle of H1 and X1. In the figure, status quo
consumption occurs below the satiation point for thermal comfort. Thus, when the price
of thermal comfort falls, demand for heating services increases by H1 − H0. The positive
income effect also increases consumption of the numeraire by X1 −X0.
The paper’s empirical challenge is to measure the welfare gains conferred by weatheriza-
tion investments. Our empirical setting allows us to develop a measure of willingness to pay
(WTP) for weatherization that accounts for both reductions in energy expenditures and the
increased consumption of space heating. The effect of energy efficiency improvements on
other consumption (X) can be measured using data on monthly energy expenditures which
vary one for one with the consumption of the numeraire. Put another way, a $1 decrease in
energy expenditures allows for a $1 increase in consumption of all other goods. Measuring
willingness to pay for the increase in heating services (i.e, the direct rebound effect) is more
challenging because demand for energy services (such as heating) is not readily observable
in household energy consumption or expenditure data.
To obtain an estimate of the efficiency-induced increase in demand for heating (H1−H0 in
the figure), we conduct a survey of indoor temperatures in weatherized and non-weatherized
homes. With this estimate in hand, we can construct bounds for the welfare consequences
8
of this increase in warmth if we impose some structure on the relationship between heating
energy demand and temperature.4
The bottom quadrant of Figure 1 plots a representative building-specific relationship be-
tween indoor temperature and the energy required to achieve that temperature E, holding
constant outdoor temperatures, W , and building characteristics, Z. 5 Efficiency improve-
ments to the building envelope (e.g. insulation improvements, window sealing, a furnace
upgrade) should reduce the energy required to deliver any given level of heating services.
This implies that the slope of the relationship between temperature and energy consumption
becomes less steep following an efficiency improvement. We can use rich data on energy con-
sumption and temperature variation over the study period to estimate these relationships
between temperature and heating demand in weatherized and unweatherized homes.
Building on the top quadrant, the relationship between heating demand and energy
consumption can be used to construct bounds on the utility gains from the efficiency-induced
increase in the demand for indoor temperature using revealed preference logic. Since the
agent chooses to increase heating services from H0 to H1following the efficiency improvement,
it follows that a lower bound for the increased utility derived from this increase in heating
services is the associated increase in heating costs incurred after weatherization– this is
represented by PE*(E1-E′1) in the figure. Note that this agent chose not to choose H1prior
to the efficiency improvement. Thus, the cost of this change in heating services prior to
weatherization, measured by PE*(E′0 − E0), provides an upper bound on the welfare gain.
4Our measure of the returns to weatherization investments accounts for a household’s valuation of in-creased warmth, but we do not account for any increase in comfort conditional on indoor air temperature.Researchers have noted that improvements in insulation enhances comfort by reducing drafts and heat ra-diation while retaining moisture (Schwarz and Taylor, 1995). The enhanced comfort allows the consumerto achieve comfort at a lower thermostat setting. Consequently, what we measure here is the net effect ofefficiency improvements on heating demand. We return to this point in our final discussion.
5The figure assumes a linear relationship between air temperature and energy demand over the relevantrange of indoor air temperatures. In the literature that analyzes energy use in buildings, this linearityassumption is fairly standard (e.g. Friedman (1987), Dyson et al.,2014). Other researchers have argued thatthis relationship is better described as non-linear (e.g. Dewees and Wilson (1990)). In section 5, we estimatemore flexible specifications and provide empirical support for this linearity assumption in our data.
9
Preferences revealed prior to the efficiency improvement suggests that the agent values the
increase in heating services less than this incremental cost.
The resulting measure of WTP can be compared to the costs of these investments to
estimate net returns and to gain insight into individuals’ decisions to make energy efficiency
investments. Specifically, the net present value of an energy efficiency investment is given
by:
NPV = C −T∑t=1
WTPt
(1 + r)t, (1)
whereC measures the private costs to the household of the energy efficiency improvement;
T represents the investment time horizon for the improvement; r is the discount rate. Will-
ingness to pay for an energy efficiency investment in period t is denoted WTPt which is
the sum of the change in energy expenditures and the monetized value of the change in the
consumption of energy services
A simple model of households’ investment choices assumes that a household will invest if
NPV > 0.6 It is noteworthy that the standard approach in the energy efficiency literature is
to ignore the utility value of any change in the consumption of energy services and effectively
set WTPt equal to the value of energy savings assuming no energy demand response.
Previous work points out several reasons why households’ investment choices may devi-
ate from this simple model. For one, this equation assumes households are fully informed
both about all costs associated with the retrofits and their energy usage with and without
retrofits. Homeowners may make systematic optimization mistakes if they are inattentive
to energy costs, unable to convey the precise energy usage of an improved house or apart-
ment they are selling or renting to others (Allcott and Greenstone 2012; Myers, 2014), or
6For simplicity, we are assuming that there is single energy efficiency improvement with a distinct lifetimeT . The home energy efficiency retrofits we will ultimately be interested in analyzing comprise several differentmeasures (e.g., insulation as well as duct sealing). In these investment scenarios, projected energy savingsaccount for interactions between interventions when forecasting post-retrofit energy use.
10
if they face imperfect credit markets. Households also may value a home with retrofits dif-
ferently, meaning there is some utility loss or gain associated with the retrofit that is not
captured by Equation 1. Finally, Equation 1 assumes risk neutrality, which may not hold
given uncertainty about future energy prices and future energy consumption.
With this paper, we do not attempt to disentangle the relative importance of these
explanations. We are primarily concerned with estimating the return on investments in
efficiency. With these estimates in hand, we conduct a series of exercises that shed light on
the cost effectiveness of these efficiency investments and the extent to which an efficiency
gap exists in this context. Specifically, we calculate the internal rate of return implied by
estimating equation 1that justifies these investments under a range of assumptions. Further,
we will calculate the cost per ton of CO2 abated.
3 Background and study design
3.1 Weatherization Assistance Program
The Weatherization Assistance Program (WAP) is the nation’s largest residential energy-
efficiency program. The WAP program supports improvements in the energy efficiency of
dwellings occupied by low-income families. Since its inception in 1976, over 7 million low-
income households had received weatherization assistance through the program. Proponents
credit the program with saving energy, creating jobs, reducing emissions, and assisting low-
income households. The American Reinvestment and Recovery Act PL111-5 (ARRA) dra-
matically increased the scale and scope of WAP.7 Our analysis seeks to estimate the impacts
of weatherization assistance over the ARRA-funded time period.
WAP funds are distributed to states based on a formula tied to a state’s climate, the num-
7Funding increased from $450 million annually in 2009 to almost $5 billion for the 2011-2012 programyears. Under the ARRA-funded program, all owner-occupied households at or below 200 percent of thepoverty line were presumptively eligible for assistance.
11
ber of low-income residents, and their typical energy bills. The states distribute WAP money
to over 1000 local sub-grantees, which are typically community action agencies (CAAs) or
similar nonprofit groups. These sub-grantees are then tasked with identifying and serving
eligible households. Participating WAP households receive free energy audits and a home
retrofit that typically includes some combination of insulation, window replacements, fur-
nace replacement, and infiltration reduction. The average value of measures installed at
participating households in our data is approximately $5000 per home.
The process of applying for weatherization is highly onerous and time intensive, at least
partially to prevent fraud. Extensive paperwork documenting income eligibility must be
completed and submitted in a timely fashion. This paperwork includes utility bills, earnings
documentation, social security cards for all residents of the home and deeds to the home.
Once the paperwork is completed, households are put on a list where the waiting times
can exceed one year. After rising to the top of the list, homeowners must accommodate
scheduling of energy audits and construction crews who do the retrofits.
The energy audits are a critical step in the weatherization process. Program auditors visit
an applicant’s home to collect detailed information about the building structure, heating and
cooling systems, appliances, ventilation, etc. Taking into account local weather conditions,
retrofit measure costs, fuel costs, and specific construction details of the home, auditors es-
timate energy savings and costs of different efficiency measures. The WAP program requires
that all recommended measures return a minimum of $1.00 in incremental savings for every
$1.00 expended in labor and material costs.
The process that determines which households receive weatherization assistance is not
purely random. Local agencies often identify potential applicants from the pool of households
that are receiving other social services, although walk-in clients are routinely admitted. Local
agencies screen potential applicants for eligibility. Eligible applicants are then prioritized
following guidelines that recommend CAAs rank applicants highly if the household includes
12
elderly, persons with disabilities or families with children, or where the occupants typically
face a high energy burden (energy as a share of income) or have high residential energy use
(see 10 CFR 440.16(b) (1-5)).8 Consequently, any comparisons of energy consumption across
weatherized and un-weatherized households risks confounding the effect of the program with
pre-existing differences in determinants of energy consumption.
3.2 Research Design
The paper’s empirical challenge is to obtain causal estimates of the effect of participation
in the WAP program on energy consumption and indoor heating. The difficulty is that
households that participate in WAP are not chosen randomly from the universe of households.
This paper uses quasi-experimental and experimental approaches to address the potentially
confounding effects of non-random selection into the WAP program.
We collect data from a sample of Michigan households. Michigan is one of the largest
recipients of WAP program funding on account of its cold winters and large low-income
population. Further, we were able to secure collaborative agreements with a major Michigan
utility and five WAP agencies working in this utility’s service territory. This allowed us
access to detailed, household-level energy consumption and weatherization program data.
Michigan received over $200 million in ARRA funding for weatherization assistance.
Figure 2 plots the number of weatherizations completed at households in our data set, since
2010. The solid black line in the figure shows how activity ramped up dramatically in early
2011 when ARRA funds became available. All stimulus funds had to be spent by March
2012. After that point (marked by the solid vertical line), the pace of weatherization activity
dropped precipitously.
8Given the high ARRA funding levels during our study period, the prioritization scheme was less bindingas compared to lower funding periods.
13
Figure 2: Household weatherizations (monthly) 2010-2014
Notes: This figure plots the number of weatherization retrofits completed per month. Thebroken line marks both the start of our encouragement intervention and the point at whichstimulus funding began to stimulate additional weatherization activities. The vertical solidline marks the deadline for spending stimulus funds.
14
3.3 Quasi-experimental design
The quasi-experimental research design compares energy consumption at homes weatherized
after March 2011 with energy consumption at homes that applied for weatherization during
our study period, but have yet to receive any efficiency improvements.
Among the households applying for weatherization after March 2011, only 40 percent
actually receive weatherization assistance during our study period. The road from applica-
tion to energy efficiency investments is long and there are many potential off-ramps. The
explanations for why applicant households fail to received these investments include: failure
to complete the necessary paperwork or to schedule/complete an energy audit, risks to WAP
employees (e.g., due to the presence of asbestos in the home), and a long waiting list such
that many applicant households do not receive assistance during the period covered by our
data. Some of these explanations for program participation are orthogonal to household
characteristics that determine energy consumption patterns, while others clearly are not.
We estimate the following equation designed to control for the effects of unobservable
factors that determine energy consumption trajectories at households that apply for weath-
erization assistance:
yimt = β1{WAP}imt + αim + αmt + εimt, (2)
where yimt measures the energy consumption (natural gas, electricity, or a combined
measure)of household i in month m and year t. The WAP indicator variable switches from
zero to one in the month after a household’s weatherization retrofit is complete. The equation
includes household-by-month fixed effects, αim, to account for permanent differences in a
household’s energy consumption in a given month. It is possible to include such a rich set
of fixed effects that account for permanent household-specific seasonal variation in energy
consumption due to the multiple years that households are observed in our data. The model
15
also includes month-by-year fixed effects, αmt, to adjust for the average effects of time-varying
factors (e.g., winter temperature) across households.
The parameter of interest is β and it measures the mean difference in energy consumption
subsequent to the completion of WAP energy efficiency investments, after adjustment for the
fixed effects. It is a difference-in-differences estimator that compares the change in energy
consumption after weatherization to before, relative to consumption among households that
have either not yet weatherized through WAP or never will.
In equation 2, the primary threat to consistent estimation of β is the possibility that time-
varing factors that affect household demand for energy also influence WAP participation. For
instance, households may choose to aggressively push forward their WAP application when
they anticipate an increase in their demand for energy as would be the case when the number
of people in their household increases or they lose a job and expect to spend more time at
home. While the quasi-experimental approach does not have a direct solution to this threat
to identification, we can do more to balance observable characteristics and trends across
weatherized and unweatherized households.
To evaluate the robustness of our findings, we re-estimate Equation 2, using alternative
sets of controls. In one specification, we drop all voided applicants out of concern that these
households may differ from successful applicants in ways that affect energy trajectories over
time. Our preferred quasi-experimental estimates re-weight control observations in order to
achieve covariate balance across weatherized and un-weatherized controls. We estimate the
propensity score, or the conditional probability of receiving weatherization assistance, using
a logit specification. Estimated participation probabilities are a function of historic natural
gas and electricity consumption, trends in energy consumption over the pre-treatment period,
household income, county of residence, and other factors that can play a role in determining
whether a household receives weatherization assistance (e.g. number of children). We then
re-estimate Equation 2 using estimated inverse probability weights for all unweatherized
16
applicants.
Randomized encouragement design
The paper’s primary empirical estimates are derived from an experimental design. The basis
of the experiment is a randomly assigned encouragement intervention that aims to increase
the probability of treatment household’s participation in WAP through recruitment and
significant application assistance. This research required close collaboration with both our
partner utility and community action agencies that serve households in the utility’s service
territory. Having established these partnerships, we confined our attention to households
living in the counties served by our partner agencies. Within these counties, we identified
census blocks that are associated with high rates of home ownership, natural gas heating,
and household incomes that would quality for weatherization assistance. Households living
in these census blocks comprise our population. From this population, we randomly selected
34,161 households for our study sample. Approximately one quarter of these households
were randomly assigned to our encouragement “treatment”. These treated households were
encouraged to apply for WAP and offered extensive application assistance. We provide no
recruitment, outreach, or assistance to the remaining 25,513 households. For households
assigned to the control groups, we simply observe energy consumption and program partici-
pation decisions.
To implement the recruitment and assistance, we issued a request for proposals from
organizations specializing in grassroots organizing and mobilization. We selected Field-
Works LLC, a private company that specializes in designing communications strategies,
running neighborhood canvassing operations, and managing outreach campaigns. Prior to
our project, their staff had generated millions of phone calls and knocked on millions of
doors in previous engagements. We found them to be highly focused and innovative in their
approach to educating households about weatherization assistance, and helping individuals
17
to navigate the application and enrollment process.
We worked closely with FieldWorks to develop a persuasive recruit and assist strategy.
Fieldworks personnel, together with our field manager, coordinated field operations in Michi-
gan. Individuals selected from the targeted communities were hired to conduct the bulk of
the door-to-door canvassing and outreach activities. The encouragement campaign ran from
March to May, 2011. Enrollment assistance activities lasted through February 2012. Figure
2 illustrates how our encouragement campaign got underway around the same time that
stimulus funds began accelerating the pace of weatherization activity in Michigan.
Table 1 summarizes these encouragement and enrollment activities in detail.During the
encouragement phase, field staff made almost 7,000 initial, in-person house visits.9 . These
ground operations were complemented with 23,500 targeted “robo-calls” to raise awareness
of both the weatherization program and our encouragement campaign.
After an initial encouragement phase designed to generate interest in and awareness of
the program, we transitioned to an enrollment phase. The application process required
households to provide extensive documentation, including utility bills, social security cards
for all household members, and the deed for the house. Further, a minority of applicants
were required to provide further documentation or to correct something on their initial
application. Our staff made over 9,000 personal phone calls to provide assistance and to
coordinate in-person meetings. Over the course of 2,720 home visits, our field staff helped
individuals assemble documentation and complete paperwork. In some cases, our field staff
provided transportation to and from the program agency offices.
The final row of Table 1 reports that we spent around $445, 000 on the encouragement or
roughly $50 per household in the treatment group. It is noteworthy that we did not initially
intend to devote such extensive efforts and resources to the encourage and enrollment phases.
9Most- but not all- houses assigned to the treated group were contacted. A small fraction were deemedinaccessible (e.g., because of a locked gate).
18
Table 1: Randomized encouragement intervention
Encouragement activity
Encouraged group (households) 8,648
Initial home visits 6,694
Robo-calls 23,500
Personal calls 9,171
Follow up appointments 2,720
Average cost/encouraged hh $51.50
Note: The table summarizes efforts to encourage a group of Michigan households to take up weatherizationassistance. These households were selected randomly from a sub-population of households who were locatedin the service territory of our partner utility and presumptively eligible based on ex ante available incomeinformation.
However, the early results suggested that we were failing to have a substantial impact on
applications and concerns about the ultimate precision of our estimated treatment effects
motivated us to raise additional monies to extend this phase of the project.10
Figure 2 provides some insight into how this randomized encouragement intervention af-
fected program participation over time. The dotted line shows monthly enrollment in the ex-
perimental control group. The broken line illustrates enrollment in the (smaller) encouraged
group. We use random assignment to encouragement as an instrument for weatherization
status. In a first stage, we estimate:
10On the one hand, these costs may have been higher than necessary. This was the first time that anencouragement program for WAP has been attempted; there was some learning by doing. On the otherhand, these costs do not reflect the time that the research team devoted to overseeing the encouragementeffort.
19
1{WAP}imt = θ1{Encouraged}imt + δim + δmt + ηimt, (3)
where the dependent variable is as described in the previous subsection and there are
also household by month, δim, and month by year, δmt, fixed effects. The indicator variable,
1{Encouraged}imt is set to zero for all households prior to the encouragement intervention
which began. After March 2011, this indicator switches to 1 for the 25% of the experimental
sample that are randomly assigned to the treatment group. The estimated impact of reduc-
ing barriers to participation (e.g. information and process costs) on program uptake is of
independent interest both to policymakers and researchers.
We substitute ˆ1{WAP}imt from the estimation of equation (3) to fit equation (??) to ob-
tain ˆβIV . In this instrumental variables (IV) framework, ˆβIV is identified using the variation
in program participation that is generated via the random assignment of encouragement;
this guarantees a consistent estimate of β.
If there is heterogeneity in how weatherization retrofits impact residential energy con-
sumption, the expectation of unbiased quasi-experimental and experimental estimates β need
not be equivalent.11 Note that our quasi-experimental approach is designed to provide an
estimate of the average treatment effect on all treated households (ATET), or an average
across the full distribution of treatment effects. In contrast, the randomized encouragement
design is used to estimate the so-called local average treatment effect (LATE), or the av-
erage effect for the subset of the population who must be encouraged to participate in the
program (i.e. the compliers) (Angrist, Imbens, and Rubin 1996). In other words, differences
in the quasi-experimental and experimental estimates could be due to bias in the former or
differences in the LATE and ATET.
11Note that this will only strictly be true when we limit the quasi-experimental sample to the same twoCAAs that comprise the experimental sample.
20
4 Data Sources and Summary Statistics
4.1 Data Sources
The data summarized below correspond to two overlapping groups of households. The first
group comprises the 34,160 households in our experimental sample. To conduct this field
experiment, we worked closely with two large agencies serving households in the service ter-
ritory of our partner utility. Our experimental sample is thus drawn from the counties served
by these two agencies. The second group of households corresponds to our quasi-experimental
research design. This design did not require the same degree of coordination with our agency
partners; we were able to expand the scope of our efforts to include five implementing agen-
cies. The quasi-experimental sample therefore includes the 7,304 households that applied for
weatherization assistance at these five agencies. This quasi-experimental sample is smaller
overall but has a larger number of applicants and weatherized households, relative to the
experimental sample. At the intersection of these two groups are the 1,773 of these applicant
households who are also are part of our experimental sample.
4.1.1 Energy Consumption Data
We obtained monthly natural gas and electricity consumption data over the period January
2008 to May 2014. This period includes at least three years of pre-retrofit data for all weath-
erized households in our sample.12 The utility data track monthly kilowatt-hours (kWh)
of electricity and thousand cubic feet (Mcf) of gas used at the dwelling. We convert both
of these variables to million British thermal units (MMBtu) using the standard conversion
factors employed by the WAP program.13 .
12We identify a household as a particular account number at an address. For some addresses, we havedata on multiple accounts, if one household moved out and another moved in, but we are working with theutility to fill in data for more addresses where a single account does not cover the whole time period.
13We also reconfigure the data to account for the fact that consumption records correspond to billingcycles or segments versus calendar months. The utility assigns households to one of 21 billing portions. In a
21
Energy consumption records obtained from the utility are merged with households in our
experimental sample and the applicant data which comprise our quasi-experimental sample.
Data are merged using detailed name and address information. Not all households find an
exact match in consumption records. Match rates are 85 and 69 percent in our experimental
and quasi-experimental samples, respectively.14
4.1.2 Energy efficiency measures
Before implementing a weatherization retrofit, CAA program staff conduct an energy audit
of the home. Detailed data collected during the audit are used to calibrate a computer-based
audit tool: the National Energy Audit Tool (NEAT). This tool uses engineering algorithms
to model the energy use of single-family and small multi-family residential units. NEAT is
the most widely used tool for weatherization audits; it is used by state and local WAP sub
grantees, utility companies, and home energy auditors.(EERE (2013))
The purpose of the audit is to make recommendations regarding which efficiency im-
provements (i.e. building envelope, space-heating and space-cooling system, and baseload
energy efficiency measures) should be implemented at the home. Auditors first estimate the
impacts of different combinations of efficiency measures on household energy consumption.
Projected energy savings are then translated into annual bill savings. The present value of
energy savings are calculated using a discount rate of 3 percent and an engineering estimate
of the lifespan of the measures.15
given month, each portion maps to a different set of calendar days. For example, one household’s June billmay reflect consumption in all of June, whereas another household’s June bill captures the last half of Apriland the first half of June. The utility provided us with a detailed mapping of billing segments to calendardays over the duration of our study period. For each customer and for each billing cycle, we divide meterreads evenly across days in the cycle. These “meter-day” measures are then aggregated by calendar monthto construct estimates of monthly consumption at the household.
14There are several possible reasons why our match rates are not perfect. For example, the householdmember holding the utility account may have a different surname than the household applying for weath-erization. There were also several applicant households located outside the service territory of our partnerutility.
15The 3% discount rate is consistent with OMB guidance on how to evaluate benefits of federal spending
22
Importantly, whereas program auditors are careful to calibrate efficiency audit analysis to
the characteristics of the dwelling, they do not calibrate their analysis to reflect the behavior
of the occupants. The rationale is that calibrating savings estimates to the behaviors of
the occupants would over-allocate program resources to more intensive users.16 Also, the
engineering estimates of savings do not aim to account for any behavioral responses to the
increased efficiency of energy services, such as the rebound effects outlined above.
For each audited household, auditors generate a report of recommended energy efficiency
measures that identifies—both individually and cumulatively—the projected energy savings,
installation cost, and the savings:investment ratio of the recommended measures. We have
obtained all of the information used to calibrate the simulations and generate the report for
all audited households in our sample. We have also obtained the work summaries filed once
the weatherization is complete. This provides information about what the measures actually
cost to install.
4.1.3 Indoor Temperature Data
Two years after our encouragement effort was initiated, a subset of weatherized and un-
weatherized households were selected randomly for a field survey. The primary purpose of
the survey was to collect measurements of indoor temperatures that can be used to test for
a direct “rebound effect.”
Michigan field staff attempted to contact 6,400 households on cold days (temperatures
below 45 degrees Fahrenheit) in March and early April 2013. Survey questions were de-
signed to collect information about thermostat set points and, where applicable, household’s
experience with weatherization. With the home owner’s permission, surveyors entered the
home, closed the door, moved to the center of the room, and recorded multiple indoor air
but is substantially lower than the cost of borrowing for most households, especially low income ones.16For example, a furnace replacement would appear to generate higher energy savings for a household
that was keeping its thermostat at 80 degrees during the winter than for a household that was keeping itsthermostat at 65 degrees.
23
temperature measurements using two different devices. Whenever possible, temperature
measurements were recorded in multiple rooms. Of our initially targeted sample, surveyors
spoke with1,658 home owners. Of these, 688 allowed us to enter their homes, close the door,
and collect two or more indoor thermometer readings. Survey results are discussed in detail
in Section X.
4.2 Summary Statistics
Table 2 summarizes pre-treatment information on the households in both the RED and quasi-
experimental samples. The top panel summarizes monthly energy consumption during the
immediate pre-treatment period: 2010-2011. The first two columns summarize means and
standard deviations (in parentheses) for the randomized encouragement and experimental
control groups, respectively. The third column reports differences between the control and
treatment groups. The fourth column summarizes these data for all weatherized households.
The final three columns correspond to the quasi-experimental controls.
Winter natural gas consumption (which is dominated by space heating) is significantly
higher than summer gas use (comprised primarily of hot water heating and cooking). Elec-
tricity usage, also measured in MMBtu, is fairly consistent across seasons.17
Households in the experimental sample were randomly assigned to the encouraged and
control groups. It is therefore to be expected that the differences in natural gas and electricity
consumption across these two groups, reported in column 3, are all small and statistically
indistinguishable from zero.
The table also provides an opportunity to judge the credibility of the comparisons that
underlie the quasi-experimental estimates that complement the paper’s experimental esti-
17Overall, these lower income households in our sample consume less energy on average as compared tothe larger Michigan population. In our sample, average monthly natural gas consumption is approximately7.33 MMBtu. Across the entire service territory, the annual residential natural gas consumption averages7.75 MMBtu per month.
24
Tab
le2:
Diff
eren
ces
insa
mple
mea
ns
bet
wee
ngr
oups
ofhou
sehol
ds
Exp
erim
enta
lE
xp
erim
enta
l(1
)-
(2)
All
Unw
eath
eriz
ed(3
)-
(4)
(3)
-m
atch
eden
cou
rage
dco
ntr
olw
eath
eriz
edap
pli
cants
app
lica
nts
(1)
(2)
(3)
(4)
(5)
(6)
(7)
Pre
-tre
atm
entperiod
month
lyenerg
yconsu
mption
Win
ter
gas
(MM
Btu
)10.
4010
.38
0.02
9.88
11.6
3-1
.75∗∗
0.08
(5.3
6)(5
.23)
(0.0
7)(3
.76)
(5.1
1)(0
.16)
(0.1
5)
Su
mm
ergas
(MM
Btu
)2.8
42.
790.
061.
802.
16-0
.36∗
∗0.
04(3
.87)
(1.9
3)(0
.03)
(1.7
5)(1
.94)
(0.0
5)(0
.06)
Win
ter
elec
tric
ity
(MM
Btu
)2.1
22.
100.
022.
242.
30-0
.06
0.00
(1.1
7)(1
.20)
(0.0
2)(1
.39)
(1.3
9)(0
.04)
(0.0
0)
Su
mm
erel
ectr
icit
y(M
MB
tu)
2.17
2.17
0.00
2.23
2.20
0.03
0.00
(1.3
0)(1
.28)
(0.0
2)(1
.39)
(1.3
9)(0
.04)
(0.0
0)
Demogra
phicsand
dwellingch
ara
cteristics
Hou
seh
old
inco
me
($)
19,6
1717
,509
2,10
8∗∗
-965
(12
,018
)(
11,9
48)
(417
.16)
(522
.31)
Per
cent
of
pov
erty
(%)
115
104
11∗∗
-4(
55)
(64
)(1
.96)
(2.6
7)
Hou
seh
old
size
(#)
2.56
2.47
0.09
-0.0
8(1
.68)
(1.
51)
(0.0
6)(0
.07)
Ch
ild
ren
0.24
0.15
0.10
∗∗-0
.02
(sh
are
of
hou
seh
old
s)(0
.43)
(0.
35)
(0.0
1)(0
.02)
Rep
orte
dd
isab
ilit
y0.
040.
030.
01∗
0.00
(sh
are
of
hou
seh
old
s)(0
.19)
(0.
16)
(0.0
0)(0
.01)
Eld
erly
0.23
0.13
0.09
∗∗-0
.00
(sh
are
of
hou
seh
old
s)(0
.42)
(0.
34)
(0.0
1)(0
.02)
Hig
hsc
hool
edu
cati
on
0.46
0.36
0.10
∗∗0.
08∗∗
(sh
are
of
hou
seh
old
s)(0
.50)
(0.
48)
(0.0
1)(0
.02)
Age
of
hom
e(y
ears
)59
.15
62.0
8-2
.93
2.02
(26.
63)
(21
3.28
)(5
.34)
(7.8
1)
Hou
seh
old
s7,
549
21,3
392,
074
2,97
3
Not
e:C
olu
mn
snu
mb
ered
(1),
(2),
(3),
and
(4)
rep
ort
aver
age
valu
esan
dst
an
dard
dev
iati
on
s(in
pare
nth
eses
).O
ther
colu
mn
sre
port
diff
eren
ces
inm
eans
(sta
nd
ard
erro
rsar
ein
par
enth
eses
).F
inal
colu
mn
rep
ort
sd
iffer
ence
sin
mea
ns
bet
wee
nw
eath
eriz
edh
ou
shold
san
dp
rop
ensi
ty-s
core
wei
ghte
dco
ntr
ols
(con
dit
ion
ing
onap
pli
cati
on).
Sig
nifi
cant
atth
e5
per
cent
leve
l*
Sig
nifi
cant
atth
e1
per
cent
leve
l
25
mates.Column 5 of the table summarizes data from the pool of possible quasi-experimental
controls: households that have applied for weatherization but have not received weather-
ization assistance. The mean differences in column 6 show that weatherized households
have historically consumed significantly less natural gas than the un-weatherized applicants
during both winter and summer months.
The significant differences between the weatherized and unweatherized applicants mo-
tivate us to re-weight observations in the control group so that observable factors are dis-
tributed similarly in the weatherized and unweatherized applicant groups. We use an es-
timated propensity score to balance covariates that presumably play a role in determining
program take-up such as historic energy use, income, and other demographics. This reweight-
ing exercise is discussed in detail in Section 5. The final column in Table 2 reports differences
in average covariate values across weatherized households and propensity-score weighted con-
trols. Reassuringly, this reweighting eliminates all significant differences in pre-period energy
consumption.
The lower panel in Table 2 summarizes the detailed demographic information and dwelling
characteristics that are collected as part of the application process. These data are not avail-
able for the majority of households in our experimental sample that choose not to apply to the
program. Weatherized households have higher incomes in an absolute and relative sense.18
Weatherized households report having more children, are more likely to report an elderly
resident, and less likely to report a disabled resident.
Table 3 reports on the detailed information collected during the household energy ef-
ficiency audits which inform the implementation of the weatherization assistance program.
The table summarizes data from 1,630 households receiving weatherization assistance during
18Program eligibility is based on the “percent of poverty”. The Census Bureau uses a set of moneyincome thresholds that vary by family size and composition to determine who is in poverty. To qualify forthe program, a household’s income cannot exceed 200 percent of poverty. Applicants fall well below thisthreshold on average. Incomes among applicants who ultimately receive weatherization are farther abovethe poverty line on average.
26
the period covered by our data.19
The top panel of Table 3 summarizes the projected energy savings. The simulations
predict that the average household will save over 50 MMBtu of natural gas each year as
a result of the efficiency improvements installed. Using auditor’s estimates of households’
baseline energy consumption, this translates into a quite substantial 46 percent reduction
in gas use. Weatherization investments are projected to have much smaller impacts on
electricity consumption with NEAT predicting a 16% decline in electricity consumption.
The middle panel of Table 3 summarizes information on the costs and projected savings
for the WAP projects. The average project involved more than $4,400 in direct expenditures
on energy saving measures. This includes materials, labor, and construction costs, but
does not include any program overhead. If costs on non-energy measures (such as safety
improvements, carbon monoxide detectors, etc) are included, average costs exceed $5,100.
Ex post realized expenditures are somewhat lower than expected. .
Using a 3% discount rate, the projected net present value of energy bill savings average
$10,680, yielding an average projected savings to investment ratio of over 2.4. Recall that
the program will only fund efficiency measures with projected savings to investment ratio
exceeding 1.
The bottom panel provides summarizes the measures recommended and implemented at
weatherized houses. Attic insulation occurs at 85% of households, while furnace replacement
(34%) and wall insulation (44%) are less frequent.
19Only 1,630 of the 2,194 NEAT data files could be confidently matched with weatherized households. Weapplied fairly strict matching criteria so as not to mis-match weatherized households with audit information.Occupant names were often not included in the audit files. Addresses appear to have been miscoded inseveral instances.
27
Table 3: Projected costs, savings, retrofit measures at weatherized households
Weatherizedhouseholds
Energy savings (projected)
Natural gas savings: heating (annual MMBtu) 50.37(46.03)
Heating energy savings/ baseline 0.46(0.20)
Electricity savings: cooling (annual MMBtu) 4.29(50.53)
Cooling energy savings/ baseline 0.16(0.12)
Investment costs and projected savings
Investment cost: reported ($) 4,146(2,692)
Investment cost: projected ($) 4,406(2,491)
Projected NPV savings ($) 10,680(11,862)
Projected savings:investment ratio 2.47(2.92)
Key measures
Furnace replacement 0.34(0.47)
Attic insulation 0.85(1.06)
Wall insulation 0.44(1.16)
Infiltration reduction 0.76(0.43)
Households 1,630
Notes: This table summarizes data from all weatherized households that could be exactly matched withaudit data.Average values reported, standard deviations appear in parentheses.
28
5 ResultsQuasi-experimental Estimates of Energy Sav-
ings
Table 4 presents the quasi-experimental estimates. The dependent variable in all regressions
summarized by this table is the log of monthly energy consumption (measured in MMBtu).
The first two columns use data from all weatherization applicants. Specifications differ only
in terms of the fixed-effects included, with the second specification allowing time period
effects to vary across counties. Columns (3) and (4) drop voided applications from the
sample. Columns (5) and (6) report the propensity score reweighted estimates.20 Standard
errors are clustered by household in all specifications.
The first row in Table 4 reports the estimated average treatment effect. Because the
dependent variable has been log transformed, these estimates can be interpreted in terms
of percentage reductions in monthly energy consumption. Estimates range from approxi-
mately 8-10 percent across specifications. The second row reports average monthly energy
consumption in the corresponding control group during the post-encouragement period. Im-
plied energy reductions range from 0.6 MMBtu/month (implied by specifications (3) and
(4)) to 0.92 MMBtu/month (specification (6)).
Panel B of Table 4 computes the present value of the estimated energy savings under
alternative assumptions about investment time horizons and discount rates. To express our
estimates of monthly energy savings into dollar terms, we decompose the estimates reported
in Table 4 into fuel-specific savings. More precisely, we re-estimate equation (?? ) for nat-
ural gas and electricity, respectively (See Appendix). Estimates of average monthly natural
gas savings are multiplied by the average residential retail price of natural gas (in $2013)
charged by this utility over the post-weatherization period ($7.98/MMBtu). Similarly, aver-
20Because some of the covariates included in the propensity score estimating equation (e.g. reporteddisability and number of children) are not reported by all households, this sample is somewhat smaller.
29
Table 4: Quasi-experimental estimates of the impact of weatherization on household energyconsumption
Dependent variable is monthly energy consumption (in logs)
All applicants Drop voids Matched controls(1) (2) (3) (4) (5) (6)
WAP -0.08∗∗ -0.09∗∗ -0.08∗∗ -0.08∗∗ -0.09∗∗ -0.10∗∗
(0.01) (0.01) (0.01) (0.01) (0.01) (0.01)
Average consumption 8.13 7.28 9.27(MMbtu/month) [ 7.80] [7.09] [8.27]in control group
month-of-sample FE Y N Y N Y Nmonth-of-sample x county FE N Y N Y N YP-score matched sample N N N N Y Y
R-squared 0.85 0.86 0.85 0.86 0.80 0.81
Households 5,012 5,012 4,330 4,330 3,403 3,403Observations 282,196 282,196 247,,742 247742 188,235 188,235
Present value of (discounted) savings
Time Horizon Discount rate3 percent 6 percent 10 percent
10 years $1,297 $1,119 $99415 years $1,815 $1,476 $1,15620 years $2,261 $1,743 $1,294
Note: Panel A reports estimates of the reduction in monthly energy consumption following weatherization. The unit ofobservation is a household-month. The dependent variable is monthly household energy consumption (electricity andnatural gas) measured in MMBtu. The coefficient reported is the coefficient on the weatherization indicator. Standarderrors (in parentheses) are clustered at the household level. Panel B reports the net present value of energy savingsimplied by the preferred estimate reported in. Panel B computes the net present value of estimated savings. Reductionsin energy bills associated with the estimates in column (6) are assumed to accrue over the life of the measure. using arange of discount rates and assumed time horizons.* Significant at the 5 percent level** Significant at the 1 percent level
30
age electricity savings are multiplied by the average retail electricity price ($0.11/kWh).21We
estimate that weatherized households save $7.82 per month on average in avoided gas ex-
penditures and $4.84 per month in electricity costs. This implies an average savings of $152
per year. 22
To compute the net present value of these energy savings, we assume that the effect of
weatherization on energy consumption - and energy prices- do not vary over the life of the
measure. Discounted benefits are calculated at discount rates of 3%, 6%, and 10% (see rows).
The NEAT program specifies life spans for all efficiency measures. Projected lifespans
range from 3 years (for a furnace tune-up) to 20 years (for attic insulation). The energy
savings-weighted average lifespan for installed measures in our dataset is 16 years. In the
table, we report discounted benefits over a range of time horizons(10-20 years). Estimates of
the net present value of energy savings range from $994 (high discount rate and short time
horizon) to $2,261 (low discount rate and long time horizon). Notably, all of these estimates
of discounted benefits fall below the average cost of the measures installed ($4,146). We
explore this finding in more detail in the next section.
5.1 Experimental Estimates of Energy Savings
5.1.1 First-stage: Program Take-Up
It may seem straightforward to encourage households to participate in a program that pro-
vides free efficiency retrofits worth over $4,000 that are designed to significantly reduce
energy expenditures. In our experience, that was hardly the case.
Table 5summarizes program take-up at three separate stages. In the first stage, our goal
21The NEAT program audits assume an electricity price of $0.11/kWh and a natural gas price of$11.46/MMBtu. The higher gas price is presumably based on 2006 prices which averaged around$11.50/MMBtu in this service territory in 2006.
22An alternative approach redefines the dependent variable in the IV regressions in monetary terms. Weobtained nominal retail prices for electricity and gas over the study period and impute monthly energyexpenditures using these prices. This approach yields very similar results
31
Table 5: Randomized encouragement: Return on effort
Application Efficiency Weatherizationaudit complete
Base rate 0.02∗∗ 0.01∗∗ 0.01∗∗
(0.00) (0.00) (0.00)
Encouragement 0.13∗∗ 0.05∗∗ 0.05∗∗
(0.00) (0.00) (0.00)
Households 28,889 28,889 28,889
Note: The table shows the effect of our encouragement on program applications, efficiency audits, andweatherization. Indicators of program participation status are regressed on an encouragement indicator anda constant. The unit of observation is a household. Significant at the 5 percent level* Significant at the 1 percent level
was simply to increase the share of households filing applications. The column (1) entry
shows that the encouragement intervention increased the rate of application to the program
by 11 percentage points from the control group mean of 2 percent.
Once a household’s application is approved, the energy audit is the critical next step.
Column (2) reveals that the fraction of households who received an energy audit was 4
percentage points higher in the encouraged group (off a base of about 1 percent). The
considerable drop off between submitting an application and receiving an energy audit is
due to several factors. For example, households asked for additional information to complete
their application sometimes neglected to follow through. Many households that submitted
applications during our study period are still waiting to schedule their audit as the pace of
the program dropped off precipitously after the ARRA funding ended. Figure 2 shows that
weatherization rates have dropped to fewer than ten weatherizations per month; households
32
waiting for weatherization assistance likely face a very long wait.
Column (3) of Table 5 documents that the treatment increased the fraction of households
that were successfully weatherized by about 4 percentage points, against a 1 percent rate
in the control group.23 The encouragement treatment is a sufficiently powerful predictor of
weatherization such that it can be used to instrument for program participation.
Before proceeding, it is worth underscoring that the low take-up rates in the encouraged
group are striking. Participants in the program incur no direct monetary costs. Successful
applicants receive substantive home improvements. All households in our encouraged group
received some information about the program via a phone call or door hanger. A majority of
households (i.e. those who spoke with our canvassers in person or by phone) received further
information about our offer of application assistance. It seems reasonable to surmise that
some combination of high perceived costs of applying for the program, low expectation of an
application leading to a weatherization, high unmeasured process costs, and low expected
benefits of participating in the program are impediments to WAP participation. In the end,
the average cost of encouragement per completed weatherization was about $1050, which is
more than 20% of the average costs of measures installed.
5.1.2 Instrumental Variables Estimates of Energy Savings
Panel A of Table 6 presents the experimental estimates of the impact of weatherization on
energy consumption, using random assignment to the encouragement group as an instru-
mental for program participation. In the first two columns, the dependent variable is the log
of total energy consumption (MMBtu/month). The third and fourth columns report results
separately for natural gas and electricity, respectively.
In the , monthly energy consumption is regressed directly on the WAP indicator. This
serves as a non-experimental differences-in-differences benchmark to be compared with the
23A small fraction of households get audited but not weatherized, primarily because the auditors deemthe home a possible danger to weatherization contractors (e.g., due to the presence of asbestos).
33
IV estimates. This estimate is very similar to the quasi-experimental estimates discussed in
the preceding section..
The IV estimate is reported in column (2). We estimate that weatherization reduces
monthly energy consumption by approximately 20 percent among households induced to
participate in WAP by the treatment. The experimental point estimate of energy savings is
twice as large - and statistically distinguishable from - the non-experimental estimate.
Columns (4) and (5) report local average treatment effects for natural gas and electric-
ity, respectively. Natural gas accounts for 94 percent of projected savings (measured in
MMBtu), so it is not surprising that natural gas consumption is more significantly impacted
by weatherization.24 . The effect of weatherization on gas consumption is also more precisely
estimated. This is also to be expected. Natural gas consumption is driven primarily by the
end uses targeted by weatherization (space and water heating), whereas electricity consump-
tion is derived from many end uses that are unaffected by weatherization (e.g. lighting and
appliances).
Panel B of Table 6 computes the present value of the estimated energy savings under
alternative assumptions about investment time horizons and discount rates. The approach we
take is similar to that described in section 5. One difference is that we cannot directly observe
average energy consumption among compliers assigned to the control group. To obtain a
consistent estimate of energy savings in levels, we re-estimate equation separately for natural
gas and electricity. Estimation results are summarized in the Appendix. The product of fuel
savings estimates and the corresponding fuel prices summed across natural gas and electricity
yield average annual savings of $222.40 per household. Discounted savings estimates range
from $1367 (high discount rate and short time horizon) to $3309 (low discount rate and long
time horizon).
24Note that the samples differ slightly across the columns and that the coefficients are not constrained tobe equal across columns (2) and (3) so the total energy estimate is not mechanically a weighted average ofthe gas and electricity coefficients.
34
Table 6: Estimated impacts of weatherization on household energy consumption
Panel A: Dependent variable is monthly energy consumption (log MMBtu)
Total Energy Gas Electricity(1) (2) (3) (4)
OLS-FE IV-FE IV-FE IV-FE
WAP -0.11∗∗ -0.21∗ -0.22∗∗ -0.10(0.01) (0.09) (0.08) (0.11)
month-of-sample FE Y Y Y Y
F-statistic . 263.3** 256.0** 262.3**encouragementHouseholds 27,229 27,229 25,505 26,571
Observations 1,653,583 1,653,583 1,515,117 1,624,030
Panel B: Present value of (discounted) savings
Time Horizon Discount rate3 percent 6 percent 10 percent
10 years $1928 $1663 $138815 years $2628 $2195 $171920 years $3362 $2592 $1924
Note: Dependent variable measures monthly household energy consumption. Panel A reports regressioncoefficients. With the exception of the first column, all specifications are estimated using 2SLS. Standarderrors (in parentheses) are clustered by household. Panel B reports savings projections generated by NEATaudit. See text for details. Standard deviations are in parentheses.
∗ Significant at the 5 percent level∗∗ Significant at the 1 percent level
35
The roughly 20 percent savings in overall energy consumption is more than twice as large
as the quasi-experimental estimates (and a simple DID constructed using data from the
experimental sample). These larger impacts imply larger reductions in energy expenditures
over the life of the measures. Ideally, we would like to understand what drives the differences
between the quasi-experimental and experimental estimates.
When we compare households receiving weatherization across the experimental encour-
aged and control groups, we find that WAP contractors installed fewer measures at encour-
aged households on average, and the total cost of the investment was significantly lower.
This rules out the possibility that differences in savings rates can be explained by higher
investment rates among compliers. We cannot, however, rule out the possibility is that
weatherization impacts (holding investment levels constant) are significantly larger among
compliers as compared to the households that seek out weatherization without our added
encouragement. Notably, weatherized households in the encouraged group do differ along
observable dimensions. In particular, they have significantly higher incomes and larger house-
holds. The houses do not differ significantly along observable dimensions (such as size or
age). Of course, another potential explanation for the difference in estimated savings rates
is a violation of the identifying assumption for the quasi-experiment.It may be that the tra-
jectory of average energy consumption in the quasi-experimental control groups provides a
biased (down) estimate of the counterfactual energy consumption at weatherized households
post-weatherization.
5.2 Household re-optimizing behavior, building thermal proper-
ties, and the welfare implications of rebound
In Section 2 we introduced the concept of bounding the average willingness to pay for an
efficiency-induced increase in energy services. In this section, we implement this bounding
36
exercise in three steps.
5.2.1 Does weatherization lead to temperature “take back”?
The first step tests for an effect of weatherization on household demand for space heating.
Table 7 summarizes the results from our survey of indoor air temperatures collected during
the winter of 2013. All weatherized households surveyed had received efficiency improvements
at least one year before the survey was administered, allowing plenty of time for residents to
observe how the retrofit affected winter heating costs.
Approximately half (347) of the 688 households that allowed us to enter their home
to take temperature measurements had received weatherization assistance. Anticipating
some measurement error, we used two different devices to measure indoor temperatures at
each home. We regress thermometer readings on a binary variable indicating whether the
household has been weatherized. Standard errors are clustered at the household level.
Table 7 summarizes the regression results. Column (1) reports the base specification.
The constant term measures the average indoor air temperature at unweatherized house-
holds: 67.3 degrees Fahrenheit.25 The coefficient on the weatherization dummy suggests
that weatherized homes are kept 0.73 degrees Fahrenheit warmer. This estimate borders on
statistical significance at conventional levels.
25Notably, this estimate of the baseline indoor temperature is very close 68 degrees, the set point assumedby the NEAT audit tool.
37
Table 7: Indoor temperature survey results
Dependent variable is indoor air temperature (measured) in ◦F
(1) (2) (3) (4)
Base temperature 67.27** 67.31** 72.33** 72.48**(0.36) (0.35) (0.94) (1.12)
Weatherized home 0.73 0.78 0.63 0.71(0.42) (0.44) (0.41) (0.43)
HDD . . −0.16∗∗ −0.15∗∗
(0.03) (0.04)
Largest implied 1.54 1.65 1.44 1.57increase (95%)
Propensity score N Y N Yweights
R-squared 0.003 0.003 0.03 0.03
Note: The table reports measured indoor temperature differentials across weatherized (WAP) andunweatherized households. All specifications estimated on 1359 observations across 688 households basedon two thermometer reads at most households. Columns (2) and (4) are weighted so that surveyedpopulation better represents total quasi-experimental sample. Standard errors clustered at a householdlevel. Significant at the 5 percent level* Significant at the 1 percent level
38
Survey respondents comprise a non-random subset of our sample. Only a fraction of
the targeted household were home and/or willing to open the door to receive our surveyors.
Only 41 percent of households who opened the door to our surveyors allowed us to come
in and collect temperature measurements. Pairwise comparisons of observable household
and dwelling characteristics across surveyed households and the larger quasi-experimental
sample reveal that the surveyed sub-sample is observationally similar among most- but not
all- dimensions. Survey respondents are more likely to report having children or elderly
family members, less likely to be unemployed, and are more likely to use gas as their primary
source of heat, as compared to the larger quasi-experimental sample. Among unweatherized
households, surveyed households are larger.
Given these observable differences between survey respondents and the larger sample,
we also estimate a weighted regression where household observations are weighted so as
to match the covariate distributions in the larger quasi-experimental sample of weatherized
households. These results are reported in columns (2) and (4). Weighted regression estimates
do not differ significantly. 26
Columns (3) and (4) control for the outdoor temperature on the day of the survey. The
coefficient on outdoor temperature is statistically significant, suggesting that our indoor
temperatures measure lower on colder days. Although we asked surveyors to wait several
minutes before recording temperatures, this finding suggests that cold air brought with the
surveyor could be affecting our measurements. Point estimates of the average impact of
weatherization on indoor temperatures remain small and imprecisely estimated.
In sum, we find some suggestive - but noisy- evidence of a small effect of the weatheriza-
tion intervention on indoor air temperatures. We believe this is the first field measurement
26To re-weight observations, we take a similar approach as was deployed in the quasi-experimental esti-mation of average treatment effects. In a first stage, we estimate the probability of responding to the surveyconditional on being in our quasi-experimental sample. Inverse propensity scores are then used to weightobservations for survey responders in the regression.
39
of household’s re-optimizing response to exogenous variation in heating efficiency improve-
ments.
5.2.2 Building energy performance
In order to estimate how a given increase in demand for indoor space heating translates
into an increase in monthly gas consumption, we need to estimate the thermal properties
a representative building. We use the so-called “degree day method”, quite standard in
building energy performance analysis, to model the energy required to increase temperatures
in an average home in our data (Thorpe, 2013).
In residential buildings where building envelope losses are the major determinant of
heating energy requirements, it is standard to summarize the technical relationship between
energy consumption and heating demand by regressing energy consumption against heating
degree days. Heating degree days (HDDs) represent the number of degrees over a period in
which the ambient temperature is lower than a specified base level. We collect hourly HDD
data for each county in our data and aggregate HDDs up to the county-month level.27 We
estimate the following equation:
Cimt = αi + β11{WAP}imt + β2HDDmt + β3HDDmt ∗ 1{WAP }imt (4)
+β4HDD2mt + β5HDD
2mt ∗ 1{WAP}imt,
where Cimt measures the natural gas consumption at household i in month m and year
t. We estimate this equation using data collected from program applicants during winter
27We adopt a standard base temperature of 65°F. This is consistent with the base temperature estimatedin the previous section less the typical heat gain that comes from other sources such as people and appliancesin the home.
40
months (September-March) over the entire sample period. Panel data allows us to include
household-level fixed effects in this regression. 28 We include both a linear and a quadratic
HDD term, allowing each coefficient to vary with weatherization status.
Estimates of all coefficients in equation (4 ) are highly statistically significant and very
precisely estimated. The R-squared value is 0.73. Figure 3 summarizes the relationship
between energy consumption and HDD separately for weatherized and unweatherized ob-
servations. This figure is analogous to the bottom panel of Figure 1 (flipped into the top
quadrant). As expected, the slope of this relationship (which is approximately linear) is less
steep among weatherized homes.
Before proceeding to use these estimated relationships to translate changes in indoor
temperatures into changes in energy consumption, several caveats are in order. First, this
approach to estimating the relationship between indoor heating demand and building en-
ergy consumption implicitly assumes that indoor air temperatures are set at a constant level
during the winter months. Our temperature survey data calls this assumption into question,
although another explanation for the relationship between our measured indoor tempera-
tures and outdoor temperatures could be cold air coming into the home with the surveyor.
Second, these graphs are generated using the data from our quasi-experimental sample and
are vulnerable to the selection biases discussed above.
5.2.3 Bounding the average valuation of increased indoor heat
Figure 3 shows that weatherization effectively reduces the marginal cost of space heating.
To get a sense of the average impact of weatherization on the marginal cost of heating, we
multiply the slope of the (approximately linear) relationships in Figure 3 by the average
28Although it is standard to assume a linear relationship between heating demand and energy consumption,researchers have noted the potential for a non-linear relationship. For example, Dewees and Wilson (1990)note that while heat losses per unit of air will rise linearly with the temperature difference, air exchangelosses are increasing in temperature. This implies that the marginal cost of a degree of indoor temperaturecan increase as the outdoor temperature falls.
41
natural gas price in the post-encouragement period. We obtain an estimate of $0.072 per
heating degree day (or $2.17 per heating degree month) at unweatherized households and
$0.056 per heating degree day (or $1.67 per heating degree month) at weatherized homes.
This implies a marginal cost reduction of approximately 20 percent. 29
If we take as given our point estimate of the efficiency-induced rebound in heating demand
(0.78 °F) and the summarized relationships between heating demand and energy consump-
tion (Figure 3), we can bound the welfare gain from re-optimizing heating demand. The
lower bound is given by 0.78 °F * $1.67/degree-month. At this lower bound, the utility
gains from increased warmth are exactly offset by the increase in the energy expenditures
incurred to achieve the temperature increase. To define the upper bound, we note that by re-
vealed preference, households are not willing to pay 0.78 °F * $2.17/degree-month to achieve
this incremental increase in temperature in the unweatherized state. It follows that average
marginal benefits from this temperature increase cannot exceed $2.17. So the average net
gain from the increase in warmth following weatherization (net of increased expenditures to
achieve the increase) should not exceed $0.39per winter month. In sum, this bounding exer-
cise suggests that the welfare gains from any efficiency-induced rebound in heating demand
are quite small.
6 Interpretation
In this section, we evaluate the returns to energy efficiency investments through the WAP
program from both a private and social perspective. We also provide an alternative summary
measure of the program’s cost effectiveness, which is the cost per ton of carbon abated.
29These estimates of incremental heating costs are comparable to a “rule of thumb” popularized by theAmerican Council for an Energy-Efficient Economy. This rule states that a household will pay approximately3% on their gas bill for a degree increase in winter thermostat settings. Average natural gas bills duringwinter months are $85.95 and $57.96 at weatherized and un-weatherized homes, respectively.
43
6.1 Cost effectiveness
With our estimates of energy savings and households’ valuation of efficiency-induced increase
in winter warmth in hand, we turn to an evaluation of the overall cost effectiveness of these
weatherization investments. Panel A of Table 8 evaluates the internal rate of return (IRR) on
investment from a private perspective. More precisely, we report the discount rate at which
the discounted value of average avoided energy expenditures exactly equals the average
upfront investment. Although households in our sample did not actially incur installation
or materials costs in this program, this IRR metric provides a useful summary statistic for
thinking about the cost effectiveness of these weatherization improvements.
Column (1) computes this internal rate of return using the average installation and ma-
terials cost ($4,991 per household) and the average reductions in annual energy expenditures
projected by the WAP program audit. Over a range of time horizons, returns are quite
high; all exceed 10 percent. In other words, efficiency investments supported under the
Weatherization Assistance Program appear highly cost effective based on the savings and
cost projections that informed program implementation.
The second column of Panel A replaces the natural gas price assumed by the WAP effi-
ciency audit calculations (i.e. $11.46/MMBtu) with the average volumetric price paid by re-
tail customers in this service territory over the post-weatherization period ($7.82/MMBtu).30
This reduction in the valuation of avoided natural gas consumption notwithstanding, esti-
mated rates of return on investment are still positive across all time horizons.
The third column of Table 8 replaces the ex ante projected savings generated ex ante
during the WAP efficiency audit with the experimental estimates generated ex post using
our randomized encouragement design. The experimental estimate of average energy savings
amount to less than a third of the average projected savings. Consequently, this adjustment
30No change is made to electricity rates because the retail electricity price used in the efficiency auditsmatches the average observed retail price.
44
Table 8: Estimated returns on investments in energy efficiency
Time (1) (2) (3) (4)horizon
Panel A: Private internal rate of return on investment
10 years 13% 1% -12% -12%15 years 17% 6% -5% -4%20 years 18% 8% -1% -1%
Panel B: Social rate of return on investment
10 years 17% 2% -11% -10%15 years 20% 7% -3% -3%20 years 21% 9% -1% 0%
Panel C: CO2 abatement cost - 3 percent discount ($/ton CO2)
10 years -$87 $48 $525 $51715 years -$132 -$3 $304 $29620 years -$154 -$19 $195 $187
Panel D: CO2 abatement cost - 7 percent discount ($/ton CO2)
10 years -$53 $82 $692 $68415 years -$96 -$38 $472 $46820 years -$117 -$18 $375 $367
Energy savings projected projected experimental experimentalestimate (NEAT) (NEAT) estimate estimate
Energy value NEAT post-period post-period post-periodassumed average observed average observed average observed
Rebound N N N Yadjustment
Note: Calculations summarized by column (1) uses average retrofit costs of $4991 and average engineeringprojections of annual savings ($907.84). In Panel B, column (1) also incorporates the value of estimatedemissions reductions valued (using a social cost of carbon value of $38 per ton CO2). Column (2) replacesthe value of natural gas used in the WAP program audits with the average retail price in 2012-2104 (PanelA) or the average gas recovery charge over this period (Panels B and C). Columns (3) and (4) replaceengineering estimates of energy savings with our experimental estimates of energy savings. Column (4)incorporates the upper bound on the net welfare gain from increased heating demand.
45
to the savings estimates pushes the rate of return on these investments into the negative
domain. The final column incorporates our estimate of the upper bound on welfare gains
from the rebound in heating demand (net of the associated increase in energy costs). This
small welfare gain has little impact on estimated returns.
Panel B conducts a very similar exercise from a social (versus private) perspective. These
calculations differ from those in Panel A in two important ways. First, estimated benefits
include the value of avoided emissions.31 Avoided emissions estimates are predicated on
the assumption that burning natural gas emits 116.39 lbs CO2 per mmbtu and a marginal
operating emissions rate of 0.916 lbs CO2 per kWh in the Midwest power sector (Callaway
et al. 2014). Avoided emissions are valued at $38 per ton. The second difference between
our private versus social rate of return calculation pertains to the valuation of avoided
fuel consumption in columns (2) through (4). The retail electricity and natural gas prices
used in Panel A include a fixed cost component; these fixed costs are not avoided when
natural gas and electricity consumption is reduced. The gas recovery charge reported by the
utility in regulatory proceedings provides a more accurate measure of the economic value
per unit of natural gas over the post-weatherization period ($5.54). To value reductions
in electricity consumption, we use the average wholesale electricity price. The net effect of
these two opposing effects (a reduction in the value of reduced energy consumption, but the
added value of avoided emissions) is a slight increase in rates of return vis a vis the private
rates. Note that these calculations do not account for the social cost of public funds, the
administrative costs of the program, or the costs of our encouragement intervention.
It is important to note that there may be additional benefits to households that received
weatherization assistance that we are not measuring. For example, consumers may realize
health benefits associated with reduced draftiness. While consumers who pay out of pocket
31We consider only the value of avoided carbon dioxide emissions. Criteria pollutants such as NOx andSO2 from electricity generation in Michigan are subject to a (barely) binding cap.
46
for an energy efficiency program would internalize those expected benefits, we cannot measure
them in a government-supported program. At the same time, there are speculations in the
public health literature that the program may increase health risks as less drafty homes may
trap indoor air pollutants such as radon.
Panels C and D estimate the average cost per ton of CO2avoided at two different discount
rates (3 and 7 percent). To construct these average abatement cost estimates, the net cost
of the investments (i.e. levelized annual cost less the value of annual energy cost savings)
is divided by estimated annual emissions reductions. Using energy savings values generated
by the NEAT program audits, abatement costs are negative because emissions reducing
efficiency investments appear highly cost effective from a private perspective. In contrast,
when the experimental estimates of natural gas and electricity savings are valued using
utility-reported gas recovery charge and average wholesale electricity prices, abatement costs
turn positive. Our preferred estimates of the average cost per ton of CO2 avoided accounts
for the small welfare gain associated with increased heating demand (column (4)).
6.2 What explains the low rate of return on these efficiency in-
vestments?
It is natural to ask why the returns to energy efficiency investments through WAP are so
low. After all, the program is designed so that the only measures implemented are ones with
projected savings to costs ratios of greater than 1.
Table 8 shows that low realized returns can be partly explained by the fact that pro-
gram auditors over-value avoided natural gas consumption; auditors assume a value of
$11.46/MMBtu, almost 50 percent above the average retail price of natural gas since 2011.
This over-valuation explains the difference between columns (1) and (2) of Table 8.
Another important factor leading to negative returns on investment is the incomplete
47
realization of projected energy savings. Our preferred experimental estimates amount to less
than a third of projected savings. The quasi-experimental estimates of energy savings are
between 23 and 28 percent of the corresponding average savings projections. Importantly,
we are not the first to report ex post realized savings that fall short of ex ante projections.
Low realization rates are fairly ubiquitous in both the academic literature (see, for example,
Davis et al. (2013); Dubin et al. (1986)) and utility program evaluation reports.32
In the academic literature, low realization rates are often attributed to t behavioral
responses to efficiency improvements are rarely captured by building simulation models (see,
for example, Dubin et al. 1986; Davis et al. 2013). However, our estimates of weatherization-
induced increases in demand for heating account for less than 5 percent of the gap between
projected and ex post realized energy savings.
Having ruled out behavioral responses as the primary explanation, we conclude that the
efficiency audit tool must systematically overstate the real returns to these investments.
Along these lines, we explore some alternative sources of this projection bias. First, we com-
pare the distribution of temperatures observed during our study period against the typical
weather patterns on which engineering calculations are based. Although we do observe some
moderate spells in our time frame, on average we observe colder than average temperatures
and higher than average degree day measures in our sample.
A second source of bias concerns the over-statement of baseline energy use. Energy
efficiency program analysts and practitioners have documented major errors in standard
software for estimating natural gas use in older, inefficient homes.33 Program audit data allow
us to isolate predictions of baseline energy consumption for each audited household. When
we compare these projections with the actual billing data, wefind that a typical household is
32For example, a recent summary of realization rates documented in Massachussets reports realiza-tion rates ranging from below 10 percent to over 100 percent. See http://ma-eeac.org/wordpress/wp-content/uploads/Home-Energy-Services-Realization-Rate-Results-Memo-6-28-12.pdf.
33See, for example. http://2011.acinational.org/sites/default/files/session/81087/aci11chal1blasnikmichael.pdf.
48
projected to use 10.25 MMBtus of natural gas per month before weatherization. In the data,
we observe audited households consuming 8.11 MMBtu per month prior to weatherization.
Thus on average, the heating-related natural gas consumption assumed by the efficiency
audits are more than 25 percent higher than the observed consumption.This suggests that
the auditing tool could be under-estimating the efficiency properties of the average home
prior to weatherization, which would lead to an over-statement of the benefits of upgrading
to a given efficiency standard.
In sum, our findings suggest that the NEAT audit tool is over-estimating returns on
the energy efficiency investments we analyze by a significant margin. This is an important
finding; NEAT is widely used by state and local WAP sub grantees, utility companies, and
home energy audit firms. While more sophisticated building simulation models exist, it is
important to remember that thousand of implementers with a range of skills and technical
training must use whatever program informs WAP or similar large-scale energy efficiency
programs. The DOE cites one of the primary benefits of NEAT as its accessibility to non-
technical users (DOE-EERE).
7 Conclusion
This paper presents results from one of the first large-scale experiments designed to measure
the real-world returns on investments in residential energy efficiency improvements. We
implement a randomized encouragement design to evaluate the benefits of weatherization
retrofits provided to low-income consumers. We also present corroborating evidence from a
quasi-experimental analysis covering over twice as many weatherizations.
We find that the program led to economically significant reductions in energy expendi-
tures for weatherized households. On average, weatherization reduces energy consumption
by 10-20 percent at participating households. Neoclassical economic theory suggests that
49
consumers facing lower prices for energy services, such as lower prices to heat their homes
after weatherization, will re-optimize and potentially consume more services. As a result,
the total direct consumer welfare of an energy efficiency program includes the utility from
increased consumption, such as increased indoor heating. We implemented a follow-up sur-
vey of program participants and non-participants to measure their indoor temperatures and
find modest increases in temperatures among participants. We use a revealed preference
approach to bound the associated increases in welfare and find very small welfare gains.
Overall, we show that the net present value of the savings, accounting for both reduced
energy expenditures and increased utility from indoor heating, are lower than the direct
program costs under a range of plausible assumptions on both the average lifetimes of the
energy efficiency measures and consumer discount rates. Accounting for program overhead
costs or the marginal cost of public funds would lead to even lower estimated returns on
investment. Our preferred estimates of the average cost per ton of CO2 avoided significantly
exceed the value that federal agencies use to estimate the climate benefits associated with
avoided emissions ($38/ton).
A natural instinct is to try to generalize our experimental estimates to other contexts,
including weatherization programs in other states, and investments in energy efficiency im-
provements more generally. Any such attempt comes with an important caveat. Our results
are specific to a population of low-income households in Michigan who were encouraged to
participate in the federally funded weatherization assistance program. From a research de-
sign perspective, analyzing a federally funded program provides several benefits. For one, we
have systematic, household-specific measures of implementation costs, savings projections,
etc. Given the low-income population served by the program, we are relatively confident
that our encouragement intervention did not inspire contemporaneous complementary in-
vestments in efficiency improvements. However, the external validity or our findings is
limited by the fact that households in our study are not representative of U.S. households
50
in other income segments or other regions of the country. Our estimates of the extent to
which realized savings fall short of ex ante projections are also specific to the audit tool that
is used to inform investment decisions in the WAP program. Although this tool is widely
used by state and local WAP sub grantees, utility companies, and home energy audit firms,
the savings projections it generates may not be representative of other modeling approaches
that inform efficiency investments in other contexts.
Widely publicized engineering estimates, such as the iconic McKinsey curve, suggest
that consumers are systematically bypassing opportunities to invest in cost-effective energy
efficiency improvements that lower their energy expenditures and reduce externalities asso-
ciated with energy production. As a result, many policymakers are championing programs
designed to encourage energy efficiency as a cost-effective strategy to confront climate change
and reduce other harmful pollutants. This paper provides evidence that field savings may
be very different from engineering estimates. These findings underscore the importance of
field-testing of projected returns on energy efficiency investments, particularly given the
increasing role of efficiency programs as a policy tool.
References
Allcott, H. and Greenstone, M. (2012). Is there an energy efficiency gap? The Journal ofEconomic Perspectives, pages 3–28. 1
Davis, L., Fuchs, A., and Gertler, P. (2013). Cash for coolers: Evaluating a large-scaleappliance replacement program in Mexico. American Economic Journal: Economic Policy,forthcoming. 6.2
Dewees, D. N. and Wilson, T. A. (1990). Cold houses and warm climates revisited: Onkeeping warm in chicago, or paradox lost. Journal of Political Economy, 98(3):pp. 656–663. 5
Dubin, J. A., Miedema, A. K., and Chandran, R. V. (1986). Price effects of energy-efficienttechnologies: A study of residential demand for heating and cooling. The RAND Journalof Economics, 17(3):pp. 310–325. 1, 6.2
EERE, D. (2013). Review of selected home energy auditing tools: In support of the de-velopment of a national building performance assessment and rating program. Technical
51
Report MSU-CSE-05-29, U.S. Department of Energy, Energy Efficiency & Renewable En-ergy. 4.1.2
Friedman, D. (1987). Cold Houses in Warm Climates and Vice Versa: A Paradox of RationalHeating. Journal of Political Economy, University of Chicago Press, 95(5):1089–97. 5
Gillingham, K., Kotchen, M. J., Rapson, D. S., and Wagner, G. (2013). Energy policy: Therebound effect is overplayed. Nature, 493(7433):475–476. 1
Gillingham, K. and Palmer, K. (2014). Bridging the Energy Efficiency Gap: Policy Insightsfrom Economic Theory and Empirical Evidence. Review of Environmental Economics andPolicy, 8(1):18–38. 1
Jacobsen, G. D. and Kotchen, M. J. (2013). Are building codes effective at saving energy?Evidence from residential billing data in florida. Review of Economics and Statistics,95(1):34–49. 1
McKinsey & Company (2009). Unlocking energy efficiency in the U.S. economy. McKinsey& Company. 1
Schwarz, P. M. and Taylor, T. N. (1995). Cold hands, warm hearth?: Climate, net takeback,and household comfort. The Energy Journal, 16(1):pp. 41–54. 4document
52